Hello,
I have just deployed a new version of BridgeDB[0]. Now BridgeDB uses rdsys[0] as
backend so it has changed how bridges are managed internally, but all the user
facing pieces are (almost) the same. The distributors (email, https[1] and moat)
hasn't changed and should work as before.
If you are a bridge user you should not see any difference. The bridges you are
already using should stay working and requesting new bridges should work as
before.
The move to rdsys implies that all the bridges might have changed their
distribution mechanism. If you are bridge operator you might see your bridge
having a different distributor mechanism in the coming hours, if you had
configured your bridge with the "BridgeDistribution" option your distribution
mechanism should not change and stay on what you configured.
There are more few more distribution mechanisms now, this is the full list:
* https. The web interface of bridgedb[0].
* moat. The API used by Tor Browser to request bridges directly from the
settings.
* email. Bridges distributed by bridges(a)torproject.org.
* telegram. Bridges that will be distributed by the telegram bot @GetBridgesBot.
* settings. A new API used to autoconfigure the circumvention mechanism
depending on your geolocation.
* reserved. Bridges reserved for experiments or future needs.
People with bridges assigned to telegram and settings will not see much usage of
their bridges yet, as those are projects we are working on but will be soon in
use.
If you notice any problems with the bridge distribution don't hesitate to open
an issue[2] or contact me.
[0] https://bridges.torproject.org
[1] https://gitlab.torproject.org/tpo/anti-censorship/rdsys/
[2] https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-02-24-15.59.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Next meeting: Thursday February 24th 16:00 UTC
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress)
== Goal of this meeting ==
Weekly checkin about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at Tor.
== Links to Useful documents ==
Our anti-censorship roadmap:
Roadmap: https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
The anti-censorship team's wiki page:
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
Past meeting notes can be found at:
https://lists.torproject.org/pipermail/tor-project/
Tickets that need reviews: from sponsors we are working on:
All needs review tickets: https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
Sponsor 30
https://gitlab.torproject.org/groups/tpo/-/milestones/4https://gitlab.torproject.org/groups/tpo/-/milestones/7https://gitlab.torproject.org/groups/tpo/-/milestones/5https://gitlab.torproject.org/groups/tpo/-/milestones/6
Sponsor 28
must-do tickets: https://gitlab.torproject.org/groups/tpo/-/milestones/10
possible tickets: https://gitlab.torproject.org/groups/tpo/-/issues?scope=all&utf8=%E2%9C%93&…
== Announcements ==
meskio plans to deploy rdsys+bridgedb next Monday, 2022-02-28.
many bridges will change distribution mechanism
== Discussion ==
obfs4 fail to connect issue
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40804
upstream maintainer recommends getting servers to upgrade
our own testing shows that new client -> new server solves the problem
no reply yet to the query sent to Aaron Johnson following last week's conversation
obfs4proxy is on its way to Debian testing, as soon as it hits testing we should ask acute to upload it to backports, to make it easier for server operators to upgrade
the package is expected to reach testing in <10 days
an upgrade of all obfs4 bridges will likely take time
start with the limited goal of upgrading all Tor Browser default bridges? https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Default-Brid…
bridges currently don't report the version of the PTs they run, it would help track the progress of upgrading if they did report
https://gitlab.torproject.org/tpo/core/tor/-/issues/11101#note_2770219
Past discussion on using the PT STATUS message for version reporting:
https://lists.torproject.org/pipermail/tor-project/2021-October/003194.htmlhttp://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-09-30-16.00.log…
uTLS in Snowflake Client Broker Communication
we have some concerns on the uTLS fingerprints being old and "easier" to censor
psiphon devs see uTLS working right for them
is a client side change, we can update it quickly
we don't enable uTLS by default, but make it an option from the SOCKS interface, so we can enable it by changing the bridge line from the circumvention settings
== Actions ==
== Interesting links ==
https://archive.org/details/ptim2021 Pluggable Transports Implementers Meeting 2021 videos
== Reading group ==
We will discuss "Throttling Twitter: an emerging censorship technique in Russia" on 10 March
https://dl.acm.org/doi/pdf/10.1145/3487552.3487858https://censorbib.nymity.ch/#Xue2021a
Questions to ask and goals to have:
What aspects of the paper are questionable?
Are there immediate actions we can take based on this work?
Are there long-term actions we can take based on this work?
Is there future work that we want to call out, in hopes that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
anadahz: 2022-01-27
Last week:
- Increase timeout check cycles for default-bridge-felix-1 and default-bridge-felix-2 as they have been generating too many alerts: https://gitlab.torproject.org/tpo/anti-censorship/monit-configuration/-/mer…
cecylia (cohosh): last updated 2022-02-17
Last week:
- tried out snowflake experiments in shadow's preload mode
- https://github.com/shadow/shadow/issues/1549
- deployed snowflake server fixes
- reached out to default bridge operators
- finished s28 prep and documentation
- wrote some follow up code for Snowflake event channel
- https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
This week:
- mostly afk
Needs help with:
dcf: 2022-02-24
Last week:
- further helped debug obfs4proxy-0.0.12 connection failures https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40804
- investigated a snowflake bridge outage and restarted the server
Next week:
- post summary of 2022-02-18 snowflake bridge outage
Help with:
agix: 2021-02-10
Last week:
- Continued work on gettor-twitter
Next week:
- Hopefully finish the task
Help with:
-
arlolra: 2022-01-20
Last week:
- [added 2022-01-20 by dcf] ALPN support for pion DTLS https://github.com/pion/dtls/pull/415
Next week:
- Figure out where in pion/webrtc ALPN should be configured and used
- Maybe add Chacha20Poly1305 to pion/dtls
https://github.com/pion/dtls#planned-featureshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
-
maxb: 2021-09-23
Last week:
- Worked on https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… re: utls for broker negotiation
- Had conversation with someone about upstream utls http round tripper https://github.com/refraction-networking/utls/pull/74
- Too busy with work :/
Next week:
- _Really_ want to get a PR for utls round tripper
meskio: 2022-02-24
Last week:
- validate opendkim headers in bridgedb (bridgedb!36)
- review concurrency problems in rdsys (rdsys#90)
- fix rdsys bridge status site, it was not reporting if they are working for non-vanilla bridges (rdsys#88)
- handle bridgedb reconnections (bridgedb!34)
- make easier to develop bridgedb with a fake rdsys (bridgedb#40034)
- coordinate the deployment of rdsys + bridgedb
Next week:
- deploy rdsys + bridgedb in production (rdsys#12)
- use rdsys bridges in circumvention settings (bridgedb#40045)
Shelikhoo: 2022-02-24
Last Week:
- [Merge Request Awaiting] Add SOCKS5 forward proxy support to snowflake (snowflake!64)
- [Merge Request Done] Add verbosity switch to suppress diagnostic output (snowflake#40079, snowflake!74)
- [Merge Request] uTLS for broker negotiation
- [Discussion] Implement metrics to measure snowflake churn (snowflake#34075)
- [Discussion] Proposal: Support for Dynamic IP obfs4 bridges with unattended proxy information update (aka "Subscription")
- [Discussion] Proposal: Push Notification Based Signaling Channel
- [Discussion] Proposal: Centralized Probe Result Collector (anti-censorship/team#54)
- [Discussion] HTTPT & Websocket (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/http…)
- [Investigate] Is there a better moat/snowflake SNI than cdn.sstatic.net? (snowflake#40068)
- [Investigate] Multi-instance Load Balanced Tor - Snowflake Deployment
- [Investigate] China "Anti-Fraud" Webpage Redirection Censorship (censorship-analysis#40026)
- [Investigate] S96 Bridge Performance
Next Week:
- [Discussion] Implement metrics to measure snowflake churn (snowflake#34075)
- [Discussion] Proposal: Push Notification Based Signaling Channel
- [Coding] Proposal: Centralized Probe Result Collector (anti-censorship/team#54)
HackerNCoder: 2021-12-16
This week:
Last/done:
Setup web mirror on tor.encryptionin.space
Next:
Get (new VPs with) new IP and setup new web mirror on new domain
hanneloresx: 2021-3-4
Last week:
- Submitted MR for bridgestrap issue #14
Next week:
- Finish bridgestrap #14
- Find new issue to work on
Help with:
-
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
Greetings,
Network team will be releasing on Friday a new alpha (0.4.7.4-alpha) but this
one is very special, it is the first alpha with the gigantic congestion
control work in it.
https://gitlab.torproject.org/tpo/core/tor/-/issues/40405https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/324-rt…
There are also couple important fixes for relay operators running alphas and
so we will recommend to upgrade.
With this alpha, we hope to get to a release candidate very soon and then onto
the long awaited 0.4.7 stable in around 2-3 months.
Upcoming versions:
- 0.4.7.4-alpha
@network-team: It is _now_ a good time to start reviewing changes/ files:
https://gitlab.torproject.org/tpo/core/tor/-/tree/main/changes
Last, we've asked the dirauth to recommend these versions few minutes ago.
Cheers!
David
--
hn/Y6tIdc7o83Kubm4tb7kwGg3cVfoop1AQ3QAX8UkA=
Hi!
We are going to have our next demo day at the end of March! Please
submit your proposals to
https://nc.torproject.net/apps/forms/MT5NTyWnGwDsJ4r8
Tor Demo Day - March 23rd 2022
==============================
What is a Demo Day?
-------------------
An hour long session at an all hands meeting where different people
present 5 minutes ideas, tools, hacking or anything that they think it
may be interesting for the Tor community.
How?
----
- We call for presenters a month before the demo day.
- The agenda will be set before the meeting and will be shared with
tor-project.
- We want 5 to 8 minutes length presentations. We will handle questions
at the end of all presentations.
- Slides are allowed, but not mandatory. You may share your screen or a
pre-recorded video if that makes you less anxious.
- We like the adrenaline of the live presentations, but if they wish,
presenters can share pre-recorded videos with us. They should be sent a
few hours in advance and will be lined up and shared with assistants
during the session.
- This is a place where people should feel safe to engage, share their
point of view, and participate. Read our Code of Conduct:
https://gitweb.torproject.org/community/policies.git/tree/code_of_conduct.t…
Who?
----
- Anyone who wants to share what they have been hacking on. It doesn't
need to be a finished feature but a work in progress. It can be small
like a Tor Browser feature or something you have been hacking around Tor
not officially.
- Presenters have 5 to 8 minutes. People can ask questions via text in
the pad for 3 minutes.
Where?
------
The link to a room in our BBB instance will be sent a few hours before
starting the demo day.
When
----
Wednesday March 23rd at 1600 UTC
Agenda
------------------------------
- 5 min for the presentation
- 3 min for questions
------------------------------
Questions will be written in text on the pad in any time during the
presentation. Presenters will answer them in audio.
--
pronouns she/her/they
GPG Fingerprint EE3F DF5C AD91 643C 21BE 8370 180D B06C 59CA BD19
Hey everyone!
Here are our meeting logs:
ttp://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-02-17-15.59.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Next meeting: Thursday February 17th 16:00 UTC
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress)
== Goal of this meeting ==
Weekly checkin about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at Tor.
== Links to Useful documents ==
Our anti-censorship roadmap:
Roadmap: https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
The anti-censorship team's wiki page:
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
Past meeting notes can be found at:
https://lists.torproject.org/pipermail/tor-project/
Tickets that need reviews: from sponsors we are working on:
All needs review tickets: https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
Sponsor 30
https://gitlab.torproject.org/groups/tpo/-/milestones/4https://gitlab.torproject.org/groups/tpo/-/milestones/7https://gitlab.torproject.org/groups/tpo/-/milestones/5https://gitlab.torproject.org/groups/tpo/-/milestones/6
Sponsor 28
must-do tickets: https://gitlab.torproject.org/groups/tpo/-/milestones/10
possible tickets: https://gitlab.torproject.org/groups/tpo/-/issues?scope=all&utf8=%E2%9C%93&…
== Announcements ==
== Discussion ==
obfs4proxy-0.0.12 causing connection failure warnings, before eventually succeeding
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40804
It looks like an obfs4proxy-0.0.12 client connecting to a pre-0.0.12 server fails to connect some fixed fraction of the time, requiring multiple connection attempts before finally succeeding
The Elligator2 fix in obfs4proxy-0.0.12 is documented to be backward compatible, but perhaps it is not fully
https://gitlab.com/yawning/obfs4/-/blob/77af0cba934d73c4baeb709560bcfc9a9fb…
"Representatives created by this implementation will correctly be decoded by existing implementations." "As the representative to public key transform should be identical, this change is fully-backward compatible"
It could be that something about the altered (fixed) mathematical transform of Elligator2 is responsible for the handshake failures. Posts about Elligator2 and the cofactor distinguishability bug:
https://loup-vaillant.fr/articles/implementing-elligatorhttps://loup-vaillant.fr/tutorials/cofactorhttps://www.reddit.com/r/crypto/comments/fd9t3m/elligator_with_x25519_is_br…
Aaron Johnson has looked into it, and says (according to immediate memory) that the Elligator2-encoded public keys of past versions of obfs4proxy are distinguishable from random in 7/8 cases. When both side have not upgraded, it is 63/64 cases. Or something along those lines.
Roger will put us in touch with Aaron to understand the encoding bug and whether it could be leading to the connection failures.
Shall we roll back the obfs4proxy version in Tor Browser until we figure it out?
It seems to be a small number of users who are bothered enough by the log messages to post on #40804 and on the forum https://forum.torproject.net/t/bridges-work-but-with-error-messages-in-log-…
No rollback for now.
We need to test a 0.0.12-to-0.0.12 connection, to see whether upgrading the server solves the connection failures. If it does, then (separately from fixing the problem in the client) we can push for server operators to upgrade.
obfs4proxy-0.0.13 is already in Debian sid, it will take some time to work its way into testing and backports
A non-upgraded server is detectable (with 7/8 probability) at the client. We could have the client log a message when that occurs, so the user can choose to use a different bridge, or encourage the bridge operator to upgrade.
Tails's test suite caught the connection errors: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40804#n…
uTLS for broker negotiation. Should we enable by default?
the fingerprints in uTLS are pretty old, is it better to use them than go TLS?
we are reachinging out to psiphon to see what they thing about uTLS maintenance
V2 uses a "bring your own browser" model: the proxy forwards through a browser installed on the system to get a good TLS fingerprint.
meek in Tor Browser used to do something similar, using the Firefox that is part of the bundle. Later meek got support for uTLS: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek…
obfs4proxy's meek_lite implementation also got uTLS support, and the Tor Browser maintainers decided to switch to that (from mainline meek with browser forwarding) so that there would be only one binary to ship: https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/29430
== Actions ==
== Interesting links ==
== Reading group ==
We will discuss "Weaponizing Middleboxes for TCP Reflected Amplification" on 2022-02-17
https://censorbib.nymity.ch/#Bock2021b
Questions to ask and goals to have:
What aspects of the paper are questionable?
Are there immediate actions we can take based on this work?
Are there long-term actions we can take based on this work?
Is there future work that we want to call out, in hopes that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
anadahz: 2022-01-27
Last week:
- Increase timeout check cycles for default-bridge-felix-1 and default-bridge-felix-2 as they have been generating too many alerts: https://gitlab.torproject.org/tpo/anti-censorship/monit-configuration/-/mer…
cecylia (cohosh): last updated 2022-02-17
Last week:
- tried out snowflake experiments in shadow's preload mode
- https://github.com/shadow/shadow/issues/1549
- deployed snowflake server fixes
- reached out to default bridge operators
- finished s28 prep and documentation
- wrote some follow up code for Snowflake event channel
- https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
This week:
- mostly afk
Needs help with:
dcf: 2022-02-17
Last week:
- opened issues for some default obfs4 bridges being offline (smallerRichard) https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/44#note_277… (KauBridge{Pale,Blue,Dot}) https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/64
- investigated "general SOCKS server failure" errors from obfs4proxy-0.0.12 https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40804#n…
- did some review of uTLS for snowflake-client https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- posted snowflake bridge VPS cost estimates https://lists.torproject.org/pipermail/anti-censorship-team/2022-February/0…
Next week:
Help with:
agix: 2021-02-10
Last week:
- Continued work on gettor-twitter
Next week:
- Hopefully finish the task
Help with:
-
arlolra: 2022-01-20
Last week:
- [added 2022-01-20 by dcf] ALPN support for pion DTLS https://github.com/pion/dtls/pull/415
Next week:
- Figure out where in pion/webrtc ALPN should be configured and used
- Maybe add Chacha20Poly1305 to pion/dtls
https://github.com/pion/dtls#planned-featureshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Help with:
-
maxb: 2021-09-23
Last week:
- Worked on https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… re: utls for broker negotiation
- Had conversation with someone about upstream utls http round tripper https://github.com/refraction-networking/utls/pull/74
- Too busy with work :/
Next week:
- _Really_ want to get a PR for utls round tripper
meskio: 2022-02-17
Last week:
- make easier to test bridgedb after rdsys change (bridgedb#40034)
- bridgedb reconnection issues with rdsys (bridgedb!34)
- bridgestrap is not displaying the results (bridgestrap#31)
- staging bridgedb configuration (bridgedb-admin!4)
- remove bridgedb blockade to distribute bridges in certain contries (rdsys#89)
- review snowflake uTLS roundtripper (snowflake!76)
Next week:
- circumvention settings fallback (bridgedb#40045)
Shelikhoo: 2022-02-17
Last Week:
- [Merge Request Awaiting] Add SOCKS5 forward proxy support to snowflake (snowflake!64)
- [Merge Request Done] Add verbosity switch to suppress diagnostic output (snowflake#40079, snowflake!74)
- [Merge Request] uTLS for broker negotiation
- [Discussion] Implement metrics to measure snowflake churn (snowflake#34075)
- [Discussion] Proposal: Support for Dynamic IP obfs4 bridges with unattended proxy information update (aka "Subscription")
- [Discussion] Proposal: Push Notification Based Signaling Channel
- [Discussion] Proposal: Centralized Probe Result Collector (anti-censorship/team#54)
- [Discussion] HTTPT & Websocket (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/http…)
- [Investigate] Is there a better moat/snowflake SNI than cdn.sstatic.net? (snowflake#40068)
- [Investigate] Multi-instance Load Balanced Tor - Snowflake Deployment
- [Investigate] China "Anti-Fraud" Webpage Redirection Censorship (censorship-analysis#40026)
- [Investigate] S96 Bridge Performance
Next Week:
- [Discussion] Implement metrics to measure snowflake churn (snowflake#34075)
- [Discussion] Proposal: Push Notification Based Signaling Channel
- [Coding] uTLS for broker negotiation
- [Coding] Proposal: Centralized Probe Result Collector (anti-censorship/team#54)
HackerNCoder: 2021-12-16
This week:
Last/done:
Setup web mirror on tor.encryptionin.space
Next:
Get (new VPs with) new IP and setup new web mirror on new domain
hanneloresx: 2021-3-4
Last week:
- Submitted MR for bridgestrap issue #14
Next week:
- Finish bridgestrap #14
- Find new issue to work on
Help with:
-
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
Hey Everyone,
The Tor Browser meeting for (only) next week is moved to Tuesday
(2022-02-22) at the usual time (1500UTC) in #tor-meeting in the usual
place (#tor-meeting on OFTC). We will resume the following week on the
usual day (Mondays).
Meeting pad: https://pad.riseup.net/p/tor-tbb-keep
best,
-Richard
Hi everyone,
We had our monthly sysadmin meeting (a bit late for various reasons)
for February, and here are the minutes.
# Roll call: who's there and emergencies
No emergencies: we have an upcoming maintenance about chi-san-01 which
will require a server shutdown at the end of the meeting.
Present: anarcat, gaba, kez, lavamind
# Storage brainstorm
The idea is to just throw ideas for this ticket:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40478
anarcat went to explain the broad strokes around the current storage
problems (lack of space, performance issues) and solutions we're
looking for (specific to some service, but also possibly applicable
everywhere without creating new tools to learn)
We specifically focused on the storage problems on gitlab-02,
naturally, since that's where the problem is most manifest.
lavamind suggested that there were basically two things we could do:
1. go through each project one at a time to see how changing certain
options would affect retention (e.g. "keep latest artifacts")
2. delete all artifacts older than 30 or 60 days, regardless of
policy about retention (e.g. keep latest), could or could not
include job logs
other things we need to do:
* encourage people to: "please delete stale branches if you do have
that box checked"
* talk with jim and mike about the 45GB of old artifacts
* draft new RFC about artifact retention about deleting old artifacts
and old jobs (option two above)
We also considered unchecking the "keep latest artifacts" box at the admin level, but this would disable the feature in all projects with no option to opt-in, so it's not really an option.
We considered the following technologies for the broader problem:
* S3 object storage for gitlab
* ceph block storage for ganeti
* filesystem snapshots for gitlab / metrics servers backups
prototype with image/cache storage on runners? safer because easy to rollback
We'll look at setting up a VM with minio for testing. We could first
test the service with the CI runners image/cache storage backends,
which can easily be rebuilt/migrated if we want to drop that test.
This would disregard the block storage problem, but we could pretend
this would be solved at the service level eventually (e.g. redesign the
metrics storage, split up the gitlab server). Anyways, migrating away
from DRBD to Ceph is a major undertaking that would require a lot of
work. It would also be part of the largest "[trusted high performance
cluster][]" work that we recently de-prioritized.
[trusted high performance cluster]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/2
# Other discussions
We should process the pending TPA-RFCs, particularly TPA-RFC-16, about
the i18 lektor plugin rewrite.
# Next meeting
Our regular schedule would bring us to March 7th, 18:00UTC.
# Metrics of the month
* hosts in Puppet: 88, LDAP: 88, Prometheus exporters: 143
* number of Apache servers monitored: 25, hits per second: 253
* number of self-hosted nameservers: 6, mail servers: 8
* pending upgrades: 0, reboots: 0
* average load: 2.10, memory available: 3.98 TiB/5.07 TiB, running processes: 722
* disk free/total: 35.81 TiB/83.21 TiB
* bytes sent: 296.17 MB/s, received: 182.11 MB/s
* planned bullseye upgrades completion date: 2024-12-01
* [GitLab tickets][]: 166 tickets including...
* open: 1
* icebox: 149
* needs information: 2
* backlog: 7
* next: 5
* doing: 2
* (closed: 2613)
[Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards
Upgrade prediction graph lives at:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/
# Number of the month
-3 months. Since the last report, our bullseye upgrade completion date
moved backwards by three months, from 2024-09-07 to 2024-12-01. That's
because we haven't started yet, but it's interesting that it's seems
to be moving back faster than time itself... We'll look at deploying a
perpetual movement time machine on top of this contraption in the next
meeting.
--
Antoine Beaupré
torproject.org system administration