Hi,
I have released CollecTor 1.17.0.
CollecTor fetches data from various nodes and services in the public Tor network and makes it available to the world.
This release is the sum of over 11 years of development, with the latest changes:
Medium changes
* Clean up descriptors written to the out/ directory by deleting files that are older than seven weeks.
* Correctly index files that are moved away and back.
* Include microdescriptors, certs, and OnionPerf analysis files when syncing from another CollecTor instance.
* Update to metrics-lib 2.16.0.
You can find the release at:
https://dist.torproject.org/collector/1.17.0/
You can find the sources at:
https://gitlab.torproject.org/tpo/metrics/collector/-/tree/collector-1.17.0/
Thanks,
Iain.
P.S. the dates on the changelog and signatures will not be today, as the release was ready to go before my access was in place to upload the release, this is now fixed and future releases will be uploaded once they are available.
Fellow translators!
I hope you enjoyed MayDay, and also Ramadan Kareem to our Muslim translators!
Here some information about the l10n efforts in the Tor Project
Tor Browser in Burmese
----------------------
This month's more important achievement, as usual with the help of Localization Lab, was to release Tor Browser in Burmese.
We send a big shout out for our contributors that under dire circumstances and risks took the time to translate the browser and are still translating the Manual and other important documents. Keep strong!
We had some problems[1] that were undetected before we published the stable release, so along with the Tor Browser team, we have refined a bit the process, to allow more time between the alpha releases and the stable releases, so the information can get back to the team and be fixed before the definitive stable release.
We have also released Snowflake and its website in Burmese. Check out the beautiful Burmese letters! https://snowflake.torproject.org/?lang=my
Checks and tests
----------------
We love our translators and we are happy to help them learn how to translate software in exchange of their work. But the lack of experience makes some errors happen (well, even experienced translators may overlook errors!)
On the l10n repository I am adding some scripts that check our translations for errors. Tests are run once per day with the gitlabCI infrastructure.
The code lives at https://gitlab.torproject.org/tpo/community/l10n/-/tree/main/ and it generates the pages at https://tpo.pages.torproject.net/community/l10n/
I plan to add more tests to find out errors, and hopefully we will be able to catch them before they reach our websites.
Thanks to all translators that reacted to our announcement, it was a very big list and is reducing fast!
Stats
-----
For our translation efforts to scale, we need better stats, and arthuredelstein (our stats genius) is working on this.
We have new stats for the Community portal: https://torpat.ch/communitytpo-contentspot and we are working on even more:
https://gitlab.torproject.org/tpo/community/l10n/-/issues/40031 is the ticket and we are currently working on the specs.
Please comment on the ticket if you have any ideas!
Documentation
-------------
I am still improving the documentation for translators at:
https://gitlab.torproject.org/tpo/community/l10n/-/wikis/Localization-for-t… and I will move some of the things currently at:
https://wiki.localizationlab.org/index.php/Tor to our wiki, to have a centralized documentation section.
I also plan to update the localization section on the community portal:
https://community.torproject.org/localization/ . I will remove some duplicated sections, and move stuff to the wiki,as it is easier to update and uses less resources (for example, no translation is needed)
Let me know if you have any ideas about this, or information you are missing in the docs.
Proposal: l10n sessions
-----------------------
After a suggestion from Erinm, we want to start hosting localization sessions, maybe once a month.
Do you have any preferred day of the week to do this? Will the third Friday of the month be OK? I plan to host them on our oftc irc channel: #tor-l10n [2]
Regarding timezones, being open minded, I could join them at 10:00am UTC, and the idea is to hang out while translating, comment on strings or new resources, and hopefully help each other while translating. I am scared to commit to more days of the month, but we could also decide to do them every week if my presence is not required.
What do you think?
Plans for May
-------------
Keep working on more gitlabCI checks and the stats.
Put the donation page up for translation.
Have a look to the comments in transifex.
Plant the veggie garden.
Thanks for your attention, and see you around!
emmapeel
Localization Coordinator
Tor Project
[1] The Tor Browser Windows installer has no Burmese language option, although you get Burmese if you select English. More information:
https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/4…
[2] https://support.torproject.org/get-in-touch/irc-help/ (join channel #tor-l10n instead of #tor)
Hi,
I have released metrics-lib 2.16.0.
metrics-lib, which also goes by the name DescripTor, is a Java API that
fetches Tor descriptors from a variety of sources like cached
descriptors and directory authorities/mirrors. The DescripTor API is
useful to support statistical analysis of the Tor network data and for
building services and applications.
This release is the sum of over 9 years of development, with the latest
changes:
Medium changes
* Parse new NAT-based Snowflake lines.
* Added a new parser for the IP Fire GeoIP database files used in core
Tor.
* Tests now depend on objenesis 2.2 (previously 2.1).
Minor changes
* A lib folder is now included in the source distribution and does not
need to be created prior to using the ant resolve task.
You can find the release at:
https://dist.torproject.org/metrics-lib/2.16.0/
You can find the sources at:
https://gitlab.torproject.org/tpo/metrics/library/-/tree/metrics-lib-2.16.0
Thanks,
Iain.
P.S. the dates on the changelog and signatures will not be today, as the
release was ready to go before my access was in place to upload the
release, this is now fixed and future releases will be uploaded once
they are available.
Hi!
We are starting to meet again to discuss releases of Tor Browser. The
next meeting will be on May 4th at 19UTC in irc.oftc.net #tor-meeting
The pad for this meetings is at
http://kfahv6wfkbezjyg4r6mlhpmieydbebr5vkok5r34ya464gqz6c44bnyd.onion/p/tor…
cheers,
gaba
--
pronouns she/her/they
GPG Fingerprint EE3F DF5C AD91 643C 21BE 8370 180D B06C 59CA BD19
As with the previous month, I figured I would show a sign of life here
and try to keep you up to date with what's happening in sysadmin-land,
even though we're not having regular meetings. I'm still experimenting
with structure here, and this is totally un-edited, so please bear
with me.
# Important announcements
You might have missed this:
* Jenkins will be retired in December 2021, and it's time to move
your jobs away
* if you want old Trac wiki redirects to go to the right place, do
let us know, see [ticket 40233][]
* we do not have ARM 32 builders anymore, the last one was shut down
recently ([ticket 32920][]) and they had been removed from CI
(Jenkins) anyways before that. the core team is looking at
alternatives for building Tor on armhf in the future, see [ticket
40347][]
* we have setup a Prometheus Alertmanager during the hack week, which
means we can do alerting based on Prometheus metrics, see the
[altering documentation][] for more information
As usual, if you have any questions, comments, or issues, please do
contact us following this "how to get help" procedure:
<https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-2-support…>
Yes, that's a terrible URL. Blame GitLab. :)
[altering documentation]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/prometheus#alerting
[ticket 40347]: https://gitlab.torproject.org/tpo/core/tor/-/issues/40347
[ticket 32920]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/32920
[ticket 40233]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40233
# Crash of the month
Your sysadmin crashed a Ganeti node, creating a split-brain scenario
([ticket 40229][]). He would love to say that was planned and a
routine exercise to test the documentation but (a) it wasn't and (b)
the document had to be made up as he went, so that was actually a
stressful experience.
Remember kids: never start a migration before the weekend or going to
bed unless you're willing and ready to stay up all night (or
weekend).
[ticket 40229]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40229
# Metrics of the month
* hosts in Puppet: 86, LDAP: 89, Prometheus exporters: 140
* number of Apache servers monitored: 28, hits per second: 147
* number of Nginx servers: 2, hits per second: 2, hit ratio: 0.86
* number of self-hosted nameservers: 6, mail servers: 7
* pending upgrades: 1, reboots: 0
* average load: 0.68, memory available: 2.00 TiB/2.77 TiB, running processes: 552
* bytes sent: 276.43 MB/s, received: 162.75 MB/s
* [GitLab tickets][]: ? tickets including...
* open: 0
* icebox: 109
* backlog: 15
* next: 2
* doing: 2
* (closed: 2266)
[Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards
# Ticket analysis
Here's an update of the ticket table, which we last saw in February:
| date | open | icebox | backlog | next | doing | closed | delta | sum | new | spill |
|------------|------|--------|---------|------|-------|--------|-------|--------|------|-------|
| 2020-11-18 | 1 | 84 | 32 | 5 | 4 | 2119 | NA | 2245 | NA | NA |
| 2020-12-02 | 0 | 92 | 20 | 9 | 8 | 2130 | 11 | 2259 | 14 | -3 |
| 2021-01-19 | 0 | 91 | 20 | 12 | 10 | 2165 | 35 | 2298 | 39 | -4 |
| 2021-02-02 | 0 | 96 | 18 | 10 | 7 | 2182 | 17 | 2313 | 15 | 2 |
| 2021-03-02 | 0 | 107 | 15 | 9 | 7 | 2213 | 31 | 2351 | 38 | -7 |
| 2021-04-07 | 0 | 106 | 22 | 7 | 4 | 2225 | 12 | 2364 | 13 | -1 |
| 2021-05-03 | 0 | 109 | 15 | 2 | 2 | 2266 | 41 | 2394 | 30 | 11 |
|------------|------|--------|---------|------|-------|--------|-------|--------|------|-------|
| total | NA | NA | NA | NA | NA | NA | 147 | NA | 149 | -2 |
|------------|------|--------|---------|------|-------|--------|-------|--------|------|-------|
| mean | 0.1 | 97.9 | 20.3 | 7.7 | 6.0 | 2185.7 | 21.0 | 2317.7 | 21.3 | -0.3 |
I added a "delta" column which shows how many additional tickets were
closed since the previous period. April is our record so far, with a
record of 41 tickets closed in less than 30 days, more than one ticket
per day!
I also added a "new" column that shows how many new tickets, in total,
were created in the period. And the "spill" is the difference between
the two. If positive, we're winning the ticket game, if negative,
we're losing ground and more tickets are being created than we are
closing. Overall, we're slightly behind (-2), but that's only because
of the epic month of April.
And while I'm here, I went crazy with Emacs' `orgtbl-mode` and added
totals and averages.
In other news, the Icebox keeps growing, which should keep us cool and
breezy during the northern hemisphere summer that's coming up. ;) At
least the Backlog is not growing too wildly, and the actual current
queue (Next/Doing) is pretty reasonable. So things seem to be under
control, but the new hiring process is taking significant time so this
might upset our roadmap a little.
Regardless of those numbers: don't hesitate to make new tickets!
# Ticket of the month
Ticket [40218][] tracks the progress of the CI migration from Jenkins
to GitLab CI. Jenkins is scheduled for retirement in December 2021,
and progress has been excellent, with the network team actually
*asking* for the Jenkins jobs to be disabled ([ticket 40225][])
which, if it gets completed, will means the retirement of 4 virtual
machines already.
Exciting cleanup!
[40218]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40218
[ticket 40225]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40225
--
Antoine Beaupré
torproject.org system administration
Hi All,
This is my monthly status report for work complete during April 2021.
There was a ramp-up period as I got back into the flow of things, including reconfiguring GitLab access (with 2FA, SSH and GPG keys) and also my LDAP account. There are still some access issues to be resolved relating to uploading releases to dist.torproject.org but otherwise I am all set up now.
In order to catch up on what's been happening since I last worked on Tor Metrics I read through the network-health and metrics-team mailing lists since the start of the year.
Throughout this project until the end of July I am working on documenting Tor Metrics in preparation for the full time Metrics person to take over. All the documentation is being collected at:
https://gitlab.torproject.org/tpo/metrics/team/-/wikis/home
This is as opposed to documentation being scattered across various project wikis and repositories, as between the move from Trac wiki to GitLab and git.tpo migration to GitLab it's important to ensure that knowledge already collected isn't lost. This may not be the permanent home for the documentation but as the GitLab wiki is really just a collection of MarkDown files it's rather portable should moving it be desired.
The two main articles in this documentation I have worked on so far are the Java survival guide and the CollecTor development, deployment and operation documentation. Onionoo, Exonerator, and metrics-web are all still to follow. In collaboration with acute, OnionPerf's wiki has been merged into this central wiki.
In putting together a "pager playbook" for each service, I'm finding that monitoring for some features is lacking. I have spoken with anarcat about ways to improve this situation including writing simple exporters to get some monitoring data into the services Prometheus instance rather than sending emails via cron job.
In addition to documentation I have three main tasks:
1. Update the GeoIP database used by Onionoo
There are no Java bindings availabile for the IP Fire database that is now used in core Tor. As such we need a means of querying the file. A compact format for the file is used by core Tor, generated by a Rust tool. nickm has extended this tool to allow for AS numbers to be included in the generated file alongside country codes.
I have written a parser and lookup function for metrics-lib which is included in the 2.16.0 release. This will allow for Onionoo to consume the generated GeoIP files.
In order to ease access to the generated files, and not require Metrics developers to have a Rust toolchain on their machines, I have set up a scheduled CI job to generate the files: https://tpo.pages.torproject.net/metrics/geoip-data/ (it would be nice to add a date/time for the last update into this page, if anyone wants to submit a MR on the repo for that it could be an easy first contribution).
2. Updating the Metrics timeline in metrics-web
I have not yet started on this task.
3. bridgestrap/rdsys/Collector/Onionoo
This issue will require collaboration with the anti-censorship team. The first step here will be to get the state of bridgestrap into the Metrics pipeline, which is waiting on https://gitlab.torproject.org/tpo/anti-censorship/bridgestrap/-/issues/15.
Finally, I have been around in Matrix/IRC to answer Metrics questions and on occasion to provide a little end-user support in #tor. I've been attending the weekly network health meetings and will continue to do so. If you've got any questions about anything I'm working on, just send me an email or chat on Matrix/IRC.
Thanks,
Iain.