tor-project May 2024

tor-project@lists.torproject.org

13 participants
15 discussions

Nina’s Monthly Status Report: April 2024
by Nina 06 May '24

06 May '24

Hi! This is Nina and my April’24 report! In April, I resolved 424 tickets: On Telegram (@TorProjectSupportBot) - 346 On RT (frontdesk@tpo) - 73 On WhatsApp (+447421000612) - 6 and on Signal (+17787431312) - 0. There is a drop in email requests from Russian-speaking users, which will be investigated further. During that month I helped Russian-speaking users to bypass censorship, shared bridges with them, helped with troubleshooting, tested new bridges and collected user feedback. MacOS and iOS This month I got more of the usual iOS and macOS related issues. The major issues were: 1. Some of our official website mirrors were blocked in Russia, and macOS users are unable to get Tor Browser from our bot due to the issue with macOS not trusting programs obtained from Telegram[1]. 2. Tor Browserpluggable transports(PTs)do not work on older versions of macOS.[2]Go 1.20 requires macOS 10.13 or higher and 1.21 requires macOS 10.15. User needs to upgrade their OS to use PTs. 3. There were many questions on what app to install on iOS and where to get that, so I added a new template in Russian on that topic. Russian translation reviewand Tor Browser User Manual improvements In April I worked on the reviewof the Russian translation of the Tor Browser User Manual and filed several issues:[3][4][5]. I also work on improving the user experience with the Telegram user support channel (https://t.me/ <https://t.me/@TorProjectSupportBot>@TorProjectSupportBot <https://t.me/@TorProjectSupportBot>), testing the best welcome-message [6]. [1] https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/4… [2] https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42507 [3] https://gitlab.torproject.org/tpo/web/manual/-/issues/163 [4] https://gitlab.torproject.org/tpo/web/manual/-/issues/162 [5] https://gitlab.torproject.org/tpo/web/manual/-/issues/164 [6] https://gitlab.torproject.org/tpo/community/support/-/issues/40149

1 0

Rhatto's Monthly Status Report, April 2024
by rhatto 03 May '24

03 May '24

Hi all :) This is my monthly status report for April 2024 with the main relevant activities I have done during the period. ## 0. Research ## 1. Development * Onionspray Log Parser, a log file parser for proxy-based onionsites: * Released version 2.0.0: https://gitlab.torproject.org/tpo/onion-services/onionspray-log-parser/-/bl… * Onion Launchpad, a landing page for onionsites: * Released version 1.0.0 with enhancements such as a privacy policy and improvements in the optional metrics aggregation: https://gitlab.torproject.org/tpo/onion-services/onion-launchpad/-/mileston… * Onionprobe, an Onion Service monitoring suite: * Released version 1.2.0 with many fixes and enhancements: https://forum.torproject.org/t/onionprobe-release-1-2-0/12442 * Oniongroove, an upcoming Onion Services manager: * Worked on the prototype: https://gitlab.torproject.org/tpo/onion-services/oniongroove/-/issues/3 * Right now this prototype works only as a simple rewriting proxy for existing websites, but it's being helpful to test Arti and other tools. ## 2. Support * Ongoing sponsored work with deployment, maintenance and monitoring of Onion Services. * Did some other support related to setting up Onion Services. ## 3. Organization Time spent (from the total available for Tor-related work): | Category | Percentage |---------------|------------ | Research | 3 | Development | 49 | Support | 23 | Organization | 25 |---------------|------------ | Total | 100 -- Silvio Rhatto pronouns he/him

1 0

Anti-censorship team meeting notes, 2024-05-02
by meskio 02 May '24

02 May '24

Hey everyone! Here are our meeting logs: http://meetbot.debian.net/tor-meeting/2024/tor-meeting.2024-05-02-16.15.html And our meeting pad: Anti-censorship -------------------------------- Next meeting: Thursday, May 9 16:00 UTC Facilitator: shelikhoo ^^^(See Facilitator Queue at tail) Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC (channel is logged while meetings are in progress) This week's Facilitator: meskio == Goal of this meeting == Weekly check-in about the status of anti-censorship work at Tor. Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community. == Links to Useful documents == * Our anti-censorship roadmap: * Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards * The anti-censorship team's wiki page: * https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home * Past meeting notes can be found at: * https://lists.torproject.org/pipermail/tor-project/ * Tickets that need reviews: from sponsors, we are working on: * All needs review tickets: * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s… * Sponsor 96 <-- meskio, shell, onyinyang, cohosh * https://gitlab.torproject.org/groups/tpo/-/milestones/24 * Sponsor 150 <-- meskio working on it * https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na… == Announcements == == Discussion == * Snowfalke API has insufficent support for passing additional settings in constructor * trying to avoid making breaking changes to the API * one option is to use builders: https://docs.rs/http/0.2.9/http/request/struct.Builder.html * another is to use WithXXXX methods: https://pkg.go.dev/google.golang.org/grpc#WithAuthority * shelikhoo will create an issue to start the discussion on a possible version 3 API for snowflake * Snowflake broker Debian version EOL * https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… * we'll do a fresh install in a new VM * dcf1 will provision a VM for it * shelikhoo will do the setup * ed25519 keys in bridgelines * https://gitlab.torproject.org/tpo/core/torspec/-/issues/257 * there is a conversation on how to integrate ed25519 keys into bridgelines == Actions == == Interesting links == * https://github.com/ooni/utls-light by arturo, an alternative approach to modifying TLS fingerprints that requires less invasive crypto/tls library changes. It works by parsing the serialized native crypto/tls handshake messages before they are sent, and re-serializing them according to a desired schema. == Reading group == * We will discuss "" on * * Questions to ask and goals to have: * What aspects of the paper are questionable? * Are there immediate actions we can take based on this work? * Are there long-term actions we can take based on this work? * Is there future work that we want to call out in hopes that others will pick it up? == Updates == Name: This week: - What you worked on this week. Next week: - What you are planning to work on next week. Help with: - Something you need help with. cecylia (cohosh): 2024-05-02 Last week: - opened an issue in core tor about MyFamily for bridges - https://gitlab.torproject.org/tpo/core/tor/-/issues/40935 - snowflake webextension work This week: - make changes to Lox encrypted bridge table - https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/147 - follow up on reported SQS errors - update wasm-bindgen fork to fix some bugs and hopefully upstream changes - create a Lox test environment and instructions for the browser team - https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42503 Needs help with: dcf: 2024-05-02 Last week: Next week: - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow… - open issue to disable /debug endpoint on snowflake broker - move snowflake-02 to new VM Help with: meskio: 2023-05-02 Last week: - implement email distributor in rdsys (rdsys#186) Next week: - test email distributor in rdsys (rdsys#186) - improve privacy on rdsys metrics reusing snowflake techniques (snowflake#40354) Shelikhoo: 2024-05-02 Last Week: - [Merge Request WIP] Add Container Image Mirroring from Tor Gitlab to Docker Hub(https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/… - [Dev] Snowflake Performance Improvement - Fix ipv6 connectivity with restricted address support https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt… - Merge request reviews Next Week/TODO: - Revise Snowflake Performance Improvement MR(cont.) - Merge request reviews onyinyang: 2023-05-02 Last week(s): - Responding to pre-existing MR reviews and other notifications/questions in gitlab This week: - attempt hyper upgrade and tokio upgrade Probably next week and beyond: - preparing for upcoming panel (maybe) - implement some preliminary user feedback mechanism to identify bridge blocking based on Vecna's work - improve metrics collection/think about how to show Lox is working/valuable - sketch out Lox blog post/usage notes for forum (long term things were discussed at the meeting!): https://pad.riseup.net/p/tor-ac-community-azaleas-room-keep - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice Question: What makes a bridge usable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people? 1. Are there some obvious grouping strategies that we can already consider? e.g., by PT, by bandwidth (lower bandwidth bridges sacrificed to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?) 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less? theodorsm: 2023-04-25 Last weeks: - Extended Pion dtls library to support hooking of client hellos, server hellos and certificate request (PR: https://github.com/pion/dtls/pull/631). Fingerprint generation of firefox and chrome + mimicking using the client hello hook have been moved to its own repo: https://github.com/theodorsm/covertDTLS - Analyzed the snowflake captures done in "Snowflake Anonymous Network Traffic Identification" paper: no handshakes found, so not relevant for me - Verified the consistency of the fingerprint generation Next weeks: - Add hook to pion/webrtc so it can be used in snowflake. - Test and validate mimicked client hello with snowflake. - Add randomized fingerprint Help with: Facilitator Queue: shelikhoo onyinyang meskio 1. First available staff in the Facilitator Queue will be the facilitator for the meeting 2. After facilitating the meeting, the facilitator will be moved to the tail of the queue -- meskio | https://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: https://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan.

1 0

gitolite to gitlab migration completed (TPA-RFC-36)
by Antoine Beaupré 01 May '24

01 May '24

Hi again everyone! This is the last update of the Gitolite migration. It's a little more detailed than previous updates, so I made it in a blog post: https://blog.torproject.org/gitolite-gitlab-migration/ ... which I attach a copy here if you're the kind of people who prefer to read email than web. :) Enjoy! ---- Tor has finally completed a long migration from legacy Git infrastructure ([Gitolite and GitWeb][]) to our self-hosted [GitLab][] server. [GitLab]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab [Gitolite and GitWeb]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git Git repository addresses have therefore changed. Many of you probably have made the switch already, but if not, you will need to change: https://git.torproject.org/ to: https://gitlab.torproject.org/ In your Git configuration. The [GitWeb front page][] is now an archived listing of all the repositories before the migration. Inactive git repositories were archived in GitLab [legacy/gitolite namespace][] and the `gitweb.torproject.org` and `git.torproject.org` web sites now redirect to GitLab. [legacy/gitolite namespace]: https://gitlab.torproject.org/legacy/gitolite/ [GitWeb front page]: https://gitweb.torproject.org/ Best effort was made to reproduce the original gitolite repositories faithfully and also avoid duplicating too much data in the migration. But it's *possible* that some data present in Gitolite has not migrated to GitLab. User repositories are particularly at risk, because they were massively migrated, and they were "re-forked" from their upstreams, to avoid wasting disk space. If a user had a project with a matching name it was *assumed* to have the right data, which might be inaccurate. The two virtual machines responsible for the legacy service (`cupani` for `git-rw.torproject.org` and `vineale` for `git.torproject.org` and `gitweb.torproject.org`) have been shutdown. Their disks will remain for 3 months (until the end of July 2024) and their backups for another year after that (until the end of July 2025), after which point all the data from those hosts will be destroyed, with only the GitLab archives remaining. The rest of this article expands on how this was done and what kind of problems we faced during the migration. # Where is the code? Normally, nothing should be lost. All repositories in gitolite have been either explicitly migrated by their owners, forcibly migrated by the sysadmin team ([TPA][]), or explicitly destroyed at their owner's request. [TPA]: https://gitlab.torproject.org/tpo/tpa/team/ An exhaustive [rewrite map][] translates gitolite projects to GitLab projects. Some of those projects actually redirect to their *parent* in cases of empty repositories that were obvious forks. Destroyed repositories redirect to the GitLab front page. [rewrite map]: https://archive.torproject.org/websites/gitolite2gitlab.txt Because the migration happened progressively, it's technically possible that commits pushed to gitolite were lost after the migration. We took great care to avoid that scenario. First, we adopted a proposal ([TPA-RFC-36][]) in June 2023 to announce the transition. Then, in [March 2024][], we locked down all repositories from any further changes. Around that time, only a [handful of repositories][] had changes made after the adoption date, and we examined each repository carefully to make sure nothing was lost. [handful of repositories]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41214#note_2983302 "handful of repositories" [March 2024]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41213 [TPA-RFC-36]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-36-gitoli… Still, we built a [diff of all the changes in the git references][] that archivists can peruse to check for data loss. It's large (6MiB+) because a lot of repositories were migrated before the mass migration and then kept evolving in GitLab. Many other repositories were rebuilt in GitLab from parent to rebuild a fork relationship which added extra references to those clones. [diff of all the changes in the git references]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215#note_3023924 A note to amateur archivists out there, it's probably too late for one last crawl now. The Git repositories now all redirect to GitLab and are effectively unavailable in their original form. That said, the GitWeb site was crawled into the [Internet Archive][] [in February 2024][], so at least some copy of it is available in the [Wayback Machine][]. At that point, however, many developers had already migrated their projects to GitLab, so the copies there were already possibly out of date compared with the repositories in GitLab. [Wayback Machine]: https://web.archive.org/web/20240204162238/https://gitweb.torproject.org/ [in February 2024]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41218#note_2992296 [Internet Archive]: https://archive.org/ [Software Heritage][] also has a copy of all repositories hosted on Gitolite [since June 2023][] and have continuously kept mirroring the repositories, where they will be kept hopefully in eternity. There's an [issue][] where the main website can't find the repositories when you search for `gitweb.torproject.org`, instead [search for `git.torproject.org`][]. [search for `git.torproject.org`]: https://archive.softwareheritage.org/browse/search/?q=git.torproject.org&vi… [issue]: https://gitlab.softwareheritage.org/swh/devel/swh-web/-/issues/4787 [since June 2023]: https://gitlab.softwareheritage.org/swh/infra/sysadm-environment/-/issues/4… [Software Heritage]: https://www.softwareheritage.org/ In any case, if you believe data is missing, please do let us know by [opening an issue with TPA][]. [opening an issue with TPA]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new # Why? This is an old project in the making. The first [discussion about migrating from gitolite to GitLab][] started in 2020 (almost 4 years ago). But [going further back][], the first GitLab experiment was in 2016, almost a decade ago. [going further back]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/trac#history [discussion about migrating from gitolite to GitLab]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40472 The current GitLab server dates from 2019, [replacing Trac for issue tracking in 2020][]. It was originally supposed to host only mirrors for merge requests and issue trackers but, naturally, one thing led to another and eventually, GitLab had grown a container registry, continuous integration (CI) runners, GitLab Pages, and, of course, hosted most Git repositories. [replacing Trac for issue tracking in 2020]: https://blog.torproject.org/from-trac-into-gitlab-for-tor/ There were hesitations at moving to GitLab for code hosting. We had [discussions about the increased attack surface][] and [ways to mitigate that][], but, ultimately, it seems the issues were not that serious and the community embraced GitLab. [ways to mitigate that]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/98 [discussions about the increased attack surface]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81 TPA actually migrated its most critical repositories out of shared hosting entirely, into specific servers (e.g. the Puppet Git repository is just on the Puppet server now), leveraging Git's decentralized nature and removing an entire attack surface from our infrastructure. Some of those repositories are *mirrored* back into GitLab, but the authoritative copy is not on GitLab. In any case, the proposal to migrate from Gitolite to GitLab was effectively just formalizing a *fait accompli*. # How to migrate from Gitolite / cgit to GitLab The progressive migration was a challenge. If you intend to migrate between hosting platforms, we strongly recommend to make a "flag day" during which you migrate *all* repositories *at once*. This ensures a smoother transition and avoids elaborate rewrite rules. When Gitolite access was shutdown, we had repositories on both GitLab and Gitolite, without a clear relationship between the two. A priori, the plan then was to import all the remaining Gitolite repositories into the `legacy/gitolite` namespace, but that seemed wasteful, particularly for large repositories like [Tor Browser][] which uses nearly a gigabyte of disk space. So we took special care to avoid duplicating repositories. [Tor Browser]: https://gitlab.torproject.org/tpo/applications/tor-browser When the [mass migration][] started, only 71 of the 538 Gitolite repositories were `Migrated to GitLab` in the `gitolite.conf` file. So, given that we had *hundreds* of repositories to migrate:, we developed some automation to "[save time][]". We already automate similar ad-hoc tasks with [Fabric][], so we used that framework here as well. (Our normal configuration management tool is [Puppet][], which is a poor fit here.) [Puppet]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/puppet [Fabric]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/fabric/ [save time]: https://xkcd.com/1205/ [mass migration]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215 So a relatively [large amount of Python code][] was produced to basically do the following: [large amount of Python code]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… 1. check if all on-disk repositories are listed in `gitolite.conf` (and vice versa) and either add missing repositories or delete them from disk if garbage 2. for each repository in `gitolite.conf`, if its category is marked `Migrated to GitLab`, skip, otherwise; 3. find a matching GitLab project by name, prompt the user for multiple matches 4. if a match is found, redirect if the repository is non-empty * we have GitLab projects that *look* like the real thing, but are only present to host migrated Trac issues * in such cases we cloned the Gitolite project locally and pushed to the existing repository instead 5. otherwise, a new repository is created in the `legacy/gitolite` namespace, using the "import" mechanism in GitLab to automatically import the repository from Gitolite, creating redirections and updating `gitolite.conf` to document the change User repositories (those under the `user/` directory in Gitolite) were handled specially. First, the existing redirection map was checked to see if a similarly named project was migrated (so that, e.g. `user/dgoulet/tor` is properly treated as a fork of `tpo/core/tor`). Then the parent project was forked in GitLab and the Gitolite project force-pushed to the fork. This allows us to show the fork relationship in GitLab and, more importantly, benefit from the "pool" feature in GitLab which deduplicates disk usage between forks. Sometimes, we found no such relationships. Then we simply imported multiple repositories with similar names in the `legacy/gitolite` namespace, sometimes creating forks between user repositories, on a first-come-first-served basis from the `gitolite.conf` order. The code used in this migration is now available publicly. We encourage other groups planning to migrate from Gitolite/GitWeb to GitLab to use (and contribute to) our [fabric-tasks][] repository, even though it does have its fair share of hard-coded assertions. [fabric-tasks]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/ The main entry point is the `gitolite.mass-repos-migration` task. A typical migration job looked like: ``` anarcat@angela:fabric-tasks$ fab -H cupani.torproject.org gitolite.mass-repos-migration [...] INFO: skipping project project/help/infra in category Migrated to GitLab INFO: skipping project project/help/wiki in category Migrated to GitLab INFO: skipping project project/jenkins/jobs in category Migrated to GitLab INFO: skipping project project/jenkins/tools in category Migrated to GitLab INFO: searching for projects matching fastlane INFO: Successfully connected to https://gitlab.torproject.org import gitolite project project/tor-browser/fastlane into gitlab legacy/gitolite/project/tor-browser/fastlane with desc 'Tor Browser app store and deployment configuration for Fastlane'? [Y/n] INFO: importing gitolite project project/tor-browser/fastlane into gitlab legacy/gitolite/project/tor-browser/fastlane with desc 'Tor Browser app store and deployment configuration for Fastlane' INFO: building a new connect to cupani INFO: defaulting name to fastlane INFO: importing project into GitLab INFO: Successfully connected to https://gitlab.torproject.org INFO: loading group legacy/gitolite/project/tor-browser INFO: archiving project INFO: creating repository fastlane (fastlane) in namespace legacy/gitolite/project/tor-browser from https://git.torproject.org/project/tor-browser/fastlane into https://gitlab.torproject.org/legacy/gitolite/project/tor-browser/fastlane INFO: migrating Gitolite repository project/tor-browser/fastlane to GitLab project legacy/gitolite/project/tor-browser/fastlane INFO: uploading 399 bytes to /srv/git.torproject.org/repositories/project/tor-browser/fastlane.git/hooks/pre-receive INFO: making /srv/git.torproject.org/repositories/project/tor-browser/fastlane.git/hooks/pre-receive executable INFO: adding entry to rewrite_map /home/anarcat/src/tor/tor-puppet/modules/profile/files/git/gitolite2gitlab.txt INFO: modifying gitolite.conf to add: "config gitweb.category = Migrated to GitLab" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project project/tor-browser/fastlane to category Migrated to GitLab INFO: skipping project project/bridges/bridgedb-admin in category Migrated to GitLab [...] ``` In the above, you can see migrated repositories skipped then the [fastlane project][] being archived into GitLab. Another example with a later version of the script, processing only user repositories and showing the interactive prompt and a force-push into a fork: [fastlane project]: https://gitlab.torproject.org/legacy/gitolite/project/tor-browser/fastlane ``` $ fab -H cupani.torproject.org gitolite.mass-repos-migration --include 'user/.*' --exclude '.*tor-?browser.*' INFO: skipping project user/aagbsn/bridgedb in category Migrated to GitLab [...] INFO: skipping project user/phw/atlas in category Migrated to GitLab INFO: processing project user/phw/obfsproxy (Philipp's obfsproxy repository) in category Users' development repositories (Attic) INFO: Successfully connected to https://gitlab.torproject.org INFO: user repository detected, trying to find fork phw/obfsproxy WARNING: no existing fork found, entering user fork subroutine INFO: found 6 GitLab projects matching 'obfsproxy' (https://gitweb.torproject.org/user/phw/obfsproxy.git) 0 legacy/gitolite/debian/obfsproxy 1 legacy/gitolite/debian/obfsproxy-legacy 2 legacy/gitolite/user/asn/obfsproxy 3 legacy/gitolite/user/ioerror/obfsproxy 4 tpo/anti-censorship/pluggable-transports/obfsproxy 5 tpo/anti-censorship/pluggable-transports/obfsproxy-legacy select parent to fork from, or enter to abort: ^G4 INFO: repository is not empty: in-pack: 2104, packs: 1, size-pack: 414 fork project tpo/anti-censorship/pluggable-transports/obfsproxy into legacy/gitolite/user/phw/obfsproxy^G [Y/n] INFO: loading project tpo/anti-censorship/pluggable-transports/obfsproxy INFO: forking project user/phw/obfsproxy into namespace legacy/gitolite/user/phw INFO: waiting for fork to complete... INFO: fork status: started, sleeping... INFO: fork finished INFO: cloning and force pushing from user/phw/obfsproxy to legacy/gitolite/user/phw/obfsproxy INFO: deleting branch protection: <class 'gitlab.v4.objects.branches.ProjectProtectedBranch'> => {'id': 2723, 'name': 'master', 'push_access_levels': [{'id': 2864, 'access_level': 40, 'access_level_description': 'Maintainers', 'deploy_key_id': None}], 'merge_access_levels': [{'id': 2753, 'access_level': 40, 'access_level_description': 'Maintainers'}], 'allow_force_push': False} INFO: cloning repository git-rw.torproject.org:/srv/git.torproject.org/repositories/user/phw/obfspro… in /tmp/tmp6orvjggy/user/phw/obfsproxy Cloning into bare repository '/tmp/tmp6orvjggy/user/phw/obfsproxy'... INFO: pushing to GitLab: https://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy remote: remote: To create a merge request for bug_10887, visit: remote: https://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy/-/merge_re… remote: [...] To ssh://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy + 2bf9d09...a8e54d5 master -> master (forced update) * [new branch] bug_10887 -> bug_10887 [...] INFO: migrating repo INFO: migrating Gitolite repository https://gitweb.torproject.org/user/phw/obfsproxy.git to GitLab project https://gitlab.torproject.org/legacy/gitolite/user/phw/obfsproxy INFO: adding entry to rewrite_map /home/anarcat/src/tor/tor-puppet/modules/profile/files/git/gitolite2gitlab.txt INFO: modifying gitolite.conf to add: "config gitweb.category = Migrated to GitLab" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project user/phw/obfsproxy to category Migrated to GitLab INFO: processing project user/phw/scramblesuit (Philipp's ScrambleSuit repository) in category Users' development repositories (Attic) INFO: user repository detected, trying to find fork phw/scramblesuit WARNING: no existing fork found, entering user fork subroutine WARNING: no matching gitlab project found for user/phw/scramblesuit INFO: user fork subroutine failed, resuming normal procedure INFO: searching for projects matching scramblesuit import gitolite project user/phw/scramblesuit into gitlab legacy/gitolite/user/phw/scramblesuit with desc 'Philipp's ScrambleSuit repository'?^G [Y/n] INFO: checking if remote repo https://git.torproject.org/user/phw/scramblesuit exists INFO: importing gitolite project user/phw/scramblesuit into gitlab legacy/gitolite/user/phw/scramblesuit with desc 'Philipp's ScrambleSuit repository' INFO: importing project into GitLab INFO: Successfully connected to https://gitlab.torproject.org INFO: loading group legacy/gitolite/user/phw INFO: creating repository scramblesuit (scramblesuit) in namespace legacy/gitolite/user/phw from https://git.torproject.org/user/phw/scramblesuit into https://gitlab.torproject.org/legacy/gitolite/user/phw/scramblesuit INFO: archiving project INFO: migrating Gitolite repository https://gitweb.torproject.org/user/phw/scramblesuit.git to GitLab project https://gitlab.torproject.org/legacy/gitolite/user/phw/scramblesuit INFO: adding entry to rewrite_map /home/anarcat/src/tor/tor-puppet/modules/profile/files/git/gitolite2gitlab.txt INFO: modifying gitolite.conf to add: "config gitweb.category = Migrated to GitLab" INFO: rewriting gitolite config /home/anarcat/src/tor/gitolite-admin/conf/gitolite.conf to change project user/phw/scramblesuit to category Migrated to GitLab [...] ``` Acute eyes will notice the [bell used as a notification mechanism][] as well in this transcript. [bell used as a notification mechanism]: https://anarc.at/blog/2022-11-08-modern-bell-urgency/ A lot of the code is now useless for us, but some, like "commit and push" or [`is-repo-empty`][] live on in the [git module][] and, of course, the [gitlab module][] has grown some legs along the way. We've also found fun bugs, like a [file descriptor exhaustion in bash][], among other oddities. The [retirement milestone][] and [issue 41215][] has a detailed log of the migration, for those curious. [issue 41215]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41215 [retirement milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/11#tab-issues [file descriptor exhaustion in bash]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=642504 [gitlab module]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… [git module]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… [`is-repo-empty`]: https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/85121b4a8a293cebb… This was a challenging project, but it feels nice to have this behind us. This gets rid of 2 of the 4 remaining machines running Debian "old-old-stable", which moves a bit further ahead in our late [bullseye upgrades milestone][]. [bullseye upgrades milestone]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/5#tab-issues Full transparency: we tested GPT-3.5, GPT-4, and other large language models to see if they could answer the question "write a set of rewrite rules to redirect GitWeb to GitLab". This has become a standard LLM test for your faithful writer to figure out how good a LLM is at technical responses. None of them gave an accurate, complete, and functional response, for the record. The actual rewrite rules as of this writing follow, for humans that actually like working answers provided by expert humans instead of artificial intelligence which currently seem to be, glorified, mansplaining interns. ## git.torproject.org rewrite rules Those rules are relatively simple in that they rewrite a single URL to its equivalent GitLab counterpart in a 1:1 fashion. It relies on the [rewrite map][] mentioned above, of course. [rewrite map]: https://archive.torproject.org/websites/gitolite2gitlab.txt ``` RewriteEngine on # this RewriteMap connects the gitweb projects to their GitLab # equivalent RewriteMap gitolite2gitlab "txt:/etc/apache2/gitolite2gitlab.txt" # if this becomes a performance bottleneck, convert to a DBM map with: # # $ httxt2dbm -i mapfile.txt -o mapfile.map # # and: # # RewriteMap mapname "dbm:/etc/apache/mapfile.map" # # according to reports lavamind found online, we hit such a # performance bottleneck only around millions of entries, which is not our case # those two rules can go away once all the projects are # migrated to GitLab # # this matches the request URI so we can check the RewriteMap # for a match next # # WARNING: this won't match URLs without .git in them, which # *do* work now. one possibility would be to match the request # URI (without query string!) with: # # /git/(.*)(.git)?/(((branches|hooks|info|objects/).*)|git-.*|upload-pack|receive-pack|HEAD|config|description)?. # # I haven't been able to figure out the actual structure of # those URLs, so it's really hard to figure out the boundaries # of the project name here. I stopped after pouring around the # http-backend.c code in git # itself. https://www.git-scm.com/docs/http-protocol is also # kind of incomplete and unsatisfying. RewriteCond %{REQUEST_URI} ^/(git/)?(.*).git/.*$ # this makes the RewriteRule match only if there's a match in # the rewrite map RewriteCond ${gitolite2gitlab:%2|NOT_FOUND} !NOT_FOUND RewriteRule ^/(git/)?(.*).git/(.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$2}.git/$3 [R=302,L] # Fallback everything else to GitLab RewriteRule (.*) https://gitlab.torproject.org [R=302,L] ``` ## gitweb.torproject.org rewrite rules Those are the vastly more complicated GitWeb to GitLab rewrite rules. Note that we say "GitWeb" but we were actually *not* running [GitWeb][] but [cgit][], as the former didn't actually scale for us. [cgit]: https://git.zx2c4.com/cgit/ [GitWeb]: https://git-scm.com/docs/gitweb ``` RewriteEngine on # this RewriteMap connects the gitweb projects to their GitLab # equivalent RewriteMap gitolite2gitlab "txt:/etc/apache2/gitolite2gitlab.txt" # special rule to process targets of the old spec.tpo site and # bring them to the right redirect on the new spec.tpo site. that should turn, for example: # # https://gitweb.torproject.org/torspec.git/tree/address-spec.txt # # into: # # https://spec.torproject.org/address-spec RewriteRule ^/torspec.git/tree/(.*).txt$ https://spec.torproject.org/$1 [R=302] # list of endpoints taken from cgit's cmd.c # those two RewriteCond are necessary because we don't move # all repositories at once. once the migration is completed, # they can be removed. # # and yes, they are copied all over the place below # # create a match for the project name to check if the project # has been moved to GitLab RewriteCond %{REQUEST_URI} ^/(.*).git(/.*)?$ # this makes the RewriteRule match only if there's a match in # the rewrite map RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND # main project page, like summary below RewriteRule ^/(.*).git/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/ [R=302,L] # summary RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/summary/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/ [R=302,L] # about RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/about/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/ [R=302,L] # commit RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond "%{QUERY_STRING}" "(.*(?:^|&))id=([^&]*)(&.*)?$" RewriteRule ^/(.*).git/commit/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%2 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/commit/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD [R=302,L] # diff, incomplete because can diff arbitrary refs and files in cgit but not in GitLab, hard to parse RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/diff/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%1 [R=302,L,QSD] # patch RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/patch/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%1.patch [R=302,L,QSD] # rawdiff, incomplete because can show only one file diff, which GitLab cannot RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/rawdiff/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commit/%1.diff [R=302,L,QSD] # log RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/log/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/%1 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/log/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD [R=302,L] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/log(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD$2 [R=302,L] # atom RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/atom/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/%1 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/atom/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/commits/HEAD [R=302,L,QSD] # refs, incomplete because two pages in GitLab, defaulting to "tags" RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/refs/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tags [R=302,L] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/tag/? https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tags/%1 [R=302,L,QSD] # tree RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} id=([^&]*) RewriteRule ^/(.*).git/tree(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tree/%1$2 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/tree(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tree/HEAD$2 [R=302,L] # /-/tree has no good default in GitLab, revert to HEAD which is a good # approximation (we can't assume "master" here anymore) RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/tree/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/tree/HEAD [R=302,L] # plain RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteCond %{QUERY_STRING} h=([^&]*) RewriteRule ^/(.*).git/plain(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/raw/%1$2 [R=302,L,QSD] RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/plain(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/raw/HEAD$2 [R=302,L] # blame: disabled #RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ #RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND #RewriteCond %{QUERY_STRING} h=([^&]*) #RewriteRule ^/(.*).git/blame(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/blame/%1$2 [R=302,L,QSD] # same default as tree above #RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ #RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND #RewriteRule ^/(.*).git/blame(/?.*)$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/blame/HEAD/$2 [R=302,L] # stats RewriteCond %{REQUEST_URI} ^/(.*).git/.*$ RewriteCond ${gitolite2gitlab:%1|NOT_FOUND} !NOT_FOUND RewriteRule ^/(.*).git/stats/?$ https://gitlab.torproject.org/${gitolite2gitlab:$1}/-/graphs/HEAD [R=302,L] # still TODO: # repolist: once migration is complete # # cannot be done: # atom: needs a feed token, user must be logged in # blob: no direct equivalent # info: not working on main cgit website? # ls_cache: not working, irrelevant? # objects: undocumented? # snapshot: pattern too hard to match on cgit's side # special case, we keep a copy of the main index on the archive RewriteRule ^/?$ https://archive.torproject.org/websites/gitweb.torproject.org.html [R=302,L] # Fallback: everything else to GitLab RewriteRule .* https://gitlab.torproject.org [R=302,L] ``` The reference copy of those is available in our (currently private) Puppet git repository. -- Antoine Beaupré torproject.org system administration

2 1

PieroV's Monthly Status Report, April 2024
by Pier Angelo Vendrame 01 May '24

01 May '24

Hi everyone! Here is my status report for April 2024. This month, I explored the automation possibilities I mentioned last month. I wrote a script [0] that automatically rebases a few branches on top of gecko-dev/esr115 [1], cherry-picks commits from our default+maintenance branches, and moves them to the appropriate position of the patchset. In this way, when Mozilla tags a new release, our new branches will be ready without us doing anything. I'm still testing that everything works as expected, so I'm running the script manually. But the following step would be to run it automatically and regularly. Also, we could modify it to send us a notification when a new Firefox tag is published. I also wrote a script to automate release preparations [2], but it's still under review. This script will check for new releases of our dependencies and update our build configuration and changelogs automatically. We already had a script to create changelogs from linked GitLab issues, but we had to add updates manually. This, and a few other details, will be automated by the script, too. Then, I addressed the various comments on the MR to improve error handling in the TorConnect module [3] I mentioned in my previous report. In addition, I started working on Mullvad Browser's Windows installer [4] again. sajolida has prepared some wireframes on how to structure it, and I tried to check what's possible with NSIS. I helped with the releases for this month. I rebased both the release and the alpha channel and backported some fixes from alpha to stable. I'm also continuing my efforts to keep a branch on top of Firefox's rapid release and to (slowly but steadily) address fingerprinting issues. And since the next Firefox ESR (128) is going to be nightly in a couple of weeks [5], I restarted my uplift effort [6] as well. Finally, I made some quick fixes: a fallback mechanism to show the circuit display also when the bridge lines don't have a fingerprint (e.g., with conjure) [7], a patch for our Japanese localization that only partially worked on macOS [8], and more. Best, Pier [0] https://gitlab.torproject.org/pierov/lazy-scripts/-/blob/main/rebase-tool/r… [1] https://github.com/mozilla/gecko-dev/tree/esr115 [2] https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_re… [3] https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… [4] https://gitlab.torproject.org/tpo/applications/mullvad-browser/-/issues/200 [5] https://whattrainisitnow.com/release/?version=128 [6] https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40789#n… [7] https://gitlab.torproject.org/tpo/applications/tor-browser/-/merge_requests… [8] https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42535

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

tor-project May 2024