tor-project April 2022

tor-project@lists.torproject.org

19 participants
29 discussions

Sponsor 38 (shadow simulator) update
by Jim Newsome 05 Apr '22

05 Apr '22

I've just shared the 2022-04 update in the Shadow forum: https://github.com/shadow/shadow/discussions/2007 Also it looks like I forgot to share the previous 2021-12 update here: https://github.com/shadow/shadow/discussions/1824 Both mirrored below: 2022-04 update This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF <https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md>. Previous update: 2021-12 <https://github.com/shadow/shadow/discussions/1824>. We've merged 82 non-dependabot pull requests <https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2021-12-17..2022-…> and closed 18 issues <https://github.com/shadow/shadow/issues?q=closed%3A2021-12-17..2022-04-05+i…> since our previous update. Release status We are continuing to work on Shadow 2.1 <https://github.com/shadow/shadow/projects/5>. The biggest user-facing goal for this release is to support running golang programs in Shadow <https://github.com/shadow/shadow/milestone/30>, especially tor simulations using the snowflake pluggable transport <https://github.com/shadow/shadow/issues/1549>. We've also begun planning the Shadow 2.2 <https://github.com/shadow/shadow/projects/6> release, which will largely be a push to refactor and migrate more of the core Shadow code to Rust. Notable change since last update Emulation accuracy * Optionally move time forward in non-blocking syscalls <https://github.com/shadow/shadow/pull/1995>. Historically Shadow doesn't move time forward except when explicitly waiting for an event, such as for a deadline to pass or for data to arrive over the network. Conceptually, this emulates a system with an infinite number of infinitely fast CPUs. Normally this is sufficient for modeling networks where CPUs aren't expected to be a bottleneck. Unfortunately as we've explored running more software under Shadow, we've found a growing number of examples of code with "busy loops" <https://github.com/shadow/shadow/issues/1792>, which deadlock in this model. This new feature optionally models every syscall taking some small amount of time (e.g. a microsecond), which allows the simulation to escape most such loops. We are still testing and improving this feature, and expect some version of it to be enabled by default in the next release. * Fixed a corner case in getaddrinfo <https://github.com/shadow/shadow/pull/1998>. * Fixed several bugs in handling file descriptors: o Fixed pipe writable state when reader is closed <https://github.com/shadow/shadow/pull/1985>. o Have pipe writers return EPIPE if there are no readers <https://github.com/shadow/shadow/pull/1983>. o Fixed errno for socket with SOCK_SEQPACKET type <https://github.com/shadow/shadow/pull/1972>. o Don't fail on TCP_NODELAY <https://github.com/shadow/shadow/pull/1936>. * Reliably intercept time via vdso <https://github.com/shadow/shadow/pull/1951>. Previously we relied on intercepting calls to VDSO functions (such as |gettimeofday|) to be intercepted at the libc level via |LD_PRELOAD|. However, this doesn't work when the VDSO is used more directly, such as in golang, which would cause the program to get the real-world time instead of the simulated time. We now patch the VDSO itself at program start to reliably intercept these functions in such cases. * Implemented basic signal emulation <https://github.com/shadow/shadow/pull/1881>. This allows managed code to install signal handlers and send and receive signals within the simulation. (Sending a signal from outside the simulation to a managed process is still not supported). Notably this support is required to handle golang programs, and allows simulated processes to be shut down cleanly by scheduling |kill| processes to send appropriate signals. * Implemented the |select| system call <https://github.com/shadow/shadow/pull/1910>. Usability * Added an experimental option for strace-style logging <https://github.com/shadow/shadow/pull/1886> of syscalls made by managed code. * Stabilized the |--progress| <https://github.com/shadow/shadow/pull/1937> option, which periodically updates stderr with the simulation progress. Performance * Nightly shadow benchmarks are now being run to help detect performance regressions. We also have the ability to run the benchmark on our own development branches to investigate performance changes before merging these branches into Shadow. Benchmark results for a 5% Tor network are published publicly <https://github.com/shadow/benchmark-results/tree/master/tor>, but are only intended to be useful for Shadow developers. * Added a library for overriding crypto functions <https://github.com/shadow/shadow/pull/1853>. When enabled, this option overrides some openssl APIs with "no-op" implementations. This is a reimplementation of a feature previously available in Shadow's "tor plugin". It is intended primarily to improve the performance of tor simulations on hardware without accelerated AES operations, and is only supported on Debian 11. Happy simulating! The Shadow team 2021-12 update This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF <https://github.com/shadow/shadow/blob/main/docs/nsf_sponsorship.md>. Previous update: 2021-10 <https://github.com/shadow/shadow/discussions/1695>. We've merged 78 pull requests <https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A%3E%3D2021-09-30> and closed 17 issues <https://github.com/shadow/shadow/issues?q=closed%3A%3E%3D2021-09-30+is%3Ais…> since our previous update. Release status We have released Shadow 2.0 <https://github.com/shadow/shadow/releases/tag/v2.0.0>! Give it a try, and let us know if you run into any issues! We have begun work on Shadow 2.1 <https://github.com/shadow/shadow/projects/5>. Notable planned features are support for signals (which is needed <https://github.com/shadow/shadow/issues/1549> to reliably run golang programs under Shadow), and improved Unix socket support. Notable change since last update Emulation accuracy * Add unnamed unix sockets created with socketpair() <https://github.com/shadow/shadow/pull/1787> * Add support for the 'O_DIRECT' flag to pipes (packet mode) <https://github.com/shadow/shadow/pull/1746> * poll and epoll: wait for timeout even with no fd's <https://github.com/shadow/shadow/pull/1697> Performance improvements * Set default min runahead to 1ms <https://github.com/shadow/shadow/pull/1711>. This fixes a performance regression <https://github.com/shadow/shadow/issues/1699> in many-host simulations using multiple CPUs. Stability * Detect and handle death by signals <https://github.com/shadow/shadow/pull/1722> * Handle process exit while it has events scheduled <https://github.com/shadow/shadow/pull/1704> UI improvements * Move host heartbeat options to the experimental options, and disable by default <https://github.com/shadow/shadow/pull/1703> Shadow at Tor We have been collaborating with The Tor Project to use Shadow in its development and testing. We have been running Shadow simulations inside a Gitlab CI pipeline <https://gitlab.torproject.org/jnewsome/sponsor-61-sims> to help develop and tune improved congestion control algorithms in the upcoming 0.4.7 <https://lists.torproject.org/pipermail/tor-project/2021-December/003241.html> release. This is the first major application of Shadow inside the Tor Project itself, and we plan to use the pipeline we've developed for further testing and profiling. We have also been making progress on running Arti (the experimental new Rust implemenation of tor) under Shadow: https://gitlab.torproject.org/tpo/core/arti/-/issues/174. In the course of this work we have fixed some bugs in Arti, fixed an upstream bug in the async-io <https://github.com/smol-rs/async-io/pull/73> crate, and found and identified and fixed several emulation accuracy bugs in Shadow. Happy simulating! The Shadow team

1 0

Cecylia's Monthly Status Report, March 2022
by Cecylia Bocovich 05 Apr '22

05 Apr '22

Hi everyone, This is my status report for March 2022. My work falls into three main areas: 1. Progress on Conjure pluggable transport I have begun adapting Conjure[0] to a Tor pluggable transport, using their Go API[1]. The work in progress client is so far mostly a stub PT and lives on Gitlab[2]. I have also been working on setting up a robust local testing environment to run the Conjure refraction networking station[3] that the registration with Conjure station works as expected. 2. On-boarding and knowledge transfer for sponsor 28 deliverables Since Tor Project hired a new developer to take over the remaining sponsor 28 work, I've been documenting and helping transfer knowledge of the unique sponsor 28 requirements and evaluation environments. 3. Snowflake simulations in Shadow Not much progress on this during the last month. I spent a very small amount of time helping the Shadow team test out fixes for Go simulations in Shadow [4]. Thanks, Cecylia [0] https://dl.acm.org/doi/abs/10.1145/3319535.3363218 [1] https://pkg.go.dev/github.com/refraction-networking/gotapdance/tapdance [2] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conj… [3] https://github.com/refraction-networking/conjure [4] https://github.com/shadow/shadow/issues/1549

1 0

Rhatto's Monthly Status Report, March 2022
by rhatto 05 Apr '22

05 Apr '22

Hi all :) This is my monthly status report for February 2022. Main activities during the period: 0. Created the Onionprobe tool for testing and monitoring Onion Services: https://gitlab.torproject.org/tpo/onion-services/onionprobe It has basic functionality, including metrics exporting to Prometheus, but still needs work to address more general use cases: https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/blob/main/TOD… A summary about what it currently does was made during the last Demo Day: https://gitlab.torproject.org/rhatto/onionprobe/-/blob/main/docs/demoday.md Since that, Onionprobe got additional metrics and it's first refactor :) Now I'm interested to know what the community would like to have in such a tool! Some special reporting output? Some auditing like Onionscan does? Please let me know :) 1. Completed the initial version of Oniongroove specs for deployment and monitoring of Onion Service sites. Current specification is available at https://gitlab.torproject.org/tpo/onion-services/oniongroove/-/blob/feature… It's is about to be reviewed and merged in the main branch: https://gitlab.torproject.org/tpo/onion-services/oniongroove/-/merge_reques… Right now the main intention with this document is to make sure we're in the right path towards an automated high availability deployment/monitoring suite, making acceptable choices to ease the handling of all complexities involved in hosting Onion Services. So comments and suggestions would be gladly welcomed :) 2. Started research/summarization for existing proposals on Onion Service usability improvements, to be published soon. Related ticket: https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/64 3. Many other activities related to Sponsor 123 such as meetings, plannings and support :) -- Silvio Rhatto pronouns he/him

1 0

Network Team - Upcoming tor.git 0.4.7.6-rc release - April 2022
by David Goulet 05 Apr '22

05 Apr '22

Greetings, Network team will be releasing on Thursday (April 7th, 2022) the first release candidate of the 0.4.7.x series! Yes! Finally! Congestion control is arriving to a relay near you! We'll be stabilizing as much as we can in the coming weeks. There might be another release candidate in case of major bugfixes or not. We are quite hopeful to have the first stable before the end of the month! Upcoming versions: - 0.4.7.6-rc @network-team: It is _now_ a good time to start reviewing changes/ files: https://gitlab.torproject.org/tpo/core/tor/-/tree/main/changes Last, we've asked the dirauth to recommend this version few minutes ago. Cheers! David -- tMKneCFiLexBFETizMy77pmc72pSH0TMqJMvr/mbpx4=

1 0

Nina's Monthly Status Report, March 2022
by nina13＠riseup.net 04 Apr '22

04 Apr '22

Hi, this is my March Report. Overall, I resolved 2041 user support tickets in March: 518 requests were answered by email, and 1523 using Telegram. Most of the requests were bridge requests from the users in Russia and troubleshooting around that. There are also many questions on how to download the Tor Browser as the main domain of the Tor Project is blocked in Russia. There were also several cases of the Android App bug and a growing amount of requests about using bridges in Tails. I've summarized my experience of using cdr.link for tech support: https://gitlab.torproject.org/tpo/community/support/-/issues/40059#note_278… I also did some translations for the forums to help Russian-speaking users to stay in touch with our latest news and updates: https://ntc.party/t/in-case-snowflake-rendezvous-gets-blocked/1857 https://forum.torproject.net/t/tor-browser-11-0-7/2484 This month I also added one more support template article in Russian with the basic information on software signature. I got several questions from the users about it and see a growing tendency to download software from unsafe sources. Thanks, Nina

1 0

minutes from the TPA meeting
by Antoine Beaupré 04 Apr '22

04 Apr '22

Hi! Here's your monthly digest of TPA minutiae. # Roll call: who's there and emergencies First few minutes of the meeting were spent dealing with blocking issues with office.com, which ultimately led to disabling sender verification. See [tpo/tpa/team#40627][] for details. [tpo/tpa/team#40627]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40627 Present: anarcat, gaba, kez, lavamind, linus. # Roadmap / OKR review We reviewed our two roadmaps: * [sysadmin OKRs][] * [web OKRs][] [sysadmin OKRs]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2022 [web OKRs]: https://gitlab.torproject.org/tpo/web/team/-/wikis/roadmap/2022 ## TPA OKRs We didn't do much in the TPA roadmap, unfortunately. Hopefully this week will get us started with the bullseye upgrades, and some initiatives have been *started* but it looks like we will probably not fullfill most (let alone all) of our objectives for the roadmap inside TPA. ## web OKRs More progress was done on the web side of things: * donate: lektor frontend needs to be cleaned up, some of the settings are still set in react instead of with lektor's `contents.lr`. Vanilla JS rewrite mostly complete, possibly enough that the rest can be outsourced. Still no .onion since production is running the react version (doesn't run in tbb) and .onion might also break on the backend. We also don't have an HTTPS certificate for the backend! * translators: good progress on this front, build time blocking on the i18n plugin status ([TPA-RFC-16][]), stuck on Python 3.8 land, we are also going to make changes to the workflow to allow developers to merge MRs (but not push) * documentation: removed some of the old docs, dev.tpo for Q2? The TPA-RFC-16 proposal (rewriting the lektor-i18n plugin) was discussed a little more in depth. We will get more details about the problems kez found with the other CMSes and a rough comparison of the time that would be required to migrate to another CMS vs rewriting the plugin. See [tpo/web/team#28][] for details. [TPA-RFC-16]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-16-replac… [tpo/web/team#28]: https://gitlab.torproject.org/tpo/web/team/-/issues/28 # Dashboard review * https://gitlab.torproject.org/tpo/tpa/team/-/boards/117 * https://gitlab.torproject.org/groups/tpo/web/-/boards * https://gitlab.torproject.org/groups/tpo/tpa/-/boards Skipped for lack of time # Holidays Skipped for lack of time # Other discussions Skipped for lack of time # Next meeting May 2nd, same time. We should discuss phase 3 of bullseye upgrades next meeting, so that we can make a decision about the stickiest problems like Icinga 2 vs Prometheus, Schleuder, Mailman, Puppet 6/7, etc. # Metrics of the month * hosts in Puppet: 91, LDAP: 91, Prometheus exporters: 149 * number of Apache servers monitored: 26, hits per second: 314 * number of self-hosted nameservers: 6, mail servers: 8 * pending upgrades: 1, reboots: 23 * average load: 3.62, memory available: 4.58 TiB/5.70 TiB, running processes: 749 * disk free/total: 29.72 TiB/85.33 TiB * bytes sent: 382.46 MB/s, received: 244.51 MB/s * planned bullseye upgrades completion date: 2025-01-30 * [GitLab tickets][]: 185 tickets including... * open: 0 * icebox: 157 * backlog: 12 * next: 8 * doing: 5 * needs review: 1 * needs information: 2 * (closed: 2680) [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards Upgrade prediction graph lives at https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/ Now also available as the main Grafana dashboard. Head to <https://grafana.torproject.org/>, change the time period to 30 days, and wait a while for results to render. -- Antoine Beaupré torproject.org system administration

1 0

More resources required for Snowflake bridge
by David Fifield 04 Apr '22

04 Apr '22

It has been great to see all the support and encouragement for people running Snowflake proxies. Thank you! But there is a problem: the Snowflake bridge (which all the temporary proxies forward their traffic to) is going as fast as can on its current hardware. The server is running close to 100% on all CPUs more or less constantly. As more people use Snowflake, they each get a smaller share of the limited available performance. The limited capacity of the bridge is the cause of the [recent slowness of Snowflake](https://www.reddit.com/r/TOR/comments/t49i14)—in the past 2 weeks it's gone [from 12,000 to 16,000 users, without a proportional increase in bandwidth](https://metrics.torproject.org/rs.html#details/5481936581E23D2D1… We've spent significant engineering resources already to make the most of the hardware, such as [load balancing multiple tor instances](https://github.com/net4people/bbs/issues/103) since a few weeks ago. This effort has roughly doubled the available bandwidth of the bridge, but it's still not enough. Demand will only continue to rise. The bridge needs to be moved to faster hardware. Its current hosting is free of charge, but is already on the highest-spec VPS configuration (8 CPUs, 16 GB). Switching to a server with, say, double the CPUs will have an immediate positive effect: the proof of that is that while we were installing the load balancing on the main bridge, I paid for an only slightly higher-spec server to handle Snowflake traffic during the upgrade, and during that week the bandwidth [immediately rose to higher than where it is now](https://github.com/net4people/bbs/issues/103#issuecomment-1033067920). I used Snowflake a lot during that week, and the difference was palpable. The minimum server required has something like 16 CPUs and 32 GB of RAM. meskio found some suitable [dedicated servers for about $200/month](https://lists.torproject.org/pipermail/anti-censorship-team/2022-February/000220.html) with unlimited bandwidth. (I estimate current needs are something like [100 TB/month of bandwidth](https://bugs.torproject.org/tpo/anti-censorship/pluggable-transp…, of course expected to grow.) I'm writing this to make people aware that the current cause of poor Snowflake performance is known: it's limited CPU capacity at the bridge, not general Tor slowness or slowness of the temporary proxies. Solving the problem will cost a few hundred dollars per month, at least for the near future. I am open to suggestions about what to do. I promised myself I would not again get in the situation of paying out of pocket for important infrastructure. I've already contacted the Open Technology Fund about a possible rapid response grant, but have not gotten a response yet. I'm willing to continue administering the bridge, as I do now. * Since 2022-02-03, Tor Metrics graphs for the Snowflake bridge are 1/4 what they should be, until the fix for https://bugs.torproject.org/tpo/network-health/metrics/onionoo/40022 is deployed.

5 7

metrics releases
by Silvia/Hiro 01 Apr '22

01 Apr '22

Hi, We have released new versions of collector, onionoo and the metrics website this week. You might have noticed the changes already if you use these systems (changelog below). In collector we stopped sanitizing contact info for bridges (https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/issues…). In onionoo and the website you will notice some bridges might have a new parameter, called blocklist. Some context: bridgedb has a mechanism to not distribute certain bridges in certain country/location. If we find a bridge has been blocked in a country we can use this mechanism to not distribute it in that location. We are now exposing this information in the bridge page on relay-search. The blocklist is also exposed in the bridge-pool-assignments, as: 02l463ff20f55qyddec130419b7t3y98de0f7c18 https transport=obfs4 ip=4 blocklist=ru If some or any of your bridges do not have the blocklist parameter this means your bridge is not being blocked, i.e. is being served to all locations. Related tickets: https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40036 https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/issues… Changelog Collector version 1.18.2 - 2022-03-31 * Medium changes - Stop sanitizing contact info for bridges. Source: https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/tags/c… Archive: https://dist.torproject.org/collector/1.18.2/ Onionoo version 8.3-1.30.0 - 2022-03-31 * Medium changes: - Add support for the blocklist param in the bridge poll assignment from bridgedb. * Minor changes: - Point to network-health list instead of metrics-team Source: https://gitlab.torproject.org/tpo/network-health/metrics/onionoo/-/tags/oni… Archive: https://dist.torproject.org/onionoo/8.3-1.30.0/ Metrics-web version 1.4.5 - 2022-03-31 * Medium changes: - Add the blocklist param to the relay.js model - Show when bridge support ipv6 on relay search - Add stacked graph for V3 network fractions and V2 network fractions * Minor changes: - Replace Burma country name with Myanmar Source: https://gitlab.torproject.org/tpo/network-health/metrics/website/-/tags/web… Archive: https://dist.torproject.org/metrics-web/1.4.5/ Thanks, hiro

1 0

PieroV's Monthly Status Report, March 2022
by Pier Angelo Vendrame 01 Apr '22

01 Apr '22

Hi everyone! Here is my status report for March 2022. This month we released several versions! 11.0.7, 11.0.9, 11.5a6, 11.5a8 for desktop, and now we have just started to prepare 11.0.10. I have helped review the rebases, and I have been the second builder for some of them. Some of the fixes they include are partially my doing 🙂️: * I have fixed a small issue that broke Onion services authentication in 11.0.6 (tor-browser#40802); * I have added a patch to keep the URL fragment/hash on onion-location redirects (tor-browser#34366); * I have completed the patch to show IPv6 addresses in the circuit display that we initially received by @illia-v from our community (tor-browser#14939). For Android, I have tried to help with the crash we have in 11.0.8. But eventually, we started working on the update from Firefox 96 to 99. I have rebased Geckoview (tor-browser#40857), reviewed other rebases, and helped solve some build problems (all the details are on tor-browser-build#40446). Then, I continued with the work from the previous months. I have fixed some small details in the about:torconnect and about:preferences#connection pages. I also presented them at our latest demo day 🥳️. Yesterday I started working on the new UI to show configured bridges, and I will continue it in the first days of April. As anticipated last month, I also continued working on the HTTPS-Everywhere replacement and the TBB tests. In particular, I have added a test to monitor our font fingerprint and make sure it does not change between versions. Thank you all! Cheers, Pier

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

tor-project April 2022