Hi folks,
Micah and I did a session teaching people about Snowflake a few weeks
back, and here are the notes that I wrote up for ourselves going into
that session, in case they are useful to anybody in the future.
(It was more of a decentralized peer-to-peer session so we don't have
anything so organized as a slide deck.)
It's separated into five main sections, with the goal that you can adapt
how much time you spend on each section depending on the audience.
Snowflake topics, 2023-05-15:
(1) History and architecture
Born out of the Flashproxy idea (PETS 2012), but Flashproxy didn't do nat
piercing and didn't do webrtc
<describe intuition around design>
The basic idea is to handle DPI attacks by looking enough like WebRTC,
and to handle IP enumeration attacks by having so many diverse and changing
addresses that blocking them all is pointless.
Started with Serene Han's OTF fellowship, then leaped forward again when
we made an anti-censorship team and had Cecylia on it
multiple components (more client interactions, more surface area for problems)
- stun server for client to learn about connection info
- domain front to reach broker
- webrtc channel to volunteer proxy
early engineering tradeoff: go-pion vs chrome's libwebrtc
prateek at princeton wrote a short paper guessing some distinguishers
that attackers could use. some of them turned out to be good guesses!
used to use azure for the domain front, now use fastly
Scaling to multiple snowflake back-end servers
one in sweden one at umich
now pushing >25terabytes/day of traffic
(2) Real-world events
- Russia censorship in Dec 2021
- spike in users, but block by webrtc DPI
- Iran censorship in Oct 2022
- spike in users, but block by domain front https
- #1 app in google play store, and also #2 app, for that day
- China, works but sometimes quite slow
- packet loss over the GFW leads to poor webrtc performance?
ooni tests for snowflake, showing it working in most places including china
(3) Ways to be a Snowflake volunteer
- browser extension for Firefox, Chrome
- and edge now?
- headless go proxy for server people
- Orbot has a 'kindness mode'
- Brave ships the extension as of January
- Mullvad considering something similar
NAT piercing, kinds of volunteers
>100k volunteers but the vast majority of them are the less useful kind
gamifying the snowflake badge
(4) developer side
iptproxy wrapper in-process library for ios and android
The Orbot 17.0 prerelease offers a better experience, since it is the
first Orbot version to use snowflake-02
https://github.com/guardianproject/orbot/releases/tag/17.0.0-RC-1-tor.0.4.7…
reliability layers:
- turbotunnel
- conflux
- striping over multiple snowflakes
data channel vs media webrtc channel -- impacts both realism and reliability
domain fronting isn't the only signaling channel we could use
orbot ships with google amp cache as an alternative channel
we could also use dnstt or other ideas in the future
surprises, e.g. people in censored countries becoming snowflake volunteers
(5) future work, funding, community
Reusing our Snowflake volunteer pool for other projects?
Too risky because whichever project draws the most attention blocks
the snowflake pool for everybody else
Besides, you really (should) want the distributed-trust features of Tor,
to keep your users and your Snowflakes safe.
how to handle enumeration attacks on the broker?
tension between really simple broker and handling more attacks
future work: realistic behavior layer a la Raven
future work: better user counting
We currently suffer from the same user (under)counting issues as Tor
Funding groups to run Snowflake volunteers?
part of otf rapid response plan for iran, short term
maybe future plans to fund groups too?
but centralizing our snowflakes kind of defeats the point
Other ideas for growing the community, e.g. integrating Snowflake into
your project
Hello everyone,
Monday 29th is an official holiday so we'll be moving our weekly meeting to the
30th at 1515 (not 1500) UTC in #tor-meeting on OFTC IRC.
best,
-Richard
Hello everyone!
Here are updates from the user support team for last month (April,
2023). Most of our work has been around helping users in countries where
Tor is censored and some user support work in light of the new stable
Tor Browser release (Tor Browser 12.0.5).
With the release of Mullvad Browser[0], we also published some
documentation for our Support Portal[1].
Timeframe: 01 - 30 April 2023
# Frontdesk (email)
-584 RT tickets created
-428 RT tickets resolved
Most frequent tickets by numbers:
1. 158 RT tickets: Private Bridge requests from China.
2. 39 RT tickets: Circumventing censorship in Russia.
3. 21 RT tickets: Circumventing censorship in Turkmenistan.
4. 4 RT tickets: Circumventing censorship with Tor in Iran.
5. 3 RT tickets: Tor Browser doesn't run with Mandatory ASLR
on Windows (the issue is fixed with the 12.0.5 Tor Browser release)[2]
# Telegram, WhatsApp and Signal Support channel
-751 tickets resolved
Breakdown:
-697 tickets on Telegram
-42 tickets on WhatsApp
-12 tickets on Signal
The most frequent tickets on cdr.link have been about:
1. 209 tickets: Circumventing censorship in Russia.
2. 137 tickets: Circumventing censorship in Turkmenistan.
3. 77 tickets: Circumventing censorship in Iran.
4. 39 tickets: Circumventing censorship in China.
Thanks!
e.
[0]: https://blog.torproject.org/releasing-mullvad-browser/
[1]: https://support.torproject.org/mullvad-browser/
[2]: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/4…
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2023/tor-meeting.2023-05-11-15.58.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
------------------------------------------------------------------------------------
THIS IS A
PUBLIC PAD
------------------------------------------------------------------------------------
Anti-censorship
--------------------------------
Next meeting: Thursday, May 18 16:00 UTC
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
== Goal of this meeting ==
Weekly check-in about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the
Tor Project and Tor community.
== Links to Useful documents ==
* Our anti-censorship roadmap:
* Roadmap:
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
* The anti-censorship team's wiki page:
*
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
* Past meeting notes can be found at:
* https://lists.torproject.org/pipermail/tor-project/
* Tickets that need reviews: from sponsors, we are working on:
* All needs review tickets:
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
* Sponsor 96 <-- meskio, shell, onyinyang, cohosh
* https://gitlab.torproject.org/groups/tpo/-/milestones/24
* Sponsor 139 <-- hackerncoder, irl, joydeep, meskio, emmapeel
working on it
* https://pad.riseup.net/p/sponsor139-meeting-pad
== Announcements ==
== Discussion ==
* Update on Analysis of speed deficiency of Snowflake in China,
2023 Q1
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
* after a lot of research the proposed solution is to enable
datagram transport on webrtc to deal with the packet loss situation
* that will convert webrtc into an unreliable channel, and
snowflake will add reliablity with kcp
* (NO update from shell @ May 11, research ongoing)
*
== Actions ==
*
== Interesting links ==
* Unofficial(?) Snowflake extension for Safari in Apple App Store?
* https://apps.apple.com/us/app/torproject-snowflake/id1597501940
* Previously noted at
https://lists.torproject.org/pipermail/anti-censorship-team/2022-February/0…
* Research about designing an armored bridge line sharing URL
format(https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/126)
*
https://opencollective.com/censorship-circumvention/projects/snowflake-dail…
== Reading group ==
* We will discuss "Lox: Protecting the Social Graph in Bridge
Distribution" on 2023 May 18
* https://cypherpunks.ca/~iang/pubs/lox-popets23.pdf
* Questions to ask and goals to have:
* What aspects of the paper are questionable?
* Are there immediate actions we can take based on this work?
* Are there long-term actions we can take based on this work?
* Is there future work that we want to call out in hopes
that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): last updated 2023-05-04
Last week:
- tor meeting
This week:
- catch up on emails
- foci stuff
- open issue about archiving snowflake prometheus metrics
- go over lox notes again from meeting
- lox-wasm tor browser builds
Needs help with:
dcf: 2023-05-11 (since 2023-04-20)
Last week:
- made a merge request to add an error check to the
distinctcounter program
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- opened an issue to upgrade tor on snowflake-01
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- posted April 2023 snowflake-01 update
https://opencollective.com/censorship-circumvention/projects/snowflake-dail…
- posted a history of pluggable transport placeholder IP
addresses
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webt…
Next week:
- upgrade tor on snowflake-01
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- open issue to have snowflake-client log whenever KCPInErrors
is nonzero
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- parent:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- open issue to disable /debug endpoint on snowflake broker
Help with:
meskio: 2023-04-20
Last week:
- update PTs to use goptlib from gitlab.tpo
- distribute bridges in rdsys even if there fewer than
requested in the hashring (rdsys#162)
- add webtunnel support to BridgeDB (rdsys#142)
Next week:
- tormeeting
Shelikhoo: 2023-05-11
Last Week:
- [Merge Request Awaiting] Add SOCKS5 forward proxy support to
snowflake (snowflake!64)
- [Research] HTTPT Planning
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/http…
- Research about designing an armored bridge line sharing URL
format(https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/126)
- Snowflake Performance Analysis(Ongoing)
Next Week/TODO:
- [Research] WebTunnel planning (Continue)
- Try to find a place to host another vantage point
- logcollector alert system
- webtunnel document for proxy operator
- Snowflake Performance Analysis
onyinyang: 2023-05-11
Last week:
- finished up implementing metrics to check on flickering
resources and ratios observed (maybe) awaiting review
- Worked through changes to handle `gone` resources. rdsys changes
are tentatively implemented, Lox library changes are more hairy.
This week:
- Finished up the Lox library changes
- Added additional changes to Lox distributor
- Refactoring lox-distributor for readability
- Adding tests for both Lox library and Lox distributor
- Looking into a more reasonable way of storing Lox library
data structures:
- https://gitlab.torproject.org/onyinyang/lox/-/issues/2
- https://gitlab.torproject.org/onyinyang/lox/-/issues/3
(long term things were discussed at the meeting!):
https://pad.riseup.net/p/tor-ac-community-azaleas-room-keep
- brainstorming grouping strategies for Lox buckets (of
bridges) and gathering context on how types of bridges are
distributed/use in practice.
Question: What makes a bridge useable for a given user, and
how can we encode that to best ensure we're getting the most appropriate
resources to people?
1. Are there some obvious grouping strategies that we
can already consider?
e.g., by pt, by bandwidth (lower bandwidth bridges
sacrified to open-invitation buckets?), by locale (to be matched with a
requesting user's geoip or something?)
2. Does it make sense to group 3 bridges/bucket, so
trusted users have access to 3 bridges (and untrusted users have access
to 1)? More? Less?
Needs Help with:
- figuring out whether or not the metrics I added to rdsys
actually collect what we want them to. I can run prometheus locally but
am unsure how to match this with a realistic onbasca test that can
actually show whether the metrics are useful/correct. Is there a known
way to do such tests other than deploy and find out?
Itchy Onion: 2023-05-11
Last week:
- Work on team#112 (Some bridges are reported offline but are
online and working)
This week:
- Continue investigating offline bridges (team#112)
- Discovered bridgestrap#37 (cache gives wrong status of bridge
sometimes)
- Start working on rdsys#56 (persistent storage for certain bridge
arributes)
hackerncoder: 2023-04-20
last week:
- (py-)ooni-exporter torsf (snowflake)
- (py-)ooni-exporter web_connectivity
Next week:
- work on "bridgetester"?
- how does Iran block bridges?
Hello everyone!
Long time no see! Here's your usual dose of sysadmin minutes, sorry for
the late mail, we skipped a few...
# Roll call: who's there and emergencies
anarcat, gaba, kez, lavamind, no emergency apart from CiviCRM hogging
a CPU but that has been happening for the last month or so
# Dashboard review
We went through our normal per-user, weekly, check-in:
* https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…
* https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…
* https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&…
We do not go through the general dashboards anymore as those are done
in triage (by the star of the week for TPA, with gaba and anarcat for
web).
# Q2 prioritisation
We looked at the coming deliverables, mostly on the web side of things:
- developer portal
- repo: force-push new HUGO site into https://gitlab.torproject.org/tpo/web/dev
- staging: use pages for it until build pipeline is ready
- triage/clean issues in web/dev (gaba)
- edit/curate content (gaba)
- review by TPO
- send to production (maybe Q4 2023)
- donation page (next project meeting is on May 17th) ~ kez working on it
- self-host forum ~ wrapping up by the end of June
- download page when ux team is done with it
We also looked at the [TPA milestones][].
Out of those milestones, we hope for the [gnt-dal migration][] to be
completed shortly. It's technically done, but there's still a bunch of
cleanup work to be completed to close the milestone compeltely.
Another item we want to start completing but that has a lot of
collateral is the bullseye upgrade, as that includes upgrading Puppet,
LDAP (!), Mailman (!!), possibly replacing Nagios, and so on.
Anarcat also wants to push the gitolite retirement forward as that has
been discussed in Costa Rican corridors and there's momentum on this
now that a set of rewrite rules has been built...
[gnt-dal migration]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/2
[TPA milestones]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones
# Holidays planning
We reviewed the summer schedule to make sure everything is up to date
and there is not too much overlap.
# Metrics of the month
* hosts in Puppet: 85, LDAP: 86, Prometheus exporters: 155
* number of Apache servers monitored: 33, hits per second: 658
* number of self-hosted nameservers: 6, mail servers: 9
* pending upgrades: 0, reboots: 2
* average load: 1.17, memory available: 3.31 TiB/4.45 TiB, running
processes: 580
* disk free/total: 35.92 TiB/105.25 TiB
* bytes sent: 306.33 MB/s, received: 198.85 MB/s
* planned bullseye upgrades completion date: 2023-01-21 (!)
* [GitLab tickets][]: 192 tickets including...
* open: 0
* icebox: 143
* backlog: 22
* next: 16
* doing: 6
* needs information: 4
* needs review: 1
* (closed: 3121)
[Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards
Upgrade prediction graph lives at:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/
Note that we're late in the bullseye upgrade procedure, but for the
first time in months we've had significant progress with the
retirement of a bunch of machines and rebuilding of existing
ones.
We're also starting to deploy our first bookworm machines now,
although that is done only on a need-to basis as we can't actually
*install* bookworm machines yet: they need to be installed with
bullseye to get Puppet boostrapped and then we immediately upgrade to
bookworm.
A more detailed post-mortem of the upgrade process is under discussion in the wiki:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye#…
--
Antoine Beaupré
torproject.org system administration
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2023/tor-meeting.2023-05-04-15.59.log…
And our meeting pad:
------------------------------------------------------------------------------------
- THIS IS A PUBLIC PAD
------------------------------------------------------------------------------------
Anti-censorship
--------------------------------
Next meeting: Thursday, May 11 16:00 UTC
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
== Goal of this meeting ==
Weekly check-in about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the Tor Project and Tor community.
== Links to Useful documents ==
- Our anti-censorship roadmap:
- Roadmap: https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
- The anti-censorship team's wiki page:
- https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
- Past meeting notes can be found at:
- https://lists.torproject.org/pipermail/tor-project/
- Tickets that need reviews: from sponsors, we are working on:
- All needs review tickets:
- https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
- Sponsor 96
- https://gitlab.torproject.org/groups/tpo/-/milestones/24
- Sponsor 139 <-- hackerncoder, irl, joydeep, meskio, emmapeel working on it
- https://pad.riseup.net/p/sponsor139-meeting-pad
== Announcements ==
== Discussion ==
- Update on Analysis of speed deficiency of Snowflake in China, 2023 Q1 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- after a lot of research the proposed solution is to enable datagram transport on webrtc to deal with the packet loss situation
- that will convert webrtc into an unreliable channel, and snowflake will add reliablity with kcp
- (NO update from shell @ Apr 20)
- goptlib now lives in gitlab.torproject.org
== Actions ==
- read the safari snowflake extension code (team)
- try to reach the developer (itchyonion will write an email draft, and the team will go over it)
== Interesting links ==
- https://guardianproject.info/2023/03/04/arti-next-gen-tor-on-mobile/
- "Support for features like advanced censorship circumvention or onion services is not exactly straight forward on mobile operating systems, because they tend to be way more locked down than traditional computers. Currently, we can successfully test pluggable transports in 'managed' mode on old versions of Android. However this technique will likely not work on the latest version of Android and never worked iOS to begin with. We have shared our findings with the Arti developer team and hope they’ll work on getting us to full Pluggable Transports support, integraing with our existing IPtProxy Library soon."
- Unofficial(?) Snowflake extension for Safari in Apple App Store?
- https://apps.apple.com/us/app/torproject-snowflake/id1597501940
- Previously noted at https://lists.torproject.org/pipermail/anti-censorship-team/2022-February/0…
== Reading group ==
- We will discuss "Lox: Protecting the Social Graph in Bridge Distribution" on 2023 May 18
- https://cypherpunks.ca/~iang/pubs/lox-popets23.pdf
- Questions to ask and goals to have:
- What aspects of the paper are questionable?
- Are there immediate actions we can take based on this work?
- Are there long-term actions we can take based on this work?
- Is there future work that we want to call out in hopes that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- - Something you need help with.
cecylia (cohosh): last updated 2023-05-04
Last week:
- tor meeting
This week:
- catch up on emails
- foci stuff
- open issue about archiving snowflake prometheus metrics
- go over lox notes again from meeting
- lox-wasm tor browser builds
Needs help with:
dcf: 2023-04-20
- Last week:
- - did a haproxy security upgrade on snowflake-01 and snowflake-01 bridges https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- - moved goptlib from git.torproject.org to gitlab.torproject.orghttps://lists.torproject.org/pipermail/tor-dev/2023-April/014829.html
- - analyzed the rate of client_ip reporting since the release of snowflake-webext-0.7.2 https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- Next week:
- - open issue to have snowflake-client log whenever KCPInErrors is nonzero https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- - parent: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- - open issue to disable /debug endpoint on snowflake broker
- Help with:
meskio: 2023-04-20
Last week:
- - update PTs to use goptlib from gitlab.tpo
- - distribute bridges in rdsys even if there fewer than requested in the hashring (rdsys#162)
- - add webtunnel support to BridgeDB (rdsys#142)
Next week:
- - tormeeting
Shelikhoo: 2023-05-04
Last Week:
- - [Merge Request Awaiting] Add SOCKS5 forward proxy support to snowflake (snowflake!64)
- - [Research] HTTPT Planning https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/http…
- - Finish all the accumulated task during in person meetup AFK
Next Week/TODO:
- - [Research] WebTunnel planning (Continue)
- - Try to find a place to host another vantage point
- - logcollector alert system
- - webtunnel document for proxy operator
- - Snowflake Performance Analysis
-
onyinyang: 2023-05-04
Last week:
- Tor meeting
This week:
- - finished up implementing metrics to check on flickering resources and ratios observed (maybe) awaiting review
- Working through changes to handle `gone` resources. rdsys changes are tentatively implemented, Lox library changes are more hairy.
- (long term things were discussed at the meeting!):
- https://pad.riseup.net/p/tor-ac-community-azaleas-room-keep
- - brainstorming grouping strategies for Lox buckets (of bridges) and gathering context on how types of bridges are distributed/use in practice.
- Question: What makes a bridge useable for a given user, and how can we encode that to best ensure we're getting the most appropriate resources to people?
- 1. Are there some obvious grouping strategies that we can already consider?
- e.g., by pt, by bandwidth (lower bandwidth bridges sacrified to open-invitation buckets?), by locale (to be matched with a requesting user's geoip or something?)
- 2. Does it make sense to group 3 bridges/bucket, so trusted users have access to 3 bridges (and untrusted users have access to 1)? More? Less?
Needs Help with:
- - figuring out whether or not the metrics I added to rdsys actually collect what we want them to. I can run prometheus locally but am unsure how to match this with a realistic onbasca test that can actually show whether the metrics are useful/correct. Is there a known way to do such tests other than deploy and find out?
Itchy Onion: 2023-05-04
Last week:
- - Costa Rica
This week:
- https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/112 (Some bridges are reported offline but are online and working)
- - consulted network health team to understand what "offline" means from the Metrics's POV (and discovered a small wording inconsistency in their doc)
- - better understood the difference between "offline" and working (they are not mutually exclusive)
-
hackerncoder: 2023-04-20
last week:
- (py-)ooni-exporter torsf (snowflake)
- (py-)ooni-exporter web_connectivity
Next week:
- work on "bridgetester"?
- how does Iran block bridges?
Hi! Below is my report for April 2023.
In April, I resolved 759 tickets, which is 200 more than in March:
On Telegram (@TorProjectSupportBot) - 556
On RT (frontdesk@tpo) - 203
Additionally, I also resolved some tickets on WhatsApp (+447421000612)
and Signal (+17787431312).
The majority of the tickets I deal with are related to censorship
circumvention and issues that happen around it, like using VPN and
bridges simultaneously or incorrect use of the bridges.
Another often issue I deal with is Tor Browser and antiviruses.
During the internet shutdown in Kazakhstan, I helped with the
translation of the Twitter post related to the situation [1]
I also communicated with users a lot, gathering feedback on different
pluggable transports and the work of our services.
[1] https://twitter.com/torproject/status/1645868260624306176