Hi,
So TPA has adopted this proposal, internally, to make yet another set of
emergency changes to our mail system, to respond to critical issues
affecting delivery and sustainability of our infrastructure.
I encourage you to read the "Affected users" section and "Timeline"
below. In particular, we will be experimenting with "sender rewriting"
soon, which will involve mangling emails we forward around to try and
fix deliverability on those.
The schleuder mailing list will also move servers.
Maintenance windows for those changes will be communicated separately.
Thank you for your attention!
PS: and no, we didn't submit this for adoption to everyone, because it
was felt it was mostly technical changes that didn't warrant outside
approval, let me know if that doesn't make sense, of course.
--
Antoine Beaupré
torproject.org system administration
---
title: TPA-RFC-71: Emergency email deployments, phase B
costs: staff
approval: TPA
affected users: all torproject.org email users
deadline: 5 days, 2024-10-01
status: draft
discussion: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41778
---
Summary: deploy a new sender-rewriting mail forwarder ASAP, migrate
mailing lists off the legacy server to a new machine, migrate the
remaining Schleuder list to the Tails server, upgrade `eugeni`.
Table of contents:
- Background
- Proposal
- Actual changes
- Mailman 3 upgrade
- New sender-rewriting mail exchanger
- Schleuder migration
- Upgrade legacy mail server
- Goals
- Must have
- Nice to have
- Non-Goals
- Scope
- Affected users
- Personas
- Timeline
- Optimistic timeline
- Worst case scenario
- Alternatives considered
- References
- History
- Personas descriptions
- Ariel, the fundraiser
- Blipblop, the bot
- Gary, the support guy
- John, the contractor
- Mallory, the director
- Nancy, the fancy sysadmin
- Orpheus, the developer
# Background
In [#41773][], we had yet another report of issues with mail delivery,
particularly with email forwards, that are plaguing Gmail-backed
aliases like grants@ and travel@.
[#41773]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41773
This is becoming critical. It has been impeding people's capacity of
using their email at work for a while, but it's been more acute since
google's recent changes in email validation (see [#41399][]) as now
hosts that have adopted the SPF/DKIM rules are bouncing.
[#41399]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41399
On top of that, we're way behind on our buster upgrade schedule. We
still have to upgrade our primary mail server, `eugeni`. The plan for
that ([TPA-RFC-45][], [#41009][]) was to basically re-architecture
everything. That won't happen fast enough for the LTS retirement which
we have crossed two months ago (in July 2024) already.
[TPA-RFC-45]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-45-mail-a…
[#41009]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41009
So, in essence, our main mail server is unsupported now, and we need
to fix this as soon as possible
Finally, we also have problems with certain servers (e.g. `state.gov`)
that seem to dislike our bespoke certificate authority (CA) which
makes *receiving* mails difficult for us.
# Proposal
So those are the main problems to fix:
- Email forwarding is broken
- Email reception is unreliable over TLS for some servers
- Mail server is out of date and hard to upgrade (mostly because of
Mailman)
## Actual changes
The proposed solution is:
- **Mailman 3 upgrade** ([#40471][])
- **New sender-rewriting mail exchanger** ([#40987][])
- **Schleuder migration**
- **Upgrade legacy mail server** ([#40694][])
[#40471]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471
[#40987]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40987
[#40694]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40694
### Mailman 3 upgrade
Build a new mailing list server to host the upgraded Mailman 3
service. Move old lists over and convert them while retaining the old
archives available for posterity.
This includes lots of URL changes and user-visible disruption, little
can be done to work around that necessary change. We'll do our best to
come up with redirections and rewrite rules, but ultimately this is a
disruptive change.
This involves yet another authentication system being rolled out, as
Mailman 3 has its own user database, just like Mailman 2. At least
it's one user per site, instead of per list, so it's a slight
improvement.
This is issue [#40471][].
### New sender-rewriting mail exchanger
This step is carried over from [TPA-RFC-45][], mostly unchanged.
[Sender Rewriting Scheme]: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme
[postsrsd]: https://github.com/roehling/postsrsd
[postforward]: https://github.com/zoni/postforward
Configure a new "mail exchanger" (MX) server with TLS certificates
signed by our normal public CA (Let's Encrypt). This replaces that
part of `eugeni`, will hopefully resolve issues with `state.gov` and
others ([#41073][], [#41287][], [#40202][], [#33413][]).
[#33413]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33413
[#40202]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40202
[#41287]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41287
[#41073]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41073
This would handle forwarding mail to other services (e.g. mailing
lists) but also end-users.
To work around reputation problems with forwards ([#40632][],
[#41524][], [#41773][]), deploy a [Sender Rewriting Scheme][] (SRS)
with [postsrsd][] (packaged in Debian, but [not in the best shape][])
and [postforward][] (not packaged in Debian, but zero-dependency
Golang program).
It's possible deploying [ARC][] headers with [OpenARC][], Fastmail's
[authentication milter][] (which [apparently works better][]), or
[rspamd's arc module][] might be sufficient as well, to be tested.
[OpenARC]: https://tracker.debian.org/pkg/openarc
[not in the best shape]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017361
Having it on a separate mail exchanger will make it easier to swap in
and out of the infrastructure if problems would occur.
The mail exchangers should also sign outgoing mail with DKIM, and
*may* start doing better validation of incoming mail.
[authentication milter]: https://github.com/fastmail/authentication_milter
[apparently works better]: https://old.reddit.com/r/postfix/comments/17bbhd2/about_arc/k5iluvn/
[rspamd's arc module]: https://rspamd.com/doc/modules/arc.html
[#41524]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41524
[#40632]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40632
[ARC]: http://arc-spec.org/
### Schleuder migration
Migrate the remaining mailing list left (the Community Council) to the
Tails Shleuder server, retiring our Schleuder server entirely.
This requires configuring the Tails server to accept mail for
`(a)torproject.org`.
Note that this may require changing the addresses of the existing
Tails list to `(a)torproject.org` if Schleuder doesn't support virtual
hosting (which is likely).
### Upgrade legacy mail server
Once Mailman has been safely moved aside and is shown to be working
correctly, upgrade Eugeni using the normal procedures. This should be
a less disruptive upgrade, but is still risky because it's such an old
box with lots of legacy.
One key idea of this proposal is to keep the legacy mail server,
`eugeni`, in place. It will continue handling the "MTA" (Mail Transfer
Agent) work, which is to relay mail for other hosts, as a legacy
system.
The full eugeni replacement is seen as too complicated and unnecessary
at this stage. The legacy server will be isolated from the rewriting
forwarder so that outgoing mail is mostly unaffected by the forwarding
changes.
## Goals
This is not an exhaustive solution to all our email problems,
[TPA-RFC-45][] is that longer-term project.
### Must have
- Up to date, supported infrastructure.
- Functional legacy email forwarding.
### Nice to have
- Improve email forward deliverability to Gmail.
### Non-Goals
- **Clean email forwarding**: email forwards *may* be mangled and
rewritten to appear as coming from `(a)torproject.org` instead of the
original address. This will be figured out at the implementation
stage.
- **Mailbox storage**: out of scope, see [TPA-RFC-45][]. It is hoped,
however, that we *eventually* are able to provide such a service, as
the sender-rewriting stuff might be too disruptive in the long run.
- **Technical debt**: we keep the legacy mail server, `eugeni`.
- **Improved monitoring**: we won't have a better view in how well we
can deliver email.
- **High availability**: the new servers will not add additional
"single point of failures", but will not improve our availability
situation (issue [#40604][])
[#40604]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40604
## Scope
This proposal affects the all inbound and outbound email services
hosted under `torproject.org`. Services hosted under `torproject.net`
are *not* affected.
It also does *not* address directly phishing and scamming attacks
([#40596][]), but it is hoped the new mail exchanger will provide
a place where it is easier to make such improvements in the future.
[#40596]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40596
## Affected users
This affects all users which interact with `torproject.org` and its
subdomains over email. It particularly affects all "tor-internal"
users, users with LDAP accounts, or forwards under `(a)torproject.org`,
as their mails will get rewritten on the way out.
### Personas
Here we collect a few "personas" and try to see how the changes will
affect them, largely derived from [TPA-RFC-45][], but without the
alpha/beta/prod test groups.
For *all* users, a common impact is that emails will be rewritten by
the sender rewriting system. As mentioned above, the impact of this
still remains to be clarified, but at least the hidden `Return-Path`
header will be changed for bounces to go to our servers.
Actual personas are in the Reference section, see [Personas
descriptions][].
| Persona | Task | Impact |
|---------|-------------|--------------------------------------------------------------------------|
| Ariel | Fundraising | Improved incoming delivery |
| Blipbot | Bot | No change |
| Gary | Support | Improved incoming delivery, new moderator account on mailing list server |
| John | Contractor | Improved incoming delivery |
| Mallory | Director | Same as Ariel |
| Nancy | Sysadmin | No change in delivery, new moderator account on mailing list server |
| Orpheus | Developer | No change in delivery |
[Personas descriptions]: #personas-descriptions
## Timeline
### Optimistic timeline
- Late September (W39): issue raised again, proposal drafted (now)
- October:
- W40: proposal approved, installing new rewriting server
- W41: rewriting server deployment, new mailman 3 server
- W42: mailman 3 mailing list conversion tests, users required for testing
- W43: mailman 2 retirement, mailman 3 in production
- W44: Schleuder mailing list migration
- November:
- W45: `eugeni` upgrade
### Worst case scenario
- Late September (W39): issue raised again, proposal drafted (now)
- October:
- W40: proposal approved, installing new rewriting server
- W41-44: difficult rewriting server deployment
- November:
- W44-W48: difficult mailman 3 mailing list conversion and testing
- December:
- W49: Schleuder mailing list migration vetoed, Schleuder stays on
`eugeni`
- W50-W51: `eugeni` upgrade postponed to 2025
- January 2025:
- W3: `eugeni` upgrade
# Alternatives considered
We decided to not just run the sender-rewriting on the legacy mail
server because too many things are tangled up in that server. It is
just too risky.
We have also decided to not upgrade Mailman in place for the same
reason: it's seen as too risky as well, because we'd first need to
upgrade the Debian base system and if that fails, rolling back is too
hard.
# References
- [discussion issue][]
[discussion issue]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41778
## History
This is the the *fifth* proposal about our email services, here are
the previous ones:
* [TPA-RFC-15: Email services][] (rejected, replaced with TPA-RFC-31)
* [TPA-RFC-31: outsource email services][] (rejected, in favor of
TPA-RFC-44 and following)
* [TPA-RFC-44: Email emergency recovery, phase A][] (standard, and
mostly implemented except the sender-rewriting)
* [TPA-RFC-45: Mail architecture][] (still draft)
[TPA-RFC-15: Email services]: policy/tpa-rfc-15-email-services
[TPA-RFC-31: outsource email services]: policy/tpa-rfc-31-outsource-email
[TPA-RFC-44: Email emergency recovery, phase A]: policy/tpa-rfc-44-email-emergency-recovery
[TPA-RFC-45: Mail architecture]: policy/tpa-rfc-45-mail-architecture
## Personas descriptions
### Ariel, the fundraiser
Ariel does a lot of mailing. From talking to fundraisers through
their normal inbox to doing mass newsletters to thousands of people on
CiviCRM, they get a lot done and make sure we have bread on the table
at the end of the month. They're awesome and we want to make them
happy.
Email is absolutely mission critical for them. Sometimes email gets
lost and that's a major problem. They frequently tell partners their
personal Gmail account address to work around those problems. Sometimes
they send individual emails through CiviCRM because it doesn't work
through Gmail!
Their email forwards to Google Mail and they now have an LDAP account
to do email delivery.
### Blipblop, the bot
Blipblop is not a real human being, it's a program that receives
mails and acts on them. It can send you a list of bridges (bridgedb),
or a copy of the Tor program (gettor), when requested. It has a
brother bot called Nagios/Icinga who also sends unsolicited mail when
things fail.
There are also bots that sends email when commits get pushed to some
secret git repositories.
### Gary, the support guy
Gary is the ticket overlord. He eats tickets for breakfast, then
files 10 more before coffee. A hundred tickets is just a normal day at
the office. Tickets come in through email, RT, Discourse, Telegram,
Snapchat and soon, TikTok dances.
Email is absolutely mission critical, but some days he wishes there
could be slightly less of it. He deals with a lot of spam, and surely
something could be done about that.
His mail forwards to Riseup and he reads his mail over Thunderbird
and sometimes webmail. Some time after TPA-RFC_44, Gary managed to
finally get an OpenPGP key setup and TPA made him a LDAP account so he
can use the submission server. He has already abandoned the Riseup
webmail for TPO-related email, since it cannot relay mail through the
submission server.
### John, the contractor
John is a freelance contractor that's really into privacy. He runs his
own relays with some cools hacks on Amazon, automatically deployed
with Terraform. He typically run his own infra in the cloud, but
for email he just got tired of fighting and moved his stuff to
Microsoft's Office 365 and Outlook.
Email is important, but not absolutely mission critical. The
submission server doesn't currently work because Outlook doesn't allow
you to add just an SMTP server. John does have an LDAP account,
however.
### Mallory, the director
Mallory also does a lot of mailing. She's on about a dozen aliases
and mailing lists from accounting to HR and other unfathomable
things. She also deals with funders, job applicants, contractors,
volunteers, and staff.
Email is absolutely mission critical for her. She often fails to
contact funders and critical partners because `state.gov` blocks our
email -- or we block theirs! Sometimes, she gets told through LinkedIn
that a job application failed, because mail bounced at Gmail.
She has an LDAP account and it forwards to Gmail. She uses Apple Mail
to read their mail.
### Nancy, the fancy sysadmin
Nancy has all the elite skills in the world. She can configure a
Postfix server with her left hand while her right hand writes the
Puppet manifest for the Dovecot authentication backend. She browses
her mail through a UUCP over SSH tunnel using mutt. She runs her own
mail server in her basement since 1996.
Email is a pain in the back and she kind of hates it, but she still
believes entitled to run their own mail server.
Her email is, of course, hosted on her own mail server, and she has
an LDAP account. She has already reconfigured her Postfix server to
relay mail through the submission servers.
### Orpheus, the developer
Orpheus doesn't particular like or dislike email, but sometimes has
to use it to talk to people instead of compilers. They sometimes have
to talk to funders (`#grantlyfe`), external researchers, teammates or
other teams, and that often happens over email. Sometimes email is
used to get important things like ticket updates from GitLab or
security disclosures from third parties.
They have an LDAP account and it forwards to their self-hosted mail
server on a OVH virtual machine. They have already reconfigured their
mail server to relay mail over SSH through the jump host, to the
surprise of the TPA team.
Email is not mission critical, and it's kind of nice when it goes
down because they can get in the zone, but it should really be working
eventually.
--
Antoine Beaupré
torproject.org system administration
--
tpa-team mailing list
tpa-team(a)lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tpa-team
Hey everyone!
Here are our meeting logs:
http://meetbot.debian.net/tor-meeting/2024/tor-meeting.2024-10-31-16.00.html
And our meeting pad:
Anti-censorship work meeting pad
--------------------------------
Anti-censorship
--------------------------------
Next meeting: Thursday, October 07 16:00 UTC
Facilitator: onyinyang
^^^(See Facilitator Queue at tail)
Weekly meetings, every Thursday at 16:00 UTC, in #tor-meeting at OFTC
(channel is logged while meetings are in progress)
This week's Facilitator: shelikhoo
== Goal of this meeting ==
Weekly check-in about the status of anti-censorship work at Tor.
Coordinate collaboration between people/teams on anti-censorship at the
Tor Project and Tor community.
== Links to Useful documents ==
* Our anti-censorship roadmap:
*
Roadmap:https://gitlab.torproject.org/groups/tpo/anti-censorship/-/boards
* The anti-censorship team's wiki page:
*
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/home
* Past meeting notes can be found at:
* https://lists.torproject.org/pipermail/tor-project/
* Tickets that need reviews: from projects, we are working on:
* All needs review tickets:
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/merge_requests?s…
* Project 158 <-- meskio working on it
*
https://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues/?label_na…
== Announcements ==
* eclips.is (Snowflake broker hosting) is switching to a partially
paid model (it has heretofore been free for users)
*
https://lists.torproject.org/pipermail/anti-censorship-team/2024-October/00…
== Discussion ==
* Snowflake broker transition still pending.
* Adding a snowflake transport to lyrebird
*
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…
* Squash PTs into Lyrebird:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyre…
* TODOs: add proxy support and event logging
* We decide is a good idea to integrate the snowflake client
into lyrebird
* we are not removing the snowflake client from the snowflake
proyect for now, we'll keep both until it becomes a burden.
* only the snowflake client will be integrated in lyrebird, not
the server
* pion/webrtc v4 release is out
* it has support to integrate with covert-dtls
* on the beta testing there were some issues, we'll need to
check if it works with the release or it needs work
* split broker into components
* there is an old MR with a WIP version of this change
* we are still interested on it
* cohosh will check with arlo if he can continue that work or
we'll close it for now
(Oct 31 New:)
== Actions ==
== Interesting links ==
*
== Reading group ==
* We will discuss "" on
*
* Questions to ask and goals to have:
* What aspects of the paper are questionable?
* Are there immediate actions we can take based on this work?
* Are there long-term actions we can take based on this work?
* Is there future work that we want to call out in hopes
that others will pick it up?
== Updates ==
Name:
This week:
- What you worked on this week.
Next week:
- What you are planning to work on next week.
Help with:
- Something you need help with.
cecylia (cohosh): 2024-10-31
Last week:
- finished adding snowflake transport to lyrebird (lyrebird!63)
- updating Ask Tor domain front for Orbot
- added a follow up for snowflake proxy support in lyrebird
(lyrebird!64)
- discussed Lox integration with Tor Browser, and removing wasm
dependency
This week:
- finish snowflake dependency upgrades that were causing problems
- take a look at snowflake web and webext translations and best
practices
- make changes to Lox encrypted bridge table
-
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/147
Needs help with:
dcf: 2024-10-31
Last week:
- made more comments on snowflake webextension Manifest V3
run-when-closed
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- commented on patch for multiple SnowflakeConn.Close
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
Next week:
- comment on updates to unreliable snowflake transport
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- open issue to have snowflake-client log whenever KCPInErrors
is nonzero
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- parent:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- open issue to disable /debug endpoint on snowflake broker
Help with:
- tell me when to restart the brokers for
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
meskio: 2024-10-24
Last week:
- bring back QR coes for email bridges (rdsys#244)
- add QR codes to Telegram distributor (rdsys#243)
- debug gettor telegram issues (onionsproutsbot#63)
Next week:
- update snowflake proxy debian package
Shelikhoo: 2024-10-31
Last Week:
- [Next Action Pending] snowflake broker update/reinstall(cont.):
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- [Awaiting Review] Unreliable+unordered WebRTC data channel
transport for Snowflake rev2 (cont.)(
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
) improvements
- [Awaiting Input] Review CI: fix `latest` container image
tag.
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
- Prepare State of the Onion script/slide
- Merge request reviews
Next Week/TODO:
- Merge request reviews
- Prepare State of the Onion script/slide
- Work on finishing snowflake container release(and fix the
comments)
- Work on Parpare DeploymentTool for Publishing
https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/…
onyinyang: 2024-10-31
Last week(s):
- MR for troll-patrol integration for Lox bridge blockage detection
-
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/merge_requests/263
- improved on the graceful exit and blockage detection
elements from last week
- work is ongoing from vecna so will update this further
when that's in a more complete state
- discussed plans for dropping lox-wasm with browser team
- test distributor implementation
Next week:
- Finish up test distributor implementation and deploy test
distributor
- update lox protocols to return duplicate responses for an
already seen request
- add issuer improvements to lox protocols
- Work on outstanding milestone issues:
in particular:
https://gitlab.torproject.org/tpo/anti-censorship/lox/-/issues/69
- key rotation automation
Later:
pending decision on abandoning lox wasm in favour of some kind
of FFI?
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/43096):
- add pref to handle timing for pubkey checks in Tor browser
- add trusted invitation logic to tor browser integration:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/42974
- improve metrics collection/think about how to show Lox is
working/valuable
- sketch out Lox blog post/usage notes for forum
(long term things were discussed at the meeting!):
- brainstorming grouping strategies for Lox buckets (of
bridges) and gathering context on how types of bridges are
distributed/use in practice
Question: What makes a bridge usable for a given user, and
how can we encode that to best ensure we're getting the most appropriate
resources to people?
1. Are there some obvious grouping strategies that we
can already consider?
e.g., by PT, by bandwidth (lower bandwidth bridges
sacrificed to open-invitation buckets?), by locale (to be matched with a
requesting user's geoip or something?)
2. Does it make sense to group 3 bridges/bucket, so
trusted users have access to 3 bridges (and untrusted users have access
to 1)? More? Less?
theodorsm: 2024-10-24
Last weeks:
- Adjusting to post-student life
- Testing out beta releases of pion dtls and webrtc
Next weeks:
- Update Snowflake to use latest pion upstream releases
- Test Snowflake fork with covert-dtls
- Condensing thesis into paper
Help with:
- Feedback on thesis
Facilitator Queue:
onyinyang meskio shelikhoo
1. First available staff in the Facilitator Queue will be the
facilitator for the meeting
2. After facilitating the meeting, the facilitator will be moved to the
tail of the queue
Hello everyone!
We have a new job opening for a User Support Specialist in Farsi!
https://www.torproject.org/about/jobs/user-support-farsi-specialist/
If you are or know someone who would be a good fit and wants to join our team, please apply/share.
Thank you for helping us spread the word! :)
Cheers,
Erin
Hello,
This email shares OONI's monthly report for September 2024.
*# OONI Monthly Report: September 2024*
Throughout September 2024, the OONI team’s work can be tracked through the
various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
*## New partnership with Digital Rights Nepal*
In September 2024, we established a new partnership with Digital Rights
Nepal (https://digitalrightsnepal.org/), a leading non-profit organization
dedicated to safeguarding and advancing digital rights in Nepal.
As part of our partnership, we will collaborate on studying internet
censorship in Nepal. We published a page featuring Digital Rights Nepal as
a new OONI partner and highlighting their important work:
https://ooni.org/partners/digital-rights-nepal/
*## Research report on internet censorship in Kazakhstan*
On 19th September 2024, in collaboration with our partners Internet Freedom
Kazakhstan (IFKZ) and Eurasian Digital Foundation, we co-published a new
research report documenting TLS MITM attacks and the blocking of news
media, human rights, and circumvention tool sites in Kazakhstan.
We published the research report in both:
* English: https://ooni.org/post/2024-kazakhstan-report/
* Russian: https://ooni.org/ru/post/2024-kazakhstan-report/
Our partner, Internet Freedom Kazakhstan (IFKZ), published the following
article about our joint research report:
https://ifkz.org/ru/article/internet-censorship-in-kazakhstan
Our report shares censorship findings based on the analysis of OONI data
collected from Kazakhstan over the past year, as well as legal analysis and
interviews with a few media representatives.
Our analysis of OONI data from Kazakhstan reveals:
* TLS Man-In-The-Middle (MITM) attacks
* Blocking of at least 17 news media websites
* Blocking of petition sites and of the Russian language edition of Amnesty
International's website
* Blocking of at least 73 circumvention tool websites
In almost all cases, the blocks appear to be implemented by means of TLS
interference, as OONI data shows that the TLS handshakes result in timeout
errors after the Client Hello message. This is observed uniformly on all
tested networks in Kazakhstan during the analysis period.
Notably, we documented the use of the latest government-mandated root
certificate authority (CA) – and its use to emit 6 distinct intermediate
certificates – that were used to carry out TLS MITM attacks, targeting at
least 14 domains on at least 19 networks in Kazakhstan. We found that these
intermediate certificates were even being used to perform MITM attacks
during periods of certificate invalidity.
Overall, as the timing and types of blocked URLs are consistent across
networks, ISPs in Kazakhstan likely implement blocks in a coordinated
manner. Coordination among ISPs is further suggested by the fact that we
found the same certificate used by 19 distinct ISPs to implement TLS MITM
attacks. These TLS MITM attacks raise concerns because such practices
weaken the online privacy and security of internet users in Kazakhstan.
Our report received press coverage from the following outlets:
* FactCheck Kazakhstan:
https://factcheck.kz/novosti/internet-tsenzura-v-kazahstane-rezultaty-issle…
* Ulysmedia Kazakhstan:
https://ulysmedia.kz/rassledovaniya/38144-ramki-rukopozhatiia-i-lichnye-dan…
* SecurityLab Russia: https://www.securitylab.ru/news/552299.php
* Sledstvie Info:
https://sledstvie.info/news/45234-informatcionnaja_izoljatsija_kazahstana_k…
*## Report on the blocking of OONI Explorer in Russia*
In September 2024, Russia started blocking access to OONI Explorer.
We published a report, documenting the blocking of OONI Explorer in Russia
based on OONI data: https://ooni.org/post/2024-russia-blocked-ooni-explorer/
On 11th September 2024, we received an email from Roskomnadzor, informing
us of their decision to block access to OONI Explorer. On the same day,
OONI data shows that ISPs in Russia started implementing the block.
While Roskomnadzor mentioned their intention to restrict access to the
Russian translation of our circumvention tool reachability measurements, in
practice, the restriction is far-reaching. The block restricts access to
all OONI data hosted on OONI Explorer.
On some networks in Russia, we are able to automatically confirm the
blocking of OONI Explorer based on fingerprints. For example, OONI data
shows that DNS resolution returns an IP that hosts a block page.
As part of this report, we made use of the data analysis capabilities of
our upcoming OONI pipeline v5 to produce a chart with the breakdown of
failure types and errors that enable the characterization of the block. On
most networks in Russia, access to OONI Explorer appears to be blocked by
means of TLS interference, as many measurements resulted in timeout errors
and connection reset errors right after the Client Hello message during the
TLS handshake.
On 18th September 2024, our Russian partner, Roskomsvoboda (
https://roskomsvoboda.org/), shared news of the blocking of OONI Explorer
with Russian communities via Telegram: https://t.me/ru_tech_talk/560
*## Report on the blocking of Twitter/X in Tanzania*
On 30th August 2024, Tanzania blocked access to Twitter/X.
In early September 2024, we published a short report on our Censorship
Findings platform, documenting the block through OONI data.
Our report on the (temporary) blocking of Twitter/X in Tanzania is
available here: : https://explorer.ooni.org/findings/188763810301
It’s worth noting that our community members also independently reported on
the blocking of Twitter/X in Tanzania through the use of OONI tools and
data:
https://x.com/MelamiVictoria/status/1829502734078185879https://x.com/ZainaFoundation/status/1829536688890085645
*## Report on the blocking of Twitter/X in Brazil*
On 31st August 2024, Brazil blocked access to Twitter/X.
In early September 2024, we published a short report on our Censorship
Findings platform, documenting the block through OONI data.
Our report on the blocking of Twitter/X in Brazil is available here:
https://explorer.ooni.org/findings/174962608001
It’s worth noting that our community members also independently reported on
the blocking of Twitter/X in Brazil through the use of OONI tools and data:
https://x.com/vesinfiltro/status/1830262921789669543https://x.com/vinifortuna/status/1830349458384486599https://x.com/OliverLinow/status/1829846237203333282
*## Presenting thematic censorship findings on OONI Explorer*
In September 2024, we started developing the new thematic censorship
findings pages for OONI Explorer based on the user research and mockups
designed in previous months. These pages will focus on OONI measurements
pertaining to News Media (https://github.com/ooni/explorer/issues/940),
Social Media (https://github.com/ooni/explorer/issues/939) and
Circumvention Tools (https://github.com/ooni/explorer/issues/941) and will
offer users a way to explore OONI data focused on these specific themes. We
also added support for theme tags that will enable the display of relevant
reports on each thematic page of OONI Explorer (
https://github.com/ooni/explorer/pull/965). The launch date of these new
pages will be determined in the coming weeks.
*## Automating censorship detection and characterization based on OONI
measurements*
We released the OONI Pipeline v5.0.0-alpha4:
https://github.com/ooni/data/pull/83
As part of this release, we:
* Added a web interface for viewing observations;
* Added an API for returning aggregates of observations;
* Added a web view for plotting aggregates of observations;
* Added support for performing observation generation using multiple
cores (instead of multiple threads since it's CPU bound);
* Separated the observation activities into distinct smaller activities
allowing for more narrowly scoped scheduling and retry policies;
* Changed the type of PrevRange so that it's possible to serialize it in
JSON allowing to pass it as a parameter to activities;
* Moved the update_assets into the observation activity;
* Added support for passing config file via `CONFIG_FILE` environment
variable;
* Made improvements to the CLI commands;
* Dropped several CLI arguments that should only be read from the config
file;
* Made other improvements related to typing.
Following this release, we made some important improvements to the schema
of the observation tables. Specifically, we:
* Replaced observation_id with observation_idx (
https://github.com/ooni/data/issues/87);
* Used the PARTITION KEY for deduplication instead of running deletes (
https://github.com/ooni/data/issues/88).
These improvements are mainly targeted towards improving the performance of
update operations and making them more robust to reprocessing since
deduplication is handled natively using the MergeTree table engine
deduplication.
*## Data analysis for upcoming research report*
As part of an upcoming research report on internet censorship in Russia, we
analyzed OONI measurements collected from Russia over the past year. We
completed this data analysis in September 2024, and further details about
the analysis are available here: https://github.com/ooni/backend/issues/847
*## Activities supported by OTF FOSS### OONI Explorer*
Notably, we launched an improved navigation menu for OONI Explorer (
https://explorer.ooni.org/). This work is available here:
https://github.com/ooni/explorer/pull/950
Based on community feedback shared through our user research in previous
months, we improved the navigation menu of OONI Explorer to enhance the
discoverability of resources and to enable us to add upcoming new pages in
the next months.
*### OONI Probe Mobile*
We continued to make progress on our multi-platform project that aims to
refactor the OONI Probe mobile app. After making good progress on our
internal MVP, we turned our attention to leveraging our initial work to
start developing the iOS version of Deutsche Welle’s News Media Scan
application. This includes tasks like creating the onboarding flow (
https://github.com/ooni/probe-multiplatform/issues/104), building the
results summary view (https://github.com/ooni/probe-multiplatform/issues/109),
and adding the ability to filter results (
https://github.com/ooni/probe-multiplatform/issues/98). Additionally, we
worked on the ability to update OONI Run v2 tests for our own internal MVP (
https://github.com/ooni/probe-multiplatform/issues/53).
Here is a list of all issues completed in September 2024 for our
multi-platform project:
https://github.com/ooni/probe-multiplatform/issues?q=is%3Aissue+is%3Aclosed…
*### OONI Run*
As part of our final preparation for the launch of OONI Run v2, we took
steps to ensure that by releasing OONI Run v2 we would not accidentally
introduce any bugs that cause a drop in measurements. We improved our
ability to filter measurements by different release channels, ensuring we
can filter measurements by our open testing or “beta” channel for our
Android application on the Google Play store. This way, we can more
accurately compare different versions of our applications as we make
changes and enhancements so we can increase our confidence in not
introducing issues (https://github.com/ooni/probe/issues/2803).
*### OONI Backend Maintenance & DevOps*
We worked on switching api.ooni.org to be served from AWS (
https://github.com/ooni/devops/issues/94), focusing first on what was
necessary for the OONI Run v2 project so that both the mobile application
and the web-based dashboard use the production API. As part of that work,
we had to move our test helpers back to Digital Ocean as AWS was proving
too costly (https://github.com/ooni/devops/issues/91). We also worked on
several other items related to this overall task. (
https://github.com/ooni/devops/issues/93,
https://github.com/ooni/devops/issues/95).
*## Hiring process for OONI Junior Backend Developer job opening*
As part of the ongoing hiring process for a new OONI Junior Backend
Developer (https://ooni.org/post/2024-job-opening-ooni-backend-developer/),
we continued to review incoming applications and interview shortlisted
candidates.
*## Test list updates*
Throughout September 2024, we did multiple minor updates to the test lists
for Kenya, Algeria, Iran, Armenia, Georgia, and Uganda, as well as to the
Global test list. All of these updates have been merged (
https://github.com/citizenlab/test-lists/pulls?q=is%3Apr+is%3Aclosed).
We also reviewed and merged a more extensive update to the Cambodian test
list submitted by the iMAP project:
https://github.com/citizenlab/test-lists/pull/1699/files
*## Collaboration with agency to boost OONI’s social media presence*
In September 2024, we started collaborating with Latte (
https://www.lattecreative.com/en/), an agency in Rome which supports
organizations (including many nonprofit organizations, such as Amnesty
International and Greenpeace) on improving their communication, branding,
advocacy, and fundraising efforts. We are collaborating with Latte on
designing an end-of-year fundraising strategy with the goal of boosting
OONI’s donations, as well as on improving OONI’s communication and social
media presence.
*## Fellowship at the Berkman Klein Center for Internet and Society*
In September 2024, OONI’s Maria started a research fellowship at the
Berkman Klein Center for Internet and Society at Harvard University. As
part of her year-long fellowship, Maria will explore how internet
censorship changed globally over the past eight years through OONI data.
She will also carry out interviews to explore the role of advocacy and
circumvention tool groups in responding to emergent censorship events.
More information about the 2024-2025 Berkman Klein Center fellowship cohort
is available here:
https://cyber.harvard.edu/story/2024-07/incoming-2024-25-bkc-fellows
*## Rapid response### Blocking of Telegram in El Salvador*
On 15th September 2024, El Salvador blocked access to Telegram. On the same
day, we rapidly responded by sharing relevant OONI data and findings on
social media: https://x.com/OpenObservatory/status/1835360393906074078
The information we shared included a chart produced by OONI data analysis
that we performed to examine the reachability of Telegram IPs in El
Salvador by probe ASN and target. We found that access to Telegram was
blocked on at least 5 networks in El Salvador (starting from around 4am UTC
on 15th September 2024), with some ISPs blocking access to Telegram IPs,
while others blocked access to Telegram by means of TLS interference.
This blocking event resulted in a significant OONI measurement spike in El
Salvador on 15th September 2024, as well as in ongoing measurement coverage
thereafter (suggesting increased OONI Probe adoption and use of automated
testing following the block). This is evident through aggregated OONI
measurement coverage in El Salvador:
https://explorer.ooni.org/chart/mat?probe_cc=SV&since=2024-08-12&until=2024…
*## Community use of OONI tools and data### Sinar Project Blocked or Not
tool*
Notably, our long-term Malaysian partner, Sinar Project (
https://sinarproject.org/), launched a new “Blocked or Not” tool, which
makes use of our miniooni research client and submits data to OONI.
Their tool is available here: https://blockedornot.sinarproject.org/
Sinar Project’s Blocked or Not tool is a web service that enables users to
easily and quickly check if a website is blocked or not in Malaysia.
*### Sinar Project report on the blocking of an entertainment platform*
On 20th September 2024, our partner, Sinar Project (
https://sinarproject.org/), published a report documenting the blocking of
ArtStation.com, a prominent platform for showcasing games, film, media, and
entertainment art. As part of their report, Sinar Project made use of OONI
data and encouraged further OONI Probe testing in Malaysia.
Read their report here:
https://imap.sinarproject.org/resources/internet-censorship-update-blocking…
*### Access Now’s press statement on the blocking of Twitter/X in Tanzania*
In response to the blocking of Twitter/X in Tanzania, Access Now published
a press release condemning the blocking of the platform. Their press
release cites OONI data as technical evidence on the block.
Read their press statement here:
https://www.accessnow.org/press-release/civil-society-asks-who-blocked-x-ta…
*### Cloudflare blog post on a global assessment of third-party connection
tampering*
In September 2024, Cloudflare published a blog post providing a global
assessment of third-party connection tampering:
https://blog.cloudflare.com/connection-tampering/
As part of this post, they provide case studies through which they compare
Cloudflare TCP connection anomalies with OONI reports of connection
tampering. Specifically, they compared anomalous Cloudflare TCP connection
data with relevant OONI data from our reports on connection tampering cases
in Tanzania and Ethiopia, and found that relevant Cloudflare data was
consistent with OONI data. This is very interesting because by publishing
data on TCP connection anomalies (
https://radar.cloudflare.com/security-and-attacks#tcp-resets-and-timeouts),
Cloudflare enable researchers to have stronger signals of connection
tampering when compared (and corroborated) with OONI data (and other
relevant datasets).
Learn more about the launch of Cloudflare Radar’s new dashboard on TCP
resets and timeouts here: https://blog.cloudflare.com/tcp-resets-timeouts/
*## Community activities### Workshop for human rights defenders in Nepal*
On 19th September 2024, OONI’s Elizaveta facilitated an online OONI
workshop for human rights defenders in Nepal. This workshop was hosted in
coordination with our new partner, Digital Rights Nepal (
https://ooni.org/partners/digital-rights-nepal/).
*### Quarterly OONI Partner Meeting*
On 20th September 2024, we hosted the quarterly OONI Partner Meeting via an
online video platform.
As part of this meeting, we presented and discussed the OONI training
calendar for October 2024 and November 2024, which will involve a series of
online OONI workshops that we will facilitate for our partners.
These workshops include:
* Introduction to internet censorship (9th October 2024)
* How to use OONI Probe (16th October 2024)
* OONI Run v2 demo (23rd October 2024)
* Maintaining/updating the Citizen Lab test lists (6th November 2024)
* OONI Explorer #1 (13th November 2024)
* OONI Explorer #2 (20th November 2024)
As part of these upcoming workshops, we aim to share relevant skills and
knowledge to enable our partners to participate in OONI censorship
measurement activities in their countries and regions. As an outcome, we
hope that our partners will be equipped to share such knowledge further
with their communities.
As part of our Quarterly Partner Meeting, we also discussed updates to our
partnership MoUs, plans for censorship monitoring during the upcoming 2025
elections around the world, as well as plans for other future partner
events.
*### Global Gathering 2024*
Between 27th-29th September 2024, OONI’s Elizaveta and Jessie traveled to
Portugal to attend the Global Gathering 2024. The detailed agenda of the
event is available here:
https://wiki.digitalrights.community/index.php?title=Global_Gathering_Agend…
As part of their participation, Elizaveta and Jessie:
Hosted an OONI booth, during which they provided a live demo of our
upcoming OONI Run v2 tool and shared OONI swag (27th and 29th September
2024)
Facilitated a discussion on rapid response (28th September 2024)
*### OONI Community Meeting*
On 24th September 2024, we hosted the monthly OONI Community Meeting on our
Slack channel (https://slack.ooni.org/).
As part of this meeting, we provided updates from the OONI team, and we
discussed the recent blocking of OONI Explorer in Russia, as well as the
(global) community’s need to measure the availability of more VPN services
and protocols.
*## Measurement coverage*
In September 2024, 56,049,250 OONI Probe measurements were collected from
3,182 networks in 176 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ OONI team.