Hello,
Throughout October 2020, the OONI team worked on the following sprints:
* Sprint 23 - Ægir (Sep 28 - Oct 11, 2020)
* Sprint 24 - Kelpie (Oct 12 - Oct 25, 2020)
* Sprint 25 - Näkki (Oct 26 - Nov 8, 2020)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
## Completed migration of OONI infrastructure to Amsterdam
In October 2020, we completed the process of migrating critical OONI
infrastructure (hosting the OONI data processing pipeline) to servers in
Amsterdam.
This process involved:
* Building a new OONI PostgreSQL metaDB (powered by the latest version
of PostgreSQL 11) which uses a different set of tables based on our
improved, fasthpath pipeline (https://github.com/ooni/backend/issues/322);
* Refactoring the OONI API codebase to use the fastpath pipeline
(https://github.com/ooni/backend/issues/437);
* Implementing a fastpath pipeline-based API to support OONI Explorer
(https://github.com/ooni/backend/issues/442);
* Replacing the batch OONI data processing pipeline entirely with our
new OONI fastpath pipeline;
* Discontinuing all of our hosts on the Hong Kong data centre
(https://github.com/ooni/backend/issues/390);
* Consolidating our infrastructure on a single host on DigitalOcean
(https://github.com/ooni/backend/issues/400);
* Improving our infrastructure to include better monitoring and alerting
(https://github.com/ooni/backend/issues/439,
https://github.com/ooni/backend/issues/398);
* Extensive testing to ensure that nothing breaks during (and after) the
migration (https://github.com/ooni/backend/issues/372).
We successfully replaced the batch OONI data processing pipeline with
our new fastpath pipeline, and now all OONI measurements collected from
around the world are processed and openly published in near real-time.
Overall, the migration to new servers in Amsterdam was successful, as
nothing broke and our services have significantly improved performance.
## Improved OONI Explorer performance
As a result of the migration, OONI Explorer (https://explorer.ooni.org/)
now has significantly improved performance!
OONI Explorer fetches OONI measurements from the OONI API, which
previously depended on both our batch data processing pipeline and our
fastpath pipeline. In previous months, several OONI Explorer queries
didn’t work and OONI Explorer itself faced several performance issues as
a result of relying on both the new database tables and the legacy table
for running queries. By consolidating our infrastructure and API, we
have managed to significantly boost the performance of OONI Explorer.
Now that the batch OONI data processing pipeline has been deprecated and
replaced entirely by our new fastpath pipeline, all OONI measurements
collected from around the world are processed and openly published in
near real-time, and OONI Explorer queries work fast and reliably. This
is further supported by our work involving the refactoring of the OONI
API, switching to a new database schema, as well as upgrading to
PostgreSQL 11.
## Discontinued PostgreSQL metaDB
While we are excited to have migrated over to a new and improved
pipeline (particularly since this significantly enhanced the performance
of OONI Explorer!), this unfortunately affected the OONI PostgreSQL
MetaDB, which was powered by an older version of PostgreSQL and which
depended on the batch OONI data processing pipeline.
As a result, the OONI PostgreSQL metaDB (which was powered by an older
version of PostgreSQL) was inevitably discontinued in late October 2020.
This means that users of the old OONI PostgreSQL metaDB would still have
access to all previous OONI measurements, but they would not receive any
updates once the migration (to servers in Amsterdam) was completed.
We therefore reached out to OONI community members (particularly those
who we knew relied on the OONI PostgreSQL metaDB for their projects) to
share these updates, encourage them to share details about their use
case, and we offered relevant help.
## Published report on censorship events amid Tanzania’s 2020 general
election
Starting from the eve of Tanzania’s 2020 general election, we started to
observe the blocking of social media apps and websites.
We published a report which shares OONI measurements and details on the
blocking of social media in Tanzania on their election day. This report
is available here:
https://ooni.org/post/2020-tanzania-blocks-social-media-tor-election-day/
## Worked on research report on LGBTIQ website blocking
In collaboration with OutRight Action International and the Citizen Lab,
we have been working on a joint research report which examines the
blocking of LGBTIQ websites in 6 countries based on OONI data.
Throughout October 2020, we made significant progress on the writing of
this report (which we expect to co-publish in early 2021).
## Internet Shutdown Measurement Training for Advocates
On 12th October 2020, we started a 6-week training program on internet
shutdown measurement training for human rights defenders in Sub-Saharan
Africa.
This training program was organized by Internews, and OONI served as the
lead partner on the program. Information about this program is available
here:
https://internews.org/call-applications-internet-shutdown-measurements-advo…
As part of this training program, we led and facilitated the following 2
modules:
* Introduction to Network Measurement
* Detecting Blocking of Websites & Applications with OONI Probe
For each of these two modules, we provided participants with a
pre-recorded webinar for asynchronous learning.
During the week of 12th October 2020, we provided several hours of
synchronous training and mentorship for the module on “Introduction to
Network Measurement”.
During the week of 26th October 2020, we provided several hours of
synchronous training and mentorship for the module on “Detecting
Blocking of Websites & Applications with OONI Probe”.
We also provided a live demo on how to use OONI Explorer in order to
find OONI measurements on the blocking of social media in Tanzania amid
its 2020 general election (which took place that week).
Meanwhile, during the week of 19th October 2020, we helped facilitate
the synchronous training provided by IODA for the module on “Detecting
Internet Blackouts”.
Throughout the training program, we also reviewed the homework
assignments of the training participants, shared feedback and relevant
resources.
## New OONI experiment for measuring encrypted DNS blocking
India's Centre for Internet and Society (CIS) implemented a new OONI
experiment (called `dnscheck`) for measuring the blocking of encrypted
DNS transports.
This experiment is currently available via the `miniooni` research
client
(https://github.com/ooni/probe-engine/tree/v0.19.0#building-miniooni)
and we plan to integrate it as part of the Websites card in the OONI
Probe apps.
CIS India published a research report ("Investigating Encrypted DNS
Blocking in India") based on this test, which is available here:
https://cis-india.org/internet-governance/blog/investigating-encrypted-dns-…
We also published a short blog post about this new experiment and CIS
India’s report, which is available here:
https://ooni.org/post/2020-encrypted-dns-blocking-india/
## Code review of new nettests written by community members
We are thrilled that community members are writing new experiments for
OONI Probe!
In October 2020, we reviewed the code of the following new nettests
(written by community members):
* RiseupVPN experiment;
* HTTP host experiment;
* DoH/DoT blocking experiment;
* HTTP3 experiment.
We hope to eventually ship such tests as part of the OONI Probe apps.
## Published OONI Probe ASN Incident Report
In October 2020, we discovered an ASN-related bug in OONI Probe.
In response, we published an Incident Report which shares details about the
bug, what we did to fix it, and we document our next steps (as well as
measures for limiting the possibility of similar bugs recurring in the
future).
Our Incident Report is available here:
https://ooni.org/post/2020-ooni-probe-asn-incident-report/
## OONI Probe Mobile
### Released OONI Probe Mobile 2.7.0
We fixed the ASN-related bug in the OONI Probe Mobile 2.7.0 release for:
* Android: https://github.com/ooni/probe-android/releases/tag/v2.7.0
* iOS: https://github.com/ooni/probe-ios/releases/tag/v2.7.0
More specifically, this release includes the following improvements:
* Fixed a bug to ensure that the ASN is not leaked in the report ID of
measurements when users have opted out of ASN collection;
* Removed all dependencies from the C++ Measurement Kit engine;
* Made the app rely entirely on the new Go probe engine;
* General improvements and minor bug fixes
We are excited that following the 2.7.0 release, the OONI Probe mobile
app relies entirely on our go-based probe engine!
### Progress on adding support for automated regular testing
We continued to make progress towards adding support in the OONI Probe
mobile app for automated regular testing, as documented through this
ticket: https://github.com/ooni/probe/issues/916
## OONI Probe Desktop
We fixed the ASN-related bug on OONI Probe desktop through the following
release: https://github.com/ooni/probe-cli/releases/tag/v3.0.8
We released OONI Probe 3.0.4 for macOS and Windows, which includes
several bug fixes, UI improvements, and other changes:
https://github.com/ooni/probe-desktop/releases/tag/v3.0.4
We also added support to OONI Probe Desktop to enable users to customize
their testing based on the Citizen Lab category codes:
https://github.com/ooni/probe/issues/1022
## Removed AS0 measurements from the OONI API and from OONI Explorer
As an extra measure in addressing the ASN-related bug, we also worked on
removing all AS0 measurements from the OONI API. We also made
adjustments to the fastpath pipeline so that it stops processing AS0
measurements (as well as measurements which don’t have a country code).
This work is tracked through the following tickets:
https://github.com/ooni/pipeline/pull/326https://github.com/ooni/api/pull/194https://github.com/ooni/api/pull/195
OONI Explorer measurements are fetched from the OONI API, so by removing
the AS0 measurements from the OONI API, we limited the possibility of
accessing affected measurements. We also made relevant adjustments to
OONI Explorer by preventing users from searching measurements based on AS0.
This is tracked through the following tickets:
https://github.com/ooni/explorer/pull/501https://github.com/ooni/explorer/pull/502
## OONI Explorer releases
We made a series of improvements to OONI Explorer, we addressed the
ASN-related bug (by hiding AS0 measurements), and we added support to
OONI Explorer for using the new factored OONI API.
These changes are documented through the following OONI Explorer releases:
* 2.0.9: https://github.com/ooni/explorer/releases/tag/v2.0.9
* 2.0.10: https://github.com/ooni/explorer/releases/tag/v2.0.10 (which
addressed the AS0 measurements)
* 2.1.0: https://github.com/ooni/explorer/releases/tag/v2.1.0 (which
uses the new refactored OONI API)
We also added more end-to-end tests for OONI Explorer
(https://github.com/ooni/explorer/pull/493), and we upgraded OONI
Explorer to use the latest ooni-components
(https://github.com/ooni/explorer/issues/509).
## OONI Probe engine
We made a series of improvements to the OONI Probe engine (which powers
the OONI Probe apps) through the following releases:
https://github.com/ooni/probe-engine/releases/tag/v0.18.0https://github.com/ooni/probe-engine/releases/tag/v0.19.0
## OONI team peer review
To improve our practices and grow as a team, we designed a peer review
feedback form for the OONI team. The goal of this feedback form was to
offer all OONI team members an opportunity to share feedback (in a
structured way) on their colleagues’ performance, as well as on their own.
## Ford Foundation Communications Training
OONI’s Maria participated in the Ford Foundation’s Communications
Training program during the last week of October 2020.
We were offered this great opportunity because we are a grantee of the
Ford Foundation, who support OONI’s community-related work. The
knowledge and skills gained throughout this week-long communications
training program will help support our work in the long-run.
## Community use of OONI data
### MIT Policy Hackathon 2020
The theme of the MIT Policy Hackathon 2020
(https://www.mitpolicyhackathon.org/) was based on OONI data!
MIT students who participated in this hackathon explored OONI data to
answer a variety of questions, which are detailed here:
https://docs.google.com/document/d/1DCK_7djZvJrd41ls-2ThN0cWPbszDQ5EAJZqcij…
### Access Now report on censorship events in Tanzania amid elections
Access Now published a report on the blocking of social media in
Tanzania amid its 2020 general election. Their report, which cites OONI
data, is available here:
https://www.accessnow.org/tanzania-votes-government-forces-telcos-escalate-…
## Community activities
### CensorWatch
India’s Centre for Internet and Society (CIS) released a new research
tool designed to perform censorship measurements, called CensorWatch:
https://cis-india.github.io/censorwatch/
Over the past months we have collaborated with CIS India developers and
CensorWatch builds upon OONI Probe methodologies.
### Created OONI Probe testing guides for October 2020 elections
We created (and shared) 3 documents with relevant OONI Probe testing
instructions for local communities in Tanzania, Guinea, and Cote
d’Ivoire so that they can participate in OONI Probe censorship
measurement leading up to, during, and after their respective October
2020 elections.
## Coordinated Telegram testing in Cuba and Thailand
Starting from 15th October 2020, OONI measurements suggest the blocking of
Telegram in Cuba, and we were told that 4 Telegram IPs were recently
added to the
Thai blocklist (though we have not seen Telegram blocked in Thailand yet).
We therefore coordinated the testing of Telegram in Cuba and Thailand to
collect more measurements throughout October 2020.
In Cuba, community members reported that they were unable to use the
OONI Probe mobile app (following the blocking of Telegram), but the OONI
Probe desktop app continued to work (which is how many measurements
continued to be collected). We spent resources in October 2020
investigating why the OONI Probe mobile app didn’t work in Cuba.
### SMEX resource including OONI Probe
SMEX published a post titled “Website Blocking in the Arab Region:
Monitoring and Counteraction Techniques” (a translation of the Arabic
resource published previously), which is available here:
https://smex.org/website-blocking-in-the-arab-region-monitoring-and-counter…
Among other tools, OONI Probe is mentioned in this post as a tool for
measuring internet censorship in the Arab region.
### OONI Community Meeting
On 27th October 2020, we hosted the monthly OONI Community Meeting on
our Slack channel (https://slack.ooni.org/), during which we discussed
the following topics:
1. Updates from the OONI team
2. Integrating the DNScheck test into the OONI Probe mobile and desktop
apps (https://github.com/ooni/probe-engine/tree/master/experiment/dnscheck)
3. Analyzing OONI data.
## Userbase
In October 2020, 5,324,739 OONI Probe measurements were collected from
5,333 networks in 204 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
Hi!
I forgot to add a fancy header like this like month, but I want to say
"hi!" to everyone, and "welcome back to our monthly reports from the
sysadmin team"! :)
Hopefully everyone can manage to stay safe in this crazier-than-usual
holiday season!
**Agenda**
- Roll call: who's there and emergencies
- Roadmap review
- Triage rotation
- Holiday planning
- TPA survey review
- Other discussions
- New intern
- Next meeting
- Metrics of the month
# Roll call: who's there and emergencies
anarcat, hiro, gaba, no emergencies
The meeting took place on IRC because anarcat had too much noise.
# Roadmap review
Did a lot of cleanup in the dashboard:
https://gitlab.torproject.org/tpo/tpa/team/-/boards
In general, the following items were priotirized:
* [GitLab CI][]
* finish setting up the Cymru network, especially the [VPN][]
* [BTCpayserver][]
* [tor browser build boxes][]
* small tickets like the [git stuff][] and triage (see below)
[git stuff]: https://gitlab.torproject.org/tpo/tpa/team/-/boards?&label_name[]=Git
[tor browser build boxes]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/34122
[BTCpayserver]: https://bugs.torproject.org/tpo/tpa/team/33750
[VPN]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40097
[GitLab CI]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095
The following items were punted to the future:
* SVN retirement (to January)
* password management (specs in January?)
* Puppet role account and verifications
We briefly discussed Grafana authentication, because of a request to
[create a new account on grafana2][]. anarcat said the current model
of managing the htpasswd file in Puppet doesn't scale so well because
we need to go through this process every time we need to grant access
(or do a password reset) and identified 3 alternative authentication
mechanisms:
[create a new account on grafana2]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40102
1. htpasswd managed in Puppet (status quo)
2. Grafana users (disabling the htpasswd, basically)
3. LDAP authentication
The current authentication model was picked because we wanted to
automate user creation in Puppet, and because it's hard to create
users in Grafana from Puppet. When a new Grafana server is setup,
there's a small window during which an attacker could create an admin
account, which we were trying to counter. But maybe those concerns are
moot now.
We also discussed passord management but that will be worked on in
January. We'll try to set a roadmap for 2021 in January, after the
results of the survey have come in.
# Triage rotation
Hiro brought up the idea of rotating the triage work instead of having
always the same person doing it. Right now, anarcat looks at the board
at the beginning of every week and deals with tickets in the "Open"
column. Often, he just takes the easy tickets, drops them in ~Next,
and just does them, other times, they end up in ~Backlog or get closed
or at least have some response of some sort.
We agreed to switch that responsability every two weeks
# Holiday planning
anarcat off from 14th to the 26th, hiro from 30th to jan 14th
# TPA survey review
anarcat is [working on a survey][] to get information from our users
to plan the 2021 roadmap.
[working on a survey]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40061
People like the survey in general, but the "services" questions were
just too long. It was suggested to remove services TPA has nothing to
do with (like websites or metrics stuff like check.tpo). But anarcat
pointed out that we need to know which of those services are
important: for example right now we "just know" that check.tpo is
important, but it would be nice to have hard data that confirms it.
Anarcat agreed to separate the table into teams so that it doesn't
look that long and will submit the survey back for review again by the
end of the week.
# Other discussions
## New intern
[MariaV][] just started as an Outreachy intern to work on Anonymous
Ticket System. She may be joining the `#tpo-admin` channel and may
join the gitlab/tooling meetings.
Welcome MariaV!
[MariaV]: https://mviolante.com/
# Next meeting
Quick check-in on December 29th, same time.
# Metrics of the month
* hosts in Puppet: 79, LDAP: 82, Prometheus exporters: 133
* number of apache servers monitored: 28, hits per second: 205
* number of nginx servers: 2, hits per second: 3, hit ratio: 0.86
* number of self-hosted nameservers: 6, mail servers: 12
* pending upgrades: 1, reboots: 0
* average load: 0.34, memory available: 1.80 TiB/2.39 TiB, running
processes: 481
* bytes sent: 245.34 MB/s, received: 139.99 MB/s
* [GitLab tickets][]: 129 issues including...
* open: 0
* icebox: 92
* backlog: 20
* next: 9
* doing: 8
* (closed: 2130)
[Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards
The upgrade prediction graph has been retired since it keeps
predicting the upgrades will be finished in the past, which no one
seems to have noticed from the last report (including me).
Metrics also available as the main Grafana dashboard. Head to
<https://grafana.torproject.org/>, change the time period to 30 days,
and wait a while for results to render.
--
Antoine Beaupré
torproject.org system administration