Hello,

Throughout October 2020, the OONI team worked on the following sprints:

* Sprint 23 - Ægir (Sep 28 - Oct 11, 2020) * Sprint 24 - Kelpie (Oct 12 - Oct 25, 2020) * Sprint 25 - Näkki (Oct 26 - Nov 8, 2020)

Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni

Highlights are shared in this report below.

## Completed migration of OONI infrastructure to Amsterdam

In October 2020, we completed the process of migrating critical OONI infrastructure (hosting the OONI data processing pipeline) to servers in Amsterdam.

This process involved:

* Building a new OONI PostgreSQL metaDB (powered by the latest version of PostgreSQL 11) which uses a different set of tables based on our improved, fasthpath pipeline (https://github.com/ooni/backend/issues/322); * Refactoring the OONI API codebase to use the fastpath pipeline (https://github.com/ooni/backend/issues/437); * Implementing a fastpath pipeline-based API to support OONI Explorer (https://github.com/ooni/backend/issues/442); * Replacing the batch OONI data processing pipeline entirely with our new OONI fastpath pipeline; * Discontinuing all of our hosts on the Hong Kong data centre (https://github.com/ooni/backend/issues/390); * Consolidating our infrastructure on a single host on DigitalOcean (https://github.com/ooni/backend/issues/400); * Improving our infrastructure to include better monitoring and alerting (https://github.com/ooni/backend/issues/439, https://github.com/ooni/backend/issues/398); * Extensive testing to ensure that nothing breaks during (and after) the migration (https://github.com/ooni/backend/issues/372).

We successfully replaced the batch OONI data processing pipeline with our new fastpath pipeline, and now all OONI measurements collected from around the world are processed and openly published in near real-time.

Overall, the migration to new servers in Amsterdam was successful, as nothing broke and our services have significantly improved performance.

## Improved OONI Explorer performance

As a result of the migration, OONI Explorer (https://explorer.ooni.org/) now has significantly improved performance!

OONI Explorer fetches OONI measurements from the OONI API, which previously depended on both our batch data processing pipeline and our fastpath pipeline. In previous months, several OONI Explorer queries didn’t work and OONI Explorer itself faced several performance issues as a result of relying on both the new database tables and the legacy table for running queries. By consolidating our infrastructure and API, we have managed to significantly boost the performance of OONI Explorer.

Now that the batch OONI data processing pipeline has been deprecated and replaced entirely by our new fastpath pipeline, all OONI measurements collected from around the world are processed and openly published in near real-time, and OONI Explorer queries work fast and reliably. This is further supported by our work involving the refactoring of the OONI API, switching to a new database schema, as well as upgrading to PostgreSQL 11.

## Discontinued PostgreSQL metaDB

While we are excited to have migrated over to a new and improved pipeline (particularly since this significantly enhanced the performance of OONI Explorer!), this unfortunately affected the OONI PostgreSQL MetaDB, which was powered by an older version of PostgreSQL and which depended on the batch OONI data processing pipeline.

As a result, the OONI PostgreSQL metaDB (which was powered by an older version of PostgreSQL) was inevitably discontinued in late October 2020. This means that users of the old OONI PostgreSQL metaDB would still have access to all previous OONI measurements, but they would not receive any updates once the migration (to servers in Amsterdam) was completed.

We therefore reached out to OONI community members (particularly those who we knew relied on the OONI PostgreSQL metaDB for their projects) to share these updates, encourage them to share details about their use case, and we offered relevant help.

## Published report on censorship events amid Tanzania’s 2020 general election

Starting from the eve of Tanzania’s 2020 general election, we started to observe the blocking of social media apps and websites.

We published a report which shares OONI measurements and details on the blocking of social media in Tanzania on their election day. This report is available here: https://ooni.org/post/2020-tanzania-blocks-social-media-tor-election-day/

## Worked on research report on LGBTIQ website blocking

In collaboration with OutRight Action International and the Citizen Lab, we have been working on a joint research report which examines the blocking of LGBTIQ websites in 6 countries based on OONI data.

Throughout October 2020, we made significant progress on the writing of this report (which we expect to co-publish in early 2021).

## Internet Shutdown Measurement Training for Advocates

On 12th October 2020, we started a 6-week training program on internet shutdown measurement training for human rights defenders in Sub-Saharan Africa.

This training program was organized by Internews, and OONI served as the lead partner on the program. Information about this program is available here: https://internews.org/call-applications-internet-shutdown-measurements-advoc...

As part of this training program, we led and facilitated the following 2 modules:

* Introduction to Network Measurement * Detecting Blocking of Websites & Applications with OONI Probe

For each of these two modules, we provided participants with a pre-recorded webinar for asynchronous learning.

During the week of 12th October 2020, we provided several hours of synchronous training and mentorship for the module on “Introduction to Network Measurement”.

During the week of 26th October 2020, we provided several hours of synchronous training and mentorship for the module on “Detecting Blocking of Websites & Applications with OONI Probe”.

We also provided a live demo on how to use OONI Explorer in order to find OONI measurements on the blocking of social media in Tanzania amid its 2020 general election (which took place that week).

Meanwhile, during the week of 19th October 2020, we helped facilitate the synchronous training provided by IODA for the module on “Detecting Internet Blackouts”.

Throughout the training program, we also reviewed the homework assignments of the training participants, shared feedback and relevant resources.

## New OONI experiment for measuring encrypted DNS blocking

India's Centre for Internet and Society (CIS) implemented a new OONI experiment (called `dnscheck`) for measuring the blocking of encrypted DNS transports.

This experiment is currently available via the `miniooni` research client (https://github.com/ooni/probe-engine/tree/v0.19.0#building-miniooni) and we plan to integrate it as part of the Websites card in the OONI Probe apps.

CIS India published a research report ("Investigating Encrypted DNS Blocking in India") based on this test, which is available here: https://cis-india.org/internet-governance/blog/investigating-encrypted-dns-b...

We also published a short blog post about this new experiment and CIS India’s report, which is available here: https://ooni.org/post/2020-encrypted-dns-blocking-india/

## Code review of new nettests written by community members

We are thrilled that community members are writing new experiments for OONI Probe!

In October 2020, we reviewed the code of the following new nettests (written by community members):

* RiseupVPN experiment; * HTTP host experiment; * DoH/DoT blocking experiment; * HTTP3 experiment.

We hope to eventually ship such tests as part of the OONI Probe apps.

## Published OONI Probe ASN Incident Report

In October 2020, we discovered an ASN-related bug in OONI Probe.

In response, we published an Incident Report which shares details about the bug, what we did to fix it, and we document our next steps (as well as measures for limiting the possibility of similar bugs recurring in the future).

Our Incident Report is available here: https://ooni.org/post/2020-ooni-probe-asn-incident-report/

## OONI Probe Mobile

### Released OONI Probe Mobile 2.7.0

We fixed the ASN-related bug in the OONI Probe Mobile 2.7.0 release for:

* Android: https://github.com/ooni/probe-android/releases/tag/v2.7.0 * iOS: https://github.com/ooni/probe-ios/releases/tag/v2.7.0

More specifically, this release includes the following improvements:

* Fixed a bug to ensure that the ASN is not leaked in the report ID of measurements when users have opted out of ASN collection; * Removed all dependencies from the C++ Measurement Kit engine; * Made the app rely entirely on the new Go probe engine; * General improvements and minor bug fixes

We are excited that following the 2.7.0 release, the OONI Probe mobile app relies entirely on our go-based probe engine!

### Progress on adding support for automated regular testing

We continued to make progress towards adding support in the OONI Probe mobile app for automated regular testing, as documented through this ticket: https://github.com/ooni/probe/issues/916

## OONI Probe Desktop

We fixed the ASN-related bug on OONI Probe desktop through the following release: https://github.com/ooni/probe-cli/releases/tag/v3.0.8

We released OONI Probe 3.0.4 for macOS and Windows, which includes several bug fixes, UI improvements, and other changes: https://github.com/ooni/probe-desktop/releases/tag/v3.0.4

We also added support to OONI Probe Desktop to enable users to customize their testing based on the Citizen Lab category codes: https://github.com/ooni/probe/issues/1022

## Removed AS0 measurements from the OONI API and from OONI Explorer

As an extra measure in addressing the ASN-related bug, we also worked on removing all AS0 measurements from the OONI API. We also made adjustments to the fastpath pipeline so that it stops processing AS0 measurements (as well as measurements which don’t have a country code).

This work is tracked through the following tickets: https://github.com/ooni/pipeline/pull/326 https://github.com/ooni/api/pull/194 https://github.com/ooni/api/pull/195

OONI Explorer measurements are fetched from the OONI API, so by removing the AS0 measurements from the OONI API, we limited the possibility of accessing affected measurements. We also made relevant adjustments to OONI Explorer by preventing users from searching measurements based on AS0.

This is tracked through the following tickets: https://github.com/ooni/explorer/pull/501 https://github.com/ooni/explorer/pull/502

## OONI Explorer releases

We made a series of improvements to OONI Explorer, we addressed the ASN-related bug (by hiding AS0 measurements), and we added support to OONI Explorer for using the new factored OONI API.

These changes are documented through the following OONI Explorer releases:

* 2.0.9: https://github.com/ooni/explorer/releases/tag/v2.0.9 * 2.0.10: https://github.com/ooni/explorer/releases/tag/v2.0.10 (which addressed the AS0 measurements) * 2.1.0: https://github.com/ooni/explorer/releases/tag/v2.1.0 (which uses the new refactored OONI API)

We also added more end-to-end tests for OONI Explorer (https://github.com/ooni/explorer/pull/493), and we upgraded OONI Explorer to use the latest ooni-components (https://github.com/ooni/explorer/issues/509).

## OONI Probe engine

We made a series of improvements to the OONI Probe engine (which powers the OONI Probe apps) through the following releases:

https://github.com/ooni/probe-engine/releases/tag/v0.18.0 https://github.com/ooni/probe-engine/releases/tag/v0.19.0

## OONI team peer review

To improve our practices and grow as a team, we designed a peer review feedback form for the OONI team. The goal of this feedback form was to offer all OONI team members an opportunity to share feedback (in a structured way) on their colleagues’ performance, as well as on their own.

## Ford Foundation Communications Training

OONI’s Maria participated in the Ford Foundation’s Communications Training program during the last week of October 2020.

We were offered this great opportunity because we are a grantee of the Ford Foundation, who support OONI’s community-related work. The knowledge and skills gained throughout this week-long communications training program will help support our work in the long-run.

## Community use of OONI data

### MIT Policy Hackathon 2020

The theme of the MIT Policy Hackathon 2020 (https://www.mitpolicyhackathon.org/) was based on OONI data!

MIT students who participated in this hackathon explored OONI data to answer a variety of questions, which are detailed here: https://docs.google.com/document/d/1DCK_7djZvJrd41ls-2ThN0cWPbszDQ5EAJZqcijf...

### Access Now report on censorship events in Tanzania amid elections

Access Now published a report on the blocking of social media in Tanzania amid its 2020 general election. Their report, which cites OONI data, is available here: https://www.accessnow.org/tanzania-votes-government-forces-telcos-escalate-c...

## Community activities

### CensorWatch

India’s Centre for Internet and Society (CIS) released a new research tool designed to perform censorship measurements, called CensorWatch: https://cis-india.github.io/censorwatch/

Over the past months we have collaborated with CIS India developers and CensorWatch builds upon OONI Probe methodologies.

### Created OONI Probe testing guides for October 2020 elections

We created (and shared) 3 documents with relevant OONI Probe testing instructions for local communities in Tanzania, Guinea, and Cote d’Ivoire so that they can participate in OONI Probe censorship measurement leading up to, during, and after their respective October 2020 elections.

## Coordinated Telegram testing in Cuba and Thailand

Starting from 15th October 2020, OONI measurements suggest the blocking of Telegram in Cuba, and we were told that 4 Telegram IPs were recently added to the Thai blocklist (though we have not seen Telegram blocked in Thailand yet).

We therefore coordinated the testing of Telegram in Cuba and Thailand to collect more measurements throughout October 2020.

In Cuba, community members reported that they were unable to use the OONI Probe mobile app (following the blocking of Telegram), but the OONI Probe desktop app continued to work (which is how many measurements continued to be collected). We spent resources in October 2020 investigating why the OONI Probe mobile app didn’t work in Cuba.

### SMEX resource including OONI Probe

SMEX published a post titled “Website Blocking in the Arab Region: Monitoring and Counteraction Techniques” (a translation of the Arabic resource published previously), which is available here: https://smex.org/website-blocking-in-the-arab-region-monitoring-and-countera...

Among other tools, OONI Probe is mentioned in this post as a tool for measuring internet censorship in the Arab region.

### OONI Community Meeting

On 27th October 2020, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:

1. Updates from the OONI team

2. Integrating the DNScheck test into the OONI Probe mobile and desktop apps (https://github.com/ooni/probe-engine/tree/master/experiment/dnscheck)

3. Analyzing OONI data.

## Userbase

In October 2020, 5,324,739 OONI Probe measurements were collected from 5,333 networks in 204 countries around the world.

This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/

~ OONI team.