Our new Signal test also contributes ongoing measurements on the blocking of the Signal app in Iran (
https://explorer.ooni.org/search?until=2021-07-30&since=2021-06-30&probe_cc=IR&test_name=signal) and China (
https://explorer.ooni.org/search?until=2021-07-30&since=2021-06-30&probe_cc=CN&test_name=signal), among other countries.
We recently added support to our backend for the adjustment of URL priorities (as part of URL prioritization work). This allows us to specify in which countries certain URLs should have higher priority for testing and empowers us to dynamically adjust these in response to censorship events.
Following the social media blockade in Cuba, we bumped up the priority of the GRP (Social Media) category code (from the Citizen Lab test lists:
https://github.com/citizenlab/test-lists/blob/master/lists/00-LEGEND-new_category_codes.csv) on 14th July 2021 for Cuba. This enabled us to detect (and collect measurements on) the temporary blocking of
facebook.com in Cuba (
https://explorer.ooni.org/search?until=2021-07-30&since=2021-06-30&probe_cc=CU&domain=www.facebook.com&only=anomalies). It also enabled us to detect (and collect measurements on) the blocking of
tiktok.com in Cuba (
https://explorer.ooni.org/search?until=2021-07-30&since=2021-06-30&probe_cc=CU&domain=www.tiktok.com). Following the prioritization of URLs from this category, we were able to collect a total of 960 measurements for the GRP category code from Cuba.
OONI’s response to the recent social media censorship events in Cuba has received the following press coverage:
https://www.nbcnews.com/tech/tech-news/cubans-protest-government-cracks-internet-access-messaging-apps-rcna1400https://www.wsj.com/articles/internet-powered-mass-protests-in-cuba-then-the-government-pulled-the-plug-11626358893?page=1https://cpj.org/2021/07/journalists-detained-and-harassed-internet-disrupted-amid-cuban-protests/Cuban organization, YucaByte, published a report documenting the censorship events in Cuba amid the protests (citing OONI data):
https://www.yucabyte.org/2021/07/12/11j-internet-cuba/## OONI Probe MobileWe released OONI Probe Android 3.2.0 (
https://github.com/ooni/probe-android/releases/tag/v3.2.0) and OONI Probe iOS 3.2.0 (
https://github.com/ooni/probe-ios/releases/tag/v3.2.0).
This release includes:
* A reminder encouraging users to enable automated testing
* A badge to indicate that a proxy is being used
* Support for minimizing a running test
* Support for validating URLs shared via OONI Run (
https://github.com/ooni/probe-android/pull/443)
* Font size fix on iPhone SE (
https://github.com/ooni/probe-ios/commit/8f3fcf90b286e7e9ffee18c4b8ea223f2f3470e4)
These features were added to encourage users to enable automated testing and contribute measurements regularly. In response to community requests, we also added a badge to indicate that a proxy is being used, and we added support for minimizing a running test (enabling users to make use of other features of the OONI Probe app while a test is running).
## OONI Probe DesktopWe released OONI Probe Desktop 3.5.2:
https://github.com/ooni/probe-desktop/releases/tag/v3.5.2This release fixed a critical bug in autoupdate which prevented downloading and installing updates.
In July 2021, we provided mentorship to Google Summer of Code (GSoC) student Shivam Shekhar, who worked on implementing end-to-end tests for OONI Probe Desktop. In particular, he implemented tests for the code responsible for running the OONI Probe command line tool (
https://github.com/ooni/probe-desktop/pull/231). Moreover, tests were implemented for several UI components as part of the following pull request:
https://github.com/ooni/probe-desktop/pull/229## Progress on browser-based OONI ProbeWe made some progress on the browser based implementation of OONI Probe. Specifically, we integrated React into the project so that we can reuse the existing components we developed for web based projects (see:
https://github.com/ooni/probe-web/commit/a9da79b9384f7f496bb3278a4a00cacc39973e11). We also did some research into how to tweak the fetch API calls so that we minimise the amount of sensitive data transmitted to the measured web hosts (see:
https://github.com/ooni/probe-web/commit/5bf0ba41a9e2464da070ba21f4463ddce23a3be5).
## Expanding OONI censorship measurement methodologiesWe looked into alternative TLS stacks that we could use for performing measurements, but potentially also for communicating with our backend services. This was prompted by the fact that we have noticed that some countries (such as Iran) will implement blocking of all TLS connections that present a specific fingerprint.
In particular, we looked into
https://github.com/refraction-networking/utls and researched how we could go about integrating it into the OONI codebase, as well as what work would be necessary to update it to the latest version of the golang TLS library. This work is documented through the following ticket:
https://github.com/ooni/probe/issues/1702In July 2021, we provided mentorship to Google Summer of Code (GSoC) student Kathrin Elmenhorst, with whom we collaborated on making several improvements to how errors are handled inside of our measurement engine. Specifically, we ensured that it’s possible to accurately map more classes of errors to OONI error strings, as documented through the following ticket:
https://github.com/ooni/probe/issues/1505We also made some progress on comparing the new golang-written Web Connectivity test helper to the old python-based one. The goal of this comparison is to determine how well-suited the new test helper is at replacing the legacy one and if, in doing the switch from one to the other, we are introducing any inaccuracies in the resulting measurements.
## Created mockups for new OONI Probe Snowflake test resultsWe implemented mockups for displaying the test results of our new Snowflake pluggable transport test inside of OONI Probe. This has been tracked as part of the following issue:
https://github.com/ooni/ooni.org/issues/759## Research on whether to ship PTs as a separate binaryWe carried out research to explore whether to ship pluggable transports as separate binaries or link them with the ooniprobe binary. Based on this research, we wrote an internal document which explains our design choices, PT spec considerations, and conclusions. In the end, we chose to integrate our new Snowflake experiment into ooniprobe as a Go library leveraging version 2 of the pluggable-transports specification:
https://github.com/ooni/ooni.org/issues/765## OONI backendIn July 2021, we worked on the following OONI backend activities:
* Fixed certbot nginx configuration bug (server/port overlap)
* Made improvements to the API rate limiting system and developed new integration tests to ensure it’s working as expected:
https://github.com/ooni/api/pull/256* Added support for the global prioritized testing of popular social media URLs:
https://github.com/ooni/backend/issues/507* Added support for the prioritized testing of the GRP Citizen Lab category code in Cuba
* Worked on the test helper rotation tool which automatically deploys on cloud infrastructure machines for running backend infrastructure. These are rotated automatically on a periodic basis:
https://github.com/ooni/backend/issues/496* As part of the development of the rotating test helpers, we also did a major rework to how TLS certificates are generated. Specifically we integrated it with DNS and certbot and deployed it to production.
* Test with probes on test API.
* API: Deployed test.ooni domains in CORS:
https://github.com/ooni/api/pull/259* API: Switched URL submission to account_id:
https://github.com/ooni/api/pull/260 * Reprocessed legacy measurements and created JSONL files
* Monitored measurement coverage by unattended runs
* Updated internal ASN coverage charts
* Developed charts that make predictions on the expected measurement volume based on the submission API. We have these charts implemented in our internal backend dashboards and have configured alerts to notify us if the expected measurement volume falls below a certain threshold.
* Implemented support for running end-to-end integration tests of the API and fastpath on our continuous integration platform. Through this setup we can ensure that a large part of the backend infrastructure is working as expected and that the API and fastpath are run on real OONI measurement data:
https://github.com/ooni/api/pull/258.
* Refactored scripts to run tests on different environments
## Building a web platform for test list updatesWe deployed the private beta of our new web platform for test list updates to production (
https://test-lists.ooni.org/login). We worked on fixing issues reported as part of internal team testing of the platform (
https://github.com/ooni/test-lists-ui/pull/2). We also shared the private beta with a few community members to collect their feedback, based on which we worked on further improvements, which are documented as issues here:
https://github.com/ooni/test-lists-ui/issuesOnce the private beta is further polished, we will share it more extensively to collect further community feedback before launching the platform.
## Collaboration with NetaliticaNetalitica researchers continued to do excellent work in reviewing and updating more Citizen Lab test lists. In July 2021, we reviewed their updates to the test lists for India and Burundi and shared feedback for improvements.
## Test list updatesIn July 2021, we contributed updates to the following Citizen Lab test lists:
* Russia:
https://github.com/citizenlab/test-lists/pull/805* Ethiopia:
https://github.com/citizenlab/test-lists/pull/804* Armenia:
https://github.com/citizenlab/test-lists/pull/803## New partnership with Internet Society (ISOC)In July 2021, we established a new partnership with Internet Society (ISOC).
As part of this partnership, OONI is now a data partner (
https://pulse.internetsociety.org/partners) for ISOC’s Pulse project on Internet shutdowns:
https://pulse.internetsociety.org/shutdownsAs a data partner, we will provide ISOC with relevant OONI data in support of their Pulse project which maps internet shutdowns around the world.
## Community activities
### Zaina Foundation Internet Shutdown TrainingOn 12th and 13th July 2021, Zaina Foundation organized and facilitated an Internet Shutdown Training for human rights defenders in Tanzania. One of the core objectives of this training was to train coalition members on OONI censorship measurement methodologies and to explore OONI data.
As part of this training, Zaina Foundation facilitated the following 2 OONI workshops:
1) Introduction to OONI Probe and OONI Explorer
2) OONI practical session
Following the training, Zaina Foundation shared community feedback with us and coordinated with local communities on translating OONI Probe to Swahili.
### OONI presentation at the Nexa Center for Internet and SocietyOn 28th July 2021, OONI’s Arturo presented OONI at the Nexa Center for Internet and Society of the Polytechnic University of Turin. As part of his presentation, Arturo provided OONI updates on the state of internet censorship worldwide in 2020.
Information about the event is available here:
https://nexa.polito.it/lunch-90The presentation can be viewed through the following link:
https://www.youtube.com/watch?v=fO53FLTu4a8### Access Now webinar on internet shutdownsOn 30th July 2021, OONI’s Maria presented OONI as part of Access Now’s webinar titled: “#KeepItOn during an internet shutdown: How to measure, document, and circumvent network interference”. In particular, she discussed how OONI’s tools and data can be used for measuring partial internet shutdowns (involving the blocking of websites and apps).
This webinar was live-streamed on YouTube:
https://youtu.be/aFIBrRXKm2E### OONI documentation in FarsiIranian community members created detailed and comprehensive documentation which explains OONI tools and methodologies in Farsi.
This documentation is available here:
https://wikicensorship.github.io/fa/docs/measure-internet-censorship/OONI/The Farsi documentation covers the following areas:
* Investigate Internet censorship through the OONI probe
* How Websites Test Works
* Directly enter custom addresses in the application
* Create a custom test list
* Analysis of results
* Simple analysis of results
* Advanced results analysis
* Diagnosis of sanctions
* Internal breaker setting only for backend connection
* Data mining in OONI results data
### OONI Probe video tutorial in SpanishOur Bolivian partners, Fundación Internet Bolivia (
https://ooni.org/partners/fundacion-internet-bolivia.org/), created a video tutorial in Spanish where they present OONI and explain OONI Probe, providing a live demo of using the tool.
This is available here:
https://peertube.tv/videos/watch/e6991978-a3d7-4718-b688-482a9a4de525### OONI Community MeetingOn 27th July 2021, we hosted the monthly OONI Community Meeting on our Slack channel (
https://slack.ooni.org/), during which we discussed the following topics:
1) New web platform for updating the Citizen Lab test lists: Request for community feedback
2) Adding Signal under instant messaging tests for Debian/Ubuntu OONI Probe users
3) Backend proxy for OONI Probe Desktop and OONI Probe CLI
## UserbaseIn July 2021, 13,522,345 OONI Probe measurements were collected from 4,969 AS networks in 195 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/~ OONI team.