Hello Tor world,
Happy new year!
In this email I share OONI's status report for November 2019, and our December 2019 report will follow next week. Apologies for the delay!
# OONI Monthly Report: November 2019
In November 2019, OONI started working on the DRL Anti-censorship grant in collaboration with The Tor Project.
We also started using 2 week cycles to track progress, effort, and priority estimation in order to improve how we estimate and track the work we do on projects.
In particular, the work related to the DRL Anti-censorship grant is being tracked on the following public GitHub project board: https://github.com/orgs/ooni/projects/18
## OONI measurements are now near real-time
We are very excited to announce that starting from November 2019, OONI measurements are getting published on OONI Explorer and in the API in near real-time!
This means that from the moment an OONI Probe user runs a test to it appearing in OONI Explorer, the delay has gone down from a maximum of 24h to an order of minutes.
## Report on internet blackout in Iran
In collaboration with IODA and Iranian community members (Kandoo), we co-published a research report examining the internet blackout in Iran: https://ooni.org/post/2019-iran-internet-blackout/
Community members translated the report to Persian: https://paskoocheh.com/blog/posts/2019-11-29-iran-internet-shutdown-ooni-rep...
## Psiphon test integration
The primary focus during November was the integration of the Psiphon test into probe-engine, probe-cli and consequently the OONI Probe Desktop app.
We picked this as the first item as we had already done a bit of cursory research into the problem and had a clear idea of how we would go about integrating and supporting it.
The integration work done for the Psiphon test is also very propaedeutic towards doing other golang based circumvention tests, such as obfs4proxy and other golang based Tor pluggable transports.
Our goal is that of getting Psiphon integrated inside of probe-engine, so that we are able to run the test from probe-cli which is in turn run by the OONI Probe Desktop app.
This issue is tracked as part of: https://github.com/ooni/probe-engine/issues/89
Throughout November 2019, we made a lot of progress and completed most of the items necessary to achieve this goal.
In particular, we ensured we’re using the latest version of Psiphon dependencies (see: https://github.com/ooni/probe-engine/issues/141), made sure the data format of the test result matches the old implementation (see: https://github.com/ooni/probe-engine/issues/128) and re-enabled the ability to build the Psiphon test inside of probe-cli (see: https://github.com/ooni/probe-cli/issues/79), making it possible to build the Psiphon test from the command line interface used by the OONI Probe desktop app.
Other completed issues relevant to Psiphon integration include:
* httpx implement NewClientWithProxy (https://github.com/ooni/netx/issues/130), which allows us to test that Psiphon is working by using it * HTTPDoConfig: add support for proxy function (https://github.com/ooni/netx/issues/131), which enables using Psiphon as a proxy * porcelain: add proxy and byte counting support (https://github.com/ooni/netx/pull/134), which allows us to measure the amount of bytes sent and received over the Psiphon proxy * x/logger: mimic cURL logging (https://github.com/ooni/netx/issues/137), which has been useful for more easily debugging the new Psiphon test * psiphon: use netx so we fill the HTTP template (https://github.com/ooni/probe-engine/issues/129), which allows us to collect low-level events and generate the Psiphon test data format * orchestra: implement authenticated requests (https://github.com/ooni/probe-engine/issues/148), which allows us to fetch the Psiphon configuration from the OONI orchestra services
On the OONI Probe desktop user interface front, with support from Antonela from the Tor UX team, we designed and implemented the Circumvention test card which will allow us to trigger the Psiphon test, but also all the circumvention tests we will be developing.
For reference, see: * Add circumvention tool tests icon: https://github.com/ooni/design-system/issues/72 * Mockups for the Performance test card: https://github.com/ooni/probe/issues/804#issuecomment-559824046
Based on these mockups, we worked on adding support for triggering the circumvention tests inside of the OONI Probe Desktop app, see: https://github.com/ooni/probe-desktop/pull/88.
## OONI testing engine improvements
We made a series of improvements related to the OONI Probe testing engine such as adding support for serialising binary header values in probe-engine (https://github.com/ooni/probe-engine/issues/110) and adding documentation for it in ooni/spec (https://github.com/ooni/spec/issues/164).
When performing HTTP requests in the Telegram golang test rewrite, we are now sure to include the Accept and Accept-Language headers to look more similar to a real browser https://github.com/ooni/probe-engine/issues/113.
We also dealt with several other low hanging fruit in the probe-engine and ooni-spec.
See: * porcelain: confusing that two errors are returned https://github.com/ooni/netx/issues/141 * dash: write tests and reach 100% coverage https://github.com/ooni/probe-engine/issues/93 * Bump data format to v0.3.0: https://github.com/ooni/spec/issues/163 * Bump data format to v0.3.0: https://github.com/ooni/probe-engine/issues/111 * Update to ooni/probe-assets@20191129161951: https://github.com/ooni/probe-engine/issues/116 * Update well know OONI hosts we use: https://github.com/ooni/probe-engine/pull/117
### OONI Probe Desktop app
Beyond the Psiphon test specific improvements listed above, we also made several smaller improvements to the OONI Probe Desktop app, mostly related to UX and App stability.
We added a link to the risks documentation in the onboarding screen (https://github.com/ooni/probe-desktop/issues/60), fixed a UI issue with some icons not being aligned (https://github.com/ooni/probe-desktop/issues/71) and made the macOS dmg disk image more pretty (https://github.com/ooni/probe-desktop/issues/31).
We also did some brainstorming on how we could improve the usability of the card descriptions in the home page of the desktop app (https://github.com/ooni/probe-desktop/issues/63 & https://github.com/ooni/probe-desktop/issues/91).
### OONI Explorer and OONI API
On the OONI Explorer front, our focus was on improving the application performancing by dealing with low hanging fruit we are sure is going to make improvements to it, while at the same time collecting more metrics to evaluate where we should be directing our other efforts related to app performance.
In OONI Explorer, we fixed an issue with the OONI API requests being launched twice during search operations: https://github.com/ooni/explorer/issues/350.
## Infrastructure and backend improvements
During this month we dealt with several improvements and incidents affecting our backend infrastructure, such as:
* An incident related to the collector running out of disk space: https://github.com/ooni/pipeline/issues/265. * We implemented a pattern that allows us to deploy application using debian packages: https://github.com/ooni/pipeline/pull/264 * We resolved an issue with the input field missing from the fastpath: https://github.com/ooni/pipeline/pull/255 * We merged and deployed the event detector in production: https://github.com/ooni/pipeline/pull/228
## Test list updates
We collaborated with OutRight International on updating multiple test lists to include (more) LGBTQI websites: https://github.com/citizenlab/test-lists/pull/539
We also reviewed several test list updates by Netalitica researchers and opened pull requests for the following test lists:
* Pakistan: https://github.com/citizenlab/test-lists/pull/541 * Tunisia: https://github.com/citizenlab/test-lists/pull/543 * Sudan: https://github.com/citizenlab/test-lists/pull/544 * United Arab Emirates: https://github.com/citizenlab/test-lists/pull/545
## Community use of OONI data
### Freedom on the Net 2019 reports
OONI data/reports were cited in the following Freedom on the Net 2019 reports: * Venezuela: https://www.freedomonthenet.org/country/venezuela/freedom-on-the-net/2019
* Zimbabwe: https://www.freedomonthenet.org/country/zimbabwe/freedom-on-the-net/2019
* Zambia: https://www.freedomonthenet.org/country/zambia/freedom-on-the-net/2019
* Nigeria: https://www.freedomonthenet.org/country/nigeria/freedom-on-the-net/2019
* Kenya: https://www.freedomonthenet.org/country/kenya/freedom-on-the-net/2019
* Jordan: https://www.freedomonthenet.org/country/jordan/freedom-on-the-net/2019
* Iran: https://www.freedomonthenet.org/country/iran/freedom-on-the-net/2019
* Indonesia: https://www.freedomonthenet.org/country/indonesia/freedom-on-the-net/2019
* Ethiopia: https://www.freedomonthenet.org/country/ethiopia/freedom-on-the-net/2019
* Egypt: https://www.freedomonthenet.org/country/egypt/freedom-on-the-net/2019
* Cuba: https://www.freedomonthenet.org/country/cuba/freedom-on-the-net/2019
OONI data is also indirectly cited via the following reports (which cite the reports of community members that use OONI data): * Ukraine: https://www.freedomonthenet.org/country/ukraine/freedom-on-the-net/2019
* Malaysia: https://www.freedomonthenet.org/country/malaysia/freedom-on-the-net/2019
## Community activities
### OTF Summit
OONI's Arturo and Maria traveled to Taiwan to attend the OTF Summit (4th & 5th November 2019) where they facilitated OONI workshops. They also participated in the Open Taiwan Day on 6th November 2019.
### Blog post by Karisma Foundation
Our Colombian partners, Karisma Foundation, published a blog post where they encourage the use of OONI Probe during the November 21st strike: https://stats.karisma.org.co/ayudanos-a-establecer-y-documentar-posibles-blo...
### Community meeting
We facilitated the monthly OONI Community Meeting on Tuesday, 26th November 2019, on the OONI Slack channel (https://slack.ooni.org/).
As part of the meeting, we discussed: 1. Comprehensive testing of protocols
2. Testing the new OONI Probe Desktop apps: https://github.com/ooni/probe-desktop/releases/tag/v3.0.0-rc.3
3. Measuring internet blackouts from the edge of the network
4. Participating at the IGF
5. Request for feedback on OONI website re-design, FAQ, and glossary
## Userbase
In November 2019, OONI Probe was run 406,846 times from 5,417 different vantage points in 205 countries around the world.
This information can also be found through our stats on the OONI Explorer landing page: https://explorer.ooni.org/
~ The OONI team