Hello Oonitarians and Divisionists,
I would love to have your feedback on what you believe to be the most important topics for the future of OONI.
I have made a list of what I believe are all possible and interesting tasks to perform, but we can't do them all and for sure we can't do them all at once.
For this reason it would be very useful if you could express a vote from 1-5:
1: Nah, this is boring and pointless 2: Not really super important, I would give priority to other stuff 3: Useful, I would do it 4: This would be awesome 5: Epic!
Feel free to also expand these topics with questions and feedback (or new ones). At the end of the list I will give you a table to cast your vote.
# Get daily OONI measurements from 50 countries
This means focusing on getting a large and diverse reliable OONI userbase. It does not simply mean to get at least 1 measurement in 50 countries, but it means either establishing relationships with trusted parties in 50 countries, or expanding our userbase to a point where we have at least 1 measurement per day for every test per country (from the same network).
Ways of achieving this are:
* Rent a VPS in X number of countries * Running the adopt an ooniprobe program (see below for more details) * Establish an agreement with operators in 50 countries to host the probing infrastructure (as previously stated)
# Develop OONI tests for censorship circumvention tools
This involves devising a methodology for testing the reliability of censorship circumvention tools in various countries. It means testing censorship circumvention tool both open source and propertairy.
A cursory list of the protocols/tools we could be interested includes, but is not limited to:
* Tor * VPN * Web proxies (hidemyass, etc.) * SSH tunnel * Freegate * Psiphon * Ultrasurf * alkasir
If you think we should be testing some other tools too, please add to this list.
Useful resources: http://cyber.law.harvard.edu/publications/2011/2011_Circumvention_Tool_Evalu... http://cyber.law.harvard.edu/publications/2010/Circumvention_Tool_Usage
# Develop scheme for orchestrating ooni-probes
This means coming up with a protocol that allows an OONI test developer to schedule a measurement to be run with a certain input they decide on a set of probes in country X.
Obviously security considerations need to be taken into account and access will be in the initial stage only limited to a very restricted set of OONI developers that will be made public.
# Implement data analytics and visualization for OONI tests
We have a bunch of data and we would like to give meaning to it. This would involve writing tools for querying the data in the database and extract useful analytical information from it.
Based on this data we can then start looking at historical OONI data and provide some sample visually supported reports.
This is a list of tests we should develop analytics for:
* HTTP requests * DNS Consistency * DNS Injection * TCP Connect * HTTP Invalid Request Line * HTTP Header Field Manipulation * Multi protocol port traceroute
# Implement pub-sub system for ooni collectors
Currently OONI collectors (the things you send your ooniprobe measurement results to) keep in sync thanks to a bunch of shell scripts and cronjobs. To have more real time data it would be useful to have a pub-sub mechanism that allows the pipeline to subscribe to all the collectors and the collectors will then publish the collected reports to it, as soon as they are submitted.
This will allow the OONI data to go through the data pipeline much faster (instead of ~2 hours, perhaps just some minutes or even less potentially).
# Reach production quality ooni rasperry-pi (beagle-board) images
This involves implementing what is specified in the lepidopter specification: https://github.com/anadahz/lepidopter/blob/master/specification.md
We should then provide scripts for building the image yourself or how to download and burn it to an SD card on Windows, OSX, Linux (with screenshots).
As a bonus we could also offer shipping of pre-made raspberry pi images already burn to an SD card, similar to what is done with rasbpian images.
# Promote and further develop OONI on mobile (Android, iOS)
This involves improving the GUI of OONI on mobile and getting it into the Google Play store and the Apple App store.
We should also work on making it easier for developers of existing iOS and Android apps to add internet measurement capabilities to their app by linking to libight.
# Do research based on OONI
This would involve doing some research on internet censorship based on OONI probe or on internet measurement in general and publishing them in peer reviewed venues.
# Publish monthly reports about the status of internet censorship in a country
This would be sort of like a monthly e-zine, where every month we analyse the status of internet censorship in a given country. It should be backed by OONI data, but the core of it should be an editorial type piece explaining the context of the country and how censorship is affecting them.
I expect most of this work to be carried out by somebody that has a background in journalism and social sciences.
# Run "adopt an ooni-probe" campaign
This is basically a program where people make a donation to support getting somebody in an interesting country an ooniprobe or receiving an ooniprobe themselves.
# Integration with other censorship measurement projects
This means finding better ways of using data from other projects as part of OONI research or getting them to use our data.
# Reaching out to communities inside of censored regions
This means getting a better understanding of what "benefit" we can give to people that are running the tool. On this topic also see: https://trac.torproject.org/projects/tor/ticket/14760
# Redesign the website for ooni
This means removing the horrible bootstrap basic template of the current website and restructuring it so people actually understand what OONI is and how it can be useful to them.
# Hold an international internet censorship conference
This means running an event where we invite a bunch of people that are active in the field of internet censorship and discuss ideas and possible collaborations.
# Implement a GUI for ooniprobe
This means picking up where we left off with the OONI web gui and finish implementing it.
This could potentially also be useful for the orchestration task as we could implement access control on the HTTP API that the web interface consumes.
Goal | Vote ---------------------------------- Get daily OONI measurements | from 50 countries | 0 ---------------------------------- Develop OONI tests for | censorship circumvention | tools | 0 ---------------------------------- Develop scheme for | orchestrating ooni-probes | 0 ---------------------------------- Implement data analytics | and visualization for OONI | tests | 0 ---------------------------------- Implement pub-sub system | for ooni collectors | 0 ---------------------------------- Reach production quality | ooni rasperry-pi | (beagle-board) images | 0 ---------------------------------- Promote and further | develop OONI on mobile | (Android, iOS) | 0 ---------------------------------- Publish monthly reports | about the status of | internet censorship in | a country | 0 ---------------------------------- Run "adopt an ooni-probe" | campaign | 0 ---------------------------------- Integration with other | censorship measurement | projects | 0 ---------------------------------- Reaching out to communities| inside of censored regions | 0 ---------------------------------- Redesign the website for | ooni | 0 ---------------------------------- Hold an international | internet censorship | conference | 0 ---------------------------------- Implement a GUI for | ooniprobes | 0 ---------------------------------- Do research based on | OONI | 0 ----------------------------------
Thanks for taking the time to go through this long email, you can as always find us on IRC #OONI irc.oftc.net.
Have fun!
~ Arturo
Dear Arturo and the other OONI-devs,
I have a couple ideas I wanted to share about possible new projects or directions for OONI.
1. a traceroute idea
I recently wrote a novel TCP traceroute implementation in golang called ParasiticTraceroute. It uses Linux NFQueue to mangle local TCP flows... altering the TTL and thus effecting a traceroute. Thus it does forward and reverse TCP traceroute... and if you traceroute your own server with this tool you might be able to learn interesting things about it clients... like for instance penetrate their NAT devices and learn the RFC1918 addresses of those NAT device interfaces. Perhaps. I haven't tried to do this yet.
Leif Ryge, Aaron Gibson and I came up with this traceroute idea last year... and Leif recently suggested I write a traceroute server: it traceroutes the clients that connect to it... and then sends the traceroute results to the client... while concurrently the client performs a traceroute to the server. This results in the client procuring a traceroute for both directions; which is useful because asymmetrical routing.
These traceroute-helper servers could be distributed around the world... and we could have an ooniprobe test query them to compare the current traceroutes with previous traceroutes to detect BGP route changes which under some circumstances but useful information about a censorship event... and perhaps help us identify which BGP ASNs are involved.
Ethan Katz-Bassett and other researchers at University of Washington have done some really excellent research into designing an even more sophisticated reverse traceroute system. Their system does not require cooperative servers or clients... I highly recommend watching Ethan's video presentation or reading their paper:
http://research.cs.washington.edu/networking/astronomy/reverse-traceroute.ht...
2. cooperate with ISPs/transit providers and Tor exit relay operators to collection statistics about interesting TCP "events"
These events in my mind fall into 3 categories of interesting:
- DOS attacks: SYN flood etc - censorship events: injected RST or FIN packets - injection attacks: segment veto, handshake hijack etc.
The most interesting to me of course is the TCP injection attacks...
It should be possible to gather very interesting statistics with help from the people running the network infrastructure even if there are strict telecommunications laws that prohibit the capturing of the content. I have started to work on a tool called HoneyBadger: it detects TCP injection attacks and performs full-take logging of all the data... however an anonymized or metadata only statistics gathering mode for HoneyBadger could be used by ISPs for instance.
https://honeybadger.readthedocs.org/en/latest/
By the way... I'd really rather collaborate and receive peer review on these types of low-level network programming projects because it would be more fun and more effective to develop things like this with help and critical feedback. =-)
Would OONI be interested in receiving statistics from "network infrastructure providers" about how often their transit users get TCP injected? Might it also be interesting to know how often users receive an injected RST or FIN from an automatic censorship device?
btw I am very inspired by the work journalistic reporting work of Jake Appelbaum, Leif Ryge, Aaron Gibson, Morgan Mayhem who are writing articles to explain to the general public that plaintext protocols are especially vulnerable to these TCP injection attacks... AND that these attacks are often used for targeted surveillance by various groups around the world... and they are even selling ready made devices that automate these attacks.
Would Tor Project/OONI be interested in helping to raise awareness of these issues?
Sincerely,
David Stainton
On Tue, Feb 10, 2015 at 2:50 PM, Arturo Filastò art@torproject.org wrote:
Hello Oonitarians and Divisionists,
I would love to have your feedback on what you believe to be the most important topics for the future of OONI.
I have made a list of what I believe are all possible and interesting tasks to perform, but we can't do them all and for sure we can't do them all at once.
For this reason it would be very useful if you could express a vote from 1-5:
1: Nah, this is boring and pointless 2: Not really super important, I would give priority to other stuff 3: Useful, I would do it 4: This would be awesome 5: Epic!
Feel free to also expand these topics with questions and feedback (or new ones). At the end of the list I will give you a table to cast your vote.
# Get daily OONI measurements from 50 countries
This means focusing on getting a large and diverse reliable OONI userbase. It does not simply mean to get at least 1 measurement in 50 countries, but it means either establishing relationships with trusted parties in 50 countries, or expanding our userbase to a point where we have at least 1 measurement per day for every test per country (from the same network).
Ways of achieving this are:
- Rent a VPS in X number of countries
- Running the adopt an ooniprobe program (see below for more details)
- Establish an agreement with operators in 50 countries to host the
probing infrastructure (as previously stated)
# Develop OONI tests for censorship circumvention tools
This involves devising a methodology for testing the reliability of censorship circumvention tools in various countries. It means testing censorship circumvention tool both open source and propertairy.
A cursory list of the protocols/tools we could be interested includes, but is not limited to:
- Tor
- VPN
- Web proxies (hidemyass, etc.)
- SSH tunnel
- Freegate
- Psiphon
- Ultrasurf
- alkasir
If you think we should be testing some other tools too, please add to this list.
Useful resources: http://cyber.law.harvard.edu/publications/2011/2011_Circumvention_Tool_Evalu... http://cyber.law.harvard.edu/publications/2010/Circumvention_Tool_Usage
# Develop scheme for orchestrating ooni-probes
This means coming up with a protocol that allows an OONI test developer to schedule a measurement to be run with a certain input they decide on a set of probes in country X.
Obviously security considerations need to be taken into account and access will be in the initial stage only limited to a very restricted set of OONI developers that will be made public.
# Implement data analytics and visualization for OONI tests
We have a bunch of data and we would like to give meaning to it. This would involve writing tools for querying the data in the database and extract useful analytical information from it.
Based on this data we can then start looking at historical OONI data and provide some sample visually supported reports.
This is a list of tests we should develop analytics for:
- HTTP requests
- DNS Consistency
- DNS Injection
- TCP Connect
- HTTP Invalid Request Line
- HTTP Header Field Manipulation
- Multi protocol port traceroute
# Implement pub-sub system for ooni collectors
Currently OONI collectors (the things you send your ooniprobe measurement results to) keep in sync thanks to a bunch of shell scripts and cronjobs. To have more real time data it would be useful to have a pub-sub mechanism that allows the pipeline to subscribe to all the collectors and the collectors will then publish the collected reports to it, as soon as they are submitted.
This will allow the OONI data to go through the data pipeline much faster (instead of ~2 hours, perhaps just some minutes or even less potentially).
# Reach production quality ooni rasperry-pi (beagle-board) images
This involves implementing what is specified in the lepidopter specification: https://github.com/anadahz/lepidopter/blob/master/specification.md
We should then provide scripts for building the image yourself or how to download and burn it to an SD card on Windows, OSX, Linux (with screenshots).
As a bonus we could also offer shipping of pre-made raspberry pi images already burn to an SD card, similar to what is done with rasbpian images.
# Promote and further develop OONI on mobile (Android, iOS)
This involves improving the GUI of OONI on mobile and getting it into the Google Play store and the Apple App store.
We should also work on making it easier for developers of existing iOS and Android apps to add internet measurement capabilities to their app by linking to libight.
# Do research based on OONI
This would involve doing some research on internet censorship based on OONI probe or on internet measurement in general and publishing them in peer reviewed venues.
# Publish monthly reports about the status of internet censorship in a country
This would be sort of like a monthly e-zine, where every month we analyse the status of internet censorship in a given country. It should be backed by OONI data, but the core of it should be an editorial type piece explaining the context of the country and how censorship is affecting them.
I expect most of this work to be carried out by somebody that has a background in journalism and social sciences.
# Run "adopt an ooni-probe" campaign
This is basically a program where people make a donation to support getting somebody in an interesting country an ooniprobe or receiving an ooniprobe themselves.
# Integration with other censorship measurement projects
This means finding better ways of using data from other projects as part of OONI research or getting them to use our data.
# Reaching out to communities inside of censored regions
This means getting a better understanding of what "benefit" we can give to people that are running the tool. On this topic also see: https://trac.torproject.org/projects/tor/ticket/14760
# Redesign the website for ooni
This means removing the horrible bootstrap basic template of the current website and restructuring it so people actually understand what OONI is and how it can be useful to them.
# Hold an international internet censorship conference
This means running an event where we invite a bunch of people that are active in the field of internet censorship and discuss ideas and possible collaborations.
# Implement a GUI for ooniprobe
This means picking up where we left off with the OONI web gui and finish implementing it.
This could potentially also be useful for the orchestration task as we could implement access control on the HTTP API that the web interface consumes.
Goal | Vote
Get daily OONI measurements | from 50 countries | 0
Develop OONI tests for | censorship circumvention | tools | 0
Develop scheme for | orchestrating ooni-probes | 0
Implement data analytics | and visualization for OONI | tests | 0
Implement pub-sub system | for ooni collectors | 0
Reach production quality | ooni rasperry-pi | (beagle-board) images | 0
Promote and further | develop OONI on mobile | (Android, iOS) | 0
Publish monthly reports | about the status of | internet censorship in | a country | 0
Run "adopt an ooni-probe" | campaign | 0
Integration with other | censorship measurement | projects | 0
Reaching out to communities| inside of censored regions | 0
Redesign the website for | ooni | 0
Hold an international | internet censorship | conference | 0
Implement a GUI for | ooniprobes | 0
Do research based on | OONI | 0
Thanks for taking the time to go through this long email, you can as always find us on IRC #OONI irc.oftc.net.
Have fun!
~ Arturo _______________________________________________ ooni-dev mailing list ooni-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
Arturo Filastò art@torproject.org writes:
Hello Oonitarians and Divisionists,
I would love to have your feedback on what you believe to be the most important topics for the future of OONI.
I have made a list of what I believe are all possible and interesting tasks to perform, but we can't do them all and for sure we can't do them all at once.
For this reason it would be very useful if you could express a vote from 1-5:
1: Nah, this is boring and pointless 2: Not really super important, I would give priority to other stuff 3: Useful, I would do it 4: This would be awesome 5: Epic!
Feel free to also expand these topics with questions and feedback (or new ones). At the end of the list I will give you a table to cast your vote.
<snip>
Hello,
instead of the above ranking system, I think my ranks mainly represent how much priority each item should receive IMO. I think all the ideas stated are good and aligned with the OONI agenda.
Also, some tasks are easier than others, so even though I think that the "pub-sub system" would be a good thing to have, I think improving the OONI website will probably take much less time and its equally important.
Also some tasks are connected. So for example, if you go ahead with the plan to make good quality rasperry-pi images, then it should be followed by the "adopt an ooni-probe" idea.
I mainly gave priority to tasks that give more visibility to the data collected so far and in the near future (visualizations, better website, monthly reports, etc.). At the same time it's important to get more vantage points and *stable* measurement sources.
Hopefully this is useful, and I will be glad to talk about this more in person.
Have a good development time!
(5: high priority, 0: no priority)
Goal | Vote ---------------------------------- Get daily OONI measurements | from 50 countries | 4 ---------------------------------- Develop OONI tests for | censorship circumvention | tools | 4 ---------------------------------- Develop scheme for | orchestrating ooni-probes | 3 ---------------------------------- Implement data analytics | and visualization for OONI | tests | 5 ---------------------------------- Implement pub-sub system | for ooni collectors | 3 ---------------------------------- Reach production quality | ooni rasperry-pi | (beagle-board) images | 4 ---------------------------------- Promote and further | develop OONI on mobile | (Android, iOS) | 3 ---------------------------------- Publish monthly reports | about the status of | internet censorship in | a country | 4 ---------------------------------- Run "adopt an ooni-probe" | campaign | 3 ---------------------------------- Integration with other | censorship measurement | projects | 3 ---------------------------------- Reaching out to communities| inside of censored regions | 3 ---------------------------------- Redesign the website for | ooni | 4 ---------------------------------- Hold an international | internet censorship | conference | 2 ---------------------------------- Implement a GUI for | ooniprobes | 3 ---------------------------------- Do research based on | OONI | ? ----------------------------------
Thanks for taking the time to go through this long email, you can as always find us on IRC #OONI irc.oftc.net.
Have fun!
~ Arturo _______________________________________________ ooni-dev mailing list ooni-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
Arturo Filastò wrote:
Hello Oonitarians and Divisionists,
I would love to have your feedback on what you believe to be the most important topics for the future of OONI.
I have made a list of what I believe are all possible and interesting tasks to perform, but we can't do them all and for sure we can't do them all at once.
For this reason it would be very useful if you could express a vote from 1-5:
1: Nah, this is boring and pointless 2: Not really super important, I would give priority to other stuff 3: Useful, I would do it 4: This would be awesome 5: Epic!
Feel free to also expand these topics with questions and feedback (or new ones). At the end of the list I will give you a table to cast your vote.
Hi Arturo,
I gave 5 to the monthly reports because I think it would be great to explain to a broad audience how censorship is working in a country.
Since I believe that reports are so important, I have framed my answers mostly in terms of them, hence I gave 4:
- to OONI tests for circumvention tools, because I think people would be very interested to read about that in the reports;
- to reaching out to communities and to running measurements in 50 countries, because doing that is functional to prepare the reports;
- to the Raspberry PIs image and to the mobile Apps because they could be two ways to lower the barrier to installing OONI and running tests, which is again functional to all the above.
I gave 3 to all the rest, except the international conference to which I gave 2, because, well, "I would give priority to other stuff" :-).
Thanks,
Simone
- - -
Goal | Vote ---------------------------------- Get daily OONI measurements| from 50 countries | 4 ---------------------------------- Develop OONI tests for | censorship circumvention | tools | 4 ---------------------------------- Develop scheme for | orchestrating ooni-probes | 3 ---------------------------------- Implement data analytics | and visualization for OONI | tests | 4 ---------------------------------- Implement pub-sub system | for ooni collectors | 3 ---------------------------------- Reach production quality | ooni rasperry-pi | (beagle-board) images | 4 ---------------------------------- Promote and further | develop OONI on mobile | (Android, iOS) | 4 ---------------------------------- Publish monthly reports | about the status of | internet censorship in | a country | 5 ---------------------------------- Run "adopt an ooni-probe" | campaign | 3 ---------------------------------- Integration with other | censorship measurement | projects | 3 ---------------------------------- Reaching out to communities| inside of censored regions | 4 ---------------------------------- Redesign the website for | ooni | 3 ---------------------------------- Hold an international | internet censorship | conference | 2 ---------------------------------- Implement a GUI for | ooniprobes | 3 ---------------------------------- Do research based on | OONI | 3 ----------------------------------
Arturo,
I appreciate you taking the lead on this and found it extremely helpful for understanding some of the possible trajectories of OONI's development (especially in appreciation for the resource gap that comes naturally to this space). Like George, my votes reflect prioritization -- and in retrospect I am biased toward infrastructure maturity. The activities "Implement a GUI for ooniprobes" and "Implement data analytics and visualization for OONI tests" received highest marks, because I think they feed into the research work that I think everyone is eager to take on. I think they will also help you find the 50 volunteers, keep them engaged in the long term and help them understand what they are doing. I also believe that doing this now will provide a closer feedback loop than doing so after probes are shipped.
Missing from the list: informed consent research :)
The future is bright.
Goal | Vote
Get daily OONI measurements | from 50 countries | 2
Develop OONI tests for | censorship circumvention | tools | 3
Develop scheme for | orchestrating ooni-probes | 5
Implement data analytics | and visualization for OONI | tests | 5
Implement pub-sub system | for ooni collectors | 4
Reach production quality | ooni rasperry-pi | (beagle-board) images | 4
Promote and further | develop OONI on mobile | (Android, iOS) | 4
Publish monthly reports | about the status of | internet censorship in | a country | 2
Run "adopt an ooni-probe" | campaign | 2
Integration with other | censorship measurement | projects | 3
Reaching out to communities| inside of censored regions | 2
Redesign the website for | ooni | 1
Hold an international | internet censorship | conference | 1
Implement a GUI for | ooniprobes | 5
Do research based on | OONI | 3
On Tue, Feb 10, 2015 at 03:50:33PM +0100, Arturo Filastò wrote:
# Implement data analytics and visualization for OONI tests
I would love to see some work on this. To my understanding, OONI already has a lot of interesting data but you have to be a programmer to explore it. Having a simple GUI for exploring or at least some command line tools would go a long way and encourage people to dig into the data.
Cheers, Philipp
Hi,
Sorry for taking a while to get to this.
On Tue, Feb 10, 2015 at 03:50:33PM +0100, Arturo Filastò wrote:
# Develop OONI tests for censorship circumvention tools
This is something I'm particularly interested in.
My current work includes adding support for OONI to detect whether or not UDP and other transports can be used from a network, and this work would allow for testing other application layer protocols over these transports, such as VPN protocols etc.
Goal | Vote
Get daily OONI measurements | from 50 countries | 5
Develop OONI tests for | censorship circumvention | tools | 5
Develop scheme for | orchestrating ooni-probes | 5
Implement data analytics | and visualization for OONI | tests | 5
Implement pub-sub system | for ooni collectors | 4
Reach production quality | ooni rasperry-pi | (beagle-board) images | 5
Promote and further | develop OONI on mobile | (Android, iOS) | 5
Publish monthly reports | about the status of | internet censorship in | a country | 4
Run "adopt an ooni-probe" | campaign | 5
Integration with other | censorship measurement | projects | 5
Reaching out to communities| inside of censored regions | 5
Redesign the website for | ooni | 3
Hold an international | internet censorship | conference | 4
Implement a GUI for | ooniprobes | 4
Do research based on | OONI | 5
Iain.
Thank you all for the feedback and very useful comments!
I will quote replies from another thread and comment on some of the topics there.
At the end based on the feedback received I will make a list of what are the next main areas of focus for the next 6 months, some of the relevant tickets and what new tickets should be created.
If you have ideas on some specific topic/issue/feature that you believe should be tackled, please do append to this list.
On 2/17/15 5:28 PM, Nick Feamster wrote:
There are many interesting ideas in here. Thanks!
Off the top of my head, one could get a better global picture of censorship using data from the “global censorship measurement” tools (Sam’s Encore tool, Roya and Ben’s “Censored Planet” project) to trigger measurements from OONI, and vice versa.
Tools such as Encore and CP can get a signal of filtering from a larger and more diverse set of clients than an OONI deployment could (and, at lower risk); the drawback is a lack of detailed information (typically the information is a binary “yes/no” about whether filtering is taking place in TCP/IP, DNS, or HTTP, but not much else).
I could imagine an OONI deployment using information about observed filtering from Encore or CP to trigger a more extensive and detailed set of tests from the OONI nodes. Likewise, blocking observed at one OONI node at one layer could be fed back into Encore or CP to see whether any observable filtering behavior is observed across a broader range of sites.
I view this as a specific “research use case” for the integration, orchestration, and data analysis points that Arturo lists below.
I think Roya and Ben are probably not quite ready for this with CP (I’d want to ask them), but Encore is very stable and we could look into good ways of passing information back and forth, either to OONI or to a common visualization engine (or both).
Some of the other efforts look interesting, but the research payoff isn’t as readily apparent to me. (I’m suspicious of monthly reports without broader, more global baseline coverage like Encore or CP could provide, but we are also quite interested in visualization tools and reports for that data, so integrating all of that into a single “dashboard” would be something that could be very useful, based on what political scientists and others seem to be asking for.)
This is indeed a very interesting concept. With respect to using ooniprobe data to trigger other tests, this will be easier once we finish implementing the API to the pipeline. We now have all the data collected with ooniprobe inside of a database and will be working (mainly to provide analytics and visualization tools) on exposing access and query functionality.
Work on this has only recently started, but you can find code on it here: https://github.com/hellais/ooni-app.
With resepect to triggering ooniprobe measurements based on data collected by encore, censored planet, etc. I think we would need to support: https://trac.torproject.org/projects/tor/ticket/12551 unless we consider it acceptable to have to wait some time before all deployed probes run the measurement.
On 2/17/15 5:36 PM, Phillipa Gill wrote:
I would say the daily measurements from stable nodes is less interesting for us since ICLab is already 99% of the way there on those (ie., running baseline tests from VPN + deployed endpoints).
I think the idea of detecting different censorship circumvention tools could be interesting and if the tests are well specified this could be something that is ported/run in ICLab as well.
Yes we plan on openly specifying all the tests that we will be deploying in a study in a specific country.
I am not sure how many details I can disclose at the moment on this, but will be sure to update the list when more is known.
On 2/17/15 8:53 PM, Meredith Whittaker wrote:
The biggest issues that I see for OONI, and all efforts in this nascent space are around:
- *Getting consistent measurements at scale, over time, across broad
geographies.* - Consistent = from a stable set of tests that, *if they do change, are clearly documented as changed*, and this versioning is reflected in collected data and elsewhere. The less change, the better (which isn't to say that new tests are bad, at all, but that there needs to be set "cannon" of core tests that work to set a baseline. Without this, much of the research etc. is much, much less valuable.) - at scale = a lot -- a significant number of representative links, mapping diurnal patterns, etc. (omg obvs, but still) - over time = for as long as possible -- letting us compare the results of Stable Text X for Place Y in 2015 vs 2017, etc.. - Across broad geographies = how *are* UK censorship techniques different from those used in Liberia? Etc.
- *Dealing with issues around user consent and risk. *
measurement points can be decoupled from individuals (i.e. can
- This is huge. We all know that. For now, the more OONI probes and
be deployed without requiring a user to download, install, flash, access), the better. Impersonal Pis, or similar, seem like the best option for this currently. But, even these pose risk if they can be linked to an individual (scapegoat) in a hostile country. The bigger OONI's impact, the bigger this problem.
I would advocate that any work undertaken focus on these two goals -- which I believe to be fundamental to many of the great other goals that have been put forward here.
With that declaration, my specific comments on the individuals ideas in Arturo's thoughtful email:
- *Getting daily OONI measurements from 50 countries*
This is clearly a laudable goal. I am concerned with the means suggested for achieving this. How can it be done without placing users in danger? How can an answer to that question be obtained for 50 countries (and orders of magnitude more regions, factions, climates within each country)? This is also ambitious practically/logistically -- this is an increase in vantage points that will likely require localization, maintenance, support, troubleshooting, and consistent and well-documented updates, such that apples can be compared to apples and data can stand as "proof." A roadmap that narrows the focus, from 50 to "a pilot of 5," and that addresses the above issues, would be welcome.
- *Tests of circumvention tools*
This seems cool, but could as easily be titled The How Well is My Censorship Working Index" Answering the question "whom does this serve, and how?" would seem to be the next step to assessing the value of this proposition.
- *Orchestrating OONI probes*
This seems HUGELY problematic WRT privacy, user consent, and security. I am also concerned with how this butts up agains the need for consistent, and durational testing (which, IMO, is more important as a research tool and a means to understand censorship than novel tests run a couple times during a given month).
- *Data analytics and visualizations*
Is there enough consistent available data (and a roadmap that would guarantee a consistent pipeline of consistent data over time) to make this useful and worthwhile currently? If/when yes, I would suggest bringing in people who have professional experience with data visualizations and analysis. With M-Lab this goal has been a continual challenge -- there aren't accepted statistical ways of working with network data (more on that if you want in another thread), and visualizations need to be gorgeous, need to be written in a browser-friendly language, need to be maintained and updated, need to not have [too many] gaps, etc.. You are, in this effort, producing a user-facing product. All of the forever-work that attends a product attends this effort.
- *Pub system*
This seems potentially useful (I don't know enough to be concrete here), but again, my question is, Whose needs does this serve, specifically, and how does serving those needs further a longer-term OONI strategy? More generally, shared storage and transport mechanisms for measured data are something this space could use, for sure. Would this potentially help build those systems?
- *Production-quality OONI Pis. *
I like this, and I like the idea of partnering with CI lab (hey!) or others. Deployment and maintenance is expensive. The more work these Pis can do once they're deployed, before they die, the better. I would be more enthusiastic about this if it involved a deployment partner, as I don't see a huge value in spending dev time to ensure that the handful of people who'd flash a Pi can.
- *OONI on mobile*
I vote to have a more stable OONI before launching a mobile test. We at M-Lab have explored mobile at length, and it's tricky (I'm not sure tests written for a non-mobile environment would be as relevant to a mobile environment), deployment is hard (marketing is key! and, who wants to use their data cap on something that isn't whatsapp? (etc.).)
- *Research based on OONI*
If there's enough consistent data, it could be interesting. But, not sure that there is (?)
- *Monthly reports*
As above, I think this is premature. Getting good data, and getting enough of it, should come first.
- *Adopt an OONI probe*
I worry about consent and permissions here. Once those are figured out, I would suggest getting a big donor to adopt (deploy) a bunch of OONIs, instead of a smaller campaign.
- *Integration with other censorship measurement projects*
Really like the sound of this! The more resources can be shared, the better. I'll let y'all discuss...
- *Reaching out to communities inside censored regions (like the UK?)*
I'm all for this, but I think this should be led by groups like Citizen Lab, maybe Amnesty, and others that have experience in qualitative user studies and access to networks on the ground.
- *Redesign the OONI website*
Definitely necessary. Not clear on its priority. I would suggest that any redesign minimize the focus on "censorship" (and the use of the term). For all the reasons we've discussed. And that a technical writer and someone with some communications training be employed in drafting and tinkering.
- *Internet censorship conference*
What would the goals be? How would things be better/different when these goals were achieved? Without a concrete motive, I worry that this could be another "fly the same people somewhere new and pretend we're innovating" model.
- *Implement a GUI for OONI*
I would prioritize this after the backend is stable, and the consent and permissions issues have been worked through. (I also think this is something that should engage designers and UX experts outside of the OONI core team, because all the reasons
I agree fully with all that you say. In particular I think, although I had not put it amongst the options to vote on, that we should dedicate quite a bit of time working on sorting out the informed consent issue.
On 2/19/15 12:08 AM, Jed Crandall wrote:
Sorry for chiming in late, have been a bit under the weather this week. I'll just second Philipp's comment:
https://lists.torproject.org/pipermail/ooni-dev/2015-February/000253.html
I.e., a "me too" for giving "Implement data analytics and visualization for OONI tests" a 5. I don't mind writing a bit of code to start looking at some data, but there's so much data out there and I'm more likely to start looking at data (or suggest that a research or networking class student do so) if I already have some idea of what's there.
I'll also add that a "wish list" of problems you'd like solved could be helpful, like:
https://research.torproject.org/ideas.html https://www.torproject.org/getinvolved/volunteer.html.en#Research
For example, I've found IP geolocation services like MaxMind to be pretty bad in certain places. It says something is in a country and it's not, which makes debugging why that data point doesn't behave like others that are supposed to be in that country a pain. Is that a problem that would help OONI if solved? If not, is something else?
FWIW, this is a step in the direction of doing IP geolocation better in parts of the world other than the U.S. and Europe:
http://www.cs.unm.edu/~crandall/infocom2015rtt.pdf
Lastly, in terms of baselines, even very specific baslines like "Tor bridge reachability in Country X" can be very illuminating if the amount of space and time the data is taken over is broad enough. Measuring everything everywhere all the time would be nice, of course, but like Salvador Dali said, "Have no fear of perfection - you'll never reach it."
The suggestion for the wish list of problems we would like solved research wise is very good.
There are a few of those and I think it would be a good idea to write them down somewhere.
The MaxmMind issue is indeed a problem of ours. Luckily we will collect by default also the ASN and in most cases you will be able to identify the inconsistency manually by looking up the details of the ASN (in a whois database or similar).
Regarding reachability of Tor brides we have at this point about 3-4 months of data (daily measurements) for obfs2/obfs3/scramblesuit/fte bridges in Iran, China, Russia, Ukraine.
The latest data is not yet public, because it needs to be scrubbed of the metadata and I have not yet finished re-setting up the pipeline (after our other machine ran out of disk space).
We also have some visualization, that needs to be completed, for it and if somebody is interested in hacking on this I could give them access to the data and code.
....
And now onto the result of the voting session:
Implement data analytics and visualization for OONI tests 4.75
Reach production quality ooni rasperry-pi (beagle-board) images 4.25
Develop OONI tests for censorship circumvention tools 4
Develop scheme for orchestrating ooni-probes 4
Promote and further develop OONI on mobile (Android, iOS) 4
Get daily OONI measurements from 50 countries 3.75
Publish monthly reports about the status of internet censorship in a country 3.75
Implement a GUI for ooniprobes 3.75
Do research based on OONI 3.666666667
Implement pub-sub system for ooni collectors 3.5
Integration with other censorship measurement projects 3.5
Reaching out to communities inside of censored regions 3.5
Run "adopt an ooni-probe" campaign 3.25
Redesign the website for ooni 2.75
Hold an international internet censorship conference 2.25
To me these results are not surprising at all and it reflects more or less what have already been the main areas of work for the past couple of months.
I think therefore we should continue in this direction and hence focus on the the three main areas of "Implement data analytics and visualization for OONI tests", "Reach production quality ooni rasperry-pi (beagle-board) images", "Informed consent research", moreover we will also be working on "Develop OONI tests for censorship circumvention tools" for a project in a specific country.
Here is a list of what are the existing tickets on these areas and what are some potentially new tickets to be created.
# Implement data analytics and visualization for OONI tests
## Existing tickets
Add generation of reports index to the export task of the pipeline https://trac.torproject.org/projects/tor/ticket/13842
Migrating OONI data-pipeline containers and server configuration on a different server https://trac.torproject.org/projects/tor/ticket/13825
Better and more efficient database schema https://trac.torproject.org/projects/tor/ticket/13803
Mongodb queries for the nettest visualization https://trac.torproject.org/projects/tor/ticket/13759
Brainstorm ideas for possible visualisations https://trac.torproject.org/projects/tor/ticket/13731
Investigate possible performance improvements to the ooni-pipeline https://trac.torproject.org/projects/tor/ticket/13720
Align the dates in the visual timeline https://trac.torproject.org/projects/tor/ticket/13639
Better tokening in the output json data format for bridge reachability visualisation https://trac.torproject.org/projects/tor/ticket/13638
## New tickets
### Design and implement OONI reports explorer
This will allow users of OONI to explore the data that we have so far collected, by filtering and searching it.
# Reach production quality ooni rasperry-pi (beagle-board) images
## Existing tickets
OONI on Raspberry Pi https://trac.torproject.org/projects/tor/ticket/13870
## New tickets
### Embedded device configuration wizarcd
Setup a OONI wifi network on the raspberry pi to configure the device at first start. This will allow the user to configure how ooni-probe should connect to the internet and what measurements should be run.
It would also be useful to provide an informed consent information page.
# Informed consent research
## Existing tickets
Write documentation of benefits for running ooniprobe https://trac.torproject.org/projects/tor/ticket/14760
Brainstorm on possible ways of minimizing the risks involved with running ooniprobe while keeping the benefits https://trac.torproject.org/projects/tor/ticket/14761
Redesign how we inform the user of the risks of running ooniprobe and get informed consent from them https://trac.torproject.org/projects/tor/ticket/14762
## New tickets
Get legal feedback for the risks of running ooniprobe in a set of specific countries
Thanks for taking the time to reach this.
I will soon send out an email to schedule next weeks IRC meeting, since we have skipped it this week.
Have fun!
~ Arturo
Hi Arturo
Thanks for starting this thread - sorry it's taken me a while to respond.
I declare an interest: I am responsible for https://www.blocked.org.uk/. We are currently working towards integrating ooni probes as our preferred method of measuring censorship. This will allow us to deploy our system outside the UK.
I've added my votes in the table below.
On 10/02/15 15:50, Arturo Filastò wrote:
...it would be very useful if you could express a vote from 1-5:
1: Nah, this is boring and pointless 2: Not really super important, I would give priority to other stuff 3: Useful, I would do it 4: This would be awesome 5: Epic!
Goal | Vote
Get daily OONI measurements | from 50 countries | 2
Develop OONI tests for | censorship circumvention | tools | 2
Develop scheme for | orchestrating ooni-probes | 5
Implement data analytics | and visualization for OONI | tests | 3
Implement pub-sub system | for ooni collectors | 5
Reach production quality | ooni rasperry-pi | (beagle-board) images | 4
Promote and further | develop OONI on mobile | (Android, iOS) | 3
Publish monthly reports | about the status of | internet censorship in | a country | 2
Run "adopt an ooni-probe" | campaign | 2
Integration with other | censorship measurement | projects | 3
Reaching out to communities| inside of censored regions | 4
Redesign the website for | ooni | 1
Hold an international | internet censorship | conference | 2
Implement a GUI for | ooniprobes | 2
Do research based on | OONI | 2
Regards
Richard
-
Arturo Filastò -
Collin Anderson -
David Stainton -
George Kadianakis -
Iain R. Learmonth -
Philipp Winter -
Richard King -
Simone Basso