As agreed with Aleksejs we are going to move this discussion onto the list.
On 1/4/15 8:40 PM, Aleksejs Popovs wrote:
Hi Arturo,
First of all, sorry for contacting you directly. Ooni-talk seems to be quite dead, and I am not sure that this is appropriate for ooni-dev. Feel free to redirect me somewhere else.
Secondly, great job on the 31C3 OONI presentation!
Now, onwards to what I wanted to tell you about. Here in Latvia, DPI-based filtering is used to block HTTP(S) connections to online gambling websites, as mandated by the law on gambling. However, there is also speculation originating from ISPs on the possibility of this being implemented for unlicensed online mass media, which to me sounds scary as hell. There don't appear to be any reports from Latvia in either OONI's report repos or Open Net Initiative's lists.
Blocking of gambling sites is in fact something very common in greedy western countries.
How are they implementing blocking for HTTPS sites? It is quite unusual to see that happening, but having information on that would be interesting.
I wanted to create an OONI report that would demonstrate this censorship in my ISP's (Lattelecom, one of the biggest ones) network. Lattelecom uses DPI on port 80 to find requests containing "Host: <blockedhost>" and serve them a page like this: https://b.popovs.lv/images/blocked_website.png (they also do something similar for HTTPS with self-signed certs). I picked a random blocked URL, unibet.net http://unibet.net, put both HTTP and HTTPS versions of it into a text file, and then put a URL of a page on my personal website, popovs.lv http://popovs.lv (which isn't blocked), to use as a baseline.
I ran the test, and it reported some errors and that "censorship is probably not happening" (which applies to my homepage, I guess). Here's the ooniprobe log and the report: https://popovs.lv/crap/ooni/ooni_run.txt https://popovs.lv/crap/ooni/report-http_requests-2015-01-04T165420Z.yamloo
Looking at the report, I saw that, while requests to my homepage went through just fine (and, as expected, were not censored), requests to the censored pages didn't show the censorship message, but instead showed various errors. I got confused as to why I could receive a parsing error, but it all cleared up when I tried looking at the plain headers using netcat: https://popovs.lv/crap/ooni/netcat.txt . That's right, there were no HTTP headers at all — their censorship setups just spits HTML out right away. I'm genuinely surprised that browsers actually render that. The same idiocy seems to be happening with HTTPS.
Oh my, that is some super ghetto censorship equipment at work.
We are relying on twisted's HTTP parsing library so it appears that it does not support very well responses that are out of spec.
There is in the making a new HTTP test template in this branch: https://github.com/thetorproject/ooni-probe/tree/feature/http-template
and it may be a good idea to support in it also logging HTTP responses that are out of spec.
In the meantime what you can do to overcome this limit of ooniprobe is that you could run the http_filtering_bypassing experimental test. If they are doing blocking based on HTTP Host header field that will trigger the blocking when running the "test_normal_request", but will also identify some possible ways to bypass the filter by doing some slightly modified requests (that is requests that a normal web server would accept, but may be erroneously matched by the filter).
With this test we were able to detect some filtering bypassing techniques in Turkmenistan and Uzbekistan: https://ooni.torproject.org/tab-tab-come-in-bypassing-internet-blocking-to-c...
Since this test does not use the full HTTP library, but just uses plain TCP to form the HTTP request and simply logs the HTTP response as a string without parsing it.
So, I'm not even sure about what I want from you: I guess I just wanted you to know about this situation. I don't know how exactly are the OONI reports analysed — do you consider errors like this one to be cases of censorship? I guess you wouldn't want to implement some hacks to support my ISPs stupid quirks, but I just want to know if I can help in any further way to report on the net censorship here in Latvia.
As I said above I think it's a good idea to support these sorts of weird behaviors ISP filtering equipment has. We may see this behavior in the future and it's useful to be able to link it to the filtering technology used by Latvia.
Huge thanks to you for all of your work on OONI and other net freedom and privacy-related projects!
Best regards, Aleksejs Popovs
Thanks for your email
~ Arturo