As agreed with Aleksejs we are going to move this discussion onto the list.
On 1/4/15 8:40 PM, Aleksejs Popovs wrote:
Hi Arturo,
First of all, sorry for contacting you directly. Ooni-talk seems to be quite dead, and I am not sure that this is appropriate for ooni-dev. Feel free to redirect me somewhere else.
Secondly, great job on the 31C3 OONI presentation!
Now, onwards to what I wanted to tell you about. Here in Latvia, DPI-based filtering is used to block HTTP(S) connections to online gambling websites, as mandated by the law on gambling. However, there is also speculation originating from ISPs on the possibility of this being implemented for unlicensed online mass media, which to me sounds scary as hell. There don't appear to be any reports from Latvia in either OONI's report repos or Open Net Initiative's lists.
Blocking of gambling sites is in fact something very common in greedy western countries.
How are they implementing blocking for HTTPS sites? It is quite unusual to see that happening, but having information on that would be interesting.
I wanted to create an OONI report that would demonstrate this censorship in my ISP's (Lattelecom, one of the biggest ones) network. Lattelecom uses DPI on port 80 to find requests containing "Host: <blockedhost>" and serve them a page like this: https://b.popovs.lv/images/blocked_website.png (they also do something similar for HTTPS with self-signed certs). I picked a random blocked URL, unibet.net http://unibet.net, put both HTTP and HTTPS versions of it into a text file, and then put a URL of a page on my personal website, popovs.lv http://popovs.lv (which isn't blocked), to use as a baseline.
I ran the test, and it reported some errors and that "censorship is probably not happening" (which applies to my homepage, I guess). Here's the ooniprobe log and the report: https://popovs.lv/crap/ooni/ooni_run.txt https://popovs.lv/crap/ooni/report-http_requests-2015-01-04T165420Z.yamloo
Looking at the report, I saw that, while requests to my homepage went through just fine (and, as expected, were not censored), requests to the censored pages didn't show the censorship message, but instead showed various errors. I got confused as to why I could receive a parsing error, but it all cleared up when I tried looking at the plain headers using netcat: https://popovs.lv/crap/ooni/netcat.txt . That's right, there were no HTTP headers at all — their censorship setups just spits HTML out right away. I'm genuinely surprised that browsers actually render that. The same idiocy seems to be happening with HTTPS.
Oh my, that is some super ghetto censorship equipment at work.
We are relying on twisted's HTTP parsing library so it appears that it does not support very well responses that are out of spec.
There is in the making a new HTTP test template in this branch: https://github.com/thetorproject/ooni-probe/tree/feature/http-template
and it may be a good idea to support in it also logging HTTP responses that are out of spec.
In the meantime what you can do to overcome this limit of ooniprobe is that you could run the http_filtering_bypassing experimental test. If they are doing blocking based on HTTP Host header field that will trigger the blocking when running the "test_normal_request", but will also identify some possible ways to bypass the filter by doing some slightly modified requests (that is requests that a normal web server would accept, but may be erroneously matched by the filter).
With this test we were able to detect some filtering bypassing techniques in Turkmenistan and Uzbekistan: https://ooni.torproject.org/tab-tab-come-in-bypassing-internet-blocking-to-c...
Since this test does not use the full HTTP library, but just uses plain TCP to form the HTTP request and simply logs the HTTP response as a string without parsing it.
So, I'm not even sure about what I want from you: I guess I just wanted you to know about this situation. I don't know how exactly are the OONI reports analysed — do you consider errors like this one to be cases of censorship? I guess you wouldn't want to implement some hacks to support my ISPs stupid quirks, but I just want to know if I can help in any further way to report on the net censorship here in Latvia.
As I said above I think it's a good idea to support these sorts of weird behaviors ISP filtering equipment has. We may see this behavior in the future and it's useful to be able to link it to the filtering technology used by Latvia.
Huge thanks to you for all of your work on OONI and other net freedom and privacy-related projects!
Best regards, Aleksejs Popovs
Thanks for your email
~ Arturo
I have completed a few more manual tests.
First of all, I remembered that in the early days of this law's implementation Lattelecom were using DNS spoofing. I decided to check if this was still the case (https://popovs.lv/crap/ooni/nslookup.txt), and it wasn't.
Then, I checked whether I can connect to port 80 on the IPs of banned websites and issue a request with a Host header unrelated to that website. It seems to have worked (https://popovs.lv/crap/ooni/plain_fake_host.txt), as the request was passed to the actual server (which seems to be configured to answer all requests on port 80 with a redirect to the same Host via HTTPS). Sending a request with "Host: unibet.net" still returns the censorship page.
Now, I decided to use openssl's s_client (basically nc through TLS) to check out what was happening with HTTPS. This test (https://popovs.lv/crap/ooni/s_client.txt) was weird, because its results were inconsistent with what I saw in my browser (no response was served to me at all) — I am not sure how to explain that. By the way, when I visit the censorship page in my browser, Chromium, it fails to display response headers (https://popovs.lv/crap/ooni/chrome_https.png — the panel in the bottom should normally display both request and response headers), so, whenever this thing works in my browser, it fails to serve headers just like its HTTP counterpart.
Also interesting is the HTTPS cert they're using, which you can find in the log. It is issued by Fortinet, an American company that's apparently known for this sort of thing, and the issuer's CN (FGT1KC3913801932) looks like a serial number.
I will try running some other ooniprobe-based tests later.
On Wed, Jan 7, 2015 at 11:25 AM, Aleksejs Popovs popoffka@gmail.com wrote:
Also interesting is the HTTPS cert they're using, which you can find in the log. It is issued by Fortinet, an American company that's apparently known for this sort of thing, and the issuer's CN (FGT1KC3913801932) looks like a serial number.
Are we sure that the Fortinet cert has originated from a network intermediary? Interestingly, the HTTP header of "Server: BigIP" is associated with Fortinet as well (I see the HTTP header but not the SSL cert).
On 7 January 2015 at 18:42, Collin Anderson collin@averysmallbird.com wrote:
Are we sure that the Fortinet cert has originated from a network intermediary? Interestingly, the HTTP header of "Server: BigIP" is associated with Fortinet as well (I see the HTTP header but not the SSL cert).
"Server: BigIP" seems to be a coincidence. I just tried the same things (nc with fake Host and s_client) on another blocked domain with the same results: https://popovs.lv/crap/ooni/888games.txt (and exactly the same cert)
On Wed, 7 Jan 2015 18:25:24 +0200 Aleksejs Popovs popoffka@gmail.com wrote:
I have completed a few more manual tests.
First of all, I remembered that in the early days of this law's implementation Lattelecom were using DNS spoofing. I decided to check if this was still the case (https://popovs.lv/crap/ooni/nslookup.txt), and it wasn't.
Then, I checked whether I can connect to port 80 on the IPs of banned websites and issue a request with a Host header unrelated to that website. It seems to have worked (https://popovs.lv/crap/ooni/plain_fake_host.txt), as the request was passed to the actual server (which seems to be configured to answer all requests on port 80 with a redirect to the same Host via HTTPS). Sending a request with "Host: unibet.net" still returns the censorship page.
Now, I decided to use openssl's s_client (basically nc through TLS) to check out what was happening with HTTPS. This test (https://popovs.lv/crap/ooni/s_client.txt) was weird, because its results were inconsistent with what I saw in my browser (no response was served to me at all) — I am not sure how to explain that. By the way, when I visit the censorship page in my browser, Chromium, it fails to display response headers (https://popovs.lv/crap/ooni/chrome_https.png — the panel in the bottom should normally display both request and response headers), so, whenever this thing works in my browser, it fails to serve headers just like its HTTP counterpart.
Also interesting is the HTTPS cert they're using, which you can find in the log. It is issued by Fortinet, an American company that's apparently known for this sort of thing, and the issuer's CN (FGT1KC3913801932) looks like a serial number.
I will try running some other ooniprobe-based tests later.
Do they MITM connections to any cloud providers such as Amazon, CloudFlare, Google, etc?
I wrote a simple OONI test for fetching TLS certificate chains of resources specified in an input list. https://github.com/TheTorProject/ooni-probe/tree/feature/tls_certificates
There are quite a few issues with using the OpenSSL bindings, as many methods (e.g. to learn the negotiated cipher suite) are non-functional.
It would probably be interesting to see which other python TLS implementations exist that could better mimic the handshake of various applications (or write an OONI test to that passively sniffs observed certificate chains off the wire).
--Aaron
On 8 January 2015 at 16:46, Aaron Gibson aagbsn@extc.org wrote:
Do they MITM connections to any cloud providers such as Amazon, CloudFlare, Google, etc?
I have just parsed the official list of blocked domains (http://www.iaui.gov.lv/images/Blokesana/Block_domain.pdf -> https://popovs.lv/crap/ooni/domains.txt) and resolved them and checked the ASs of their IPs (also comparing the reports from my network with reports from a host in Croatia: I didn't see any differences in IPs that weren't due to multiple A DNS records, although it's important to note I'm using Google Public DNS instead of Lattelecom DNS) (https://popovs.lv/crap/ooni/domains_resolved.txt). There are a bunch of websites hosted/proxied by Cloudflare and Amazon (AS13335 and AS16509, respectively), I took a random one: lucky31.com, hosted on 198.41.249.151 by Cloudflare. http://www.tcpiputils.com/browse/ip-address/198.41.249.151 told me that unitedcostumes.com.au was also hosted on that IP. unitedcostumes.com.au is available to me over HTTP, and seems to not have any gambling-related content. Accessing it over HTTPS, however, is impossible from the Lattelecom network, because the TLS connection times out, in exactly the same manner as I previously observed while attempting to query blocked HTTPS servers while requesting a non-blocked server name. Also, I was able to access unitedcostumes.com.au via HTTPS through Tor. To conclude, this is a real case of a non gambling-related page being blocked, although almost definitely by accident. I will notify Lattelecom about this.
Thank you for your suggestion to check this.
On Thu, Jan 8, 2015 at 3:47 PM, Aleksejs Popovs popoffka@gmail.com wrote:
To conclude, this is a real case of a non gambling-related page being blocked, although almost definitely by accident. I will notify Lattelecom about this.
Cloudflare-hosted sites are included as Subject Alternative Names in the certificate they share between hosted sites. I wonder if the intermediary parses the cert to match against the blacklisted domains, and terminates any connection that lists the name. That identifier also provides you a list of all of the sites that are overblocked as a result in your work, e.g.
openssl s_client -connect lucky31.com:443 2>&1 | openssl x509 -text
X509v3 Subject Alternative Name:
DNS:ssl3055.cloudflare.com, DNS:myriotravelguide.com, DNS:*. americandreamhomeimprovement.com, DNS:*.thermosystemsinc.info, DNS:*. evasi0ndownload.com, DNS:*.fraglive.cl, DNS:thealtitudecompany.com, DNS: cmonsite.fr, DNS:*.weekcal.com, DNS:cu2nite.com.au, DNS:*. genesisenergyinternational.net, DNS:*.lucky31.com, DNS:*.starspayment.com, DNS:starspayment.com, DNS:*.loppis.me, DNS:loppis.me, DNS: unitedcostumes.com.au, DNS:2ch.hk, DNS:thermosystemsinc.info, DNS:*. bunadformenn.info, DNS:weekcal.com, DNS:starfishmedia.com, DNS: mycareers360.com, DNS:*.casinoextra.com, DNS:peakfit.com.gt, DNS: productworld.com, DNS:*.unitedcostumes.com.au, DNS:*.habbo.as, DNS: genesisenergyinternational.net, DNS:lucky31.com, DNS:*. thealtitudecompany.com, DNS:*.timesulin.com, DNS:evasi0ndownload.com, DNS: fraglive.cl, DNS:*.2ch.hk, DNS:*.productworld.com, DNS:casinoextra.com, DNS: americandreamhomeimprovement.com, DNS:timesulin.com, DNS:*.peakfit.com.gt, DNS:*.myriotravelguide.com, DNS:bunadformenn.info, DNS:habbo.as, DNS:*. cmonsite.fr, DNS:*.starfishmedia.com, DNS:*.mycareers360.com, DNS:*. cu2nite.com.au
If it is certificate parsing, that might make for an interesting test and test helper. Neat find.
-
Aaron Gibson -
Aleksejs Popovs -
Arturo Filastò -
Collin Anderson