Greetings from India,
So I've been testing networks in Bangalore and I've noticed a few odd quirks with using a test deck.
Here is my ooniprobe.conf:
% cat ooniprobe.conf # This is the configuration file for OONIProbe # This file follows the YAML markup format: http://yaml.org/spec/1.2/spec.html # Keep in mind that indentation matters.
basic: # Where OONIProbe should be writing it's log file logfile: ooniprobe-bangalore.log privacy: # Should we include the IP address of the probe in the report? includeip: true # Should we include the ASN of the probe in the report? includeasn: true # Should we include the country as reported by GeoIP in the report? includecountry: true # Should we include the city as reported by GeoIP in the report? includecity: true # Should we collect a full packet capture on the client? includepcap: false reports: # This is a packet capture file (.pcap) to load as a test: pcap: Null advanced: # XXX change this to point to the directory where you have stored the GeoIP # database file. This should be the directory in which OONI is installed # /path/to/ooni-probe/data/ #geoip_data_dir: /usr/share/GeoIP/ geoip_data_dir: /home/a/ooni-probe/data/ debug: true # tor_binary: '/usr/sbin/tor' # For auto detection interface: auto # Of specify a specific interface #interface: wlan0 # If you do not specify start_tor, you will have to have Tor running and # explicitly set the control port and SOCKS port start_tor: true # After how many seconds we should give up on a particular measurement measurement_timeout: 30 # After how many retries we should give up on a measurement measurement_retries: 2 # How many measurments to perform concurrently measurement_concurrency: 10 # After how may seconds we should give up reporting reporting_timeout: 30 # After how many retries to give up on reporting reporting_retries: 6 # How many reports to perform concurrently reporting_concurrency: 10 tor: socks_port: 9250 control_port: 9251 # Specify the absolute path to the Tor bridges to use for testing #bridges: bridges.list # Specify path of the tor datadirectory. # This should be set to something to avoid having Tor download each time # the descriptors and consensus data. data_dir: ~/.tor/
Here is the test deck:
% cat decks/india-full.deck - options: collector: null help: 0 logfile: null pcapfile: null reportfile: null subargs: [-t, '192.168.1.1', -f, 'inputs/india-uniq-hosts-with-alexa-top-1000.txt'] test_file: nettests/blocking/dnsconsistency.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200'] test_file: nettests/manipulation/http_header_field_manipulation.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200'] test_file: nettests/manipulation/http_invalid_request_line.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200', -f, 'inputs/india-uniq-urls-with-alexa-top-1000.txt'] test_file: nettests/manipulation/http_host.py
A few things happen when I attempt to use this deck.
Tor fails to return my IP: 2013-06-01 00:44:15+0530 [TorControlProtocol,client] [D] 100%: Done 2013-06-01 00:44:15+0530 [TorControlProtocol,client] [D] Building a TorState 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Successfully bootstrapped Tor 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] We now have the following circuits: 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 1 BUILT [194.132.32.43 165.225.132.54 46.165.221.166] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 2 EXTENDED [194.132.32.43] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 3 EXTENDED [] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 4 EXTENDED [] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] Obtained our IP address from a Tor Relay None 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Unhandled Error Traceback (most recent call last): Failure: txtorcon.torcontrolprotocol.TorProtocolError: 551 Address unknown
2013-06-01 00:44:16+0530 [TorControlProtocol,client] Unable to lookup the probe IP via Tor. 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] Cannot determine the probe IP address with a traceroute, becase of insufficient priviledges 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Looking up your IP address via maxmind
Then things get a little strange - http_host.py is never executed. Another is that http_header_field_manipulation.py runs and the log file shows everything, the yamloo file shows only this:
% cat report-http_header_field_manipulation-2013-05-31T191417Z.yamloo ########################################### # OONI Probe Report for http_header_field_manipulation (0.1.3) # Sat Jun 1 00:57:40 2013 ########################################### --- options: [-b, 'http://93.95.227.200'] probe_asn: AS24560 probe_cc: IN probe_ip: 122.167.211.176 software_name: ooniprobe software_version: 0.0.11 start_time: 1370027657.776991 test_name: http_header_field_manipulation test_version: 0.1.3 ...
The debug log shows the headers being sent and the data being returned with an issue at the collector: 2013-06-01 00:57:40+0530 [SOCKS5Client,client] Creating report with OONIB Reporter. Please be patient. 2013-06-01 00:57:40+0530 [SOCKS5Client,client] This may take up to 1-2 minutes... 2013-06-01 00:57:40+0530 [SOCKS5Client,client] [D] Successfully performed report <ooni.tasks.ReportEntry object at 0x588c190> 2013-06-01 00:57:40+0530 [SOCKS5Client,client] [D] None 2013-06-01 00:57:40+0530 [Uninitialized] [!] Failed to connect to reporter backend 2013-06-01 00:57:40+0530 [Uninitialized] Traceback (most recent call last): 2013-06-01 00:57:40+0530 [Uninitialized] File "/home/io/Documents/backup/git/tor/ooni-probe/ooni/reporter.py", line 323, in createReport 2013-06-01 00:57:40+0530 [Uninitialized] bodyProducer=bodyProducer) 2013-06-01 00:57:40+0530 [Uninitialized] ConnectError: An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. 2013-06-01 00:57:40+0530 [Uninitialized] ]. 2013-06-01 00:57:40+0530 [Uninitialized] [!] Failed to open <ooni.reporter.OONIBReporter object at 0x461d710> reporter, giving up... 2013-06-01 00:57:40+0530 [Uninitialized] [!] Reporter <ooni.reporter.OONIBReporter object at 0x461d710> failed, removing from report... 2013-06-01 00:57:40+0530 [Uninitialized] [D] Starting this task <generator object generateMeasurements at 0x51906e0> 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_put 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup 2013-06-01 00:57:40+0530 [Uninitialized] [D] Performing request http://93.95.227.200 PUT {'Accept-Language': ['en-US,en;q=0.8'], 'Accept-Encoding': ['gzip,deflate,sdch'], 'Accept': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'User-Agent': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'], 'Accept-Charset': ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'], 'Host': ['XAxlpMzUMfI5Vvi.com']} 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_get_random_capitalization 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup 2013-06-01 00:57:40+0530 [Uninitialized] [D] Performing request http://93.95.227.200 gET {'accePt-lanGuAGe': ['en-US,en;q=0.8'], 'accEpT-eNcoDING': ['gzip,deflate,sdch'], 'ACCepT': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'USeR-aGEnT': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7'], 'aCcEPt-chaRseT': ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'], 'HoSt': ['l5tHomKVddWW1A4.com']} 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_post_random_capitalization 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup
In the end, I didn't have any yamloo files from the nettests/manipulation/http_invalid_request_line.py test. I had three files that updated and had some data which was basically:
report-dns_consistency-2013-05-31T191417Z.yamloo report-http_header_field_manipulation-2013-05-31T191417Z.yamloo ooniprobe-bangalore.log
I expected a few different things - one is that each test in the deck should produce a yamloo file. If the reporting back end takes the report, I suppose I might find it alright to not have the file but in the event of a failure, I really hope the data will be logged to a local .yamloo file.
When I run the following deck:
% cat decks/india.deck - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: http_host_india_bangalore_justa_hotel.log pcapfile: null reportfile: http_host_india_cis.yamloo subargs: [-b, 'http://93.95.227.200', -f, 'inputs/india-uniq-urls-with-alexa-top-1000.txt'] test_file: nettests/manipulation/http_host.py
I have the proper output for http_host.py:
% head report-http_host-2013-05-31T193306Z.yamloo ########################################### # OONI Probe Report for http_host (0.2.3) # Sat Jun 1 01:03:06 2013 ########################################### --- options: [-b, 'http://93.95.227.200', -f, inputs/india-uniq-urls-with-alexa-top-1000.txt] probe_asn: AS24560 probe_cc: IN probe_ip: 122.167.211.176 software_name: ooniprobe
% tail report-http_host-2013-05-31T193306Z.yamloo url: http://93.95.227.200 response: body: '{"headers_dict": {"Connection": ["close"], "Host": ["zustmovies.com"]}, "request_line": "\nGET / HTTP/1.1", "request_headers": [["Connection", "close"], ["Host", "zustmovies.com"]]}' code: 200 headers: [] socksproxy: null transparent_http_proxy: false ...
Note that the yamloo file is created not as http_host_india_bangalore_justa_hotel.log but as report-http_host-2013-05-31T193306Z.yamloo...
It seems that perhaps test decks are too experimental for actual use with these issues - or did I do something horribly wrong?
Thoughts?
All the best, Jacob