I wrote a program that uses the OONI API to download reports and keep a
local directory of reports up to date. It's much faster than the Wget
loop I used to use and it finishes quickly when there is nothing new to
download.
git clone https://www.bamsoftware.com/git/ooni-sync.git
For example, lately I've had to download a lot of tcp_connect reports. I
run it like this:
ooni-sync -xz -directory reports.tcp_connect/ test_name=tcp_connect
This command downloads the index of tcp_connect reports and only
downloads the ones that are not already downloaded. It compresses the
downloaded files with xz. The next time I need to update, I run the same
command again, and it only downloads reports that are new since the last
time.
You can use other query parameters supported by the API, like probe_cc,
probe_asn, since, and until. For example:
ooni-sync -xz -directory reports.is/ probe_cc=IS since=2017-01-01
ooni-sync -xz -directory reports.as25/ probe_asn=AS25
ooni-sync -xz -directory reports.tor-turkey/ test_name=vanilla_tor probe_cc=TR
ooni-sync -xz -directory reports.web_connectivity/ test_name=web_connectivity since=2017-01-01 until=2017-01-02
I prefer to keep all the reports compressed on disk, so I always use the
-xz option, but by default reports are saved unmodified.
I'm reviewing the Malaysian Citizen Lab test url report summary and
there seem to be false positives (or possible interference that is not
visible)
The ones that are obviously outdated, and redirected due to domain
parking or something I file for removal (usually short lived minor porn
sites)
Where should I report and track these?
http://www.yawa.org on-line gambling site is reported as possible http-
diff
Package: wnpp
Severity: wishlist
Owner: "Iain R. Learmonth" <irl(a)debian.org>
* Package name : measurement-kit
Version : 0.6.0
Upstream Author : OONI <ooni-dev(a)lists.torproject.org>
* URL : https://github.com/measurement-kit/measurement-kit
* License : BSD-2-clause
Programming Lang: C++
Description : Portable C++11 network measurement library
MeasurementKit is a library that implements open network measurement
methodologies (performance, censorship, etc.) and targets mobile platforms
(Android and iOS).
It is meant to be embedded by third party applications with specific network
measurement needs and/or to be used by researchers as a basis to implement
novel tools.
---
This package will be maintained by the Internet Measurement Packaging Team
<pkg-netmeasure-discuss(a)lists.alioth.debian.org>.
Thanks,
Iain.