Hi,
I ran into the technical report from George Danezis about an anomaly-based censorship-detection system for Tor. I have a few questions that I hope you can help me with.
Is there an implementation available of the approach described in the paper?
The paper talks about finding anomalies in the time series of the number of requests to directory servers from certain jurisdictions. Where can I find this data about the requests to directory servers?
--john
[1] https://research.torproject.org/techreports/detector-2011-09-09.pdf
On Tue, Jan 12, 2016 at 07:21:39AM +0000, John wrote:
I ran into the technical report from George Danezis about an anomaly-based censorship-detection system for Tor. I have a few questions that I hope you can help me with.
Is there an implementation available of the approach described in the paper?
Go to https://metrics.torproject.org/userstats-relay-country.html and check "Show possible censorship events if available".
I think the source code is here: https://gitweb.torproject.org/karsten/metrics-web.git/tree/detector
The paper talks about finding anomalies in the time series of the number of requests to directory servers from certain jurisdictions. Where can I find this data about the requests to directory servers?
https://collector.torproject.org/
I think you want the "dirreq-v3-reqs" lines. For example, here is a recently archived extra-info consensus: https://collector.torproject.org/recent/relay-descriptors/extra-infos/2016-0... (The file will expire after a few days; after that go to https://collector.torproject.org/archive/relay-descriptors/extra-infos/.)
Here's a line from the file showing the number of requests by country for one of the relays: dirreq-v3-reqs us=7808,ru=4400,de=4112,fr=2584,gb=1848,it=1352,es=1208,jp=1192,br=1184,nl=1024,ua=1008,ca=912,pl=728,au=696,md=680,in=584,se=464,ar=448,mx=448,id=384,be=336,at=320,tr=288,tw=288,ch=280,cz=264,il=264,kr=248,ir=240,th=232,kz=224,pt=224,gr=216,sg=216,fi=208,my=208,ro=192,dk=184,hu=176,ae=168,ve=168,by=160,cl=160,ie=160,vn=160,za=152,ph=144,no=136,co=128,hk=128,lv=128,nz=120,sk=120,bg=112,eg=112,pk=112,rs=104,bd=88,ma=80,ec=72,ng=72,dz=64,hr=64,pe=64,si=64,ee=56,lt=56,do=48,tn=48,az=40,ba=40,cr=40,is=40,jo=40,kw=40,qa=40,uy=40,af=32,am=32,bo=32,ci=32,cy=32,gh=32,gt=32,iq=32,ke=32,lu=32,pr=32,uz=32,et=24,ge=24,lb=24,mk=24,mn=24,sa=24,sd=24,sv=24,sy=24,tm=24,al=16,bh=16,bj=16,cu=16,gd=16,kg=16,kh=16,ly=16,ni=16,np=16,om=16,ps=16,re=16,sn=16,tt=16,??=8,ad=8,ao=8,as=8,aw=8,ax=8,bb=8,bf=8,bm=8,bn=8,bs=8,bw=8,cd=8,cg=8,cm=8,cn=8,cv=8,dm=8,gf=8,gm=8,gp=8,gq=8,gw=8,gy=8,hn=8,ht=8,im=8,jm=8,km=8,ky=8,li=8,lk=8,me=8,mg=8,mq=8,mr=8,mt=8,mu=8,mz=8,na=8,nc=8,ne=8,pa=8,pf=8,pg=8,pm=8,py=8,rw=8,sc=8,sm=8,sr=8,tj=8,tz=8,ug=8,vi=8,vu=8,ye=8,yt=8,zm=8,zw=8
Hi David,
Thank you, these pointers were very helpful. Do you know if there is some kind of resource that lists known censorship events? I'd like to see how good the approach from the paper does at identifying them.
--john
David Fifield:
On Tue, Jan 12, 2016 at 07:21:39AM +0000, John wrote:
I ran into the technical report from George Danezis about an anomaly-based censorship-detection system for Tor. I have a few questions that I hope you can help me with.
Is there an implementation available of the approach described in the paper?
Go to https://metrics.torproject.org/userstats-relay-country.html and check "Show possible censorship events if available".
I think the source code is here: https://gitweb.torproject.org/karsten/metrics-web.git/tree/detector
The paper talks about finding anomalies in the time series of the number of requests to directory servers from certain jurisdictions. Where can I find this data about the requests to directory servers?
https://collector.torproject.org/
I think you want the "dirreq-v3-reqs" lines. For example, here is a recently archived extra-info consensus: https://collector.torproject.org/recent/relay-descriptors/extra-infos/2016-0... (The file will expire after a few days; after that go to https://collector.torproject.org/archive/relay-descriptors/extra-infos/.)
Here's a line from the file showing the number of requests by country for one of the relays: dirreq-v3-reqs us=7808,ru=4400,de=4112,fr=2584,gb=1848,it=1352,es=1208,jp=1192,br=1184,nl=1024,ua=1008,ca=912,pl=728,au=696,md=680,in=584,se=464,ar=448,mx=448,id=384,be=336,at=320,tr=288,tw=288,ch=280,cz=264,il=264,kr=248,ir=240,th=232,kz=224,pt=224,gr=216,sg=216,fi=208,my=208,ro=192,dk=184,hu=176,ae=168,ve=168,by=160,cl=160,ie=160,vn=160,za=152,ph=144,no=136,co=128,hk=128,lv=128,nz=120,sk=120,bg=112,eg=112,pk=112,rs=104,bd=88,ma=80,ec=72,ng=72,dz=64,hr=64,pe=64,si=64,ee=56,lt=56,do=48,tn=48,az=40,ba=40,cr=40,is=40,jo=40,kw=40,qa=40,uy=40,af=32,am=32,bo=32,ci=32,cy=32,gh=32,gt=32,iq=32,ke=32,lu=32,pr=32,uz=32,et=24,ge=24,lb=24,mk=24,mn=24,sa=24,sd=24,sv=24,sy=24,tm=24,al=16,bh=16,bj=16,cu=16,gd=16,kg=16,kh=16,ly=16,ni=16,np=16,om=16,ps=16,re=16,sn=16,tt=16,??=8,ad=8,ao=8,as=8,aw=8,ax=8,bb=8,bf=8,bm=8,bn=8,bs=8,bw=8,cd=8,cg=8,cm=8,cn=8,cv=8,dm=8,gf=8,gm=8,gp=8,gq=8,gw=8,gy=8,hn=8,ht=8,im=8,jm=8,km=8,ky=8,li=8,lk=8,me=8,mg=8,mq=8,mr=8,mt=8,mu=8,mz=8,na=8,nc=8,ne=8,pa=8,pf=8,pg=8,pm=8
, py=8,rw=8,sc=8,sm=8,sr=8,tj=8,tz=8,ug=8,vi=8,vu=8,ye=8,yt=8,zm=8,zw=8
On Tue, Jan 12, 2016 at 11:49:19PM +0000, John wrote:
Hi David,
Thank you, these pointers were very helpful. Do you know if there is some kind of resource that lists known censorship events? I'd like to see how good the approach from the paper does at identifying them.
For Tor-specific censorship, a list that is reasonably complete and up to date is: http://www.eecs.berkeley.edu/~sa499/tor_timeline.pdf. It was composed by mining the Tor issue tracker and blog for events.
For general censorship, I'm not aware of any systematic or complete list of censorship events. There are quite a lot of research papers that study one particular place for a period of time, and from them we know of some incidents.
Here's a list of some measurement papers I'm aware of that might discuss some events. You might also look at OpenNet and Freedom House reports.
Internet censorship in Iran: A first look https://www.usenix.org/conference/foci13/workshop-program/presentation/aryan
The Anatomy of Web Censorship in Pakistan https://www.usenix.org/conference/foci13/workshop-program/presentation/Nabi
Dimming the Internet: Detecting throttling as a mechanism of censorship in Iran http://arxiv.org/abs/1306.4361
Global Network Interference Detection Over the RIPE Atlas Network https://www.usenix.org/system/files/conference/foci14/foci14-anderson.pdf
Analysis of Country-wide Internet Outages Caused by Censorship http://conferences.sigcomm.org/imc/2011/docs/p1.pdf
Characterizing Web Censorship Worldwide: Another Look at the OpenNet Initiative Data http://censorbib.nymity.ch/pdf/Gill2015a.pdf
-
David Fifield -
John