Hi Arturo, Roya,
It's great to see this discussion here. I wanted to chime in as well to end my long-time lurking around here, and largely to support what Roya's been saying. I've published a bit of research in this area, some specifically on ethics and some on getting data with a relatively light touch, and I'm involved with a few other people who are working on a set of ethical principles associated with network measurement. (That work coming out of the IMC papers that Roya mentioned.) I think OONI is a key testbed for this kind of thinking.
To clarify: in the comments below, I'm not attacking anything you've said because I appreciate that OONI even considers these issues. I did want to pick them apart a bit, though. :)
On Mon, Jan 05, 2015 at 02:27:47PM +0100, Arturo Filastò wrote:
The reason why all these projects rely on vantage points from the network point of view is that this is the way to have the most accurate measurements and in a lot of cases it is the only way to measure that particular kind of censorship.
Being from the vantage point of the censored user allows you to fully emulate what a user would be doing when accessing the censored site.
I know that you are consciously making a balance between user safety and effectiveness, but it's worth explicitly stating that sometimes there are measurements you just can't have without compromising user safety, and sometimes you have to accept a loss in accuracy.
Your research is very interesting and I believe very important for getting more data when it would just be too risky to have network vantage. I do think, though, that we can't rely only on these sorts of measurements. They complement and extend what we measure from the network vantage point of the user, but may not work as reliably in all censorship systems and only give you a subset of the information we are interested in acquiring.
I think it might be useful to try and draw a line at which the risk to users overwhelms the need for accuracy. At the moment, even though ethics are being discussed, they seem to be subordinate to functionality.
For example things that we are interested in gathering with ooniprobe are also fingerprints for the censorship equipment being used. This is something that I don't think will be as accurately measured with indirect scans.
Of course the balance swings in the other direction, and you shouldn't abandon key data just because of a vague and unspecified risk. Have you considered alternative approaches to fingerprinting censorship equipment in detail, though? It strikes me that there are possibly several approaches that wouldn't rely on end user vantage points. In a sense, the danger of OONI is that you /can/ answer your questions with end user installs, so you have less incentive to find alternative less OONI-like approaches.
Another factor here is that, as you say below, you are trying to balance user safety against impact of getting data, but your impact is equally fuzzy. How do you measure your impact? How do you judge whether having detailed information about a filtering device is worth the possible risk to an end user?
Yes I perfect agree with the fact that we should also be collecting measurements gathers using these sorts of technique using ooniprobe. It would be epic if you or somebody else were to implement ooniprobe measurements for them.
This relates back to the OONI-centric idea -- shouldn't side channel techniques like these be an alternative data source, rather than worked into the end-user installed OONI model?
I would however like to make the point that with the OONI project our main goal is not that of publishing academic papers. If that comes as part of the process then it's great, but our top priority is finding the right balance between the safety of users and impact we can reach by exposing facts about internet censorship.
This is key, and again -- what is your measure of impact? How do you weigh it against potentially unknown user risks?
This is something very tough, but I think that by not being directly affiliated with a university (hence not having to jump through the various hoops you folks have to before doing your research), we have a slight advantage. We don't have to get approval from IRBs or have to publish a certain amount of papers per year. The only people we are accountable to are our users.
I know what you mean here, and I'm sure you didn't mean this how it reads, but this is the thing that convinced me to reply to this email! The fact that you don't have to 'jump through the hoops' of IRB approval is deeply worrying to me, because you have no independent oversight to balance your wishes against the risks to other people. IRB isn't (meant to be) an adversarial process where people try to stop you doing things, it's a second set of thinking about the appropriate balance between risks and benefits of research.
I believe that you (we!) are doing this for the right reasons, but I find it a bad policy to trust anyone who thinks they're doing good things for good reasons. That way lies Jack Bauer. :)
I think that the censor would have a pretty hard job proving in a just court of law that such user was engaging in censorship measurements (assuming they consider censorship measurements to be an illegal thing). Unfortunately in some countries were we measure the courts of law are not just and we have to make all sorts of crazy assumptions on how they will interpret what we are doing. Using routers instead of real users when doing the scans could be a safer move if it does not affect your measurement.
I know that you qualify it in the second sentence, but 'proving something in a just court of law' isn't even worth mentioning in the specific field we're talking about.
I definitely think that using routers is a great idea if it can be managed, as is using alternative services where possible and trying to locate probes in organizations' networks rather than personal users' ones. The niggle I have is 'if it does not affect your measurement'. I really think that it should be 'if the balance is right between the effect on the measurement, and the risk to the user'.
I'm really happy to see these discussions happening here, and I hope that nothing above came across as an attack -- I think you're fighting the good fight. Roya has been doing some amazing work in this field, and I think there's huge potential for combining ooniprobe-ish data sources with others to maximize the 'impact' of what comes out of all these filtering measurement projects.
My wider point is that impact requires analysis as well as data, but we can have that discussion later. :)
All the best,
Joss