While brainstorming for a recent funding proposal, I wrote up this list of "Things we should measure to track our impacts/success" in the context of the Salmon bridge distribution strategy.
Or to put it another way, these are questions I'll be asking phw et al to understand how things are going, once we deploy it. Some of the questions are variations on others, i.e. a single data source can answer several of them. So it's probably better framed as "questions to track" rather than "data sources we should collect".
(1a) how many obfs4 / httpt bridges are running total?
Bridges report their existence to the bridge authority, and bridgedb/rdsys aggregate them and send them to the metrics datasets. So we should already have these numbers.
(1b) out of those, how many are we distributing via salmon?
This is a parameter that we choose, and we will choose it based on how successful Salmon is compared to our existing distribution strategies. If we choose a larger fraction of our bridges to be used for Salmon, it's a good indication that we're finding Salmon to be an effective option.
(2a) how often we are reachability-testing each bridge?
We'll probably start off doing daily testing, but we should aim to get the frequency higher, and we might end up doing more targeted just-in-time testing where whenever a user reports that a bridge is blocked, we launch a test for it right then in order to make a decision about how to respond to the user.
(2b) from how many vantage points we are doing this reachability-testing?
We will want to start with n+1 vantage points, one for each target region and one "control" outside the censored area. That might be sufficient for a long time, or we might learn that we need to split up our testing across more vantage points, or test more in particular regions.
(2c) how many of our bridges are currently reachable from each of these vantage points?
Ideally the answer will be "all of them", but the reality is that some blocking will occur. So the higher the fraction here the better, and in some sense it is a measure of the health of our plans, and/or a measure of the intensity of attention from the censor.
(2d) When a bridge stops working in China, what fraction of the time is it because the bridge went down, i.e. normal churn?
One of the tradeoffs with a community of volunteer bridges is that bridges naturally come and go over time. Understanding the dynamics of our bridge population is key because it impacts the rate at which users need fresh bridges even when there is no censorship.
(3a) how many total users have registered with the salmon system?
This long-term measure of how many people have tried to use our system lets us see how well our outreach is working.
(3b) how many of those users do we think are recently active?
Tracking how many people are still using it is a key indicator for both usability and blocking-resistance: "do people actually find it useful?"
(3c) what is the rate of new users registering with the system?
First of all this helps us understand growth in interest, for example from our outreach efforts, but also it helps us understand how many fresh bridge addresses we need to support this growth. It is tied into the next item which is the other side of the question:
(3d) how many bridges do we have in reserve (not yet filled with users)?
The trouble comes when this number reaches 0. So the target is that we always have some bridges in reserve, which means we are keeping up with the rate of new user registrations. If this number hits zero, it means we need to activate more of our partners to get fresh bridges.
(4a) how many high-reputation users are currently assigned at least one bridge that we think works?
This number summarizes Salmon's success for our high-value or established users. That is, if this number remains high, then Salmon is succeeding at providing availability to its core set of users, even if the other numbers aren't doing well.
(4b) how many low-reputation users are currently assigned at least one bridge that we think works?
This number measures the other side: what is the health of the Salmon system at adding new users, compared to the attention the censor is giving at adding fake users in order to find and block our bridges?
(5a) at what rate are we filling bridges with new salmon users?
This one is quite related to 3c above, but it is from the bridge availability side: the best number here is the highest possible number such that 3d doesn't go to 0.
(5b) at what rate are we filling bridges with existing salmon users?
By looking at the rate where established users need new bridges, we can understand how much stability we have in the system.
(5c) at what rate are users reporting failed bridges but we think the bridges are working and reachable for them?
This point aims to measure our false positives, which could stem from logic errors inside Tor and Tor Browser (e.g. reporting bridges down when actually our internet isn't on), or from non-uniform blocking within countries, or probably many other reasons. If this rate gets much above 0, it's a bug report that we need to track down and understand.
--Roger