Dear OTF, Ooni, and M-Lab,
Summary
=======
We've hashed out a design to integrate Ooni with mlab-ns on the M-Lab
deployment, and we've implemented a fully functional deployment that
approximates this by simulating mlab-ns (this is attached). This
completes Milestone D of our contract with OTF.
Design Goals
============
Our top goals for this integration are:
It does not rely on any changes to upstream Ooni. (For example,
probes still use a bouncer .onion, and the backend has stock bouncers,
collectors, and test helpers running.)
It can be disabled easily without redeploying the M-Lab backend. Our
branch's ooni-support README.md has instructions to disable the
integration, merely by editing a cron job to unset an ENABLED flag.
There's no need to redeploy different versions of ooni-support.
When enabled, it allows M-Lab operations to monitor collectors and
test_helpers status with the same infrastructure as all other M-Lab
tools.
Future Architectural Changes
----------------------------
In the future, it may be nice to augment ooni / mlab-ns integration.
For example, mlab-ns is designed to support different policies which
may be useful to tools, such as geo-location of test_helpers.
The Simulator
=============
This deployment architecture uses a simulator. While it is fully
functional and useful for testing it lacks security or robustness, so
we want to emphasize *not to deploy this* to non-test environments.
Rationale
---------
There are three rationales for this approach:
First, Least Authority didn't want to push through modifications to
mlab-ns without first creating and testing a proof-of-concept.
Second, we didn't want to block our effort on M-Lab engineering
effort, so this allows a clean division of labor.
Third, by creating and testing a working proof of concept we can help
define the necessary changes to mlab-ns in a tightly scoped and
concrete manner.
Security
--------
This system is insecure because it does not use the M-Lab nagios
system to gather data, and instead lets anyone paste any data they
want into the simulator. Nagios integration is future work captured
in this ticket:
* https://github.com/m-lab-tools/ooni-support/issues/10
Next Steps
==========
Our contract with OTF proposes our next two milestones will focus on
improving integration testing and unit test coverage. Our focus at
that time was on test automation and documentation for diagnosing
integration problems. Test automation has already been improved since
that time, and we've accomplished most of the work for documentation:
https://github.com/m-lab-tools/ooni-support/issues/60
Therefore, we propose to focus on some outstanding issues which will
improve mlab-ns integration while continuing not to block on, or
interfere with, M-Lab operations as follows:
The primary change to mlab-ns will be to allow any tool to include
arbitrary data per slivver to be gathered and distributed by mlab-ns.
Ooni will use this to distribute data such as collector `.onion`
addresses. The need for this change is discussed here:
* https://github.com/m-lab-tools/ooni-support/issues/4
This proposed change is documented in this ticket:
* https://github.com/m-lab-tools/ooni-support/issues/47
A secondary change is to implement `match=all` described in #47 above.
It may not be necessary, so there is further investigation and testing
necessary:
https://github.com/m-lab-tools/ooni-support/issues/56
Along with these changes to `mlab-ns`, we need trivial updates to our
integration scripts to work with mlab-ns rather than the simulator:
* https://github.com/m-lab-tools/ooni-support/issues/10
* https://github.com/m-lab-tools/ooni-support/issues/11
Details & Links
===============
Attached is a shortish overview of possible approaches to implement
this integration. We've implemented a deployment with a mock mlab-ns
(called mlab-ns-simulator) and the "arbitrary data" approach from the
attached design document. The pull request is here:
* https://github.com/m-lab-tools/ooni-support/pull/59
Specific details about this pull request:
* A script for gathering necessary information from collectors and
testhelpers, then updating the mlab-ns-simulator.
* A script for updating a bouncer's state based on the mlan-ns-simulator.
* A cron script to update the bouncer on an hourly schedule.
* The mlab-ns-simulator itself, which approximates the production mlab-ns.
* `.init/` script changes to automatically launch the simulator and
bouncer on `mlab1.nuq0t.measurement-lab.org`.
* Design documentation for mlab-ns integration (including this
stepping stone architecture).
* Each instructions to disable mlab-ns integration without any redeployment.
We also created a subset pull request that has bug fixes but no
mlab-ns integration features:
* https://github.com/m-lab-tools/ooni-support/pull/58
Github Milestones
-----------------
We split the mlab-ns-simulator deployment tasks out from the larger
mlab-ns integration deployment. The mlab-ns-simulator milestone is
at:
* https://github.com/m-lab-tools/ooni-support/issues?q=is%3Aissue+milestone%3…
The full mlab-ns integration milestone:
* https://github.com/m-lab-tools/ooni-support/issues?q=is%3Aissue+milestone%3…
As always, let us know if you have any feedback!
--
Nathan Wilcox
Least Authoritarian
email: nathan(a)leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993
Hello,
We are working on scripts to create the Ooni bouncer configuration from
multiple instances of Ooni running on M-Lab slices. So far, we have two
scripts. One, which collects the portion of the bouncer.yaml out of an
Ooni slice, and another script, which combines all of the parts into a
single bouncer.yaml:
https://github.com/LeastAuthority/ooni-support/blob/mlab-ns-simulator/bounc…https://github.com/LeastAuthority/ooni-support/blob/mlab-ns-simulator/bounc…
One thing we would like the "getconfig.py" script to do is to
dynamically determine which Test Helpers are enabled on the slice. I'm
not sure how to do this, because the oonib.conf always contains entries
for all the test helpers. So my question is this: How can we determine
which Test Helpers are enabled by examining the oonib.conf or other files?
Or, is this question misguided? Are *all* test helpers supposed to be in
the bouncer.yaml file, even ones which are not running, and we don't
want to have running?
(For the first M-Lab deployment, there will be only one Test Helper, the
http-return-headers one, so we could hard-code that one into the script,
but we would like to make this script keep working when new tests are
enabled).
Thanks,
--
Taylor Hornby
Dear Oonitarians,
We're about to start testing some mlab-ns integration features on the
M-Lab test slice. This will be potentially disruptive because we'll
be reinitializing the slice, installing new bits and bobs, and we'll
be using our own fork of ooni-support [1] rather than upstream
ooni-support [2].
Please let us know if we should *not* do this soon. Today I planned
to only poke mlab1.nuq0t.measurement-lab.org [3], but eventually we
want to test at least two test slivvers.
Meanwhile, what happens if we deploy some integration glue and it
doesn't work, or has unanticipated problems in the future, or M-Lab
policies change, etc?
To anticipate those kinds of issues, we want to make sure that it's
easy to toggle off the integration stuff. One of our key design goals
has been to avoid modifying upstream ooni. We've just realized a new
and related design goal: make sure it's easy to turn off the
integration glue without requiring slice re-initialization, packages
changes, repository changes, etc... So we just added a ticket for
that [4].
Note that *while* we are doing this interactive testing, the two
slivvers we're using will be pretty unstable, but *after* we reach a
stable level of development, we should be able to flip the integration
switch to off and the M-Lab test slice should operate as before
(except with some extra stuff installed).
If you have any concerns / feedback / comments, etc... feel free to
email us, or respond to specific tickets. All of the tickets related
to the mlab-ns integration in particular should live in the
`ooni-support` issue tracker with the `LeastAuthority D` label [5].
Regards,
Nathan
links:
[1] https://github.com/LeastAuthority/ooni-support
[2] https://github.com/m-lab-tools/ooni-support
[3] https://github.com/m-lab-tools/ooni-support/issues/50#issuecomment-50060748
[4] https://github.com/m-lab-tools/ooni-support/issues/54
[5] https://github.com/m-lab-tools/ooni-support/issues?direction=desc&labels=Le…
--
Nathan Wilcox
Least Authoritarian
email: nathan(a)leastauthority.com
twitter: @least_nathan
PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993
