Also requested was a description of typical package deployment and
experiment management on M-Lab.
Typical package management follows these basic steps:
1) build rpm package from source repos in http://github.com/m-lab-tools/
2) copy and sign rpm package to M-Lab yum repository
Now per-machine, the slice (i.e. experiment VM) is instantiated:
3) upon slice vm creation on an M-lab server, the first-phase of
initialization includes bootstrapping the yum configuration of the vm's
filesystem (i.e. /etc/yum.conf and /etc/yum.slice.d/slice.repo list).
These import the public signing key, and point to CentOS mirrors and the
M-Lab slice package repository.
4) the second phase of slice initialization then tries to install the
slice package:
i.e. yum install mlab_ooni
5) on success, the service starts. on failure, stop.
6) M-Lab uses external monitoring to identify failed services and inform
a directory service like mlab-ns of available servers.
Can we extend this to inform a different service that is not mlab-ns? We have concluded that it's probably ideal to not rely to heavily on m-lab for the discovery and of collectors and helpers.
We would ideally want to be able to do a HTTP POST request over Tor to a certain service to update the list of collectors and helpers that are currently running as is detailed in this ticket.
Would it be a problem to install tor, torsocks and curl on these machines and add a script to push to such a service?
Does this monitoring service also routinely check to monitor the health of the nodes?
Since the rpm packages are effectively public, they do not include any
private information. Some experiments have conditional deployment. i.e.
if site==nuq01, then run an additional service. As well, the initialize
scripts for a package could generate per-machine key pairs.
The reason why this is the case is now clear to me. If we can add some extra commands to the monitoring scripts that would be sufficient for us.
Would it be possible to service the monitoring infrastructure with private key material?