Sebastian Hahn sebastian@torproject.org writes:
On 16 Sep 2014, at 16:15, George Kadianakis desnacked@riseup.net wrote:
====How guardiness works==== The idea was that the guardiness script will be an external script that is run by Tor in a similar fashion to the bandwidth auth scripts. We chose that because we could write the script in a high-level language and because it could be modular and we could change the algorithm in the future if we wanted. Unfortunately, it seems that external scripts in dirauths are a PITA to maintain as can be seen by the lack of bwauth operators.
The problem isn't so much that this is an external script, the problem is that there are never dedicated maintainers for these things (tho it being an external script is additionally problematic). If they are in Tor proper, we have someone who cares about them when new stuff gets introduced, and it gets updated with the rest of Tor, etc.
I understand and I can maintain the guardiness script.
However, I'm also hoping for other people to review the script before deployment (#13125). It's a pretty small script so it shouldn't be too wrong.
Summary files are designed to be quick to parse (even with Python) and contain all the necessary information for guardiness. For example, parsing 2000 summary files in my laptop takes about 10 seconds.
Does this scale linearly? 9 months would be ~6500 files.
Yes, it should scale linearly. 6500 files should take about 30 to 40 seconds.
(Each consensus parsing is an independent event.)
FWIW, the guardiness scripts are ready for review and can be found here: https://gitweb.torproject.org/user/asn/hax.git/shortlog/refs/heads/guardines...
====How the guardiness script will be deployed====
The idea is that dirauths will add another script to their crontab that is called every hour (before or after the bwauth scripts).
Only 4/9 have such scripts, if at all - it is possible to run a bwauth on a different host, and scp the file over. I don't know if any of the dirauth ops actually do this currently.
This should also be possible with the guardiness stuff, as long as you can arrange fresh consensuses to arrive on the second host.
The script first calls the summarizer script, which goes to the consensus/ directory and summarizes all consensuses it finds and puts them in the summary/ directory. The summarizer script then deletes all the consensuses that got summarized.
You must not delete files which you did not create. It's not cool to delete consensuses which tor decided to put somewhere if you're not tor. This is relevant to the plan to use mvdan's script.
Yes, you are absolutely right on that. FWIW, the --delete-summarized switch of the summarizer script is disabled by default.
If we go ahead with using Daniel's tor feature, we should _not_ delete those consensuses.
Then the script calls the the guardiness script, which goes to the summary/ directory, parses all summary files it finds, and outputs a guardiness output file that gets parsed by the dirauth prior to voting.
That should be all. Easy, eh? :)
What are the failure modes? Are there version strings included, does Tor notice if the guardiness file is outdated, etc? What happens when different dirauths use different versions of the guardiness file generation script?
There are various failure modes I think.
Failure modes include dirauths considering 5 consensuses when they should be considering 3000 of them. Or dirauths considering 12000 consensuses when they should be considering 3000.
As you pointed out, in the future if the guardiness algorithm changes, failure modes could also include dirauths using different guardiness versions.
Based on your comment, I think it might be a good idea to publish the guardiness script version somewhere in the dirauth votes. Also the number of consensuses considered, and the maximum consesus age considered.
It might even be smart to make a consensus health module that ensures that guardiness versions are the same in all dirauths, and they all derive approximately the same guardiness for all relays (contrary to bwauths, two perfectly configured dirauths should derive the exact same guardiness value).