So I’ve just had a conversation with dgoulet on IRC, which I will reformat and subedit here as a conversation regarding OnionBalance and issues in 2.6 and 2.7 when a recently rebooted HS publishes a fresh descriptor:
[…]
alecm: consider OnionBalance which - being a bunch of daemons on a bunch of servers - will be a lot more prone to intermittent failures of 1+ daemons yielding a lot of republishing
alecm: we tend to move services around, and daemons will be killed in one place and resurrected elsewhere, and then we'll have to bundle up a new descriptor and ship it out
dgoulet: hrm so with that new 027 cache behavior, as long as the IP are usable, the descriptor will be kept, if they all become unusable, a new descriptor fetch is triggered and then those IPs will be tried
alecm: There's a mandatory refresh [of the descriptor] after N minutes?
dgoulet: we'll retry 3 times and after that all HSDir are in timeout for 15 minutes (I think, I'll have to validate) before retrying any HSDirs
alecm: I wonder if descriptors should publish a recommended TTL - [number of seconds to live before refresh]
dgoulet: yeah we have an idea for a "revision-counter" in the descriptor being incremented at each new version for the 24 hours period
dgoulet: a TTL could be useful for load balancing though!
alecm: so, here's a scenario: imagine that we run 10 daemons,
alecm: call these daemons: A B C D E F G H I J - they all have random onion addresses
alecm: we steal one IP from each daemon, and bundle the 10 stolen IPs together to make an onionbalance site descriptor and publish it
alecm: people pull that descriptor, it's quite popular
alecm: we then lose power in a datacentre, which takes out half of our onions - say, A through E
alecm: we reboot the datacentre and restart A-E merely 10 minutes later
alecm: everyone who has already loaded our onionbalance site descriptor tests A B C D E and finds them all dead, because the old IPs for A-E are invalid
alecm: so they all move to F G H I J - which get overloaded even though (new) A B C D E are back up
alecm: and this persists for up to 244, even though the outage was only 10 minutes
alecm: net result: large chunks of the world (anyone with an old descriptor + anyone randomly choosing F-J) have a shitty experience, which is not what high-availability is all about :-)
dgoulet: that will be what's going to happen - having a TTL in the desc. would help here indeed, I see the issue
dgoulet: TTL would be one thing to add, here we could also add a mechanism for a client retrying IPs that failed in the situation where some of the IPs are still working, or making client balance themself randomly could be also an idea
dgoulet: definitely there is some content here for tor-dev - I don't have a good answer but it should definitely be addressed
alecm: proper random selection of IP would be beneficial for load-balancing; not perfect, but in the long run, helpful
— Alec Muffett Security Infrastructure Facebook Engineering London