On Thu, Feb 9, 2023 at 11:12 PM rhatto rhatto@torproject.org wrote:
On Sun, Dec 25, 2022 at 08:14:30PM +0100, p13dz p13dz via tor-onions wrote:
Hello, We've built a solution to scale our onion server horizontally using Kubernetes horizontal autoscaler. It's basically an onionbalance & a set of wrappers to regenerate its config file depending on the set
of
running TorHiddenService instances.
Very interesting!
I wonder if you plan to release this solution as a free/open source software at some point.
That's definitely possible. We'd need to separate it from the rest of our product, clean everything up, write docs, etc.
As a side note, after running our solution in production under a few (D?)DoS attacks, we found some downsides to it (and to the Onion system limits in general):
- Very low limit on the descriptor size in the Onion protocol. Every server instance needs to have its own entry in the descriptor, and at 7-10 instances the descriptor gets saturated, which makes further scaling up impossible. - The current implementation of onionbalance doesn't support automatic adjustment of the number of intro points per server instance. As an example, when I have 1 server instance running, I might want to have 6 intro points to make sure they don't become an easy target for DoS. When we scale up the service, say 3x, 3*6=18 intro points no longer fit in the descriptor, and onionbalance breaks down and stops publishing updates at all. It could be smart enough to automatically lower the "replication factor" until the message fits in the descriptor. If the number of instances is too high, it should publish at least some of them instead of nothing.
At the level of the Onion protocol, it would be great to increase the number of intro points per descriptor to a higher number (say 10x from what we have today). If the Tor team could assist with that, we'd publish our autoscaler and together make Tor a cool platform for building services resilient to serious attacks. Let me know what you think.
Each time we scale the set of servers up, we start new Tor onion servers on random addresses (that's what onionbalance expects), and their descriptors are then merged into the single one by onionbalance. When we delete old instances, their onion addresses are abandoned. We want to be good citizens and not to clutter Tor's DHT with
abandoned
records. My question basically is whether the Tor ecosystem automatically deletes abandoned descriptors or not. If yes, we can
keep
our system simpler. If not, we'll have to make the system stateful and try to reuse our existing onions as much as possible.
TL;DR I think you can keep your system simple :)
Please others correct if I'm wrong, but as far as I understand you don't have to worry about having disposable Onion Service addresses.
This does not overload Tor's "DHT-like" HSDir structure (which in fact is a hash ring in constant renewal).
From the [rend-spec-v3]:
2.2.2. When to publish a hidden service descriptor [WHEN-HSDESC] Hidden services periodically publish their descriptor to theresponsible HSDirs. The set of responsible HSDirs is determined as specified in [WHERE-HSDESC].
Specifically, every time a hidden service publishes its descriptor,it also sets up a timer for a random time between 60 minutes and 120 minutes in the future. When the timer triggers, the hidden service needs to publish its descriptor again to the responsible HSDirs for that time period.
[...] 2.2.5. Expiring hidden service descriptors [EXPIRE-DESC] Hidden services set their descriptor's "descriptor-lifetime" fieldto 180 minutes (3 hours). Hidden services ensure that their descriptor will remain valid in the HSDir caches, by republishing their descriptors periodically as specified in [WHEN-HSDESC].
So a descriptor vanishes from the HSDirs after a while if it's not regularly published. (See also section [HASHRING] and other sections in that document for more info).
That's perfect, thanks!
-- Silvio Rhatto pronouns he/him