Greetings!
This is another thread [0] about onion service denial-of-service attacks.
It has long been suggested that onion service operators should be given the option to kill spammy rendezvous circuits at will if they feel they are causing too much damage.
Right now this is possible using the HiddenServiceExportCircuitID torrc option (introduced in #4700) and then using the CLOSECIRCUIT control port command to close circuits.
Unfortunately, we have recently got reports that this technique is not viable for busy onion services under DoS because their control port gets overwhelmed by (useful for them) events, and it's basically rendered useless to the point that any CLOSECIRCUIT command takes several seconds to become effective.
For this reason, multiple onion operators [1] have resorted to using the actual HTTP protocol as a direct channel of communication to the Tor daemon to request circuit shutdowns. This works by embedding a special string (or HTTP error code) to the HTTP responses from nginx to the Tor daemon and adding special custom code to the Tor daemon to close circuits that carry this string. This seems to work well enough for people so far.
This is a thread to discuss this approach and other alternatives since it seems a useful tool against application-layer onion service denial of service attacks.
Let me go through the positives and negatives of actuallym erging this defence upstream to little-t-tor:
---
Positives:
1) This is a solid defense that actually helps people and has been reported as a positive countermeasure in an area that has been hard to find concrete defences (also see [0]).
2) Seems like more and more people are doing this already in a custom ad-hoc fashion, so merging this upstream will at least give them a secure way of doing it (instead of writing custom C code).
3) It's actually a pretty simple patch in terms of tech-debt and maintaining it.
4) The more we address DoS vectors like this one, the less incentive will exist for DoS actors to exist. Effectively improving the long-term health of the network.
Negatives:
a) It's a dirty hotfix that blends the networking layers and might be annoying to maintain in the long-term.
b) It only works for HTTP (and without SSL?).
---
For me, point (1) is extremely important, since we've been struggling with helping onion services that are getting DoSsed and this feature offers a solid defense against practical attacks.
However, IMO the right way to do this feature, would be to improve the control port code and design so that it doesn't get so overwhelmed by multiple events. That said, I'm not sure exactly what kind of changes we would have to do to the control port to actually make it a viable option, and it seems to me like a pretty big project that serves as a medium-term to long-term solution (which we have no resources to pursue right now), whereas the hack of this thread is more of a short-term solution.
I'm looking forward to constructive feedback here, since this seems like a controversial feature that users really need.
Thanks! :)
[0]: serving as a continuation of previous classics such as: https://lists.torproject.org/pipermail/tor-dev/2019-June/013875.html https://lists.torproject.org/pipermail/tor-dev/2019-July/013923.html etc.
[1]: http://www.hackerfactor.com/blog/index.php?/archives/804-A-New-Tor-Attack.ht... https://trac.torproject.org/projects/tor/ticket/32511
On Mon, Dec 2, 2019 at 9:16 AM George Kadianakis desnacked@riseup.net wrote:
However, IMO the right way to do this feature, would be to improve the control port code and design so that it doesn't get so overwhelmed by multiple events. That said, I'm not sure exactly what kind of changes we would have to do to the control port to actually make it a viable option, and it seems to me like a pretty big project that serves as a medium-term to long-term solution (which we have no resources to pursue right now), whereas the hack of this thread is more of a short-term solution.
I think I agree with you here about this part.
A quick question that might help us: _why_ is the control port code slow in this case? Is the problem that a bunch of events are queued, and that keeps the controller from getting events that it needs? Or that the control port doesn't listen for incoming commands until the events are flushed (seems unlikely to me)? Or that a bunch of events are queued, and that keeps the controller from getting the ACK for its close command? Or something else?
For some of these cases, there might be a quick workaround in just having the controller open two connections: one for bulk and one for low-latency.
Hi,
There's also another negative we haven't considered:
On 3 Dec 2019, at 00:16, George Kadianakis desnacked@riseup.net wrote:
Negatives:
a) It's a dirty hotfix that blends the networking layers and might be annoying to maintain in the long-term.
b) It only works for HTTP (and without SSL?).
c) We'll need to make sure that this defence can't be triggered accidentally, (or maliciously via request or response content), otherwise it turns into another way of triggering a DoS.
For example, if we searched for a custom string anywhere in the data stream, then any page documenting that string would be unavailable.
T
George Kadianakis desnacked@riseup.net writes:
However, IMO the right way to do this feature, would be to improve the control port code and design so that it doesn't get so overwhelmed by multiple events. That said, I'm not sure exactly what kind of changes we would have to do to the control port to actually make it a viable option, and it seems to me like a pretty big project that serves as a medium-term to long-term solution (which we have no resources to pursue right now), whereas the hack of this thread is more of a short-term solution.
Is it definitely the Tor-side control-port code that's "overwhelmed", or is the controller-side not reading fast enough or something..? (It seems odd to me that it's somehow faster/better to inband-signal via HTTP..) What controller is being used?
-
George Kadianakis -
meejah -
Nick Mathewson -
teor