Hello,
we recently received more reports of Denial of Service attacks on hidden services. You can find our ticket here:
https://trac.torproject.org/projects/tor/ticket/16052
After some debugging, we found the attack vector and identified a few ways to fix or mitigate it. In this attack, the adversary forces a hidden service to create thousands of connections to its underlying application (e.g. the webserver), which overwhelms both Tor and the underlying application.
It's worth noting that Tor is a small network, and even simple DoS attacks can be quite effective against it. Consider that there are botnets out there with more bandwidth than the whole Tor network. Hence, completely solving DoS attacks on Tor is still an open problem; a few long-term defense ideas were stated on the trac ticket.
In any case, we have prepared an experimental short-term fix for this vulnerability that hidden service operators can deploy. It seems to mitigate the attack in our lab tests, so maybe it helps others too.
The fix, allows the hidden service operator to ignore hidden service circuits that try to open tons of connections. This is possible since normal clients will only open a few such connections, whereas the attacker was opening like 5000 connections per circuit.
Here are some instructions for the short-term fix:
== Instructions ==
Our patch is not in an official Tor release yet, so you will need to use an unofficial git branch:
https://trac.torproject.org/projects/tor/ticket/16052#comment:18
First, use one of the provided branches or apply the commits directly to your tor, and recompile your tor. You need to do this. The instructions below *won't* work with stock tor.
Next, an operator who wants to deploy this experimental fix, should first figure out how many simultaneous TCP connections a normal client would establish. For example, an IRC server would probably not need more than 1 simultaneous connection per user. A web server, depending on the use, might need something between 6 to 12 (?) simultaneous connections.
To proceed, as an example, let's say that a client of your hidden service can work fine with 20 simultaneous TCP connections.
Then add the following to your torrc, as part of the per-hidden service configuration:
HiddenServiceMaxStreams 20
and start up your new Tor. See if that helps out for you. You might still get 100% CPU when under attack, but other clients should work normally.
If that doesn't stop the attack, there is a more aggressive solution that might work. In your torrc also add the following line
HiddenServiceMaxStreamsCloseCircuit 1
That will make your hidden service *kill* the attacker's circuit, which will reduce CPU load even more. However, if you have picked a small max streams number, it might also kill normal users' connections. So be careful! Also, please don't set max streams to a tiny value while this is enabled; forcing the attacker to build tons of circuits to you is not good for your HS or for the network.
====
In any case, let us know if this works for you, by sending an email to this list, or commenting on the trac ticket. If it works for people, we might incorporate to a Tor release soon.
Hello,
Some minor notes...
On Wed, 20 May 2015 20:03:38 +0100 George Kadianakis desnacked@riseup.net wrote:
== Instructions ==
Our patch is not in an official Tor release yet, so you will need to use an unofficial git branch:
https://trac.torproject.org/projects/tor/ticket/16052#comment:18
The configuration parameters are now in master (aka 0.2.7.1-alpha-dev). It's likely that a 0.2.6.x backport will happen, but feedback would play an instrumental part in ensuring that happens (either as a reply, or by commenting on the trac ticket).
Next, an operator who wants to deploy this experimental fix, should first figure out how many simultaneous TCP connections a normal client would establish. For example, an IRC server would probably not need more than 1 simultaneous connection per user. A web server, depending on the use, might need something between 6 to 12 (?) simultaneous connections.
Per discussion with the Tor Browser developers, I have been told that 6 is the correct number for http content, and that if there are any more streams associated with a Tor Browser user accessing a site, it would be a Tor Browser bug.
Other browsers/protocols may require a higher or lower limit. A warning is logged periodically (rate limited to avoid log spam/clutter) if circuits exceed the limit, so adjusting the parameter should be relatively straight forward.
Regards,