Hi

I've implemented following mitigations:
* limit memory in queues. For my system that's a safe yet large enough setting (2gb system mem, current usage around 320mb).
    MaxMemInQueues 768 MB

* connlimit: both count & rate. Although, based on observations, only the rate limit is actually being hit, and then only for the reported suspect networks (see below on counts per /24).

    -A INPUT -p tcp -m multiport --dports 9080,9443 -m connlimit --connlimit-upto 360 --connlimit-mask 24 -m hashlimit --hashlimit-upto
20/second --hashlimit-mode srcip --hashlimit-srcmask 24 --hashlimit-name mask24 -j ACCEPT

With these settings, there are no stability issues.

Regards


Frequent offenders of rate limit are (to name a few):
37.48.86.*
51.15.161.*
95.211.95.*
212.32.226.*
199.115.112.*
213.227.137.*
54.36.51.*


On Thu, 21 Dec 2017 at 21:21 mick <mbm@rlogin.net> wrote:
On Wed, 20 Dec 2017 17:22:54 +0100
fcornu@wardsback.org allegedly wrote:

> Hi
>
> I'm the happy maintainer of wardsback :
> B143D439B72D239A419F8DCE07B8A8EB1B486FA7

And I run 0xbaddad - EA8637EA746451C0680559FDFF34ABA54DDAE831 a guard
(though whether it stays a guard depends. It keeps falling over.)
>
> As many of us have noticed, many guard nodes are beeing abused by
> extremely high numbers of connection attempts.
> Thanks to some of you guys, I manged to put some mitigation in place
> [0] and I assume many of us did as well.

I'm still looking at mitigation. I'd rather not add iptables filter
rules because it feels like the wrong thing to do (I might hurt
legitimate connections) and at the wrong end of the stack. I'd prefer
there to be mitigations available at the application end (Tor itself).
But realistically I know that that is difficult and the Tor developer
team are still working hard at this problem. (As an aside, I'd be very
grateful for any feedback from other relay operators who /have/ added
iptables "connlimit" rules. What is your view either way?)

I'm only sticking my head above the parapet now to note what I am
seeing.

So: My logs show Tor staying up for around 10 minutes at a time before
rebooting with the following sort of entries:

Dec 21 16:25:44.000 [notice] Performing bandwidth self-test...done.
Dec 21 16:35:20.000 [notice] Tor 0.3.1.9 (git-df96a13e9155c7bf) opening
log file. Dec 21 16:35:20.946 [notice] Tor 0.3.1.9
(git-df96a13e9155c7bf) running on Linux with Libevent 2.0.21-stable,
OpenSSL 1.1.0f, Zlib 1.2.8, Liblzma 5.2.2, and Libzstd 1.1.2. Dec 21
16:35:20.947 [notice] Tor can't help you if you use it wrong! Learn how
to be safe at https://www.torproject.org/download/download#warning Dec
21 16:35:20.947 [notice] Read configuration file
"/usr/share/tor/tor-service-defaults-torrc". Dec 21 16:35:20.947
[notice] Read configuration file "/etc/tor/torrc". Dec 21 16:35:20.951
[notice] Based on detected system memory, MaxMemInQueues is set to 369
MB. You can override this by setting MaxMemInQueues by hand. Dec 21
16:35:20.952 [notice] Opening Control listener on 127.0.0.1:9051 Dec 21
16:35:20.953 [notice] Opening OR listener on 0.0.0.0:9001 Dec 21
16:35:20.000 [notice] Not disabling debugger attaching for unprivileged
users. Dec 21 16:35:21.000 [notice] Parsing GEOIP IPv4
file /usr/share/tor/geoip. Dec 21 16:35:21.000 [notice] Parsing GEOIP
IPv6 file /usr/share/tor/geoip6. Dec 21 16:35:22.000 [notice]
Configured to measure statistics. Look for the *-stats files that will
first be written to the data directory in 24 hours from now. Dec 21
16:35:22.000 [notice] Your Tor server's identity key fingerprint is
'0xbaddad EA8637EA746451C0680559FDFF34ABA54DDAE831' Dec 21 16:35:22.000
[notice] Bootstrapped 0%: Starting Dec 21 16:35:31.000 [notice]
Starting with guard context "default" Dec 21 16:35:31.000 [notice]
Bootstrapped 80%: Connecting to the Tor network Dec 21 16:35:31.000
[notice] Signaled readiness to systemd Dec 21 16:35:31.000 [notice]
Opening Control listener on /var/run/tor/control Dec 21 16:35:31.000
[notice] Bootstrapped 85%: Finishing handshake with first hop Dec 21
16:35:32.000 [warn] Problem bootstrapping. Stuck at 85%: Finishing
handshake with first hop. (Connection refused; CONNECTREFUSED; count
10; recommendation warn; host CD14AE63A02686BAE838A8079449B480801A8A5F
at 195.181.208.180:443) Dec 21 16:35:32.000 [warn] 9 connections have
failed: Dec 21 16:35:32.000 [warn]  9 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000 [warn]
Problem bootstrapping. Stuck at 85%: Finishing handshake with first
hop. (Connection refused; CONNECTREFUSED; count 11; recommendation
warn; host 500FE4D6B529855A2F95A0CB34F2A10D5889E8C1 at
134.19.177.109:443) Dec 21 16:35:32.000 [warn] 10 connections have
failed: Dec 21 16:35:32.000 [warn]  10 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000 [warn]
Problem bootstrapping. Stuck at 85%: Finishing handshake with first
hop. (Connection refused; CONNECTREFUSED; count 12; recommendation
warn; host 3DE7762DD6165FD70C74BD02A6589C8C0C1B020A at
62.210.76.88:9001) Dec 21 16:35:32.000 [warn] 11 connections have
failed: Dec 21 16:35:32.000 [warn]  11 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000 [warn]
Problem bootstrapping. Stuck at 85%: Finishing handshake with first
hop. (Connection refused; CONNECTREFUSED; count 13; recommendation
warn; host 03DC081E4409631006EFCD3AF13AFAAF2B553FFC at
185.32.221.201:443) Dec 21 16:35:32.000 [warn] 12 connections have
failed: Dec 21 16:35:32.000 [warn]  12 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000 [warn]
Problem bootstrapping. Stuck at 85%: Finishing handshake with first
hop. (Connection refused; CONNECTREFUSED; count 14; recommendation
warn; host 51939625169E2C7E0DC83D38BAE628BDE67E9A22 at
109.236.90.209:443) Dec 21 16:35:32.000 [warn] 13 connections have
failed: Dec 21 16:35:32.000 [warn]  13 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000 [warn]
Problem bootstrapping. Stuck at 85%: Finishing handshake with first
hop. (Connection refused; CONNECTREFUSED; count 15; recommendation
warn; host 500FE4D6B529855A2F95A0CB34F2A10D5889E8C1 at
134.19.177.109:443) Dec 21 16:35:32.000 [warn] 14 connections have
failed: Dec 21 16:35:32.000 [warn]  14 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000 [warn]
Problem bootstrapping. Stuck at 85%: Finishing handshake with first
hop. (Connection refused; CONNECTREFUSED; count 16; recommendation
warn; host 03DC081E4409631006EFCD3AF13AFAAF2B553FFC at
185.32.221.201:443) Dec 21 16:35:32.000 [warn] 15 connections have
failed: Dec 21 16:35:32.000 [warn]  15 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:32.000
[notice] Bootstrapped 90%: Establishing a Tor circuit Dec 21
16:35:33.000 [warn] Problem bootstrapping. Stuck at 90%: Establishing a
Tor circuit. (Connection refused; CONNECTREFUSED; count 17;
recommendation warn; host 1FA8F638298645BE58AC905276680889CB795A94 at
185.129.249.124:9001) Dec 21 16:35:33.000 [warn] 16 connections have
failed: Dec 21 16:35:33.000 [warn]  16 connections died in state
connect()ing with SSL state (No SSL object) Dec 21 16:35:33.000 [warn]
Problem bootstrapping. Stuck at 90%: Establishing a Tor circuit.
(Connection refused; CONNECTREFUSED; count 18; recommendation warn;
host DAC825BBF05D678ABDEA1C3086E8D99CF0BBF112 at 185.73.220.8:443) Dec
21 16:35:33.000 [warn] 17 connections have failed: Dec 21 16:35:33.000
[warn]  17 connections died in state connect()ing with SSL state (No
SSL object) Dec 21 16:35:33.000 [notice] Tor has successfully opened a
circuit. Looks like client functionality is working. Dec 21
16:35:33.000 [notice] Bootstrapped 100%: Done

So - I get loads of CONNECTREFUSED whilst coming up (presumably because
of the attack) and then come fully back online. "netstat" then shows my
connections rising rapidly to around the 10,000-11,000 "ESTABLISHED"
mark before it all goes wrong again.

As others have noted I see multiple connections from OVH (netblock
54.36.51/24 (around 1200, when I normally only see a max of 200 or so
per /24, and a more normal dozen or so per /24). The next largest,
at around 700-800 is 144.76.175/24 (Hetzner Online). I don't recall
seeing that level of connections in the past.

If anyone wants more info, let me know.

Best

Mick

---------------------------------------------------------------------
 Mick Morgan
 gpg fingerprint: FC23 3338 F664 5E66 876B  72C0 0A1F E60B 5BAD D312
 http://baldric.net

---------------------------------------------------------------------

_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays