May I ask what your set up is?
Are you running your relays on separate VMs on the main system or are
you using a different set up like having all IP addresses on the same OS
and using OutboundBindAddress , routing, etc... to separate them? If I
know more, I might be able to make a script specific to your set up.

Thank you. Yes, of course.

Ubuntu server 22.04 runs on bare metal. Ansible-relayor manages 20 exit relays on each system. Netplan has each IP individually listed (sub-divided as a /25 per server from within a dedicated /24, similarly for v6 addresses). I believe an available IP is randomly picked by ansible-relayor and used statically in each torrc file.

Here is an example torrc:

# ansible-relayor generated torrc configuration file

# Note: manual changes will be OVERWRITTEN on the next ansible-playbook run


OfflineMasterKey 1

RunAsDaemon 0

Log notice syslog

OutboundBindAddress 23.129.64.130

SocksPort 0

User _tor-23.129.64.130_443

DataDirectory /var/lib/tor-instances/23.129.64.130_443

ORPort 23.129.64.130:443

ORPort [2620:18c:0:192::130]:443

OutboundBindAddress [2620:18c:0:192::130]


DirPort 23.129.64.130:80

Address 23.129.64.130


SyslogIdentityTag 23.129.64.130_443


ControlSocket /var/run/tor-instances/23.129.64.130_443/control GroupWritable RelaxDirModeCheck


Nickname ageis

ContactInfo url:emeraldonion.org proof:uri-rsa ciissversion:2 tech@emeraldonion.org


Sandbox 1

NoExec 1


# we are an exit relay!

ExitRelay 1

IPv6Exit 1

DirPort [2620:18c:0:192::130]:80 NoAdvertise

DirPortFrontPage /etc/tor/instances/tor-exit-notice.html



ExitPolicy reject 23.129.64.128/25:*,reject6 [2613:18c:0:192::]/64:*,accept *:*,accept6 *:*



MyFamily <snip>

# end of torrc



--
Christopher Sheats (yawnbox)
Executive Director
Emerald Onion
Signal: +1 206.739.3390
Website: https://emeraldonion.org/
Mastodon: https://digitalcourage.social/@EmeraldOnion/




On Dec 4, 2022, at 10:08 PM, Chris <tor@wcbsecurity.com> wrote:

Sorry to hear it wasn't much help. Even though the additions I suggested
didn't help they certainly couldn't cause any harm and can't be
responsible for the drops in traffic.

As for the torutils scripts, I'm sure toralf would be able to better
investigate that but I have a feeling you have a certain set up that
might not have worked with the script. May I ask what your set up is?
Are you running your relays on separate VMs on the main system or are
you using a different set up like having all IP addresses on the same OS
and using OutboundBindAddress , routing, etc... to separate them? If I
know more, I might be able to make a script specific to your set up.

On 12/3/2022 2:07 PM, Christopher Sheats wrote:
Hello,

Thank you for this information. After 24-hours of testing, these
configurations brought Tor to a halt.

At first I started with the sysctl modifications. After a few hours
with just that, there was no improvement in ~75%
inet_csk_bind_conflict utilization. I then installed Torutils for both
IPv4 and IPv6. After only a couple of hours, Tor dropped to below 15
Mbps across both servers (40 relays). 16 hours later, Tor dropped
below 2 Mbps.

I've removed all of these new settings and restarted.

--
Christopher Sheats (yawnbox)
Executive Director
Emerald Onion
Signal: +1 206.739.3390
Website: https://emeraldonion.org/
Mastodon: https://digitalcourage.social/@EmeraldOnion/




On Dec 2, 2022, at 7:30 AM, Chris <tor@wcbsecurity.com> wrote:

Hi,

As I'm sure you've already gathered, your system is maxing out trying to
deal with all the connection requests. When inet_csk_get_port is called
and the port is found to be occupied then inet_csk_bind_conflict is
called to resolve the conflict. So in normal circumstances you shouldn't
see it in perf top much less at 79%. There are two ways to deal with it,
and each method should be complimented by the other. One way is to try
to increase the number of ports and reduce the wait time which you have
somehow tried. I would add the following:

net.ipv4.tcp_fin_timeout = 20

net.ipv4.tcp_max_tw_buckets = 1200

net.ipv4.tcp_keepalive_time = 1200

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_max_syn_backlog = 8192

The complimentary method to the above is to lower the number of
connection requests by removing the frivolous connection requests out of
the equation using a few iptables rules.

I'm assuming the increased load you're experiencing is due to the
current DDos attacks and I'm not sure if you're using anything to
mitigate that but you should consider it.

You may find something useful at the following  links

[1](https://github.com/Enkidu-6/tor-ddos)

[2](https://github.com/toralf/torutils)

[background](https://gitlab.torproject.org/tpo/community/support/-/issues/40093)

Cheers.

On 12/1/2022 3:35 PM, Christopher Sheats wrote:
Hello tor-relays,

We are using Ubuntu server currently for our exit relays.
Occasionally, exit throughput will drop from ~4Gbps down to ~200Mbps
and the only observable data point that we have is a significant
increase in inet_csk_bind_conflict, as seen via 'perf top', where it
will hit 85% [kernel] utilization.

A while back we thought we solved with with two /etc/sysctl.conf
settings:
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1

However we are still experiencing this problem.

Both of our (currently, two) relay servers suffer from the same
problem, at the same time. They are AMD Epyc 7402P bare-metal servers
each with 96GB RAM, each has 20 exit relays on them. This issue
persists after upgrading to 0.4.7.11.

Screenshots of perf top are shared
here: https://digitalcourage.social/@EmeraldOnion/109440197076214023

Does anyone have experience troubleshooting and/or fixing this problem?

Cheers,

--
Christopher Sheats (yawnbox)
Executive Director
Emerald Onion
Signal: +1 206.739.3390
Website: https://emeraldonion.org/
Mastodon: https://digitalcourage.social/@EmeraldOnion/





_______________________________________________
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays