On Thu, Sep 22, 2022 at 11:25:54PM -0600, David Fifield wrote:

On Thu, Sep 22, 2022 at 09:24:47AM -0600, David Fifield wrote:

...

There is increased usage of the snowflake-01 bridge since yesterday, likely related to protests/shutdowns in Iran. The 4 load-balanced tor instances, which recently were at about 60% CPU at the steady state, are currently near 100%.

I am planning to increase the number of instances today.

I increased the number of instances without incident: https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla...

I increased the number of instances again, from 8 to 12. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla...

...

I am also seeing a lot of "no address in clientID-to-IP map (capacity 10240)" (dozens per second), so I will increase that parameter at the same time.

This is among some performance changes that I hope to deploy tomorrow. I've actually deployed them on the snowflake-02 bridge for testing already. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla...

I deployed more optimizations aimed at decreasing memory usage per client. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla...

I attached a graph of interface bandwidth for the past few days. Outgoing bandwidth reached well over 300 MB/s on September 24. At this moment, traffic is approaching the daily minimum, which is still around 200 MB/s. We'll see what tomorrow brings. If we run into more memory pressure, we have another easy mitigation, which is to decrease the size of client send queues. https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla... For CPU pressure, I don't see any quick fixes. In an emergency, we could hack the tor binary to use a static ExtORPort authentication cookie, and remove the extor-static-cookie shim from the pipeline.

On Mon, Sep 26, 2022 at 10:39:42AM +0200, Linus Nordberg via anti-censorship-team wrote:

Possibly related, the number of packets per second seems to be bound to 400 kpps so something is going full at a nice and round figure here:

More granularity shows symmetric "dips" for in- and outbound with a frequency of about 10-15 seconds:

CPU interrupts, softirq and softnet show matching dips:

I see the packets/s exceeding 400k now. But you're right, the daily peaks of the packets/s graph look unnaturally flattened the past few days, see attached graph. (Iran shutdowns marked in red.)

I don't know what those semi-periodic "dips" could be. Maybe garbage collection in the snowflake-server process?

On Mon, Sep 26, 2022 at 10:39:42AM +0200, Linus Nordberg via anti-censorship-team wrote:

It seems likely that we're hitting a limit of some sort and next thing is to figure out if it's a soft limit that we can influence through system configuration or if it's a hardware resource limit.

tor has a default bandwidth limit, but we should be nowhere close to it, especially disitributed across 12 instances:

BandwidthRate N bytes|KBytes|MBytes|GBytes|TBytes|KBits|MBits|GBits|TBits A token bucket limits the average incoming bandwidth usage on this node to the specified number of bytes per second, and the average outgoing bandwidth usage to that same value. If you want to run a relay in the public network, this needs to be at the very least 75 KBytes for a relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) — but of course, more is better; we recommend at least 250 KBytes (2 mbits) if possible. (Default: 1 GByte)

I do not see any rate limit enabled in /etc/haproxy/haproxy.cfg.

I checked the number of sockets connected to the haproxy frontend port, thinking that we may be running out of localhost 4-tuples. It's still in bounds (but we may have to figure something out for that eventually).

# ss -n | grep -c '127.0.0.1:10000\s*$' 27314 # sysctl net.ipv4.ip_local_port_range net.ipv4.ip_local_port_range = 15000 64000

According to https://stackoverflow.com/a/3923785, some other parameters that may be important are

# sysctl net.ipv4.tcp_fin_timeout net.ipv4.tcp_fin_timeout = 60 # cat /proc/sys/net/netfilter/nf_conntrack_max 262144 # sysctl net.core.netdev_max_backlog net.core.netdev_max_backlog = 1000 Ethernet txqueuelen (1000)

net.core.netdev_max_backlog is the "maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them." https://www.kernel.org/doc/html/latest/admin-guide/sysctl/net.html#netdev-ma... But if we were having trouble with backlog buffer sizes, I would expect to see lots of dropped packets, and I don't:

# ethtool -S eno1 | grep dropped rx_dropped: 0 tx_dropped: 0

It may be something inside snowflake-server, for example some central scheduling algorithm that cannot run any faster. (Though if that were the case, I'd expect to see one CPU core at 100%, which I do not.) I suggest doing another round of profiling now that we have taken care of the more obvious hotspots in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...

On Tue, Sep 27, 2022 at 08:22:21PM +0200, Linus Nordberg wrote:

David Fifield david@bamsoftware.com wrote Tue, 27 Sep 2022 08:54:53 -0600:

...

I checked the number of sockets connected to the haproxy frontend port, thinking that we may be running out of localhost 4-tuples. It's still in bounds (but we may have to figure something out for that eventually).

# ss -n | grep -c '127.0.0.1:10000\s*$'
27314
# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 15000    64000

Would more IP addresses and DNS round robin work?

By more IP addresses you mean more localhost IP addresses, I guess? All of 127.0.0.0/8 is localhost, so we can expand the range of four-tuples by using more addresses from that address range in either the source or destination address position. haproxy probably has an option to listen on multiple addresses. The trick is actually using the multiple addresses. I don't think DNS will work directly, because snowflake-server gets the address of its upstream from the TOR_PT_ORPORT environment variable, which is specified to take an IP:port, not a DNS name (and is implemented that way in goptlib). https://gitweb.torproject.org/torspec.git/tree/pt-spec.txt?id=ec77ae643f3e47... https://gitweb.torproject.org/pluggable-transports/goptlib.git/tree/pt.go?h=... You could try using more addresses from 127.0.0.0/8 in the *source* address position, by specifying the second parameter of net.DialTCP to set the source address here: https://gitweb.torproject.org/pluggable-transports/goptlib.git/tree/pt.go?h=...

...

It may be something inside snowflake-server, for example some central scheduling algorithm that cannot run any faster. (Though if that were the case, I'd expect to see one CPU core at 100%, which I do not.) I suggest doing another round of profiling now that we have taken care of the more obvious hotspots in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...

After an interesting chat with anarcat I think that we are CPU bound and in particular by handling so many interrupts from the NIC and dealing with such a high number of context switches. I have two suggestions on how to move forward with this.

First, let's patch tor to get rid of the extor processes, as suggested by David earlier when discussing RAM pressure. This should bring down context switches.

The easiest way to do this is probably to comment out the re-randomization of the ExtORPort auth cookie file on startup, and replace the existing cookie files with static files. Or even just comment out the failure case in connection_ext_or_auth_handle_client_hash. https://gitweb.torproject.org/tor.git/tree/src/feature/relay/ext_orport.c?h=...

The uncontrollable rerandomization of auth cookies is the whole reason for extor-static-cookie: https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-si...

Here's my post requesting support in core tor: https://lists.torproject.org/pipermail/tor-dev/2022-February/014695.html

On Wed, Sep 28, 2022 at 11:31:05AM +0200, Linus Nordberg wrote:

David Fifield david@bamsoftware.com wrote Tue, 27 Sep 2022 14:40:48 -0600:

...

On Tue, Sep 27, 2022 at 08:22:21PM +0200, Linus Nordberg wrote:

...

David Fifield david@bamsoftware.com wrote Tue, 27 Sep 2022 08:54:53 -0600:

...

I checked the number of sockets connected to the haproxy frontend port, thinking that we may be running out of localhost 4-tuples. It's still in bounds (but we may have to figure something out for that eventually).

# ss -n | grep -c '127.0.0.1:10000\s*$'
27314
# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 15000    64000

Would more IP addresses and DNS round robin work?

By more IP addresses you mean more localhost IP addresses, I guess?

My confusion was strong at that time yesterday. I mixed up 4-tuples on our (only) externally reachable address with 4-tuples on localhost addresses. Please ignore and thanks for clarifying.

Getting rid of extor should lower the need for localhost 4-tuples, shouldn't it?

No, not really. The problem is not the total number of 127.0.0.1 four-tuples in use — there are ≈2^32 of those — it's when one end has a fixed port number. The bottleneck in this case is the link between snowflake-server and haproxy (see diagram): https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guid...

haproxy binds to 127.0.0.1:10000 and snowflake-proxy connects to haproxy from 127.0.0.1 and an ephemeral port, so three of the four elements of the four-tuple are fixed, permitting only ≈2^16 different tuples:

(127.0.0.1, X, 127.0.0.1, 10000)

The whole pluggable transports interface is built around this model of localhost TCP sockets; I think it did not anticipate scale like this. snowflake-server gets the address 127.0.0.1:10000 from an environment variable; see in /etc/systemd/system/snowflake-server.service:

Environment=TOR_PT_EXTENDED_SERVER_PORT=127.0.0.1:10000

When snowflake-server does pt.DialOr, it's the above address that it makes a TCP connection to. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...

snowflake-server *thinks* it is talking to an upstream tor process's ExtORPort at that address, when actually the connection is intermediated by haproxy (because a single tor process can only handle a limited amount of traffic) and extor-static-cookie (because each tor instance uses a different random authentication key).

haproxy, of course, can listen on multiple ports on its frontend, but TOR_PT_EXTENDED_SERVER_PORT is specified to contain only a single address: https://gitweb.torproject.org/torspec.git/tree/pt-spec.txt?id=ec77ae643f3e47...

That said, none of the above prevents us from hacking around the pluggable transports model where it is constraining. We can free up four-tuple space by varying any of the four elements in the example above; or by using something other than TCP sockets for one or more localhost links. For example, we could hack pt.DialOr to use a random source address in the 127.0.0.0/8 range; that would give us an additional factor of 2^24 between snowflake-server and haproxy. Or we could replace that link with a Unix domain socket. It would just require an alternative means of passing the socket address into snowflake-server, because TOR_PT_EXTENDED_SERVER_PORT cannot represent such an address, and a different version of the pt.DialOr function that does not have the assumption of TCP baked in. https://pkg.go.dev/git.torproject.org/pluggable-transports/goptlib.git#DialO... https://gitweb.torproject.org/pluggable-transports/goptlib.git/tree/pt.go?h=...

Removing extor-static-cookie from the chain would not have an effect on the need for four-tuples, since each of them uses a distinct port number and only has 1/12 of the connections of the bottleneck link.

On 2022-09-27 09:49, David Fifield wrote:

On Fri, Sep 23, 2022 at 11:01:37AM -0400, Cecylia Bocovich wrote:

...

On 2022-09-22 11:24, David Fifield wrote:

...

There is increased usage of the snowflake-01 bridge since yesterday, likely related to protests/shutdowns in Iran. The 4 load-balanced tor instances, which recently were at about 60% CPU at the steady state, are currently near 100%.

I am planning to increase the number of instances today.

It looks like we're also reaching proxy capacity again for the first time in a while.

I've attached a visualization of available proxies that are compatible with all types of client NATs. You can see in the first image that the number of idle proxies has gone to zero and all available proxies are being matched. The second image shows spikes in the number of clients denied a working proxy.

The depletion of this proxy pool could be due to the high amount of mobile network usage, since these networks are likely to have complex and restrictive NAT topologies.

Besides recruiting more proxies, could we stretch the existing unrestricted proxies further? When a proxy finds its own NAT type to be unrestricted, it could increase its polling frequency and/or concurrent capacity.

That's a good idea.

I've started working on tackling the problem from the other side[0]: we have a lot of clients who may not need unrestricted proxies pulling from that pool because their NAT type is unknown (see attached image).

If we have these clients optimistically pull from the other pool, we could reduce the load substantially. However, this change will take a while to roll out because it has to be included in a Tor Browser release.

Updating proxies to poll more frequently is easier to roll out quickly.

[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...

On Tue, Sep 27, 2022 at 10:56:57AM -0400, Cecylia wrote:

On 2022-09-27 10:56, Cecylia Bocovich wrote:

...

On 2022-09-27 09:49, David Fifield wrote:

...

Besides recruiting more proxies, could we stretch the existing unrestricted proxies further? When a proxy finds its own NAT type to be unrestricted, it could increase its polling frequency and/or concurrent capacity.

That's a good idea.

I've started working on tackling the problem from the other side[0]: we have a lot of clients who may not need unrestricted proxies pulling from that pool because their NAT type is unknown (see attached image).

If we have these clients optimistically pull from the other pool, we could reduce the load substantially. However, this change will take a while to roll out because it has to be included in a Tor Browser release.

Updating proxies to poll more frequently is easier to roll out quickly.

[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...

Oops, attaching image.

Could probetest being sick also contribute to a lack of the right NAT types? It's at 100% CPU again, just now as I check it.

/etc/runit/snowflake-probetest/run currently has "timeout 7d"; what if we restarted it every hour?

On 2022-09-29 11:05, Cecylia Bocovich wrote:

Besides recruiting more proxies, could we stretch the existing unrestricted proxies further? When a proxy finds its own NAT type to be unrestricted, it could increase its polling frequency and/or concurrent capacity.

I've opened a merge request that will both increase the polling frequency and bump up the max number of clients from 1 to 2 for unrestricted proxies. This change happens once the proxies have successfully opened a datachannel with a client.

https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...

Proxies that work well with clients in their pool already poll pretty frequently (once every 60s), so I don't think increasing this to 30 seconds will make much of a difference, but doubling the capacity for unrestricted web-based proxies should. I'm hesitant to increase the maximum number of clients further without more testing.