On Thu, Dec 30, 2021 at 10:42:51PM -0700, David Fifield wrote:
One complication we'll have to work out is that ptadapter doesn't have a setting for ExtORPort forwarding. ptadapter absorbs any ExtORPort information and forwards an unadorned connection onward. The idea I had to to work around this limitation is to have ptadapter, rather than execute snowflake-server directly, execute a shell script that sets TOR_PT_EXTENDED_SERVER_PORT to a hardcoded address (i.e., to haproxy) before running snowflake-server. Though, I am not sure what to do about the extended_orport_auth_cookie file, which will be different for different tor instances.
There are a number of potential ways to deal with the complication of ExtORPort authentication, from alternative ExtORPort authentication types, to ExtORPort-aware load balancing. With a view towards deploying something in the near future, I wrote this program that enables an external pluggable transport to talk to tor's ExtORPort and authenticate as if it had an unchanging authentication cookie.
https://gitlab.torproject.org/dcf/extor-static-cookie
The difficulty with load-balancing multiple tor instances, with respect to ExtORPort, is that to authenticate with the ExtORPort you need to read a cookie from a file on disk, which tor overwrites randomly every time it starts. If you do not know which instance of tor will receive your forwarded traffic, you do not know which ExtORPort cookie to use.
The extor-static-cookie program presents an ExtORPort interface, however it reads its authentication cookie that is independent of any instance of tor, which you can write once and then leave alone. The external server pluggable transport can read from the shared authentication cookie file as well. Every instance of tor runs a copy of extor-static-cookie, all using the same authentication cookie file. The extor-static-cookie instances receive ExtORPort authentication from the external server pluggable transport, along with the USERADDR and TRANSPORT metadata, then re-authenticate and echo that information to their respective tor's ExtORPort.
So we change from this: ___ .->|tor| ________________ _______ | --- ->|snowflake-server|->|haproxy|-+->|tor| ---------------- ------- | --- '->|tor| --- to this: ___________________ ___ .->|extor-static-cookie|->|tor| ________________ _______ | ------------------- --- ->|snowflake-server|->|haproxy|-+->|extor-static-cookie|->|tor| ---------------- ------- | ------------------- --- '->|extor-static-cookie|->|tor| ------------------- ---
I have a similar setup running now on a test bridge, with one instance of obfs4proxy load-balancing to two instances of tor.
## Setup notes
Install extor-static-cookie: # apt install golang # git clone https://gitlab.torproject.org/dcf/extor-static-cookie # (cd extor-static-cookie && go build) # install -o root -g root extor-static-cookie/extor-static-cookie /usr/local/bin/
Generate a shared authentication cookie file: # mkdir -m 755 /var/lib/extor-static-cookie # extor-static-cookie/gen-auth-cookie > /var/lib/extor-static-cookie/static_extended_orport_auth_cookie
Install a first instance of tor and configure it as a bridge: # apt install tor # tor-instance-create o1 /etc/tor/instances/o1/torrc: BridgeRelay 1 PublishServerDescriptor 0 AssumeReachable 1 SocksPort 0 ORPort 127.0.0.1:auto ExtORPort auto ServerTransportPlugin extor_static_cookie exec /usr/local/bin/extor-static-cookie /var/lib/extor-static-cookie/static_extended_orport_auth_cookie ServerTransportListenAddr extor_static_cookie 127.0.0.1:10001 Notice we set `ExtORPort auto` (this is tor's own ExtORPort), and also pass `127.0.0.1:10001` to extor-static-cookie, which is the ExtORPort that the external server pluggable transport will talk to. Start the first instance, which will generate keys: systemctl start tor@o1
Install a second instance of tor and configure it as a bridge (with a different ServerTransportListenAddr port): # tor-instance-create o2 /etc/tor/instances/o2/torrc: BridgeRelay 1 PublishServerDescriptor 0 AssumeReachable 1 SocksPort 0 ORPort 127.0.0.1:auto ExtORPort auto ServerTransportPlugin extor_static_cookie exec /usr/local/bin/extor-static-cookie /var/lib/extor-static-cookie/static_extended_orport_auth_cookie ServerTransportListenAddr extor_static_cookie 127.0.0.1:10002 But before starting the second instance the first time, copy keys from the first instance: # cp -r /var/lib/tor-instances/o1/keys /var/lib/tor-instances/o2/ # chown -R _tor-o2:_tor-o2 /var/lib/tor-instances/o2/keys/ # systemctl start tor@o2
The two instances should have the same fingerprint: # cat /var/lib/tor-instances/*/fingerprint Unnamed 4808CD98E4C1D4F282DA741A860A44D755701F2F Unnamed 4808CD98E4C1D4F282DA741A860A44D755701F2F
Install haproxy and configure it to forward to the two instances of extor-static-cookie (which will then forward to the ExtORPort of their respective tor instances): # apt install haproxy /etc/haproxy/haproxy.cfg: frontend tor mode tcp bind 127.0.0.1:10000 default_backend tor-o backend tor-o mode tcp server o1 127.0.0.1:10001 server o2 127.0.0.1:10002 Restart haproxy with the new configuration: # systemctl restart haproxy
Instead of ptadapter, I found it more convenient to start the external server pluggable transport with a shell script that sets up the necessary variables: extor.sh: #!/bin/sh
# Usage: extor.sh 127.0.0.1:10000 /var/lib/extor-static-cookie/static_extended_orport_auth_cookie /usr/bin/obfs4proxy
set -e
EXTOR_ADDR="${1:?missing ExtORPort address}" EXTOR_COOKIE_FILE="${2:?missing ExtORPort auth cookie file}" shift 2
BINDADDR='[::]:443' TRANSPORT=obfs4
TOR_PT_MANAGED_TRANSPORT_VER=1 \ TOR_PT_SERVER_TRANSPORTS="$TRANSPORT" \ TOR_PT_SERVER_BINDADDR="$TRANSPORT"-"$BINDADDR" \ TOR_PT_EXTENDED_SERVER_PORT="$EXTOR_ADDR" \ TOR_PT_AUTH_COOKIE_FILE="$EXTOR_COOKIE_FILE" \ TOR_PT_STATE_LOCATION=pt_state \ TOR_PT_EXIT_ON_STDIN_CLOSE=1 \ exec "$@"
Then I run the shell script, giving the address of the haproxy frontend, the path to the shared authentication cookie file, and a command to run: # ./extor.sh 127.0.0.1:10000 /var/lib/extor-static-cookie/static_extended_orport_auth_cookie /usr/bin/obfs4proxy
On the client, make a torrc file with the information from pt_state/obfs4_bridgeline.txt: UseBridges 1 SocksPort auto Bridge obfs4 172.105.3.197:443 4808CD98E4C1D4F282DA741A860A44D755701F2F cert=1SCzqyYyPh/SiXTJa9nLFxMyjWQITVCKeICME+SwxgNcTTSUQ7+vM/ghofU7oaalIRBILg iat-mode=0 ClientTransportPlugin obfs4 exec /usr/bin/obfs4proxy DataDir datadir Then run tor with the torrc: $ tor -f torrc