On 2020-01-31 9:24 p.m., David Fifield wrote:
https://gitweb.torproject.org/user/dcf/snowflake.git/log/?h=turbotunnel&...
These are the elements of a Turbo Tunnel implementation for Snowflake. Turbo Tunnel is a name for overlaying an abstract, virtual session on top of concrete, physical network connections, such that the virtual session is not tied to any particular network connection. In Snowflake, it solves the problem of migrating a session across multiple WebRTC connections as temporary proxies come and go. This post is a walkthrough of the code changes and my design decisions.
This is good stuff, thanks for working on this!
== How to try it ==
Download the branch and build it: git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git checkout -b turbotunnel --track dcf/turbotunnel for d in client server broker proxy-go; do (cd $d && go build); done Run the broker (not changed in this branch): broker/broker --disable-tls --addr 127.0.0.1:8000 Run a proxy (not changed in this branch): proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/ Run the server: tor -f torrc.server # contents of torrc.server: DataDirectory datadir-server SocksPort 0 ORPort 9001 ExtORPort auto BridgeRelay 1 AssumeReachable 1 PublishServerDescriptor 0 ServerTransportListenAddr snowflake 0.0.0.0:8080 ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log Run the client: tor -f torrc.client # contents of torrc.client: DataDirectory datadir-client UseBridges 1 SocksPort 9250 ClientTransportPlugin snowflake exec client/client --url http://127.0.0.1:8000/ --ice stun:stun.l.google.com:19302 --log snowflake-client.log Bridge snowflake 0.0.3.0:1
I've made some updates to snowbox to easily run all of the pieces needed for Turbo Tunnel :
https://github.com/cohosh/snowbox
All you need to do is use the configuration to point the docker container towards a snowflake repo with dcf's turbotunnel branch checked out in it and run `$ build` in the docker container to compile it, and run the components. Typing `$ run-client` will start a client process that bootstraps through snowflake. The log files will all be in the home directory.
== Introduction to code changes ==
Start by looking at the server changes: https://gitweb.torproject.org/user/dcf/snowflake.git/diff/server/server.go?h...
The first thing to notice is a kind of "inversion" of control flow. Formerly, the ServeHTTP function accepted WebSocket connections and connected each one with the ORPort. There was no virtual session: each WebSocket connection corresponded to exactly one client session. Now, the main function, separately from starting the web server, starts a virtual listener (kcp.ServeConn) that calls into a chain of acceptSessions→acceptStreams→handleStream functions that ultimately connects a virtual stream with the ORPort. But this virtual listener doesn't actually open a network port, so what drives it? That's now the sole responsibility of the ServeHTTP function. It still accepts WebSocket connections, but it doesn't connect them directly to the ORPort—instead, it pulls out discrete packets (encoded into the stream using length prefixes) and feeds those packets to the virtual listener. The glue that links the virtual listener and the ServeHTTP function is QueuePacketConn, an abstract interface that allows the virtual listener to send and receive packets without knowing exactly how those I/O operations are implemented. (In this case, they're implemented by encoding packets into WebSocket streams.)
I like that this implementation is very tidy, in that it uses different layers of abstraction (like KCP) to do a lot of the work required in deciding which client each packet corresponds to. It took me a while to wrap my head around the fact that the QueuePacketConn is a single abstract connection that handles *all* incoming and outgoing traffic for *all* clients. The result is a relatively clean interface with turbotunnel in the actual server code while there's a lot going on behind the scenes.
The behaviour I am still unsure about is which websocket connection the data from the server (data going from the server to the client) is written to. From what I can tell, each new websocket connection from a proxy will pull from the OutgoingQueue that corresponds to the clientID of the connection until the connection times out. This means that, since the server is not in charge of redial, there are potentially multiple connections pulling from this queue. If a connection is dropped and a new one redialed at the client, the server may write data out to the dropped connection instead of the newer redialed connection. Presumably KCP will take care of retransmitting the dropped packet, but I'm curious about the latency cost here. It's also a bit different from an earlier proposal to do connection migration similar to Mosh: https://github.com/net4people/bbs/issues/14
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... QueuePacketConn and ClientMap are imported pretty much unchanged from the meek implementation (https://github.com/net4people/bbs/issues/21). Together these data structures manage queues of packets and allow you to send and receive them using custom code. In meek it was done over raw HTTP bodies; here it's done over WebSocket. These two interfaces are candidates for an eventual reusable Turbo Tunnel library.
+1 to this, I like the idea of QueuePacketConn and ClientMap as a part of a reusable library.
== Limitations ==
I'm still using the same old logic for detecting a dead proxy, 30 seconds without receiving any data. This is suboptimal for many reasons (https://bugs.torproject.org/25429), one of which is that when your proxy dies, you have to wait at least 30 seconds until the connection becomes useful again. That's why I had to use "--speed-time 60" in the curl command above; curl has a default idle timeout of 30 seconds, which would cause it to give up just as a new proxy was becoming available.
I think we can ultimately do a lot better, and make better use of the available proxy capacity. I'm thinking of "striping" packets across multiple snowflake proxies simultaneously. This could be done in a round-robin fashion or in a more sophisticated way (weighted by measured per-proxy bandwidth, for example). That way, when a proxy dies, any packets sent to it would be detected as lost (unacknowledged) by the KCP layer, and retransmitted over a different proxy, much quicker than the 30-second timeout. The way to do this would be to replace RedialPacketConn—which uses once connection at a time—with a MultiplexingPacketConn, which manages a set of currently live connections and uses all of them. I don't think it would require any changes on the server.
This has been the subject of discussion on https://trac.torproject.org/projects/tor/ticket/29206 as well. In fact, one of the biggest usability challenges with Snowflake right now is that if a user happens to get a bad snowflake the first time up, Tor Browser's SOCKS connection to the PT will timeout before a circuit has the chance to bootstrap a Tor connection through a new proxy (mostly because it takes 30s to realize the snowflake is bad). If this happens, the client is told that Snowflake is failing and is asked to reconfigure their network settings. It is indistinguishable through the user interface from the case in which Snowflake isn't working at all.
I agree that multiplexing is the way to go here. It is very neat that the way you've implemented this doesn't require changes on the server side to do it and mostly consists of swapping out RedialPacketConn. I'm still curious about what I said above w.r.t. which proxy the server ends up using for returning packets. There's some potential for some server-side optimizations here if we want to go the route of using a more sophisticated method of choosing proxies in the other direction.
But the situation in the turbotunnel branch is better than the status quo, even without multiplexing, for two reasons. First, the connection actually *can* recover after 30 seconds. Second, the smux layer sends keepalives, which means that you won't discard a proxy merely because you're temporarily idle, but only when it really stops working.
Yes! This is really great work. We should talk at the next anti-censorship meeting perhaps on the steps we should take to get this merged and deployed.
Minor note: - There is some old buffering code at the client side that could be rolled into the new RedialPacketConn:
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.... https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc....