https://gitweb.torproject.org/user/dcf/snowflake.git/log/?h=turbotunnel&...
These are the elements of a Turbo Tunnel implementation for Snowflake. Turbo Tunnel is a name for overlaying an abstract, virtual session on top of concrete, physical network connections, such that the virtual session is not tied to any particular network connection. In Snowflake, it solves the problem of migrating a session across multiple WebRTC connections as temporary proxies come and go. This post is a walkthrough of the code changes and my design decisions.
== How to try it ==
Download the branch and build it: git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git checkout -b turbotunnel --track dcf/turbotunnel for d in client server broker proxy-go; do (cd $d && go build); done Run the broker (not changed in this branch): broker/broker --disable-tls --addr 127.0.0.1:8000 Run a proxy (not changed in this branch): proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/ Run the server: tor -f torrc.server # contents of torrc.server: DataDirectory datadir-server SocksPort 0 ORPort 9001 ExtORPort auto BridgeRelay 1 AssumeReachable 1 PublishServerDescriptor 0 ServerTransportListenAddr snowflake 0.0.0.0:8080 ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log Run the client: tor -f torrc.client # contents of torrc.client: DataDirectory datadir-client UseBridges 1 SocksPort 9250 ClientTransportPlugin snowflake exec client/client --url http://127.0.0.1:8000/ --ice stun:stun.l.google.com:19302 --log snowflake-client.log Bridge snowflake 0.0.3.0:1
Start downloading a big file through the tor SocksPort. You will be able to see activity in snowflake-client.log and in the output of proxy-go. curl -x socks5://127.0.0.1:9250/ --location --speed-time 60 https://cdimage.debian.org/mirror/cdimage/archive/10.1.0/amd64/iso-cd/debian... > /dev/null
Now kill proxy-go and restart it. Wait 30 seconds for snowflake-client to notice the proxy has disappeared. Then snowflake-client.log will say redialing on same connection and the download will resume. It's not curl restarting the download on a new connection—from the perspective of curl (and tor) it's all one long proxy connection, with a 30-second lull in the middle. Only snowflake-client knows that there were two WebRTC connections involved.
== Introduction to code changes ==
Start by looking at the server changes: https://gitweb.torproject.org/user/dcf/snowflake.git/diff/server/server.go?h...
The first thing to notice is a kind of "inversion" of control flow. Formerly, the ServeHTTP function accepted WebSocket connections and connected each one with the ORPort. There was no virtual session: each WebSocket connection corresponded to exactly one client session. Now, the main function, separately from starting the web server, starts a virtual listener (kcp.ServeConn) that calls into a chain of acceptSessions→acceptStreams→handleStream functions that ultimately connects a virtual stream with the ORPort. But this virtual listener doesn't actually open a network port, so what drives it? That's now the sole responsibility of the ServeHTTP function. It still accepts WebSocket connections, but it doesn't connect them directly to the ORPort—instead, it pulls out discrete packets (encoded into the stream using length prefixes) and feeds those packets to the virtual listener. The glue that links the virtual listener and the ServeHTTP function is QueuePacketConn, an abstract interface that allows the virtual listener to send and receive packets without knowing exactly how those I/O operations are implemented. (In this case, they're implemented by encoding packets into WebSocket streams.)
The new control flow boils down to a simple, traditional listen/accept loop, except that the listener doesn't interact with the network directly, but only through the QueuePacketConn interface. The WebSocket part of the program now only acts as a network interface that performs I/O functions on behalf of the QueuePacketConn. In effect, we've moved everything up one layer of abstraction: where formerly we had an HTTP server using the operating system as a network interface, we now have a virtual listener using the HTTP server as a network interface (which in turn ultimately uses the operating system as the *real* network interface).
Now look at the client changes: https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel&a...
The Handler function formerly grabbed exactly one snowflake proxy (snowflakes.Pop()) and used its WebRTC connection until it died, at which point it would close the SOCKS connection and terminate the whole Tor session. Now, the function creates a RedialPacketConn, an abstract interface that grabs a snowflake proxy, uses it for as long as it lasts, then grabs another. Each of the temporary snowflake proxies is wrapped in an EncapsulationPacketConn to convert it from a stream-oriented interface to a packet-oriented interface. EncapsulationPacketConn uses the same length-prefixed protocol that the server expects. We then create a virtual client connection (kcp.NewConn2), configured to use the RedialPacketConn as its network interface, and open a new virtual stream. (This sequence of calls kcp.NewConn2→sess.OpenStream corresponds to acceptSessions→acceptStreams on the server.) We then connect (copyLoop) the SOCKS connection and the virtual stream. The virtual stream never touches the network directly—it interacts indirectly through RedialPacketConn and EncapsulationPacketConn, which make use of whatever snowflake proxy WebRTC connection happens to exist at the time.
You'll notice that before anything else, the client sends a 64-bit ClientID. This is a random number that identifies a particular client session, made necessary because the virtual session is not tied to an IP 4-tuple or any other network identifier. The ClientID remains the same across all redials in one call to the Handler function. The server parses the ClientID out of the beginning of a WebSocket stream. The ClientID is how the server knows if it should open a new ORPort connection or append to an existing one, and which temporary WebSocket connections should receive packets that are addressed to a particular client.
There's a lot of new support code in the common/encapsulation and common/turbotunnel directories, mostly reused from my previous work in integrating Turbo Tunnel into pluggable transports.
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/encapsulati... The encapsulation package provides a way of encoding a sequence of packets into a stream. It's essentially just prefixing each packet with its length, but it takes care to permit traffic shaping and padding to the byte level. (The Snowflake turbotunnel branch doesn't take advantage of the traffic-shaping and padding features.)
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... QueuePacketConn and ClientMap are imported pretty much unchanged from the meek implementation (https://github.com/net4people/bbs/issues/21). Together these data structures manage queues of packets and allow you to send and receive them using custom code. In meek it was done over raw HTTP bodies; here it's done over WebSocket. These two interfaces are candidates for an eventual reusable Turbo Tunnel library.
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... RedialPacketConn is adapted from clientPacketConn in the obfs4proxy implementation (https://github.com/net4people/bbs/issues/14#issuecomment-544747519). It's the part that uses an underlying connection for as long as it exists, then switches to a new one. Since the obfs4proxy implementation, I've decided that it's better to have this type use the packet-oriented net.PacketConn as the underlying type, not the stream-oriented net.Conn. That way, RedialPacketConn doesn't have to know details of how packet encapsulation happens, whether by EncapsulationPacketConn or some other way.
== Backward compatibility ==
The branch as of commit 07495371d67f914d2c828bbd3d7facc455996bd2 is not backward compatible with the mainline Snowflake code. That's because the server expects to find a ClientID and length-prefixed packets, and currently deployed clients don't work that way. However, I think it will be possible to make the server backward compatible. My plan is to reserve a distinguished static token (64-bit value) and have the client send that at the beginning of the stream, before its ClientID, to indicate that it uses Turbo Tunnel features. The token will be selected to be distinguishable from any protocol that non–Turbo Tunnel clients might use (i.e., Tor TLS). Then, the server's ServeHTTP function can choose one of two implementations, depending on whether it sees the magic token or not.
If I get backward compatibility working, then we can deploy a dual-mode bridge that is able to serve either type of client. Then I can try making a Tor Browser build, to make the Turbo Tunnel code more accessible for user testing.
One nice thing about all this is that it doesn't require any changes to proxies. They remain simple dumb pipes, so we don't have to coordinate a mass proxy upgrade.
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/server/server.go?h... The branch currently lacks client geoip lookup (ExtORPort USERADDR), because of the difficulty I have talked about before of providing an IP address for a virtual session that is not inherently tied to any single network connection or address. I have a plan for solving it, though; it requires a slight breaking of abstractions. In the server, after reading the ClientID, we can peek at the first 4 bytes of the first packet. These 4 bytes are the KCP conversation ID (https://github.com/xtaci/kcp-go/blob/v5.5.5/kcp.go#L120), a random number chosen by the client, serving roughly the same purpose in KCP as our ClientID. We store a temporary mapping from the conversation ID to the IP address of client making the WebSocket connection. kcp-go provides a GetConv function that we can call in handleStream, just as we're about to connect to the ORPort, to look up the client's IP address in the mapping. The possibility of doing this is one reason I decided to go with KCP for this implementation rather than QUIC as I did in the meek implementation: the quic-go package doesn't expose an accessor for the QUIC connection ID.
== Limitations ==
I'm still using the same old logic for detecting a dead proxy, 30 seconds without receiving any data. This is suboptimal for many reasons (https://bugs.torproject.org/25429), one of which is that when your proxy dies, you have to wait at least 30 seconds until the connection becomes useful again. That's why I had to use "--speed-time 60" in the curl command above; curl has a default idle timeout of 30 seconds, which would cause it to give up just as a new proxy was becoming available.
I think we can ultimately do a lot better, and make better use of the available proxy capacity. I'm thinking of "striping" packets across multiple snowflake proxies simultaneously. This could be done in a round-robin fashion or in a more sophisticated way (weighted by measured per-proxy bandwidth, for example). That way, when a proxy dies, any packets sent to it would be detected as lost (unacknowledged) by the KCP layer, and retransmitted over a different proxy, much quicker than the 30-second timeout. The way to do this would be to replace RedialPacketConn—which uses once connection at a time—with a MultiplexingPacketConn, which manages a set of currently live connections and uses all of them. I don't think it would require any changes on the server.
But the situation in the turbotunnel branch is better than the status quo, even without multiplexing, for two reasons. First, the connection actually *can* recover after 30 seconds. Second, the smux layer sends keepalives, which means that you won't discard a proxy merely because you're temporarily idle, but only when it really stops working.
== Notes ==
https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel&a... I added go.mod and go.sum files to the repo. I did this because smux (https://github.com/xtaci/smux) has a v2 protocol that is incompatible with the v1 protocol, and I don't know how to enforce the use of v2 in the build other than by activating Go modules and specifying a version in go.mod.
On Fri, Jan 31, 2020 at 07:24:48PM -0700, David Fifield wrote:
== Backward compatibility ==
The branch as of commit 07495371d67f914d2c828bbd3d7facc455996bd2 is not backward compatible with the mainline Snowflake code. That's because the server expects to find a ClientID and length-prefixed packets, and currently deployed clients don't work that way. However, I think it will be possible to make the server backward compatible. My plan is to reserve a distinguished static token (64-bit value) and have the client send that at the beginning of the stream, before its ClientID, to indicate that it uses Turbo Tunnel features. The token will be selected to be distinguishable from any protocol that non–Turbo Tunnel clients might use (i.e., Tor TLS). Then, the server's ServeHTTP function can choose one of two implementations, depending on whether it sees the magic token or not.
https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel&a...
Here's a commit that restores compatibility with current Snowflake clients. It works the way I described: the client sends a magic token at the beginning, one that non–Turbo Tunnel clients will never send. The server looks for the token, and depending on whether the token is present or not, chooses whether to handle each WebSocket connection in Turbo Tunnel mode or the old one-session-per-WebSocket mode. The magic token is 1293605d278175f5, which I got from Python os.urandom(8).
The addition of a magic token prefix very slightly constrains traffic shaping. The client is still free to shape its traffic however it likes: it doesn't have to send the whole 8-byte token at once; and it can put any amount of padding after the token, because once the server sees the token, it will know how to interpret the padding scheme that follows. The only constraint is on the server, which cannot start using padding until it receives all 8 bytes of the token and is sure that it is talking to a Tubro Tunnel–aware client. Be aware that I'm not taking advantage of any of the traffic shaping features made possible by the packet encapsulation and padding layer, so the fact that the client sends 8 bytes (the token) then another 8 bytes (the ClientID) will be visible in packet sizes.
Here is how to test it out with both kinds of clients. git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git checkout -b turbotunnel --track dcf/turbotunnel for d in client server broker proxy-go; do (cd $d && go build); done Run the broker: broker/broker --disable-tls --addr 127.0.0.1:8000 Run a proxy: proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/ Run the server: tor -f torrc.server # contents of torrc.server: DataDirectory datadir-server SocksPort 0 ORPort 9001 ExtORPort auto BridgeRelay 1 AssumeReachable 1 PublishServerDescriptor 0 ServerTransportListenAddr snowflake 0.0.0.0:8080 ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log Run a turbotunnel client: tor -f torrc.client # contents of torrc.client: DataDirectory datadir-client UseBridges 1 SocksPort 9250 ClientTransportPlugin snowflake exec client/client --url http://127.0.0.1:8000/ --ice stun:stun.l.google.com:19302 --log snowflake-client.log Bridge snowflake 0.0.3.0:1 Kill the turbotunnel client, and switch back to the master branch, keeping the broker, proxy, and server running. Rebuild the client and run it again. (cd client && go build) tor -f torrc.client
On Mon, Feb 03, 2020 at 09:22:16PM -0700, David Fifield wrote:
My plan is to reserve a distinguished static token (64-bit value) and have the client send that at the beginning of the stream, before its ClientID, to indicate that it uses Turbo Tunnel features.
[...] Here's a commit that restores compatibility with current Snowflake clients. It works the way I described: the client sends a magic token at the beginning, one that non???Turbo Tunnel clients will never send. [...] The addition of a magic token prefix very slightly constrains traffic shaping.
Hi David!
Awesome news about the turbotunnel integration. I've been pondering the backward compatibility thing ever since your first mail. Here are two hopefully useful thoughts. Let me know if they are misunderstanding things in some way.
(A) An extra 8 bytes for every new or resumed flow, forever, is kind of sad. But it doesn't have to be forever: in the future, once the old Snowflakes that don't send that static token are long gone, we can stop requiring the token. (That is, if clients send it, that's fine, we can recognize and strip it. And if they don't send it, that's fine too.)
(B) Another approach to backward compatibility might be to leave the old Snowflake bridge up, not knowing what a turbotunnel is, and to put up a new one nearby or somewhere else that expects turbotunnel connections. Then after a few releases of Tor Browser, we could shut down the old one and move on.
Approach 'B' would have been best to suggest in between your first mail and your second one. Oops. :) Better late than never maybe.
Thanks! --Roger
On Tue, Feb 04, 2020 at 01:17:33AM -0500, Roger Dingledine wrote:
Awesome news about the turbotunnel integration. I've been pondering the backward compatibility thing ever since your first mail. Here are two hopefully useful thoughts. Let me know if they are misunderstanding things in some way.
FYI, we've discussed these and other approaches at https://bugs.torproject.org/30704 and https://bugs.torproject.org/29206.
(A) An extra 8 bytes for every new or resumed flow, forever, is kind of sad. But it doesn't have to be forever: in the future, once the old Snowflakes that don't send that static token are long gone, we can stop requiring the token. (That is, if clients send it, that's fine, we can recognize and strip it. And if they don't send it, that's fine too.)
Personally I don't feel 8 bytes is anything to cry over. Let it be 4 bytes, or 1 byte, or 16 bytes. Compare it to the overhead of a TLS connection or a WebSocket negotiation. Compare it to the overhead of Tor's cell padding or the 24-byte KCP header. Especially considering that it's amortized over the life of a proxy. 8 bytes every 100 KB is hardly worth considering, in my opinion.
(B) Another approach to backward compatibility might be to leave the old Snowflake bridge up, not knowing what a turbotunnel is, and to put up a new one nearby or somewhere else that expects turbotunnel connections. Then after a few releases of Tor Browser, we could shut down the old one and move on.
This is a reasonable idea, but certain characteristics of the current Snowflake deployment make it more difficult. The key element is that it requires changes in proxy code and proxies to upgrade. The reason for this is that you'll need some proxies to serve the old bridge and some to serve the new. So you could partition your pool of proxies—each one choosing which bridge it will use randomly at startup, for example. You would then need to run two brokers too, so that Turbo Tunnel clients get matched with Turbo Tunnel proxies, and likewise with the non–Turbo Tunnel clients and proxies. Or instead of running two brokers, you could have the client, indicate in its rendezvous message, which protocol it intends to use, and then the broker, when matching the client up with a proxy, would do something like https://bugs.torproject.org/25598 and simply instruct the proxy as to what bridge it should use. But beside the fact that none of that is implemented, the protocol change is no longer an end-to-end matter between clients and the bridge; it's something that all pieces of the pipeline have to be aware of: client, broker, proxy, and bridge. And the client now will have to spend some number of bytes in its rendezvous message to indicate what protocol it's using: the overhead moves out of the client→proxy flow and into the client→broker flow. And all other things being equal, we prefer to send bytes on the client→proxy flow rather than the client→broker flow, because the domain-fronted CDN link is more expensive (in $$) than the WebRTC/WebSocket link. And on top of everything, you still end up with lingering code complexity; for example what happens if proxy joins that doesn't know how to obey the broker's instructions about what bridge to connect to? You need some logic in the client to detect the situation and fall back to the legacy protocol, or in the broker to detect and deny service to such proxies (with its own indeterminate sunset period?), or something.
Running a second copy of the bridge wouldn't require a second IP address or even a second port. Because it's an HTTP-based protocol, it could be a second URL path on the same server; say /tt for the Turbo Tunnel clients and / for the old clients, with a reverse proxy dispatching to one of two local servers. It's a feasible design. For me, when I weigh the cost of 25 lines of compatibility code living in the server itself, versus the added risk of now having to restart two things instead of one whenever doing a bridge upgrade, I tend to come down on the side of a small amount of compatibility code.
On 2020-02-04 1:24 p.m., David Fifield wrote:
On Tue, Feb 04, 2020 at 01:17:33AM -0500, Roger Dingledine wrote:
Awesome news about the turbotunnel integration. I've been pondering the backward compatibility thing ever since your first mail. Here are two hopefully useful thoughts. Let me know if they are misunderstanding things in some way.
FYI, we've discussed these and other approaches at https://bugs.torproject.org/30704 and https://bugs.torproject.org/29206.
(A) An extra 8 bytes for every new or resumed flow, forever, is kind of sad. But it doesn't have to be forever: in the future, once the old Snowflakes that don't send that static token are long gone, we can stop requiring the token. (That is, if clients send it, that's fine, we can recognize and strip it. And if they don't send it, that's fine too.)
Personally I don't feel 8 bytes is anything to cry over. Let it be 4 bytes, or 1 byte, or 16 bytes. Compare it to the overhead of a TLS connection or a WebSocket negotiation. Compare it to the overhead of Tor's cell padding or the 24-byte KCP header. Especially considering that it's amortized over the life of a proxy. 8 bytes every 100 KB is hardly worth considering, in my opinion.
(B) Another approach to backward compatibility might be to leave the old Snowflake bridge up, not knowing what a turbotunnel is, and to put up a new one nearby or somewhere else that expects turbotunnel connections. Then after a few releases of Tor Browser, we could shut down the old one and move on.
This is a reasonable idea, but certain characteristics of the current Snowflake deployment make it more difficult. The key element is that it requires changes in proxy code and proxies to upgrade. The reason for this is that you'll need some proxies to serve the old bridge and some to serve the new. So you could partition your pool of proxies—each one choosing which bridge it will use randomly at startup, for example. You would then need to run two brokers too, so that Turbo Tunnel clients get matched with Turbo Tunnel proxies, and likewise with the non–Turbo Tunnel clients and proxies. Or instead of running two brokers, you could have the client, indicate in its rendezvous message, which protocol it intends to use, and then the broker, when matching the client up with a proxy, would do something like https://bugs.torproject.org/25598 and simply instruct the proxy as to what bridge it should use. But beside the fact that none of that is implemented, the protocol change is no longer an end-to-end matter between clients and the bridge; it's something that all pieces of the pipeline have to be aware of: client, broker, proxy, and bridge. And the client now will have to spend some number of bytes in its rendezvous message to indicate what protocol it's using: the overhead moves out of the client→proxy flow and into the client→broker flow. And all other things being equal, we prefer to send bytes on the client→proxy flow rather than the client→broker flow, because the domain-fronted CDN link is more expensive (in $$) than the WebRTC/WebSocket link. And on top of everything, you still end up with lingering code complexity; for example what happens if proxy joins that doesn't know how to obey the broker's instructions about what bridge to connect to? You need some logic in the client to detect the situation and fall back to the legacy protocol, or in the broker to detect and deny service to such proxies (with its own indeterminate sunset period?), or something.
Running a second copy of the bridge wouldn't require a second IP address or even a second port. Because it's an HTTP-based protocol, it could be a second URL path on the same server; say /tt for the Turbo Tunnel clients and / for the old clients, with a reverse proxy dispatching to one of two local servers. It's a feasible design. For me, when I weigh the cost of 25 lines of compatibility code living in the server itself, versus the added risk of now having to restart two things instead of one whenever doing a bridge upgrade, I tend to come down on the side of a small amount of compatibility code.
The compatibility code at the server seems like a good idea to me, but I want to add another option to consider even though I agree that the current way forward is the least complex for now.
We have in the unmapped future a desire to prepare snowflake for another bridge (https://trac.torproject.org/projects/tor/ticket/28651) and there is some desire for client choice about which bridge is used (https://trac.torproject.org/projects/tor/ticket/31661). One option is for the client to issue something like a SOCKS connect request to a proxy when it first makes a connection and for the proxy to read the first X bytes and use that information to open a connection to a Snowflake bridge of the user's request. They could default to a known bridge if the old version of the protocol is used.
As David mentioned, this has the disadvantage of requiring a proxy upgrade. However, we've done a non-backwards compatible proxy upgrade before with https://trac.torproject.org/projects/tor/ticket/29207 and it worked pretty well. The webextension proxies updated quickly and we run (half?) of the standalone proxies now so we can restart them ourselves. The broker will send back error messages that will be logged by non-updated standalone proxies if we force the update at the broker by bumping the required broker-proxy protocol version.
Anyway, not saying this is what we should do, just adding to the pile of ideas while we're at it.
On Wed, Feb 05, 2020 at 05:03:33PM -0500, Cecylia Bocovich wrote:
The compatibility code at the server seems like a good idea to me, but I want to add another option to consider even though I agree that the current way forward is the least complex for now.
We have in the unmapped future a desire to prepare snowflake for another bridge (https://trac.torproject.org/projects/tor/ticket/28651) and there is some desire for client choice about which bridge is used (https://trac.torproject.org/projects/tor/ticket/31661). One option is for the client to issue something like a SOCKS connect request to a proxy when it first makes a connection and for the proxy to read the first X bytes and use that information to open a connection to a Snowflake bridge of the user's request. They could default to a known bridge if the old version of the protocol is used.
As David mentioned, this has the disadvantage of requiring a proxy upgrade. However, we've done a non-backwards compatible proxy upgrade before with https://trac.torproject.org/projects/tor/ticket/29207 and it worked pretty well. The webextension proxies updated quickly and we run (half?) of the standalone proxies now so we can restart them ourselves. The broker will send back error messages that will be logged by non-updated standalone proxies if we force the update at the broker by bumping the required broker-proxy protocol version.
Yeah I'm not opposed to coordinating proxy upgrades in all cases, I just don't want to do it if we don't have to, especially if it's going to require its own non-trivial design.
On 2020-01-31 9:24 p.m., David Fifield wrote:
https://gitweb.torproject.org/user/dcf/snowflake.git/log/?h=turbotunnel&...
These are the elements of a Turbo Tunnel implementation for Snowflake. Turbo Tunnel is a name for overlaying an abstract, virtual session on top of concrete, physical network connections, such that the virtual session is not tied to any particular network connection. In Snowflake, it solves the problem of migrating a session across multiple WebRTC connections as temporary proxies come and go. This post is a walkthrough of the code changes and my design decisions.
This is good stuff, thanks for working on this!
== How to try it ==
Download the branch and build it: git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git checkout -b turbotunnel --track dcf/turbotunnel for d in client server broker proxy-go; do (cd $d && go build); done Run the broker (not changed in this branch): broker/broker --disable-tls --addr 127.0.0.1:8000 Run a proxy (not changed in this branch): proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/ Run the server: tor -f torrc.server # contents of torrc.server: DataDirectory datadir-server SocksPort 0 ORPort 9001 ExtORPort auto BridgeRelay 1 AssumeReachable 1 PublishServerDescriptor 0 ServerTransportListenAddr snowflake 0.0.0.0:8080 ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log Run the client: tor -f torrc.client # contents of torrc.client: DataDirectory datadir-client UseBridges 1 SocksPort 9250 ClientTransportPlugin snowflake exec client/client --url http://127.0.0.1:8000/ --ice stun:stun.l.google.com:19302 --log snowflake-client.log Bridge snowflake 0.0.3.0:1
I've made some updates to snowbox to easily run all of the pieces needed for Turbo Tunnel :
https://github.com/cohosh/snowbox
All you need to do is use the configuration to point the docker container towards a snowflake repo with dcf's turbotunnel branch checked out in it and run `$ build` in the docker container to compile it, and run the components. Typing `$ run-client` will start a client process that bootstraps through snowflake. The log files will all be in the home directory.
== Introduction to code changes ==
Start by looking at the server changes: https://gitweb.torproject.org/user/dcf/snowflake.git/diff/server/server.go?h...
The first thing to notice is a kind of "inversion" of control flow. Formerly, the ServeHTTP function accepted WebSocket connections and connected each one with the ORPort. There was no virtual session: each WebSocket connection corresponded to exactly one client session. Now, the main function, separately from starting the web server, starts a virtual listener (kcp.ServeConn) that calls into a chain of acceptSessions→acceptStreams→handleStream functions that ultimately connects a virtual stream with the ORPort. But this virtual listener doesn't actually open a network port, so what drives it? That's now the sole responsibility of the ServeHTTP function. It still accepts WebSocket connections, but it doesn't connect them directly to the ORPort—instead, it pulls out discrete packets (encoded into the stream using length prefixes) and feeds those packets to the virtual listener. The glue that links the virtual listener and the ServeHTTP function is QueuePacketConn, an abstract interface that allows the virtual listener to send and receive packets without knowing exactly how those I/O operations are implemented. (In this case, they're implemented by encoding packets into WebSocket streams.)
I like that this implementation is very tidy, in that it uses different layers of abstraction (like KCP) to do a lot of the work required in deciding which client each packet corresponds to. It took me a while to wrap my head around the fact that the QueuePacketConn is a single abstract connection that handles *all* incoming and outgoing traffic for *all* clients. The result is a relatively clean interface with turbotunnel in the actual server code while there's a lot going on behind the scenes.
The behaviour I am still unsure about is which websocket connection the data from the server (data going from the server to the client) is written to. From what I can tell, each new websocket connection from a proxy will pull from the OutgoingQueue that corresponds to the clientID of the connection until the connection times out. This means that, since the server is not in charge of redial, there are potentially multiple connections pulling from this queue. If a connection is dropped and a new one redialed at the client, the server may write data out to the dropped connection instead of the newer redialed connection. Presumably KCP will take care of retransmitting the dropped packet, but I'm curious about the latency cost here. It's also a bit different from an earlier proposal to do connection migration similar to Mosh: https://github.com/net4people/bbs/issues/14
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... https://gitweb.torproject.org/user/dcf/snowflake.git/tree/common/turbotunnel... QueuePacketConn and ClientMap are imported pretty much unchanged from the meek implementation (https://github.com/net4people/bbs/issues/21). Together these data structures manage queues of packets and allow you to send and receive them using custom code. In meek it was done over raw HTTP bodies; here it's done over WebSocket. These two interfaces are candidates for an eventual reusable Turbo Tunnel library.
+1 to this, I like the idea of QueuePacketConn and ClientMap as a part of a reusable library.
== Limitations ==
I'm still using the same old logic for detecting a dead proxy, 30 seconds without receiving any data. This is suboptimal for many reasons (https://bugs.torproject.org/25429), one of which is that when your proxy dies, you have to wait at least 30 seconds until the connection becomes useful again. That's why I had to use "--speed-time 60" in the curl command above; curl has a default idle timeout of 30 seconds, which would cause it to give up just as a new proxy was becoming available.
I think we can ultimately do a lot better, and make better use of the available proxy capacity. I'm thinking of "striping" packets across multiple snowflake proxies simultaneously. This could be done in a round-robin fashion or in a more sophisticated way (weighted by measured per-proxy bandwidth, for example). That way, when a proxy dies, any packets sent to it would be detected as lost (unacknowledged) by the KCP layer, and retransmitted over a different proxy, much quicker than the 30-second timeout. The way to do this would be to replace RedialPacketConn—which uses once connection at a time—with a MultiplexingPacketConn, which manages a set of currently live connections and uses all of them. I don't think it would require any changes on the server.
This has been the subject of discussion on https://trac.torproject.org/projects/tor/ticket/29206 as well. In fact, one of the biggest usability challenges with Snowflake right now is that if a user happens to get a bad snowflake the first time up, Tor Browser's SOCKS connection to the PT will timeout before a circuit has the chance to bootstrap a Tor connection through a new proxy (mostly because it takes 30s to realize the snowflake is bad). If this happens, the client is told that Snowflake is failing and is asked to reconfigure their network settings. It is indistinguishable through the user interface from the case in which Snowflake isn't working at all.
I agree that multiplexing is the way to go here. It is very neat that the way you've implemented this doesn't require changes on the server side to do it and mostly consists of swapping out RedialPacketConn. I'm still curious about what I said above w.r.t. which proxy the server ends up using for returning packets. There's some potential for some server-side optimizations here if we want to go the route of using a more sophisticated method of choosing proxies in the other direction.
But the situation in the turbotunnel branch is better than the status quo, even without multiplexing, for two reasons. First, the connection actually *can* recover after 30 seconds. Second, the smux layer sends keepalives, which means that you won't discard a proxy merely because you're temporarily idle, but only when it really stops working.
Yes! This is really great work. We should talk at the next anti-censorship meeting perhaps on the steps we should take to get this merged and deployed.
Minor note: - There is some old buffering code at the client side that could be rolled into the new RedialPacketConn:
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.... https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc....
On Wed, Feb 05, 2020 at 02:46:58PM -0500, Cecylia Bocovich wrote:
I've made some updates to snowbox to easily run all of the pieces needed for Turbo Tunnel :
Thanks for doing that.
It took me a while to wrap my head around the fact that the QueuePacketConn is a single abstract connection that handles *all* incoming and outgoing traffic for *all* clients. The result is a relatively clean interface with turbotunnel in the actual server code while there's a lot going on behind the scenes.
That's right. The analogy is with a UDP server, which is also a PacketConn. You do ListenUDP once, and it does duty for all clients. In fact, that's the typical way of using KCP, you do it over UDP. But the kcp-go interface allows replacing the UDPConn with any other type implementing the PacketConn interface.
The behaviour I am still unsure about is which websocket connection the data from the server (data going from the server to the client) is written to. From what I can tell, each new websocket connection from a proxy will pull from the OutgoingQueue that corresponds to the clientID of the connection until the connection times out. This means that, since the server is not in charge of redial, there are potentially multiple connections pulling from this queue. If a connection is dropped and a new one redialed at the client, the server may write data out to the dropped connection instead of the newer redialed connection. Presumably KCP will take care of retransmitting the dropped packet, but I'm curious about the latency cost here. It's also a bit different from an earlier proposal to do connection migration similar to Mosh: https://github.com/net4people/bbs/issues/14
This is a good observation. Indeed, not only could there be an about-to-die WebSocket connection pulling packets from OutgoingQueue, losing them, and forcing another WebSocket connection to retransmit them; there could be multiple simultaneous live WebSocket connections with the same ClientID, all pulling from OutgoingQueue at once (as would happen in a setup like https://bugs.torproject.org/25723). I haven't tried it yet, but my guess is that it will all just work out. When a WebSocket dies, it will lose some number of packets, but those will be retransmitted after a timeout that I believe KCP dynamically calculates. And if there are multiple, each will deliver packets as fast as it can support, and the client will take care of reordering them. It may cause some RTT estimation algorithm to freak out, but it's not really different than TCP segments taking different routes within a connection.
I did the connection migration more straightforwardly in the obfs4proxy implementation. The clientMap type in Snowflake is a descendant of connMap in obfs4proxy, and in fact they store almost the same data: In obfs4proxy: type connMapRecord struct { Addr net.Addr LastSeen time.Time Conn net.Conn } In Snowflake: type clientRecord struct { Addr net.Addr LastSeen time.Time SendQueue chan []byte } The difference is that Snowflake stores a (shared) SendQueue instead of single Conn. The advantage is that many Conns can pull from the shared queue at once. It's like a more general version of connection migration. The obfs4proxy implementation also would have supported a limited version of multiplexing across multiple connections simultaneously, but it would be dependent on the client consciously striping its sends over the connections, in order to coax the server into updating its Conn mapping constantly.
But the situation in the turbotunnel branch is better than the status quo, even without multiplexing, for two reasons. First, the connection actually *can* recover after 30 seconds. Second, the smux layer sends keepalives, which means that you won't discard a proxy merely because you're temporarily idle, but only when it really stops working.
Yes! This is really great work. We should talk at the next anti-censorship meeting perhaps on the steps we should take to get this merged and deployed.
I was going to propose deploying the backward-compatible turbotunnel server to the public bridge, then making some Tor Browser builds that our helpers on the issue tracker can try out.
Minor note:
- There is some old buffering code at the client side that could be
rolled into the new RedialPacketConn:
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc.... https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/webrtc....
Thanks, I would not have noticed this myself. I guess the original idea was that WebRTCPeer would be not a single WebRTC connection, but a wrapper for a sequence of WebRTC connections? That's why you can Connect it, Reset it, and Connect it again. I suppose we can eliminate all those features and make WebRTCPeer less stateful by having it represent just one WebRTC connection. RedialPacketConn takes care of sequencing multiple connections.
On Fri, Jan 31, 2020 at 07:24:48PM -0700, David Fifield wrote:
https://gitweb.torproject.org/user/dcf/snowflake.git/tree/server/server.go?h... The branch currently lacks client geoip lookup (ExtORPort USERADDR), because of the difficulty I have talked about before of providing an IP address for a virtual session that is not inherently tied to any single network connection or address. I have a plan for solving it, though; it requires a slight breaking of abstractions. In the server, after reading the ClientID, we can peek at the first 4 bytes of the first packet. These 4 bytes are the KCP conversation ID (https://github.com/xtaci/kcp-go/blob/v5.5.5/kcp.go#L120), a random number chosen by the client, serving roughly the same purpose in KCP as our ClientID. We store a temporary mapping from the conversation ID to the IP address of client making the WebSocket connection. kcp-go provides a GetConv function that we can call in handleStream, just as we're about to connect to the ORPort, to look up the client's IP address in the mapping. The possibility of doing this is one reason I decided to go with KCP for this implementation rather than QUIC as I did in the meek implementation: the quic-go package doesn't expose an accessor for the QUIC connection ID.
https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel&a... This commit adds USERADDR support for turbotunnel sessions. I found a nicer way to do it than what I proposed above, that doesn't require peeking into the packet structure. Instead of using the KCP conversation ID as the common element linking an IP address and a client session, we can use the ClientID (the artificial 8-byte value that we tack on at the beginning of every WebSocket connection). The ServeHTTP function has access to the ClientID because it's what parses it out, and once you have a session you can recover the ClientID by calling the RemoteAddr method—this is an effect of kcp-go using the address returned from its ReadFrom calls as the remote address of the session, and the fact that we use the ClientID for the address in those ReadFrom calls.
To summarize: * ServeHTTP has an IP address and a ClientID but not a session. * acceptStreams has a session and a ClientID but not an IP address. * We bridge the gap using a data structure that maps a Client ID to an IP address. ServeHTTP stores an entry in the structure, and acceptStreams looks it up.
I designed a simple data structure, clientIDMap, to serve as the lookup table. In spirit it is a map[ClientID]string: you Set(clientID, addr) to store a mapping, and Get(clientID) to retrieve it. It differs from a plain map in that it expires old entries when storing new ones: it's a fixed-size circular buffer. I designed it to be proof against memory leaks. With a plain map, it would be possible for a client to get as far as sending a ClientID (storing an entry in the map), but not ultimately establish a session (leaving the entry in the map forever). But it means that the buffer has to be large enough not to expire entries before they are needed (how big depends on the rate of new session creation and delay between when a WebSocket connection starts and when a session is established). But even if an entry is expired before it is used, the worst thing that happens is that one session gets attributed to ?? rather than a certain country. I added a log message for when this happens and if it turns out to be a problem, we can design a more complicated, dynamically sized data structure.
https://gitweb.torproject.org/user/dcf/tor-browser-build.git/commit/?h=snowf... https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn...
This is a build of Tor Browser using the Turbo Tunnel–enabled Snowflake. The public Snowflake bridge is not yet Turbo Tunnel–aware, so at this point you have to run your own local broker, proxy, and bridge.
== How to try it ==
Download the branch and build all but the client: git clone https://git.torproject.org/pluggable-transports/snowflake.git cd snowflake git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git fetch dcf git checkout 2cb47769e3c83a8998fcaccf0527701c7630f048 for d in broker proxy-go server; do (cd $d && go build); done Run the broker: broker/broker --disable-tls --addr 127.0.0.1:8000 Run a proxy: proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/ Run the bridge: tor -f torrc.server # contents of torrc.server: DataDirectory datadir-server SocksPort 0 ORPort 9001 ExtORPort auto BridgeRelay 1 AssumeReachable 1 PublishServerDescriptor 0 ServerTransportListenAddr snowflake 0.0.0.0:8080 ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log
Unpack the Tor Browser package and edit the file Browser/TorBrowser/Data/Tor/torrc-defaults. Configure it to use the local broker by changing the "ClientTransportPlugin snowflake" line: ClientTransportPlugin snowflake exec ./TorBrowser/Tor/PluggableTransports/snowflake-client -url http://127.0.0.1:8000/ -ice stun:stun.l.google.com:19302 -log snowflake-client.log
Run Tor Browser. Select "Configure" then "Tor is censored in my country" then "Provide a bridge I know". Enter snowflake 0.0.3.0:1 The purpose of this step is to provide a bridge line without a fingerprint. If you just select "snowflake" from the menu, it will use a bridge line with a fingerprint of the public bridge, which won't match the fingerprint of the bridge you are running locally.
Now click "Connect" and you can surf. I think you'll agree that the experience is markedly better than currently exists with Snowflake. Part of that is because you're using a fast and reliable local proxy, but also, you can leave it running all day and come back to it, and it will still be working.
== Dependency considerations ==
While implementing the tor-browser-build changes, I was dismayed at the number of new dependencies introduced by kcp-go. And the sad thing is, almost none of them are actually used in the build--they are only needed to support features of kcp-go that we don't actually use, like the built-in crypto and forward error correction.
- snowflake - pion-webrtc - smux (new) - kcp-go (new) - goerrors - gogmsm (new) - goreedsolomon (new) - gocpuid (new) - goxcrypto - x/crypto/blowfish (new) - x/crypto/cast5 (new) - x/crypto/pbkdf2 (new) - x/crypto/salsa20 (new) - x/crypto/tea (new) - x/crypto/twofish (new) - x/crypto/xtea (new) - goxnet - x/net/ipv4 (new) - x/net/ipv6 (new) - go-templexxx-xorsimd (new) - go-templexxx-cpu (new)
It's making me have second thoughts about using kcp-go instead of quic-go. quic-go is already present in the tor-browser-build tree, as a dependency of pion-webrtc, so it wouldn't introduce a new nest of dependencies.
Switching from kcp-go to quic-go isn't very difficult, because the APIs are similar. The main thing is to decide what to do with the obligatory TLS layer in quic-go. We could just disable certificate verification in the client (because the data we send is already encrypted+authenticated), or do public key pinning as I did in the meek implementation: https://github.com/net4people/bbs/issues/21#issue-540238698
So I think I will try another build with quic-go in place of kcp-go. Then I'm thinking that we should deploy the turbotunnel branch to the public bridge, and invite testing with another Tor Browser build.
On Tue, Feb 11, 2020 at 10:17:42PM -0700, David Fifield wrote:
It's making me have second thoughts about using kcp-go instead of quic-go. quic-go is already present in the tor-browser-build tree, as a dependency of pion-webrtc, so it wouldn't introduce a new nest of dependencies.
Switching from kcp-go to quic-go isn't very difficult, because the APIs are similar. The main thing is to decide what to do with the obligatory TLS layer in quic-go. We could just disable certificate verification in the client (because the data we send is already encrypted+authenticated), or do public key pinning as I did in the meek implementation: https://github.com/net4people/bbs/issues/21#issue-540238698
So I think I will try another build with quic-go in place of kcp-go. Then I'm thinking that we should deploy the turbotunnel branch to the public bridge, and invite testing with another Tor Browser build.
https://gitweb.torproject.org/user/dcf/tor-browser-build.git/commit/?h=snowf... https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn...
Here's a build that replaces kcp-go and smux with quic-go. Instead of adding 7 new projects to tor-browser-build, it adds none. Functionally, it's the same as the kcp-go version (but incompatible, of course).
https://gitweb.torproject.org/user/dcf/snowflake.git/commit/?h=turbotunnel&a... Here are the changes to convert from kcp-go to quic-go. As you can see, it's not that bad. The main thing, as I've said, is that QUIC has its own TLS handshake which means you need a certificate for the server. I decided not to have the client verify the server's certificate (here in the inner QUIC layer only, independent of any other TLS interactions the client may do), because we don't rely on having a secure channel here, and the code necessary to support it would distract from the Turbo Tunnel changes I want to emphasize. If you want to see what that code would look like: https://gitweb.torproject.org/pluggable-transports/meek.git/commit/?h=turbot...
== How to try it ==
Download the branch and build all but the client: git clone https://git.torproject.org/pluggable-transports/snowflake.git cd snowflake git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git fetch dcf git checkout 5eff0f3ba406eb7d8cd90711edab7de7d7bda4fa for d in broker proxy-go server; do (cd $d && go build); done Run the broker: broker/broker --disable-tls --addr 127.0.0.1:8000 Run a proxy: proxy-go/proxy-go --broker http://127.0.0.1:8000/ --relay ws://127.0.0.1:8080/ Run the bridge: tor -f torrc.server # contents of torrc.server: DataDirectory datadir-server SocksPort 0 ORPort 9001 ExtORPort auto BridgeRelay 1 AssumeReachable 1 PublishServerDescriptor 0 ServerTransportListenAddr snowflake 0.0.0.0:8080 ServerTransportPlugin snowflake exec server/server --disable-tls --log snowflake-server.log
Unpack the Tor Browser package and edit the file Browser/TorBrowser/Data/Tor/torrc-defaults. Configure it to use the local broker by changing the "ClientTransportPlugin snowflake" line: ClientTransportPlugin snowflake exec ./TorBrowser/Tor/PluggableTransports/snowflake-client -url http://127.0.0.1:8000/ -ice stun:stun.l.google.com:19302 -log snowflake-client.log
Run Tor Browser. Select "Configure" then "Tor is censored in my country" then "Provide a bridge I know". Enter snowflake 0.0.3.0:1 The purpose of this step is to provide a bridge line without a fingerprint. If you just select "snowflake" from the menu, it will use a bridge line with a fingerprint of the public bridge, which won't match the fingerprint of the bridge you are running locally.
Now click "Connect" and you can surf.
Beware that with Tor Browser 9.5a5 soon to be released, these special builds will try to auto-update themselves right away, and then you won't have a special Turbo Tunnel build anymore. Set app.update.auto=false in about:config to prevent the automatic upgrade.
Cecylia reminded me that building the quic-go version is trickier because quic-go has an unstable API, and the version of quic-go currently being used in the Tor Browser build is API-incompatible with what you'll get if you just "go get" or "go build". The snowflake repo doesn't yet have a go.mod file to lock in specific versions.
So the build instructions are a little bit more involved: git clone https://git.torproject.org/pluggable-transports/snowflake.git cd snowflake git remote add dcf https://git.torproject.org/user/dcf/snowflake.git git fetch dcf git checkout 5eff0f3ba406eb7d8cd90711edab7de7d7bda4fa for d in broker proxy-go server; do (cd $d && go get); done (cd $GOPATH/src/github.com/lucas-clemente/quic-go && git checkout 907071221cf97f75398d9cf8b1174e94f56e8f96) (cd $GOPATH/src/github.com/marten-seemann/qtls && git checkout 65ca381cd298d7e0aef0de8ba523a870ec5a96fe) for d in broker proxy-go server; do (cd $d && go build); done
Alternatively, you can change IdleTimeout to MaxIdleTimeout in the quic.Config structures; I think that's the only API change.
On Feb 13, 2020, at 1:17 PM, David Fifield david@bamsoftware.com wrote:
The snowflake repo doesn't yet have a go.mod file to lock in specific versions.
Maybe it should? CI seems to be failing because it isn't getting the right version of pion/dtls
Today I deployed the turbotunnel branch to the Snowflake public bridge. It's a triple-mode bridge, compatible with current Snowflake clients, Turbo Tunnel with KCP, and Turbo Tunnel with QUIC. https://bugs.torproject.org/33336#comment:8
To go with it there are special builds of Tor Browser that use turbotunnel-kcp and turbotunnel-quic respectively. To try these, just select "Configure", then "Tor is censored in my country", then "Select a built-in bridge", then choose snowflake. Alternatively, enable snowflake at about:preferences#tor. https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn... https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn...
The purpose of having two bundles is so we can evaluate both KCP and QUIC and see if either offers a compelling advantage over the other. My thinking is that after this experimental phase, we'll choose one of the two and deploy only that one. The experimental bundles will stop working after 2020-04-15, when they will be liable to self-update to an ordinary alpha and lose their Turbo Tunnel nature.
Here are tips on running the experimental bundles simultaneously and alongside your normal Tor Browser. You just have to change some port numbers to avoid conflicts. https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking#Launchi... tar xf tor-browser-snowflake-turbotunnel-kcp-9.5a5-20200215/tor-browser-linux64-9.5a5_en-US.tar.xz mv tor-browser_en-US tor-browser_en-US_kcp cd tor-browser_en-US_kcp export TOR_SOCKS_PORT=9250 export TOR_CONTROL_PORT=9251 sed -i -e 's/Port 915/Port 925/' ./Browser/TorBrowser/Data/Tor/torrc-defaults ./start-tor-browser.desktop
tar xf tor-browser-snowflake-turbotunnel-quic-9.5a5-20200215/tor-browser-linux64-9.5a5_en-US.tar.xz mv tor-browser_en-US tor-browser_en-US_quic cd tor-browser_en-US_quic export TOR_SOCKS_PORT=9350 export TOR_CONTROL_PORT=9351 sed -i -e 's/Port 915/Port 935/' ./Browser/TorBrowser/Data/Tor/torrc-defaults ./start-tor-browser.desktop You may also want to go to the Customize... menu and choose different themes (default/light/dark) to help distinguish the browsers.
One thing I didn't think about is enabling client logging by default. If you need a log for debugging, you'll have to manually add "-log snowflake-client.log" to the "ClientTransportPlugin snowflake" line in Browser/TorBrowser/Data/Tor/torrc-defaults.
The Turbo Tunnel browsers are meant to keep working even after you've left them idle for a long time, and even if your Snowflake proxy dies. When a proxy dies, your connection will hang for at least 30 s--maybe longer if a replacement isn't immediately available. Occasional hangs are expected (at least until #25723 is done), but it should always recover within a few minutes at the outmost.
Here are second-draft Tor Browser packages with Turbo Tunnel–enabled Snowflake. These fix most of the problems I experienced with the first draft. You still run the risk of being assigned a slow proxy, the frequent disconnections and occasional CPU spikes should be fixed. https://bugs.torproject.org/33336#comment:17 summarizes the changes.
https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn... https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn...
There are two packages because we are testing two options for the inner tunnel layer. You can choose either one, or download both and compare them. To enable Snowflake the first time you run one, do this: * Click "Configure" * Click "Tor is censored in my country" * Click "Select a built-in bridge" * Select "snowflake" from the menu How to enable Snowflake if the browser is already running: * Go to about:preferences#tor (open the Preferences menu then click Tor on the left side) * Look at the "Bridges" section * Click "Use a bridge" * Click "Select a built-in bridge" * Select "snowflake" from the menu If you are curious about what's happening, you can see the Snowflake log in the file Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log. tail -F Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log
These packages are experimental and are not meant to be used permanently. They are configured so that after 2020-04-23, they will auto-update to the latest alpha release of Tor Browser, and lose their Turbo Tunnel capabilities.
The name Turbo Tunnel refers to a design that keeps your session alive even if a network connection temporarily stops working. In Snowflake, it allows a client to transition across temporary Snowflake proxies.
== How to run multiple browsers at once ==
If you want to run both browsers at the same time, or run them at the some time as your usual Tor Browser, you have to make them use different ports to avoid conflicts. The default ports are 9150 and 9151. What I did is rename the "tor-browser_en-US" directory so I could have two of them: tar xf tor-browser-snowflake-turbotunnel-kcp-9.5a5-20200223/tor-browser-linux64-9.5a5_en-US.tar.xz mv tor-browser_en-US tor-browser_en-US_kcp tar xf tor-browser-snowflake-turbotunnel-quic-9.5a5-20200223/tor-browser-linux64-9.5a5_en-US.tar.xz mv tor-browser_en-US tor-browser_en-US_quic Then you can run each one with different ports: cd tor-browser_en-US_kcp TOR_SOCKS_PORT=9250 TOR_CONTROL_PORT=9251 ./start-tor-browser.desktop cd ../tor-browser_en-US_quic TOR_SOCKS_PORT=9350 TOR_CONTROL_PORT=9351 ./start-tor-browser.desktop You may also want to go to the Customize... menu and choose different themes (light/dark) to visually distinguish them.
== How to interpret log messages ==
BrokerChannel Response: 504 Gateway Timeout This means the broker couldn't find a proxy for you. It's a temporary error and the client will try again in 10 seconds. BrokerChannel Response: 200 OK This means that you got matched up with a proxy, but it doesn't necessarily mean the proxy works yet. Traffic Bytes (in|out): 0 | 972 -- (0 OnMessages, 8 Sends) If the number on the left stays at 0, it means the proxy isn't working (you're sending but not receiving anything). If 30 second pass without receiving anything, the client will abandon that proxy and contact the broker to get another one. Traffic Bytes (in|out): 52457 | 7270 -- (47 OnMessages, 75 Sends) When you start getting numbers like this, your proxy is working. WebRTC: No messages received for 30s -- closing stale connection This means the proxy stopped working (or never worked) and the client will try another one. WebRTC: At capacity [1/1] Retrying in 10s... This is normal and means that the client has its desired number of proxies (1).
This revision of Turbo Tunnel–enabled Snowflake fixes a hang that would occur if tor opened more than one SOCKS connection through the pluggable transport client at a time: https://bugs.torproject.org/33519. What would happen is the first SOCKS connection would get the only available proxy, and the second would get none. Now, multiple SOCKS connections may share the same proxy.
https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn... https://people.torproject.org/~dcf/pt-bundle/tor-browser-snowflake-turbotunn...
To enable Snowflake the first time you run it: * Click "Configure" * Click "Tor is censored in my country" * Click "Select a built-in bridge" * Select "snowflake" from the menu How to enable Snowflake when the browser is already running: * Go to about:preferences#tor (open the Preferences menu then click Tor on the left side) * Look at the "Bridges" section * Click "Use a bridge" * Click "Select a built-in bridge" * Select "snowflake" from the menu
The browser is set not to automatically install updates, because an update would remove the Turbo Tunnel features. The browser will still download updates and ask to install them, so you'll have to click "Not Now" whenever it asks.
== How to run multiple browsers at once ==
If you want to run both browsers at the same time, or run them at the some time as your usual Tor Browser, you have to make them use different ports to avoid conflicts. The default ports are 9150 and 9151. What I did is rename the "tor-browser_en-US" directory so I could have two of them: tar xf tor-browser-snowflake-turbotunnel-kcp-9.5a8-20200223/tor-browser-linux64-9.5a8_en-US.tar.xz mv tor-browser_en-US tor-browser_en-US_kcp tar xf tor-browser-snowflake-turbotunnel-quic-9.5a8-20200223/tor-browser-linux64-9.5a8_en-US.tar.xz mv tor-browser_en-US tor-browser_en-US_quic Then you can run each one with different ports: cd tor-browser_en-US_kcp TOR_SOCKS_PORT=9250 TOR_CONTROL_PORT=9251 ./start-tor-browser.desktop cd ../tor-browser_en-US_quic TOR_SOCKS_PORT=9350 TOR_CONTROL_PORT=9351 ./start-tor-browser.desktop You may also want to go to the Customize... menu and choose different themes (light/dark) to visually distinguish them.
== How to interpret log messages ==
The Snowflake log is in Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log.
BrokerChannel Response: 504 Gateway Timeout This means the broker couldn't find a proxy for you. It's a temporary error and the client will try again in 10 seconds. BrokerChannel Response: 200 OK This means that you got matched up with a proxy, but it doesn't necessarily mean the proxy works yet. Traffic Bytes (in|out): 0 | 972 -- (0 OnMessages, 8 Sends) If the number on the left stays at 0, it means the proxy isn't working (you're sending but not receiving anything). If 30 second pass without receiving anything, the client will abandon that proxy and contact the broker to get another one. Traffic Bytes (in|out): 52457 | 7270 -- (47 OnMessages, 75 Sends) When you start getting numbers like this, your proxy is working. WebRTC: No messages received for 30s -- closing stale connection This means the proxy stopped working (or never worked) and the client will try another one. WebRTC: At capacity [1/1] Retrying in 10s... This is normal and means that the client has its desired number of proxies (1).
anti-censorship-team@lists.torproject.org
-
Arlo Breault -
Cecylia Bocovich -
David Fifield -
Roger Dingledine