The attached is a draft proposal for allowing tor to lie to an application about the SOCKS connection enabling it to send data optimistically.
It's going to need some fleshing out in ways I am not familiar with, but I wanted to get something out to start as we think that this is probably the best path forward for bringing back Tor Browser's optimistic SOCKS behavior.
-tom
I'll add that a cypherpunk has been testing a very simple patch implementing this behavior for a few months and not seen adverse effects: https://trac.torproject.org/projects/tor/attachment/ticket/5915/tor-optimist... (Although I propose to not include the error page component.)
-tom
On Sat, 22 Jun 2019 at 00:50, Tom Ritter tom@ritter.vg wrote:
The attached is a draft proposal for allowing tor to lie to an application about the SOCKS connection enabling it to send data optimistically.
It's going to need some fleshing out in ways I am not familiar with, but I wanted to get something out to start as we think that this is probably the best path forward for bringing back Tor Browser's optimistic SOCKS behavior.
-tom
On 6/21/19 8:50 PM, Tom Ritter wrote:
The attached is a draft proposal for allowing tor to lie to an application about the SOCKS connection enabling it to send data optimistically.
It's going to need some fleshing out in ways I am not familiar with, but I wanted to get something out to start as we think that this is probably the best path forward for bringing back Tor Browser's optimistic SOCKS behavior.
I am not sure what to do about it, but I think the approach you describe will break the method that Tor Browser just started to use to detect that an onion service requires client authentication (see https://trac.torproject.org/projects/tor/ticket/30000 and associated child tickets). The tldr is that we rely on receiving a new error code from the SOCKS connect request.
On Tue, 2 Jul 2019 at 13:42, Mark Smith mcs@pearlcrescent.com wrote:
On 6/21/19 8:50 PM, Tom Ritter wrote:
The attached is a draft proposal for allowing tor to lie to an application about the SOCKS connection enabling it to send data optimistically.
It's going to need some fleshing out in ways I am not familiar with, but I wanted to get something out to start as we think that this is probably the best path forward for bringing back Tor Browser's optimistic SOCKS behavior.
I am not sure what to do about it, but I think the approach you describe will break the method that Tor Browser just started to use to detect that an onion service requires client authentication (see https://trac.torproject.org/projects/tor/ticket/30000 and associated child tickets). The tldr is that we rely on receiving a new error code from the SOCKS connect request.
Hm, yes.
We could not use optimistic data for onions...
Or instead of using a SOCKs error code we could return a special type of error (encapsulated in a HTTP response) recognizable by Tor Browser. Something like "If the response to an onion request is status code 407 Proxy Authentication Required (or 4xx whatever) then the Browser should prompt for onion service client authentication and retry the request with that."
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
-tom
On Tue, 2 Jul 2019 at 09:23, Tom Ritter tom@ritter.vg wrote:
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
I never updated this thread after discussing with people on irc.
So the implementation of SOCKS-error-code-for-an-Onion-Service-needs-auth implementation is done. David (if I'm summarizing correctly) felt that the SOCKS Error code approach may not be the best choice given our desire for optimistic data; but felt it was up to the Tor Browser team to decide.
In the goal of something that works for 90%+ of use case today, the rest later, I'll propose the following:
In little-t tor, detect if we're connecting to an onion site, and if so do not early-report SOCKS connection.
Another ugly option is to early-report a successful SOCKS connection even for onion sites, and if we later receive an auth request, send an HTTP error code like 407 that we then detect over in the browser and use to prompt the user. I don't like this because it is considerably more work (I expect), horrible ugly layering violations, and I don't think it will work for https://onion links.
-tom
On Mon, 5 Aug 2019 at 18:33, Tom Ritter tom@ritter.vg wrote:
On Tue, 2 Jul 2019 at 09:23, Tom Ritter tom@ritter.vg wrote:
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
I never updated this thread after discussing with people on irc.
So the implementation of SOCKS-error-code-for-an-Onion-Service-needs-auth implementation is done. David (if I'm summarizing correctly) felt that the SOCKS Error code approach may not be the best choice given our desire for optimistic data; but felt it was up to the Tor Browser team to decide.
In the goal of something that works for 90%+ of use case today, the rest later, I'll propose the following:
In little-t tor, detect if we're connecting to an onion site, and if so do not early-report SOCKS connection.
Another ugly option is to early-report a successful SOCKS connection even for onion sites, and if we later receive an auth request, send an HTTP error code like 407 that we then detect over in the browser and use to prompt the user. I don't like this because it is considerably more work (I expect), horrible ugly layering violations, and I don't think it will work for https://onion links.
I attached an updated proposal taking this into account, and I'd like to request it be entered into torspec's proposals list.
-tom
On Fri, Sep 27, 2019 at 1:35 PM Tom Ritter tom@ritter.vg wrote:
On Mon, 5 Aug 2019 at 18:33, Tom Ritter tom@ritter.vg wrote:
On Tue, 2 Jul 2019 at 09:23, Tom Ritter tom@ritter.vg wrote:
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
I never updated this thread after discussing with people on irc.
So the implementation of SOCKS-error-code-for-an-Onion-Service-needs-auth implementation is done. David (if I'm summarizing correctly) felt that the SOCKS Error code approach may not be the best choice given our desire for optimistic data; but felt it was up to the Tor Browser team to decide.
In the goal of something that works for 90%+ of use case today, the rest later, I'll propose the following:
In little-t tor, detect if we're connecting to an onion site, and if so do not early-report SOCKS connection.
Another ugly option is to early-report a successful SOCKS connection even for onion sites, and if we later receive an auth request, send an HTTP error code like 407 that we then detect over in the browser and use to prompt the user. I don't like this because it is considerably more work (I expect), horrible ugly layering violations, and I don't think it will work for https://onion links.
I attached an updated proposal taking this into account, and I'd like to request it be entered into torspec's proposals list.
Okay! This is now proposal 309.
On Wed, Oct 2, 2019 at 5:46 PM Nick Mathewson nickm@freehaven.net wrote:
On Fri, Sep 27, 2019 at 1:35 PM Tom Ritter tom@ritter.vg wrote:
On Mon, 5 Aug 2019 at 18:33, Tom Ritter tom@ritter.vg wrote:
On Tue, 2 Jul 2019 at 09:23, Tom Ritter tom@ritter.vg wrote:
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
I never updated this thread after discussing with people on irc.
So the implementation of SOCKS-error-code-for-an-Onion-Service-needs-auth implementation is done. David (if I'm summarizing correctly) felt that the SOCKS Error code approach may not be the best choice given our desire for optimistic data; but felt it was up to the Tor Browser team to decide.
In the goal of something that works for 90%+ of use case today, the rest later, I'll propose the following:
In little-t tor, detect if we're connecting to an onion site, and if so do not early-report SOCKS connection.
Another ugly option is to early-report a successful SOCKS connection even for onion sites, and if we later receive an auth request, send an HTTP error code like 407 that we then detect over in the browser and use to prompt the user. I don't like this because it is considerably more work (I expect), horrible ugly layering violations, and I don't think it will work for https://onion links.
I attached an updated proposal taking this into account, and I'd like to request it be entered into torspec's proposals list.
Okay! This is now proposal 309.
I went for a walk and I came to the realization that we're going about this (a little bit) wrong.
The advantage of optimistic data is that application data is available when tor sends the RELAY_BEGIN cell (therefore it is able to send a RELAY_DATA cell immediately after the RELAY_BEGIN cell is sent). So, tor doesn't need to reply immediately, just early enough such that the application can start writing data on the connection.
For exit connections, Tor should probably reply a success/failure immediately (where failures result from impossible connection requests or other early failures).
For onion service connections, tor can reply much later. I might suggest as late as successfully retrieving the onion service descriptor. Of course, this will introduce a race between the application writing data and tor completing the introduction and rendezvous, but this may be worth the risk.
In particular, this allows using most of the proposed onion service error codes in the SOCKS5 reply. I think the control protocol should still emit event messages describing the errors, but this will allow using synchronous error handling, as well.
On 08 Oct (19:49:34), Matthew Finkel wrote:
On Wed, Oct 2, 2019 at 5:46 PM Nick Mathewson nickm@freehaven.net wrote:
On Fri, Sep 27, 2019 at 1:35 PM Tom Ritter tom@ritter.vg wrote:
On Mon, 5 Aug 2019 at 18:33, Tom Ritter tom@ritter.vg wrote:
On Tue, 2 Jul 2019 at 09:23, Tom Ritter tom@ritter.vg wrote:
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
I never updated this thread after discussing with people on irc.
So the implementation of SOCKS-error-code-for-an-Onion-Service-needs-auth implementation is done. David (if I'm summarizing correctly) felt that the SOCKS Error code approach may not be the best choice given our desire for optimistic data; but felt it was up to the Tor Browser team to decide.
In the goal of something that works for 90%+ of use case today, the rest later, I'll propose the following:
In little-t tor, detect if we're connecting to an onion site, and if so do not early-report SOCKS connection.
Another ugly option is to early-report a successful SOCKS connection even for onion sites, and if we later receive an auth request, send an HTTP error code like 407 that we then detect over in the browser and use to prompt the user. I don't like this because it is considerably more work (I expect), horrible ugly layering violations, and I don't think it will work for https://onion links.
I attached an updated proposal taking this into account, and I'd like to request it be entered into torspec's proposals list.
Okay! This is now proposal 309.
I went for a walk and I came to the realization that we're going about this (a little bit) wrong.
The advantage of optimistic data is that application data is available when tor sends the RELAY_BEGIN cell (therefore it is able to send a RELAY_DATA cell immediately after the RELAY_BEGIN cell is sent). So, tor doesn't need to reply immediately, just early enough such that the application can start writing data on the connection.
For exit connections, Tor should probably reply a success/failure immediately (where failures result from impossible connection requests or other early failures).
For onion service connections, tor can reply much later. I might suggest as late as successfully retrieving the onion service descriptor. Of course, this will introduce a race between the application writing data and tor completing the introduction and rendezvous, but this may be worth the risk.
I think indeed this could work!
Proposal 304 specifies errors up until a rendezvous failure so should we then only send back the SOCKS reply once the rendezvous circuit has been established (upon reception of the RENDEZVOUS2 cell)?
It could be seconds in real life before that reply is given back to Tor Browser so not sure if optimistic data will worth anything there :P ...?
Another option exists: In my experience, failing a rendezvous is much more rare than failing an introduction. In other words, if the introduction is successful, chances are very good that you'll rendezvous. Which means that we could send back the SOCKS reply after a successful introduction and drop the rdv error from prop304?
In particular, this allows using most of the proposed onion service error codes in the SOCKS5 reply. I think the control protocol should still emit event messages describing the errors, but this will allow using synchronous error handling, as well.
As we discussed the other day, we have this "Stream ID" problem as in for an emitted control event (for a stream), TB needs to associate it with the right tab basically.
We basically have two options here I believe. Either TB sends us a unique identifier that could become an exported ID on the event by encoding it in the SOCKS username/password somehow.
Else, tor sends it back to TB within the SOCKS reply. That one leads to a race where the control event is likely to be fired _before_ the SOCKS reply and thus TB will have to handle async event vs the SOCKS reply.
Else else, theoritical, maybe the current u+p from TB per tab is already enough but I doubt since I'm guessing it is the same u+p for two tabs on the same .onion?
Cheers! David
David Goulet dgoulet@torproject.org writes:
On 08 Oct (19:49:34), Matthew Finkel wrote:
On Wed, Oct 2, 2019 at 5:46 PM Nick Mathewson nickm@freehaven.net wrote:
On Fri, Sep 27, 2019 at 1:35 PM Tom Ritter tom@ritter.vg wrote:
On Mon, 5 Aug 2019 at 18:33, Tom Ritter tom@ritter.vg wrote:
On Tue, 2 Jul 2019 at 09:23, Tom Ritter tom@ritter.vg wrote:
Or... something else? Very interested in what David/asn think since they worked on #30382 ...
I never updated this thread after discussing with people on irc.
So the implementation of SOCKS-error-code-for-an-Onion-Service-needs-auth implementation is done. David (if I'm summarizing correctly) felt that the SOCKS Error code approach may not be the best choice given our desire for optimistic data; but felt it was up to the Tor Browser team to decide.
In the goal of something that works for 90%+ of use case today, the rest later, I'll propose the following:
In little-t tor, detect if we're connecting to an onion site, and if so do not early-report SOCKS connection.
Another ugly option is to early-report a successful SOCKS connection even for onion sites, and if we later receive an auth request, send an HTTP error code like 407 that we then detect over in the browser and use to prompt the user. I don't like this because it is considerably more work (I expect), horrible ugly layering violations, and I don't think it will work for https://onion links.
I attached an updated proposal taking this into account, and I'd like to request it be entered into torspec's proposals list.
Okay! This is now proposal 309.
I went for a walk and I came to the realization that we're going about this (a little bit) wrong.
Thanks for the updates Matt!
The advantage of optimistic data is that application data is available when tor sends the RELAY_BEGIN cell (therefore it is able to send a RELAY_DATA cell immediately after the RELAY_BEGIN cell is sent). So, tor doesn't need to reply immediately, just early enough such that the application can start writing data on the connection.
For exit connections, Tor should probably reply a success/failure immediately (where failures result from impossible connection requests or other early failures).
For onion service connections, tor can reply much later. I might suggest as late as successfully retrieving the onion service descriptor. Of course, this will introduce a race between the application writing data and tor completing the introduction and rendezvous, but this may be worth the risk.
So are you suggesting that we can still do SOCKS error codes? But as David said, some of the errors we care about are after the descriptor fetch, so how would we do those?
Also, please help me understand the race condition you refer to. I tried to draw this in a diagram form: https://gist.github.com/asn-d6/55fbe7a3d746dc7e00da25d3ce90268a
IIUC, for onions the advantage of opportunistic SOCKS is that we would send DATA to the service right after finishing rendezvous, whereas right now we need to do a round-trip with Tor Browser after finishing rendezvous. Is that right?
If that's the case, then sending the SOCKS reply after the rendezvous circuit is completed would be the same as the current behavior, and hence not an optimization, right?
And sending the SOCKS reply after the introduction is completed (as David is suggesting) would be an optimization indeed, but we lose errors (we lose the rendezvous failed error, which can occur if the onion service is under DoS and cannot build new circuits but can still receive introductions).
What other problems exist here?
On Thu, 10 Oct 2019 at 10:37, George Kadianakis desnacked@riseup.net wrote:
So are you suggesting that we can still do SOCKS error codes? But as David said, some of the errors we care about are after the descriptor fetch, so how would we do those?
Only 'X'F3' Onion Service Rendezvous Failed' - right?
I think David is proposing we just don't do that one because in his experience it's pretty rare.
Also, please help me understand the race condition you refer to. I tried to draw this in a diagram form: https://gist.github.com/asn-d6/55fbe7a3d746dc7e00da25d3ce90268a
I edited this: https://gist.github.com/tomrittervg/e0552ed007dbe50077528936b09a2eff
Whose first diagram (normal situation) is correct? Something looks off in yours... You obviously know tor much better; but latency gain for optimistic socks doesn't come from sending the HTTP GET to tor sooner, it comes from sending it to the destination sooner - so I think that the GET must travel to the destination before the destination replies with CONNECTED, doesn't it?
Anyway, I tried to illustrate Matts race condition as I understand it; but I am entirely unconcerned with it. There's no way it's going to take the browser more time to generate a HTTP GET and send it over SOCKS than it's going to take tor to roundtrip a rendezvous setup.
IIUC, for onions the advantage of opportunistic SOCKS is that we would send DATA to the service right after finishing rendezvous, whereas right now we need to do a round-trip with Tor Browser after finishing rendezvous. Is that right?
If that's the case, then sending the SOCKS reply after the rendezvous circuit is completed would be the same as the current behavior, and hence not an optimization, right?
Correct.
And sending the SOCKS reply after the introduction is completed (as David is suggesting) would be an optimization indeed, but we lose errors (we lose the rendezvous failed error, which can occur if the onion service is under DoS and cannot build new circuits but can still receive introductions).
Yup.
What other problems exist here?
I'll have to think about it more at a time I'm better equipped to.
-tom
On Fri, Oct 11, 2019 at 6:30 AM Tom Ritter tom@ritter.vg wrote:
On Thu, 10 Oct 2019 at 10:37, George Kadianakis desnacked@riseup.net wrote:
So are you suggesting that we can still do SOCKS error codes? But as David said, some of the errors we care about are after the descriptor fetch, so how would we do those?
Only 'X'F3' Onion Service Rendezvous Failed' - right?
Correct.
I think David is proposing we just don't do that one because in his experience it's pretty rare.
Also, please help me understand the race condition you refer to. I tried to draw this in a diagram form: https://gist.github.com/asn-d6/55fbe7a3d746dc7e00da25d3ce90268a
I edited this: https://gist.github.com/tomrittervg/e0552ed007dbe50077528936b09a2eff
Whose first diagram (normal situation) is correct? Something looks off in yours... You obviously know tor much better; but latency gain for optimistic socks doesn't come from sending the HTTP GET to tor sooner, it comes from sending it to the destination sooner - so I think that the GET must travel to the destination before the destination replies with CONNECTED, doesn't it?
I think Tom's diagram is correct. We're worried about: "Does the application send the "HTTP GET" bytes before or after tor sends the RELAY_BEGIN relay cell?" If the application payload arrives before the RELAY_BEGIN command is sent, then we can send optimistic data --else we can't. The remaining (open) question is *when* do we send the application "FAKE SUCCESSFUL SOCKS REPLY". If we send it after receiving an INTRODUCE_ACK command relay cell, then the application will almost certainly win the race but it must watch the controller events for a rendezvous failure. If we send the "fake success socks reply" after receiving a RENDEZVOUS2 command relay cell, then the application will likely lose the race because tor will immediately attach the stream on the circuit and send a RELAY_BEGIN relay cell (and the application payload will arrive after the stream is connected).
The optimistic data simply saves us time. The race condition only exists on the client-side. The application's payload (via optimistic data) doesn't need to reach the service-side tor client before the service-side replies with the CONNECTED relay cell. If the rendezvous circuit is established and the stream is attached, then the application on the service-side will receive the client-side's payload and handle it normally.
The main advantage of optimistic data is that we don't wait the full round-trip time. The RELAY_DATA relay cell is sent immediately after the RELAY_BEGIN relay cell, so it is already in-flight when the stream is connected. When this fails, then (I believe) the rendezvous circuit should be destroyed and the client application's connection with the SocksPort is closed. This is where we need the unique ID, so we can correlate the error message from the controller event with the closed TCP connection. I believe Mark and/or Kathy mentioned how handling this error case in Tor Browser will be difficult, because the controller error message may be emitted after Tor Browser handles the closed connection.
-
David Goulet -
George Kadianakis -
Mark Smith -
Matthew Finkel -
Nick Mathewson -
Tom Ritter