Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
Instead of listing all options and going in an endless loop of possibilities, I'll outline what we've discussed so far between some of us. In the following, "v2" is current hidden service and "v3" is the next generation detailed in prop224.
1) Once v3 is released, from that point on _no_ v2 service will be allowed to be created by "tor" itself. It will always be possible to do it by hand by creating an RSA key and putting it in the service directory (see 3 below).
The reasoning for this is that we really want to phase out as quickly as possible the v2 protocol for privacy and security reasons.
2) All the current torrc options will be kept for v3 as they all apply in terms of format. (One exception might be the client authorization.)
3) When tor is loading config for a service:
if a v2 key is found that is an RSA key, we treat that service as v2, print a deprecation warning and continue.
if a v3 key is found, use v3, all is good.
if no key is found, consider a new service and generate v3 (ties to 1 above).
4) Over time, extra option might appear and they will be ONLY for v3 unless for force majeur security madness.
One of those options that we want to add is this one as offline key will be a reality with v3:
"HiddenServiceOfflineKey 1"
It will indicate tor to _not_ generate the master identity key and will require the user to put the public key in the HiddenServiceDir.
Note that with all of the above it will be possible to have one of your application on both v2 and v3 address. Here is a snippet as an example:
# v2 HiddenServiceDir /srv/hs/v2 HiddenServicePort 80 localhost:80
# v3 HiddenServiceDir /srv/hs/v3 HiddenServicePort 80 localhost:80
The important part here is the different directories but the port points to the same place.
Ok here it is. Please comment, improve, or propose! :)
Cheers! David
[1] https://www.torproject.org/docs/tor-manual.html.en (see HIDDEN SERVICE OPTIONS)
David Goulet dgoulet@ev0ke.net writes:
- Once v3 is released, from that point on _no_ v2 service will be allowed to be created by "tor" itself. It will always be possible to do it by hand by creating an RSA key and putting it in the service directory (see 3 below).
+1 or +2 at least :)
Ok here it is. Please comment, improve, or propose! :)
How does ADD_ONION fit in?
I think the ephemeral interface is really good, and IME integrators prefer it -- if they're already "handing secret stuff" (which seems fairly likely, if they're aware of Tor and are integrating it in their application) then it makes more sense for them to use this API.
Also, it's right now extremely unlikely (and in any case very fragile) to actually succesffully add a service to a running Tor via "SETCONF" -- so I think anything using the control-protocol will typically prefer ADD_ONION. If you want to use the filesystem/torrc approach, txtorcon will always launch a new Tor instance for you (maybe this is the best way to go anyway for "Tor-integrating applications"?)
All that said, I'm not suggesting the torrc/"filesystem" options go away! These are already familiar to most service operators, presumably, and your approach sounds great! It's also really easy to support from a controller perspective. Also, the backwards-compatibility is very nice too.
On 23 Nov (03:12:22), meejah wrote:
David Goulet dgoulet@ev0ke.net writes:
- Once v3 is released, from that point on _no_ v2 service will be allowed to be created by "tor" itself. It will always be possible to do it by hand by creating an RSA key and putting it in the service directory (see 3 below).
+1 or +2 at least :)
Ok here it is. Please comment, improve, or propose! :)
How does ADD_ONION fit in?
I think the ephemeral interface is really good, and IME integrators prefer it -- if they're already "handing secret stuff" (which seems fairly likely, if they're aware of Tor and are integrating it in their application) then it makes more sense for them to use this API.
Also, it's right now extremely unlikely (and in any case very fragile) to actually succesffully add a service to a running Tor via "SETCONF" -- so I think anything using the control-protocol will typically prefer ADD_ONION. If you want to use the filesystem/torrc approach, txtorcon will always launch a new Tor instance for you (maybe this is the best way to go anyway for "Tor-integrating applications"?)
All that said, I'm not suggesting the torrc/"filesystem" options go away! These are already familiar to most service operators, presumably, and your approach sounds great! It's also really easy to support from a controller perspective. Also, the backwards-compatibility is very nice too.
I agree with you on the fact that ADD_ONION is nice and also crucial to hidden service as well. That will be addressed with the control port implementation of next generation. It's still an undecided part of the engineering work which is how we are going to proceed with it. Do we change the current events/commands to support both v2 and v3? Do we make new events/commands for v3? ... (#20699) It's quite a big piece of work so if anyone wants to take a stab at the specification! woot!!! :)
Thanks! David
-- meejah _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 11/23/2016 09:39 AM, David Goulet wrote:
I agree with you on the fact that ADD_ONION is nice and also crucial to hidden service as well. That will be addressed with the control port implementation of next generation. It's still an undecided part of the engineering work which is how we are going to proceed with it. Do we change the current events/commands to support both v2 and v3? Do we make new events/commands for v3? ... (#20699)
I suggest that we create new commands for v3. This will allow us to officially deprecate the old APIs and its more intuitive for developers since the distinction is clearer. This is also a golden opportunity to change "HiddenServiceX" to "OnionServiceX" if we want to officially transition the lingo.
On 24 Nov. 2016, at 06:09, Jesse V kernelcorn@torproject.org wrote:
On 11/23/2016 09:39 AM, David Goulet wrote:
I agree with you on the fact that ADD_ONION is nice and also crucial to hidden service as well. That will be addressed with the control port implementation of next generation. It's still an undecided part of the engineering work which is how we are going to proceed with it. Do we change the current events/commands to support both v2 and v3? Do we make new events/commands for v3? ... (#20699)
I suggest that we create new commands for v3. This will allow us to officially deprecate the old APIs and its more intuitive for developers since the distinction is clearer.
It also makes it easier to maintain backwards compatibility.
This is also a golden opportunity to change "HiddenServiceX" to "OnionServiceX" if we want to officially transition the lingo.
Yes, that's the plan. (We made the single onion service options start with HiddenService so we could change them all at once.) https://trac.torproject.org/projects/tor/ticket/17343
T
On Wed, 23 Nov 2016 03:12:22 +0400 meejah meejah@meejah.ca wrote:
David Goulet dgoulet@ev0ke.net writes:
- Once v3 is released, from that point on _no_ v2 service will be
allowed to be created by "tor" itself. It will always be possible to do it by hand by creating an RSA key and putting it in the service directory (see 3 below).
+1 or +2 at least :)
Ok here it is. Please comment, improve, or propose! :)
How does ADD_ONION fit in?
It's forward compatible by design, since you have to specify a key type when you handle key management, and Tor gets to do whatever it wants if you ask it to generate a key with the `BEST` algorithm.
Assuming people who use it aren't explicitly asking for RSA1024, their apps will magically switch to using Ed25519 automagically one day, when their tor is updated.
(People who expect `NEW:BEST` ADD_ONION-ed services to always give RSA1024 based HSes, should fix their code since the spec makes no guarantee that `BEST` will be RSA1024.)
Regards,
On 24 Nov. 2016, at 09:00, Yawning Angel yawning@schwanenlied.me wrote:
On Wed, 23 Nov 2016 03:12:22 +0400 meejah meejah@meejah.ca wrote:
David Goulet dgoulet@ev0ke.net writes:
- Once v3 is released, from that point on _no_ v2 service will be
allowed to be created by "tor" itself. It will always be possible to do it by hand by creating an RSA key and putting it in the service directory (see 3 below).
+1 or +2 at least :)
Ok here it is. Please comment, improve, or propose! :)
How does ADD_ONION fit in?
It's forward compatible by design, since you have to specify a key type when you handle key management, and Tor gets to do whatever it wants if you ask it to generate a key with the `BEST` algorithm.
Assuming people who use it aren't explicitly asking for RSA1024, their apps will magically switch to using Ed25519 automagically one day, when their tor is updated.
(People who expect `NEW:BEST` ADD_ONION-ed services to always give RSA1024 based HSes, should fix their code since the spec makes no guarantee that `BEST` will be RSA1024.)
+1
(I've changed my opinion, adding a new command is pointless. People who want the old ADD_ONION behaviour where BEST produces a v2 HS can use an older version of Tor, until the software that makes incorrect assumptions is updated.)
T
teor wrote:
How does ADD_ONION fit in?
It's forward compatible by design, since you have to specify a key type when you handle key management, and Tor gets to do whatever it wants if you ask it to generate a key with the `BEST` algorithm.
Assuming people who use it aren't explicitly asking for RSA1024, their apps will magically switch to using Ed25519 automagically one day, when their tor is updated.
(People who expect `NEW:BEST` ADD_ONION-ed services to always give RSA1024 based HSes, should fix their code since the spec makes no guarantee that `BEST` will be RSA1024.)
+1
(I've changed my opinion, adding a new command is pointless. People who want the old ADD_ONION behaviour where BEST produces a v2 HS can use an older version of Tor, until the software that makes incorrect assumptions is updated.)
T
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
I don't think it's productive to ask users to already support a new feature upon our first release providing the said feature.
To add some value on this point, I will bring into discussion a software that is widely used, produces significant rendezvous traffic and is important for some people:
Bitcoin Core - latest versions detect if you use Tor and automatically use ADD_ONION to create v2 services, and, important: it doesn't support yet the v3 address types because of their length. This way people behind NAT running it can be better connected by accepting incoming connections without an open port, some people don't want their upstream provider to know they are using this app, etc.
Example: my Bitcoin node (working as a dual stack both on clearnet and onion) has 146 total connections to peers, out of which onion:
~# sudo -u bitnode -i bitcoin-cli getpeerinfo | grep onion | wc -l 29
On 24 Nov. 2016, at 10:43, s7r s7r@sky-ip.org wrote:
teor wrote:
How does ADD_ONION fit in?
It's forward compatible by design, since you have to specify a key type when you handle key management, and Tor gets to do whatever it wants if you ask it to generate a key with the `BEST` algorithm.
Assuming people who use it aren't explicitly asking for RSA1024, their apps will magically switch to using Ed25519 automagically one day, when their tor is updated.
(People who expect `NEW:BEST` ADD_ONION-ed services to always give RSA1024 based HSes, should fix their code since the spec makes no guarantee that `BEST` will be RSA1024.)
+1
(I've changed my opinion, adding a new command is pointless. People who want the old ADD_ONION behaviour where BEST produces a v2 HS can use an older version of Tor, until the software that makes incorrect assumptions is updated.)
T
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
No-one is proposing we abolish ADD_ONION with v2 services straight away.
What we will do is make BEST mean v3, rather than v2. RSA1024 will continue to mean v2, as it always has.
ADD_ONION has always had an explicit BEST option, if clients don't want the BEST type of key, they should ask for a specific type they are prepared to handle.
Please read the appropriate control spec section: https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1446
I don't think it's productive to ask users to already support a new feature upon our first release providing the said feature.
This isn't what is proposed.
(We are going to stop automatically creating v2 services via HiddenServiceDir in the first v3 release, but there will always be the ability to manually create a key. And that's a separate conversation.)
To add some value on this point, I will bring into discussion a software that is widely used, produces significant rendezvous traffic and is important for some people:
Bitcoin Core - latest versions detect if you use Tor and automatically use ADD_ONION to create v2 services, and, important: it doesn't support yet the v3 address types because of their length.
Does it use ADD_ONION NEW:RSA1024 or ADD_ONION RSA1024:<String>?
Then it will be fine.
Does it use ADD_ONION NEW:BEST?
Then that's a client bug, and it should be fixed in the client.
...
T
teor wrote:
No-one is proposing we abolish ADD_ONION with v2 services straight away.
What we will do is make BEST mean v3, rather than v2. RSA1024 will continue to mean v2, as it always has.
ADD_ONION has always had an explicit BEST option, if clients don't want the BEST type of key, they should ask for a specific type they are prepared to handle.
Please read the appropriate control spec section: https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1446
I don't think it's productive to ask users to already support a new feature upon our first release providing the said feature.
This isn't what is proposed.
(We are going to stop automatically creating v2 services via HiddenServiceDir in the first v3 release, but there will always be the ability to manually create a key. And that's a separate conversation.)
Hey, I apologize, my bad. I wasn't considering the options following ADD_ONION, I thought it's straight forward. Sorry for this, my mistake.
Based on your explanation I agree with you and Yawning that ADD_ONION:BEST should produce a v3 key, and ADD_ONION:RSA1024 v2.
To add some value on this point, I will bring into discussion a software that is widely used, produces significant rendezvous traffic and is important for some people:
Bitcoin Core - latest versions detect if you use Tor and automatically use ADD_ONION to create v2 services, and, important: it doesn't support yet the v3 address types because of their length.
Does it use ADD_ONION NEW:RSA1024 or ADD_ONION RSA1024:<String>?
Then it will be fine.
Does it use ADD_ONION NEW:BEST?
Then that's a client bug, and it should be fixed in the client.
Hey: // Finally - now create the service if (private_key.empty()) // No private key, generate one private_key = "NEW:BEST"; // Request hidden service, redirect port. // Note that the 'virtual' port doesn't have to be the same as our internal port, but this is just a convenient // choice. TODO; refactor the shutdown sequence some day. _conn.Command(strprintf("ADD_ONION %s Port=%i,127.0.0.1:%i", private_key, GetListenPort(), GetListenPort()), boost::bind(&TorController::add_onion_cb, this, _1, _2));
whooops ;) filing a ticket so everyone is on the same page with us.
Thanks.
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
That is a completely orthogonal to the fact that people that have dumb limitations like:
Bitcoin Core - latest versions detect if you use Tor and automatically use ADD_ONION to create v2 services, and, important: it doesn't support yet the v3 address types because of their length. This way people behind NAT running it can be better connected by accepting incoming connections without an open port, some people don't want their upstream provider to know they are using this app, etc.
Should be using `NEW:RSA1024` explicitly. The override is there for a reason, and the key type is part of the serialized data that tor returns to you when you have it generate a key, precisely to avoid this sort of problem.
Their fix: "Replace line 473 of bitcoin/torcontrol.cpp with `private_key = "NEW:RSA1024";`".
Our fix: "Add another command, that does essentially the same thing, because people picked the wrong options, then later deal with the fallout from people getting used to the temporary command, and crying when it's deprecated."
I say "they should fix their code".
I don't think it's productive to ask users to already support a new feature upon our first release providing the said feature.
If they aren't using existing interfaces correctly, when correct behavior has been part of the interface since support for it was added, quite frankly it's their problem.
Regards,
On 24 Nov. 2016, at 11:04, Yawning Angel yawning@schwanenlied.me wrote:
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
So I think we should have an option:
OnionServiceCreateV3 0|1
Create V3 onion services by default when using HiddenServiceDir and ADD_ONION BEST.
Which defaults to 0 for at least the first release containing the new code, and then defaults to 1 for at least one release, and then is deprecated.
...
T
On Thu, 24 Nov 2016 11:13:06 +1100 teor teor2345@gmail.com wrote:
On 24 Nov. 2016, at 11:04, Yawning Angel yawning@schwanenlied.me wrote:
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
So I think we should have an option:
OnionServiceCreateV3 0|1
Create V3 onion services by default when using HiddenServiceDir and ADD_ONION BEST.
Which defaults to 0 for at least the first release containing the new code, and then defaults to 1 for at least one release, and then is deprecated.
This seems like a sensible plan to me.
On 24 Nov (11:13:06), teor wrote:
On 24 Nov. 2016, at 11:04, Yawning Angel yawning@schwanenlied.me wrote:
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
So I think we should have an option:
OnionServiceCreateV3 0|1
Few things why I'm not too keen with this even though it could actually be very useful... I'll outline the potential issues here with such an option but unfortunately I don't have a perfect solution in the end. My main concern is not really an easy or intuitive way for the user to start using v3 service but rather when the users will start using it, the network MUST be ready for it which makes things a bit more complicated to do properly I believe.
Considering a torrc option such as "OnionServiceCreateV3 0|1":
1) This is extremely non intuitive to any operator even power users. The concept of "v3" is something that very few people will understand what it's actually is. Even if we go with "OnionServiceEpicNextMoreSecureGeneration 1", we are creating two classes of onion service that will make things more confusing for the user with the question of "which one should I use?". Also see 3 below for more confusion.
2) It also means we create an option that will get deprecated once v2 is phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
3) The current plan so far is to add a consensus parameter that will control the switch for client and service to start using prop224. And every relay out there that will run a tor supporting the v3 protocol (HSDir, IP, RP) will speak the protocol no matter what. So then we can decide when the network is mature enough for client/service to start talking actively that protocol.
That switch will also be an opportunity for us to remove v0 and v1 support as well from tor making that event "The Big Onion Service Switch" or for short "The BOSS (tm)" :P.
HOWEVER, the hard part here is getting that consensus param information about the state of prop224 _before_ configuring your HS.... it's doable but will weirdly make a specific tor version bootstrap HS much slower as they'll have to wait for the consensus to be downloaded.... maybe there is a better way?
Here is an idea, we could rely on a tor version for enabling the feature instead that is we release client/service support when and only when the network is mature so then there will be no confusion, you run version X, you get v3, period, the network is ready. That is again not ideal as we then delay this important new feature by some unknown chunks of 6 months... but at least we have some assurance that it's ready.
We can ship the code in let's say 031 but make tor only use it in 032 for instance.
Thoughts?
David
Create V3 onion services by default when using HiddenServiceDir and ADD_ONION BEST.
Which defaults to 0 for at least the first release containing the new code, and then defaults to 1 for at least one release, and then is deprecated.
...
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 25 Nov. 2016, at 03:40, David Goulet dgoulet@ev0ke.net wrote:
On 24 Nov (11:13:06), teor wrote:
On 24 Nov. 2016, at 11:04, Yawning Angel yawning@schwanenlied.me wrote:
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
So I think we should have an option:
OnionServiceCreateV3 0|1
Few things why I'm not too keen with this even though it could actually be very useful... I'll outline the potential issues here with such an option but unfortunately I don't have a perfect solution in the end. My main concern is not really an easy or intuitive way for the user to start using v3 service but rather when the users will start using it, the network MUST be ready for it which makes things a bit more complicated to do properly I believe.
Considering a torrc option such as "OnionServiceCreateV3 0|1":
- This is extremely non intuitive to any operator even power users. The
concept of "v3" is something that very few people will understand what it's actually is. Even if we go with "OnionServiceEpicNextMoreSecureGeneration 1", we are creating two classes of onion service that will make things more confusing for the user with the question of "which one should I use?". Also see 3 below for more confusion.
We are already creating two classes of onion service: they have significantly different user-visible addresses, client authentication, and Tor version requirements.
This option simply controls which version is created by default when the user asks for an onion service.
But we already have a HiddenServiceVersion option, let's use that, and make the default:
For at least one release: * whatever the existing service uses, or * 2 for new services (with 3 being an alternative),
For at least one more release: * whatever the existing service uses (with 2 being deprecated), * 3 for new services (with 2 being deprecated),
For at least one more release: * whatever the existing service uses (with 2 being deprecated), * 3 for new services (with 2 being deprecated, and only able to be created manually),
For releases after that: * make the default and only version 3.
Do we support the same service publishing version 2 and version 3 descriptors?
Or do we expect people to set up two different services and direct them to the same port? Do we want to keep that option as a list for future use?
Either way, we should document that in the man page.
Since there's a default, most users will never set it, and they will just see the default for new services change in a particular release, and then another release where old services are deprecated, and then they will be gone entirely.
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option: * a developer wants to test that their software works with V3 onion services, before the default is changed, * a user wants to run their old software with V2 onion services, after the default is changed.
- The current plan so far is to add a consensus parameter that will control
the switch for client and service to start using prop224. And every relay out there that will run a tor supporting the v3 protocol (HSDir, IP, RP) will speak the protocol no matter what. So then we can decide when the network is mature enough for client/service to start talking actively that protocol.
That switch will also be an opportunity for us to remove v0 and v1 support as well from tor making that event "The Big Onion Service Switch" or for short "The BOSS (tm)" :P.
HOWEVER, the hard part here is getting that consensus param information about the state of prop224 _before_ configuring your HS.... it's doable but will weirdly make a specific tor version bootstrap HS much slower as they'll have to wait for the consensus to be downloaded.... maybe there is a better way?
Here is an idea, we could rely on a tor version for enabling the feature instead that is we release client/service support when and only when the network is mature so then there will be no confusion, you run version X, you get v3, period, the network is ready. That is again not ideal as we then delay this important new feature by some unknown chunks of 6 months... but at least we have some assurance that it's ready.
We can ship the code in let's say 031 but make tor only use it in 032 for instance.
…
I think the consensus parameter is more useful as an emergency on/off switch for descriptor publication, not service configuration.
If the code exists on the client, a user should always be able to configure a v3 onion service. Even before the consensus is downloaded. Even if they do not have access to the network.
So here's how I'd use the consensus parameter: * if the consensus parameter says v3 services are disabled, users can configure a v3 service, but it will warn and refuse to publish its descriptor. * if the consensus parameter says v3 services are enabled, users can configure a v3 service, and it will publish its descriptor. (And we should be able to deprecate v3 at some point in the future.)
And similarly for v2: * if the consensus parameter says v2 services are enabled, users can configure a v2 service, and it will publish its descriptor. * if the consensus parameter says v2 services are deprecated, users can configure a v2 service, and it will warn, and then publish its descriptor. * if the consensus parameter says v2 services are disabled, users can configure a v2 service, but it will warn and refuse to publish its descriptor. (And some future Tor version will remove support for creating v2 services, and another version will remove support for loading them.)
We should also consider how this interacts with the protocol lines in the consensus.
T
On 25 Nov (10:33:35), teor wrote:
On 25 Nov. 2016, at 03:40, David Goulet dgoulet@ev0ke.net wrote:
On 24 Nov (11:13:06), teor wrote:
On 24 Nov. 2016, at 11:04, Yawning Angel yawning@schwanenlied.me wrote:
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
So I think we should have an option:
OnionServiceCreateV3 0|1
Few things why I'm not too keen with this even though it could actually be very useful... I'll outline the potential issues here with such an option but unfortunately I don't have a perfect solution in the end. My main concern is not really an easy or intuitive way for the user to start using v3 service but rather when the users will start using it, the network MUST be ready for it which makes things a bit more complicated to do properly I believe.
Considering a torrc option such as "OnionServiceCreateV3 0|1":
- This is extremely non intuitive to any operator even power users. The
concept of "v3" is something that very few people will understand what it's actually is. Even if we go with "OnionServiceEpicNextMoreSecureGeneration 1", we are creating two classes of onion service that will make things more confusing for the user with the question of "which one should I use?". Also see 3 below for more confusion.
We are already creating two classes of onion service: they have significantly different user-visible addresses, client authentication, and Tor version requirements.
What I meant by this is I want to avoid to have a world where v2 and v3 can be created at the same time. I think we want either new v2 or new v3 but not both for a single tor instance. Of course, the network is diversed, we won't stop that over night but for a single tor release it's easy.
This option simply controls which version is created by default when the user asks for an onion service.
But we already have a HiddenServiceVersion option, let's use that, and make the default:
Yeah that's a better idea actually. Once the network is ready for v3 that is we judge that we have enough nodes for HSDir and IP (RP doesn't change), we release a tor with that value to "3".
For at least one release:
- whatever the existing service uses, or
- 2 for new services (with 3 being an alternative),
For at least one more release:
- whatever the existing service uses (with 2 being deprecated),
- 3 for new services (with 2 being deprecated),
For at least one more release:
- whatever the existing service uses (with 2 being deprecated),
- 3 for new services (with 2 being deprecated, and only able to be created manually),
I really think that we want to _avoid_ the scenario where the user has a choice to create a service between v2 or v3. I don't see any pros of having a release that does v2 by default when v3 support is supported by the tor instance.
If v3 is available in tor, you use it. From my perspective, it's a security issue and shouldn't be a transition usability issue. v2 has to be considered "broken" that is less secure and as soon as your tor supports v3, it should be dumped. We should never give a choice to our users between an insecure protocol vs a secure one.
I also want us to aim for the smallest possible transition window between v2 and v3 meaning that once v3 is ready, we want that set of descriptors to go up relatively fast and not having 5 early adopters for months making it obvious who is who in the network. Giving a choice to the user to create v2 or v3 will slow that down for an already slow process and will partition it intensely.
For releases after that:
- make the default and only version 3.
Do we support the same service publishing version 2 and version 3 descriptors?
Or do we expect people to set up two different services and direct them to the same port?
This. Both version will live in parallel so as long as you have a tor that supports v2 and tor can find a RSA key, this is possible. My original email gives out an example on how to do such a thing.
Do we want to keep that option as a list for future use?
If I understand correctly your question, if let's say v4 comes in later on, the key will have a "version tag" so this is how tor knows what version your service is and then I propose we do the same, you point your HiddenServiceDir to the key folder and then you copy that if you want different version.
Either way, we should document that in the man page.
Yes and I would like also a transition guide for sure! Just the new .onion showing up in the hostname file, I predict confusion a lot :).
Since there's a default, most users will never set it, and they will just see the default for new services change in a particular release, and then another release where old services are deprecated, and then they will be gone entirely.
Phasing out v2 will be another problem but I don't forsee this until few years after v3 ... Maybe we could speed it up but the "Debian stable" factor will make it harder to do it fast.
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option:
- a developer wants to test that their software works with V3 onion services, before the default is changed,
- a user wants to run their old software with V2 onion services, after the default is changed.
Running v2 onion is fine, it's creating v2 onion that I don't want the user to control when that tor supports v3.
- The current plan so far is to add a consensus parameter that will control
the switch for client and service to start using prop224. And every relay out there that will run a tor supporting the v3 protocol (HSDir, IP, RP) will speak the protocol no matter what. So then we can decide when the network is mature enough for client/service to start talking actively that protocol.
That switch will also be an opportunity for us to remove v0 and v1 support as well from tor making that event "The Big Onion Service Switch" or for short "The BOSS (tm)" :P.
HOWEVER, the hard part here is getting that consensus param information about the state of prop224 _before_ configuring your HS.... it's doable but will weirdly make a specific tor version bootstrap HS much slower as they'll have to wait for the consensus to be downloaded.... maybe there is a better way?
Here is an idea, we could rely on a tor version for enabling the feature instead that is we release client/service support when and only when the network is mature so then there will be no confusion, you run version X, you get v3, period, the network is ready. That is again not ideal as we then delay this important new feature by some unknown chunks of 6 months... but at least we have some assurance that it's ready.
We can ship the code in let's say 031 but make tor only use it in 032 for instance.
…
I think the consensus parameter is more useful as an emergency on/off switch for descriptor publication, not service configuration.
True. Actually, in the long run, once v3 is stable and we feel comfortable with it, I would like us to phase out that consensus param somehow as a killswitch in the consensus is something that bothers me :S ... but at first it could save us from a catastrophe. :)
If the code exists on the client, a user should always be able to configure a v3 onion service. Even before the consensus is downloaded. Even if they do not have access to the network.
Agree.
So here's how I'd use the consensus parameter:
- if the consensus parameter says v3 services are disabled, users can configure a v3 service, but it will warn and refuse to publish its descriptor.
- if the consensus parameter says v3 services are enabled, users can configure a v3 service, and it will publish its descriptor. (And we should be able to deprecate v3 at some point in the future.)
And similarly for v2:
- if the consensus parameter says v2 services are enabled, users can configure a v2 service, and it will publish its descriptor.
- if the consensus parameter says v2 services are deprecated, users can configure a v2 service, and it will warn, and then publish its descriptor.
- if the consensus parameter says v2 services are disabled, users can configure a v2 service, but it will warn and refuse to publish its descriptor. (And some future Tor version will remove support for creating v2 services, and another version will remove support for loading them.)
I could see a consensus param for publication of v2 which could allow us to phase out v2 faster than waiting for a Tor release to be broadly used which takes years.
But I just want to reiterate that my strong opinion on this is that if v3 is available (let's use HiddenServiceVersion 3), tor should ONLY use that for new service. Which means it will be controlled by a tor release and that release could also remove v0 and v1 (The Big Switch :).
We should also consider how this interacts with the protocol lines in the consensus.
Service will pick HSDir/IP based on that. See #20571 for IP. We need to do the same for HSDir actually. Hopefully, we get all this in 030.
Thanks teor! Great comments! David
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 26 Nov. 2016, at 02:25, David Goulet dgoulet@ev0ke.net wrote:
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option:
- a developer wants to test that their software works with V3 onion services,
before the default is changed,
- a user wants to run their old software with V2 onion services, after the
default is changed.
Running v2 onion is fine, it's creating v2 onion that I don't want the user to control when that tor supports v3.
But what if I have a tor client that only supports v2 onion services, (because it has not been updated yet, or is not maintained,) and I want to use it with a version of tor that only supports v3 onion services? (because it has other security fixes I want)
I would prefer we release at least one version with v3 as the default, but that will still create v2 if asked nicely.
(Alternately, we can tell people how to create v2 manually, or we can tell people to keep their old version of tor and make it create v2.)
T
On Sat, Nov 26, 2016 at 09:25:44PM +1100, teor wrote:
On 26 Nov. 2016, at 02:25, David Goulet dgoulet@ev0ke.net wrote:
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option:
- a developer wants to test that their software works with V3 onion services,
before the default is changed,
- a user wants to run their old software with V2 onion services, after the
default is changed.
Running v2 onion is fine, it's creating v2 onion that I don't want the user to control when that tor supports v3.
But what if I have a tor client that only supports v2 onion services, (because it has not been updated yet, or is not maintained,) and I want to use it with a version of tor that only supports v3 onion services? (because it has other security fixes I want)
I have a little voice whispering in my head saying "this is a bad idea!".
Similarly, "This is not supported" is the ideal solution for this scenario, but being the realist that I am, *someone* will try this and it should work for them. Specifically, I'm thinking about the Jessie and Wheezy users who are using a system tor (as was mentioned earlier). I'm torn because they are doing it wrong, but at the same time, is it their fault? The various other distros are another nightmare. Luckily most users should be using Tor Browser/Messenger, but that doesn't help the edge cases.
I agree HiddenServiceVersion is useful for providing this transition, but I'm worried this option maybe more surprising than intended, and the manpage's description would need some changes. Currently, the implementation is only asserts the v2 is set (since 2009), so luckily backwards compatibility is not actually a concern. However, the description says:
A list of rendezvous service descriptor versions to publish for the hidden service.
This isn't what we'd want. This is actually the wrong side of tor from what is being discussed. The purpose of this option would change from the descriptor publication side to being something user-facing. Although this option is not currently useful, I worry about repurposing it for specifying how an onion service should be created. Yes, the onion-service-version and the version of the descriptor that tor publishes are now tightly coupled, comparing v2 and v3. However, this may not always be the case and, indeed, was not the case previously. I'm not saying HiddenServiceVersion is not the correct torrc option, but please keep these problems in mind. Also, realize that the current torrc syntax allows multiple versions (so it would allow tor create both a v2 and v3 descriptor, and if this is used for onion service creation, it could create both v2 and v3 rend services).
I would prefer we release at least one version with v3 as the default, but that will still create v2 if asked nicely.
(Alternately, we can tell people how to create v2 manually, or we can tell people to keep their old version of tor and make it create v2.)
Ugh. None of these are good, but releasing a tor version with v3 creation as the default and hiding v2 creation behind a torrc option seems like the best and least astonishing decision. I fully agree with David that this is not ideal and v2 should die sooner than possible. However, I'm not convinced this is realistically the best move.
Luckily this isn't my decision, so take my 2 cents however you want.</bikeshed>
- Matt
On 27 Nov (00:44:39), Matthew Finkel wrote:
On Sat, Nov 26, 2016 at 09:25:44PM +1100, teor wrote:
On 26 Nov. 2016, at 02:25, David Goulet dgoulet@ev0ke.net wrote:
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option:
- a developer wants to test that their software works with V3 onion services,
before the default is changed,
- a user wants to run their old software with V2 onion services, after the
default is changed.
Running v2 onion is fine, it's creating v2 onion that I don't want the user to control when that tor supports v3.
But what if I have a tor client that only supports v2 onion services, (because it has not been updated yet, or is not maintained,) and I want to use it with a version of tor that only supports v3 onion services? (because it has other security fixes I want)
Ok, I see your point. If some HS operator has a large client base and they are all v2 and that operator updates to a tor with v3, creates new HS well all the clients can't reach that new service so a "transition" period would be desirable.
To be honest, this is a tough one to answer.
I have a little voice whispering in my head saying "this is a bad idea!".
Similarly, "This is not supported" is the ideal solution for this scenario, but being the realist that I am, *someone* will try this and it should work for them. Specifically, I'm thinking about the Jessie and Wheezy users who are using a system tor (as was mentioned earlier). I'm torn because they are doing it wrong, but at the same time, is it their fault? The various other distros are another nightmare. Luckily most users should be using Tor Browser/Messenger, but that doesn't help the edge cases.
Right, that's is an important point. If we do a switch like such that is we release a tor that only allows v3 service creation, we _need_ to make sure Tor Messenger and Browser and Tor Birdy have been released with a v3 client before that. Else, this partitioning of v2 client vs v3 service won't be fun for anyone...
I agree HiddenServiceVersion is useful for providing this transition, but I'm worried this option maybe more surprising than intended, and the manpage's description would need some changes. Currently, the implementation is only asserts the v2 is set (since 2009), so luckily backwards compatibility is not actually a concern. However, the description says:
A list of rendezvous service descriptor versions to publish for the hidden service.
This isn't what we'd want. This is actually the wrong side of tor from what is being discussed. The purpose of this option would change from the descriptor publication side to being something user-facing. Although this option is not currently useful, I worry about repurposing it for specifying how an onion service should be created.
Wait, the code makes that option specific to a HiddenServiceDir so if that value changes to "3", the configured HS has to be considered to be a v3 or creates it for v3. So I think using it is fine.
So, if we were to switch that value to 3 at once in a tor release, all new HS will be v3. The question would be after that if we also allow "2" as a possible value.
Yes, the onion-service-version and the version of the descriptor that tor publishes are now tightly coupled, comparing v2 and v3. However, this may not always be the case and, indeed, was not the case previously. I'm not saying HiddenServiceVersion is not the correct torrc option, but please keep these problems in mind. Also, realize that the current torrc syntax allows multiple versions (so it would allow tor create both a v2 and v3 descriptor, and if this is used for onion service creation, it could create both v2 and v3 rend services).
That's another possible thing we could do but it has some possible UX consequences. Let's say we have a tor version that by default creates both version for a configured HS. You would end up with two addresses for the same service and then the operator broadcast those somehow to their users.
I have this feeling that it could be more of a mayhem of confusion for clients. The 16 char onion will always work but then clients trying the 52 char onion (because they were told it's MORE secure) might fail not realizing that their client is v2 only... and then we end up with all clients using the v2 address. What will happen at the next tor version which will phase out v2? We end up with the same problem again... old clients and so on.
However, the expectation by then would be that we gave enough time to clients to upgrade but on that I have my doubts. We can make sure that TBB/TM/TB have v3 support by the time we roll out v3, we control that. After that, we rely on distros so let's do this exercise with Debian stable. We won't get v3 client everywhere until _after_ Stretch which has not even been released yet and their freeze is around Febuary in theory making it impossible to have v3 in that release considering the amount of work and pace we are implementing this. Meaning, we'll have to wait another 2 years (2019+) at the very least after that plus the time for people to actually do the OS upgrade...
So all in all, the "client upgrading" transition imo only makes sense if we make sure that TBB/TM/TB supports it and after this is out of our control. Really, v3 is a _security_ upgrade so clients have to upgrade at some point, we can't wait for distros... even if it means more pain for some users. We offer a nice deb.torproject.org alternative for this so users aren't let hanging in the desert.
I would prefer we release at least one version with v3 as the default, but that will still create v2 if asked nicely.
(Alternately, we can tell people how to create v2 manually, or we can tell people to keep their old version of tor and make it create v2.)
Ugh. None of these are good, but releasing a tor version with v3 creation as the default and hiding v2 creation behind a torrc option seems like the best and least astonishing decision. I fully agree with David that this is not ideal and v2 should die sooner than possible. However, I'm not convinced this is realistically the best move.
Ok, we discussed lots of things so far. Very GOOD feedback from everyone on the thread! I think we all realize that we are in a situation that whatever the solution, it won't be the ideal and perfect one!
What I'll do is reboot the thread replying to the original message with few points of what was 1) agreed upon and 2) one or two questions that are left for us to answer so we can move forward here as the implementation is kind of ongoing right now.
Thanks! David
Luckily this isn't my decision, so take my 2 cents however you want.</bikeshed>
- Matt
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Hi,
Yes, the onion-service-version and the version of the descriptor that tor publishes are now tightly coupled, comparing v2 and v3. However, this may not always be the case and, indeed, was not the case previously. I'm not saying HiddenServiceVersion is not the correct torrc option, but please keep these problems in mind. Also, realize that the current torrc syntax allows multiple versions (so it would allow tor create both a v2 and v3 descriptor, and if this is used for onion service creation, it could create both v2 and v3 rend services).
That's another possible thing we could do but it has some possible UX consequences. Let's say we have a tor version that by default creates both version for a configured HS. You would end up with two addresses for the same service and then the operator broadcast those somehow to their users.
I think the "multiple address window" is something we should try to minimize, and consider how to avoid entirely for future versions if possible, especially if multiple versions could be supported at once (v3, v4, and v5, for example).
As an operator, I would need to publish multiple addresses for the same service, and communicate which address corresponds to which version.
As an end user, I will either need to 1) try different addresses until one succeeds, or 2) know my current version and use this knowledge when selecting which address to use. Sharing addresses could also be difficult (I might try to share a v3 address with my friend, but it doesn't work for them because they are on v2). It might be good to do user testing to see what other edge scenarios come up, what is hard/confusing, etc.
Maybe it's impossible to avoid this for the v2 to v3 transition, but minimizing this seems like a good thing to do!
All best, Chelsea
On Sun, Nov 27, 2016 at 01:07:02PM -0500, chelsea komlo wrote:
Hi,
Yes, the onion-service-version and the version of the descriptor that tor publishes are now tightly coupled, comparing v2 and v3. However, this may not always be the case and, indeed, was not the case previously. I'm not saying HiddenServiceVersion is not the correct torrc option, but please keep these problems in mind. Also, realize that the current torrc syntax allows multiple versions (so it would allow tor create both a v2 and v3 descriptor, and if this is used for onion service creation, it could create both v2 and v3 rend services).
That's another possible thing we could do but it has some possible UX consequences. Let's say we have a tor version that by default creates both version for a configured HS. You would end up with two addresses for the same service and then the operator broadcast those somehow to their users.
I think the "multiple address window" is something we should try to minimize, and consider how to avoid entirely for future versions if possible, especially if multiple versions could be supported at once (v3, v4, and v5, for example).
As an operator, I would need to publish multiple addresses for the same service, and communicate which address corresponds to which version.
As an end user, I will either need to 1) try different addresses until one succeeds, or 2) know my current version and use this knowledge when selecting which address to use. Sharing addresses could also be difficult (I might try to share a v3 address with my friend, but it doesn't work for them because they are on v2). It might be good to do user testing to see what other edge scenarios come up, what is hard/confusing, etc.
Yeah, this is a usability nightmare, but this is a tradeoff we have by running both versions in parallel on the network. Hopefully the majority of the clients (as David said) will upgrade to a tor version that support v3 very soon after it's stable, but the edge cases are always the hardest.
System tors are potential landmines. Maybe we should consider planning a public push for Debian/Ubuntu users configuring the deb.tp.o apt source *if* they use a system tor. I worry this may confuse some Tor Browser users who use Debian or Ubuntu, so we'd somehow need to be clear that this is only if they installed tor locally.
Maybe it's impossible to avoid this for the v2 to v3 transition, but minimizing this seems like a good thing to do!
This is definitely something we should start thinking about now/soon before we start planning v4. It's easy to do "don't release v4 client support until x% of the relays support it", but we don't have any idea of when x% of clients that run onion services upgrade, and this may be something we want to seed, as well: not only relay support, but service creation/publication support. Maybe this is something the you should consider doing for v3, too. Start releasing tor versions with v3 relay (IntroPoint, HSDir, etc) support before client support, but also release the service-side support early, too. With this services can start upgrading and publishing descriptors months before clients have the functionality needed for using the new v3 addresses. This could reduce the inherent partitioning and it could minimize the thundering herd of services switching from v2 to v3 addresses all at once. But maybe this is something you already discussed, if so then great!
- Matt
On Sun, Nov 27, 2016 at 11:24:11AM -0500, David Goulet wrote:
On 27 Nov (00:44:39), Matthew Finkel wrote:
On Sat, Nov 26, 2016 at 09:25:44PM +1100, teor wrote:
On 26 Nov. 2016, at 02:25, David Goulet dgoulet@ev0ke.net wrote:
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option:
- a developer wants to test that their software works with V3 onion services,
before the default is changed,
- a user wants to run their old software with V2 onion services, after the
default is changed.
Running v2 onion is fine, it's creating v2 onion that I don't want the user to control when that tor supports v3.
But what if I have a tor client that only supports v2 onion services, (because it has not been updated yet, or is not maintained,) and I want to use it with a version of tor that only supports v3 onion services? (because it has other security fixes I want)
Ok, I see your point. If some HS operator has a large client base and they are all v2 and that operator updates to a tor with v3, creates new HS well all the clients can't reach that new service so a "transition" period would be desirable.
To be honest, this is a tough one to answer.
I have a little voice whispering in my head saying "this is a bad idea!".
Similarly, "This is not supported" is the ideal solution for this scenario, but being the realist that I am, *someone* will try this and it should work for them. Specifically, I'm thinking about the Jessie and Wheezy users who are using a system tor (as was mentioned earlier). I'm torn because they are doing it wrong, but at the same time, is it their fault? The various other distros are another nightmare. Luckily most users should be using Tor Browser/Messenger, but that doesn't help the edge cases.
Right, that's is an important point. If we do a switch like such that is we release a tor that only allows v3 service creation, we _need_ to make sure Tor Messenger and Browser and Tor Birdy have been released with a v3 client before that. Else, this partitioning of v2 client vs v3 service won't be fun for anyone...
Yeah, luckily that is a much easier transition that can be coordinated. I'm not very concerned about including tor-0.3.x in Tor {Browser,Messenger,Birdy}. Personally, I view that as a relatively simple discussion that I hope won't get much push back.
I agree HiddenServiceVersion is useful for providing this transition, but I'm worried this option maybe more surprising than intended, and the manpage's description would need some changes. Currently, the implementation is only asserts the v2 is set (since 2009), so luckily backwards compatibility is not actually a concern. However, the description says:
A list of rendezvous service descriptor versions to publish for the hidden service.
This isn't what we'd want. This is actually the wrong side of tor from what is being discussed. The purpose of this option would change from the descriptor publication side to being something user-facing. Although this option is not currently useful, I worry about repurposing it for specifying how an onion service should be created.
Wait, the code makes that option specific to a HiddenServiceDir so if that value changes to "3", the configured HS has to be considered to be a v3 or creates it for v3. So I think using it is fine.
So, if we were to switch that value to 3 at once in a tor release, all new HS will be v3. The question would be after that if we also allow "2" as a possible value.
Correct, I agree. I don't think I was precise when I said this was on the wrong side of tor. I was splitting tor in half, the user-facing side included the parts of tor with which users see and interact, like the onion address. If this changes then this is a user-facing change that could affect user experience. The other side of tor is the tor network, and how tor clients connect and interact with tor relays. Users shouldn't see this side or care how it works. HiddenServiceVersion was originally used as an option that controls what format(s) a tor client uses when it publishes a rend descriptor for a specific onion service. If HiddenServiceVersion is modified so it creates a v2 and/or v3 onion service, then now it has a user-facing effect and controls whether a user gets an 16 character address or 52 character. I don't think this is a problem, but a change like this will need to be well documented and (I think) explicitly included in the changelog.
Yes, the onion-service-version and the version of the descriptor that tor publishes are now tightly coupled, comparing v2 and v3. However, this may not always be the case and, indeed, was not the case previously. I'm not saying HiddenServiceVersion is not the correct torrc option, but please keep these problems in mind. Also, realize that the current torrc syntax allows multiple versions (so it would allow tor create both a v2 and v3 descriptor, and if this is used for onion service creation, it could create both v2 and v3 rend services).
That's another possible thing we could do but it has some possible UX consequences. Let's say we have a tor version that by default creates both version for a configured HS. You would end up with two addresses for the same service and then the operator broadcast those somehow to their users.
I have this feeling that it could be more of a mayhem of confusion for clients. The 16 char onion will always work but then clients trying the 52 char onion (because they were told it's MORE secure) might fail not realizing that their client is v2 only... and then we end up with all clients using the v2 address. What will happen at the next tor version which will phase out v2? We end up with the same problem again... old clients and so on.
Oh yes, I do not believe we want users creating both v2 and v3 addresses. I only wanted to point out that the current HiddenServiceVersion syntax *allows* specifying multiple versions.
However, the expectation by then would be that we gave enough time to clients to upgrade but on that I have my doubts. We can make sure that TBB/TM/TB have v3 support by the time we roll out v3, we control that. After that, we rely on distros so let's do this exercise with Debian stable. We won't get v3 client everywhere until _after_ Stretch which has not even been released yet and their freeze is around Febuary in theory making it impossible to have v3 in that release considering the amount of work and pace we are implementing this. Meaning, we'll have to wait another 2 years (2019+) at the very least after that plus the time for people to actually do the OS upgrade...
So all in all, the "client upgrading" transition imo only makes sense if we make sure that TBB/TM/TB supports it and after this is out of our control. Really, v3 is a _security_ upgrade so clients have to upgrade at some point, we can't wait for distros... even if it means more pain for some users. We offer a nice deb.torproject.org alternative for this so users aren't let hanging in the desert.
Yep. I don't think we have a better option than that. This is a hard problem that we've run into many times before (and will likely continue running into).
I would prefer we release at least one version with v3 as the default, but that will still create v2 if asked nicely.
(Alternately, we can tell people how to create v2 manually, or we can tell people to keep their old version of tor and make it create v2.)
Ugh. None of these are good, but releasing a tor version with v3 creation as the default and hiding v2 creation behind a torrc option seems like the best and least astonishing decision. I fully agree with David that this is not ideal and v2 should die sooner than possible. However, I'm not convinced this is realistically the best move.
Ok, we discussed lots of things so far. Very GOOD feedback from everyone on the thread! I think we all realize that we are in a situation that whatever the solution, it won't be the ideal and perfect one!
What I'll do is reboot the thread replying to the original message with few points of what was 1) agreed upon and 2) one or two questions that are left for us to answer so we can move forward here as the implementation is kind of ongoing right now.
Thanks! David
Luckily this isn't my decision, so take my 2 cents however you want.</bikeshed>
- Matt
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 28 Nov. 2016, at 03:24, David Goulet dgoulet@ev0ke.net wrote:
On 27 Nov (00:44:39), Matthew Finkel wrote:
On Sat, Nov 26, 2016 at 09:25:44PM +1100, teor wrote:
On 26 Nov. 2016, at 02:25, David Goulet dgoulet@ev0ke.net wrote:
- It also means we create an option that will get deprecated once v2 is
phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
Not if we use the existing HiddenServiceVersion option for this.
Here are the use cases for this option:
- a developer wants to test that their software works with V3 onion services,
before the default is changed,
- a user wants to run their old software with V2 onion services, after the
default is changed.
Running v2 onion is fine, it's creating v2 onion that I don't want the user to control when that tor supports v3.
But what if I have a tor client that only supports v2 onion services, (because it has not been updated yet, or is not maintained,) and I want to use it with a version of tor that only supports v3 onion services? (because it has other security fixes I want)
Ok, I see your point. If some HS operator has a large client base and they are all v2 and that operator updates to a tor with v3, creates new HS well all the clients can't reach that new service so a "transition" period would be desirable.
To be honest, this is a tough one to answer.
Also, what if the software or library you are using to create onion services only supports v2?
What if the program that creates onion services for you only supports v2?
What if the program uses v2 addresses as a persistent, self-authenticating identifier? (Ricochet is a good example of all these issues.)
...
Yes, the onion-service-version and the version of the descriptor that tor publishes are now tightly coupled, comparing v2 and v3. However, this may not always be the case and, indeed, was not the case previously. I'm not saying HiddenServiceVersion is not the correct torrc option, but please keep these problems in mind. Also, realize that the current torrc syntax allows multiple versions (so it would allow tor create both a v2 and v3 descriptor, and if this is used for onion service creation, it could create both v2 and v3 rend services).
That's another possible thing we could do but it has some possible UX consequences. Let's say we have a tor version that by default creates both version for a configured HS. You would end up with two addresses for the same service and then the operator broadcast those somehow to their users.
I have this feeling that it could be more of a mayhem of confusion for clients. The 16 char onion will always work but then clients trying the 52 char onion (because they were told it's MORE secure) might fail not realizing that their client is v2 only…
So add a warning message *now* (to 0.3.0) that recognises v3 addresses and logs a sensible message?
And discuss with Tor Browser etc. how to show it in the UI?
and then we end up with all clients using the v2 address. What will happen at the next tor version which will phase out v2? We end up with the same problem again... old clients and so on.
Yes, there is a need for a transition period where both v2 and v3 work.
T
Hi,
- It also means we create an option that will get deprecated once v2 is phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
It could be useful to have a standard transition/upgrade path when releasing new versions after v3. If we know that in the future we will have v4, v5, etc, we could design a more general process so users know what to expect, can plan for future releases, etc.
All best, Chelsea
On Thu, Nov 24, 2016 at 08:14:55PM -0500, ckomlo@thoughtworks.com wrote:
Hi,
- It also means we create an option that will get deprecated once v2 is phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
It could be useful to have a standard transition/upgrade path when releasing new versions after v3. If we know that in the future we will have v4, v5, etc, we could design a more general process so users know what to expect, can plan for future releases, etc.
Very true, it would be very useful. Many moons ago, when we first began discussing how the network would switch over to using (what is now) v3, we discussed having a flag day or a hard cutover between versions, or a soft migration. Previous us decided this was a hard problem and was a discussion for a later date. :)
Unfortunately I've lost track of how close v3 is to being ready. If it's as close as David makes it sound, then I think you're right that this specific discussion may be a little too late for v3, but really is something that should be prepared for v4+.
Hi,
David Goulet wrote:
On 24 Nov (11:13:06), teor wrote:
On 24 Nov. 2016, at 11:04, Yawning Angel yawning@schwanenlied.me wrote:
On Thu, 24 Nov 2016 01:43:15 +0200 s7r s7r@sky-ip.org wrote:
I agree that this would be "the technical way" to do it, but real world usability kind of prevents us to do it this way. The spec for ADD_ONION indeed does not say that v2 hidden services will be supported forever and it clearly SHOULD NOT, but it also doesn't make much sense to abolish it at the first Tor release supporting v3 services (because if we make ADD_ONION == v3 (best) this is what we are doing).
Even I don't think `BEST` should be changed to Ed25519 immediately, especially when the code is being stabilized.
So I think we should have an option:
OnionServiceCreateV3 0|1
Few things why I'm not too keen with this even though it could actually be very useful... I'll outline the potential issues here with such an option but unfortunately I don't have a perfect solution in the end. My main concern is not really an easy or intuitive way for the user to start using v3 service but rather when the users will start using it, the network MUST be ready for it which makes things a bit more complicated to do properly I believe.
Considering a torrc option such as "OnionServiceCreateV3 0|1":
- This is extremely non intuitive to any operator even power users. The concept of "v3" is something that very few people will understand what it's actually is. Even if we go with "OnionServiceEpicNextMoreSecureGeneration 1", we are creating two classes of onion service that will make things more confusing for the user with the question of "which one should I use?". Also see 3 below for more confusion.
I agree. We should avoid adding new torrc options with different confusing and potentially expiring versions at the end. This is why I suggested, for v3 ed25519 key based authenticated clients we will not add a torrc option, but instead we change the auth-type for HiddenServiceAuthorizeClient, example:
HiddenServiceAuthorizeClient pubkey /var/lib/tor/hs-name/authclient.dat
Where Tor will know to interpret 'pubkey' for only v3 obviously.
- It also means we create an option that will get deprecated once v2 is phased out so we are adding a "temporary" option for users to "keep creating v2 addresses" but then will be useless and we'll deprecate and get a whole lot confusing if for instance v4 or v5 happens in the next years for X reasons (unlikely but not impossible).
This is very correct.
The current plan so far is to add a consensus parameter that will control the switch for client and service to start using prop224. And every relay out there that will run a tor supporting the v3 protocol (HSDir, IP, RP) will speak the protocol no matter what. So then we can decide when the network is mature enough for client/service to start talking actively that protocol.
That switch will also be an opportunity for us to remove v0 and v1 support as well from tor making that event "The Big Onion Service Switch" or for short "The BOSS (tm)" :P.
HOWEVER, the hard part here is getting that consensus param information about the state of prop224 _before_ configuring your HS.... it's doable but will weirdly make a specific tor version bootstrap HS much slower as they'll have to wait for the consensus to be downloaded.... maybe there is a better way?
Here is an idea, we could rely on a tor version for enabling the feature instead that is we release client/service support when and only when the network is mature so then there will be no confusion, you run version X, you get v3, period, the network is ready. That is again not ideal as we then delay this important new feature by some unknown chunks of 6 months... but at least we have some assurance that it's ready.
We can ship the code in let's say 031 but make tor only use it in 032 for instance.
Thoughts?
David
teor's previous post on this thread is a good idea, to use HiddenServiceVersion. I am afraid there's no way to do it without a consensus param and little more bootstrap needed before HS is created for the benefit of not delaying the feature by some unknown time.
The HiddenServiceVersion could have priorities like:
0: value from torrc 2: default one for the Tor version, which is: a) whatever the instance is running, if it already has a HS key in the configured HiddenServiceDir. b) the consensus param c) if no consensus param, 2, until a specific version when we see the network is ready for v3. This version will have the default set to 3.
Since this version, v2 cannot be created any more nor existing keys for v2 will work any more - we have all the reasons to push for faster v3 adoption.
It becomes quite complicated if we want to allow people hosting v2 and v3 simultaneously on the same instance, so we should think carefully if we want to allow this or not.
On 11/23/2016 07:04 PM, Yawning Angel wrote:
Our fix: "Add another command, that does essentially the same thing, because people picked the wrong options, then later deal with the fallout from people getting used to the temporary command, and crying when it's deprecated."
I say "they should fix their code".
This issue with incorrect implementations reminds me of the compatibility issues that cause issues with new SSL/TLS standards. These implementations led to compatibility workarounds that introduced security issues that had to be eventually fixed by TLS_FALLBACK_SCSV.
It's not our problem if their code breaks because they made incorrect assumptions regarding the standard. The polite thing would be to submit a patch so that Bitcoin nodes can update before we make the change.
-- Jesse
On 11/24/2016 2:24 AM, Jesse V wrote:
On 11/23/2016 07:04 PM, Yawning Angel wrote:
Our fix: "Add another command, that does essentially the same thing, because people picked the wrong options, then later deal with the fallout from people getting used to the temporary command, and crying when it's deprecated."
I say "they should fix their code".
This issue with incorrect implementations reminds me of the compatibility issues that cause issues with new SSL/TLS standards. These implementations led to compatibility workarounds that introduced security issues that had to be eventually fixed by TLS_FALLBACK_SCSV.
It's not our problem if their code breaks because they made incorrect assumptions regarding the standard. The polite thing would be to submit a patch so that Bitcoin nodes can update before we make the change.
-- Jesse
Right, which is why I have already submitted just now all the relevant details and I'll keep an eye that everyone is compatible with both v2 and v3 for the transition period to also maintain working configurations.
Given that I filed it now, this should give sufficient time to be ready until our first release supporting v3.
To be frank, the spec linked to me by teor for ADD_ONION clearly states the difference between :BEST and :RSA1024 so Yawning was right from the first place.
Hello David,
David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
Instead of listing all options and going in an endless loop of possibilities, I'll outline what we've discussed so far between some of us. In the following, "v2" is current hidden service and "v3" is the next generation detailed in prop224.
Once v3 is released, from that point on _no_ v2 service will be allowed to be created by "tor" itself. It will always be possible to do it by hand by creating an RSA key and putting it in the service directory (see 3 below).
The reasoning for this is that we really want to phase out as quickly as possible the v2 protocol for privacy and security reasons.
Perfect. Fully agree.
- All the current torrc options will be kept for v3 as they all apply in terms of format. (One exception might be the client authorization.)
Very good. We can add a new torrc option for ed25519 key based authentication for clients (v3) so the current HiddenServiceAuthorizeClient (v2) will not break old torrc's until everyone upgrades.
However, we could just add an auth-type value after HiddenServiceAuthorizeClient that will only work for v3. For example:
HiddenServiceAuthorizeClient pubkey client-name,client-name,client-name,...
It's obvious where 'pubkey' comes from, but it can be changed to anything. This way we preserve the option that it's already common for users (HiddenServiceAuthorizeClient), we maintain compatibility of torrcs during transition period and also have the same option in the future when we are fully migrated to v3.
This is perfectly fine, since we do not use any auth method that works on v2 to v3 (which I am also super happy with). We also don't restrict ourselves from the possibility to add new auth-types for v3 in the future, should we need them.
When tor is loading config for a service:
if a v2 key is found that is an RSA key, we treat that service as v2, print a deprecation warning and continue.
if a v3 key is found, use v3, all is good.
if no key is found, consider a new service and generate v3 (ties to 1 above).
Great.
Over time, extra option might appear and they will be ONLY for v3 unless for force majeur security madness.
One of those options that we want to add is this one as offline key will be a reality with v3:
"HiddenServiceOfflineKey 1"
It will indicate tor to _not_ generate the master identity key and will require the user to put the public key in the HiddenServiceDir.
Note that with all of the above it will be possible to have one of your application on both v2 and v3 address. Here is a snippet as an example:
# v2 HiddenServiceDir /srv/hs/v2 HiddenServicePort 80 localhost:80 # v3 HiddenServiceDir /srv/hs/v3 HiddenServicePort 80 localhost:80The important part here is the different directories but the port points to the same place.
Ok here it is. Please comment, improve, or propose! :)
Cheers! David
[1] https://www.torproject.org/docs/tor-manual.html.en (see HIDDEN SERVICE OPTIONS)
Yes, this looks good to me as well. As for ADD_ONION, I think we should give the people that use it automatically for other apps a change until they upgrade to be compatible with v3, so for the transition period as long as v2 is supported (but not recommended) in the network, ADD_ONION controller command will create a v2 service, and we add another controller command: ADD_ONIONV3 that will create v3 services. When v2 is permanently gone, we make ADD_ONION a synonym to ADD_ONIONV3, remove the RSA1024/SHA1 code and we're all set.
On 24 Nov. 2016, at 10:18, s7r s7r@sky-ip.org wrote:
Hello David,
David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
Instead of listing all options and going in an endless loop of possibilities, I'll outline what we've discussed so far between some of us. In the following, "v2" is current hidden service and "v3" is the next generation detailed in prop224. ...
- All the current torrc options will be kept for v3 as they all apply in
terms of format. (One exception might be the client authorization.)
Very good. We can add a new torrc option for ed25519 key based authentication for clients (v3) so the current HiddenServiceAuthorizeClient (v2) will not break old torrc's until everyone upgrades.
However, we could just add an auth-type value after HiddenServiceAuthorizeClient that will only work for v3. For example:
HiddenServiceAuthorizeClient pubkey client-name,client-name,client-name,...
It's obvious where 'pubkey' comes from, but it can be changed to anything. This way we preserve the option that it's already common for users (HiddenServiceAuthorizeClient), we maintain compatibility of torrcs during transition period and also have the same option in the future when we are fully migrated to v3.
This is perfectly fine, since we do not use any auth method that works on v2 to v3 (which I am also super happy with). We also don't restrict ourselves from the possibility to add new auth-types for v3 in the future, should we need them.
This means that people who upgrade to v3 services will need to redistribute all their client auth keys. This could be a lot of work, particularly if the out-of-band channel is manual, or there are a large number of users.
Here's an alternative proposal:
Tor accepts v2 HiddenServiceAuthorizeClient and HidServAuth for v3 services, then feeds the auth-cookie into a key derivation function to produce an ed25519 private key. This key is used for authentication as specified.
I don't think this is worth the extra coding effort - I think we are better to focus on getting people to upgrade, and designing the new options to be easy to use.
But I could be convinced if a hidden service operator who uses client authentication says it would take a lot of effort to re-issue client keys.
(That said, I assume operators would re-issue keys the same way they send the updated v3 onion address.)
...
T
teor wrote:
Very good. We can add a new torrc option for ed25519 key based authentication for clients (v3) so the current HiddenServiceAuthorizeClient (v2) will not break old torrc's until everyone upgrades.
However, we could just add an auth-type value after HiddenServiceAuthorizeClient that will only work for v3. For example:
HiddenServiceAuthorizeClient pubkey client-name,client-name,client-name,...
It's obvious where 'pubkey' comes from, but it can be changed to anything. This way we preserve the option that it's already common for users (HiddenServiceAuthorizeClient), we maintain compatibility of torrcs during transition period and also have the same option in the future when we are fully migrated to v3.
This is perfectly fine, since we do not use any auth method that works on v2 to v3 (which I am also super happy with). We also don't restrict ourselves from the possibility to add new auth-types for v3 in the future, should we need them.
This means that people who upgrade to v3 services will need to redistribute all their client auth keys. This could be a lot of work, particularly if the out-of-band channel is manual, or there are a large number of users.
Here's an alternative proposal:
Tor accepts v2 HiddenServiceAuthorizeClient and HidServAuth for v3 services, then feeds the auth-cookie into a key derivation function to produce an ed25519 private key. This key is used for authentication as specified.
This is not specified anywhere until now, my suggestion was based on the current form of the proposals. For v3 we only use ed25519 key based authentication, and if we use:
HiddenServiceAuthroizeClient pubkey client1,client2
Tor will use this only for v3 of course, while:
HiddenServiceAuthorizeClient basic client1,client2 will only be used for v2.
If an user upgrades to v3, it has no other but to generate the new authentication credentials (keys) for clients, regardless of the torrc option we use.
I don't think this is worth the extra coding effort - I think we are better to focus on getting people to upgrade, and designing the new options to be easy to use.
But I could be convinced if a hidden service operator who uses client authentication says it would take a lot of effort to re-issue client keys.
No, no, it is not worth it *at all*. The biggest headache is in distributing the private keys out of band to authorized clients, not their generation (generation is done by Tor automatically, in few seconds, based on just one line with the client names). And we have no way to skip the distribution phase, better focus to make the key management as simple as possible rather than coding a derivation function that (maybe) provides a false sense of security if the passphrases have low entropy.
On 24 Nov. 2016, at 10:56, s7r s7r@sky-ip.org wrote:
teor wrote:
Very good. We can add a new torrc option for ed25519 key based authentication for clients (v3) so the current HiddenServiceAuthorizeClient (v2) will not break old torrc's until everyone upgrades.
However, we could just add an auth-type value after HiddenServiceAuthorizeClient that will only work for v3. For example:
HiddenServiceAuthorizeClient pubkey client-name,client-name,client-name,...
It's obvious where 'pubkey' comes from, but it can be changed to anything. This way we preserve the option that it's already common for users (HiddenServiceAuthorizeClient), we maintain compatibility of torrcs during transition period and also have the same option in the future when we are fully migrated to v3.
This is perfectly fine, since we do not use any auth method that works on v2 to v3 (which I am also super happy with). We also don't restrict ourselves from the possibility to add new auth-types for v3 in the future, should we need them.
This means that people who upgrade to v3 services will need to redistribute all their client auth keys. This could be a lot of work, particularly if the out-of-band channel is manual, or there are a large number of users.
Here's an alternative proposal:
Tor accepts v2 HiddenServiceAuthorizeClient and HidServAuth for v3 services, then feeds the auth-cookie into a key derivation function to produce an ed25519 private key. This key is used for authentication as specified.
This is not specified anywhere until now, my suggestion was based on the current form of the proposals. For v3 we only use ed25519 key based authentication, and if we use:
HiddenServiceAuthroizeClient pubkey client1,client2
Tor will use this only for v3 of course, while:
HiddenServiceAuthorizeClient basic client1,client2 will only be used for v2.
If an user upgrades to v3, it has no other but to generate the new authentication credentials (keys) for clients, regardless of the torrc option we use.
I don't think this is worth the extra coding effort - I think we are better to focus on getting people to upgrade, and designing the new options to be easy to use.
But I could be convinced if a hidden service operator who uses client authentication says it would take a lot of effort to re-issue client keys.
No, no, it is not worth it *at all*. The biggest headache is in distributing the private keys out of band to authorized clients
Which my proposal avoids, by re-using the v2 keys for v3 services.
, not their generation (generation is done by Tor automatically, in few seconds, based on just one line with the client names). And we have no way to skip the distribution phase, better focus to make the key management as simple as possible rather than coding a derivation function that (maybe) provides a false sense of security if the passphrases have low entropy.
The v3 keys derived from the existing v2 keys would have the same entropy as those keys: 16 bytes (REND_DESC_COOKIE_LEN).
Yes, this is inferior to the ~31 bytes of entropy in an ed25519 key.
T
On Thu, 24 Nov 2016 01:18:46 +0200 s7r s7r@sky-ip.org wrote:
Yes, this looks good to me as well. As for ADD_ONION, I think we should give the people that use it automatically for other apps a change until they upgrade to be compatible with v3, so for the transition period as long as v2 is supported (but not recommended) in the network, ADD_ONION controller command will create a v2 service, and we add another controller command: ADD_ONIONV3 that will create v3 services. When v2 is permanently gone, we make ADD_ONION a synonym to ADD_ONIONV3, remove the RSA1024/SHA1 code and we're all set.
What. Why. Anyone right now, that explicitly wants a v2 service going forward, should use `ADD_ONION` correctly. It takes the type of key for a reason.
Regards,
On 22 Nov (17:36:33), David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
1) When v3 is released in a tor stable version, it will NOT be the default version for new service, v2 will until maturity.
2) HiddenServiceVersion is used to control which version of the service you want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
3) ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is with the tor you have.
4) In order to avoid relying on a tor stable version release to switch the default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change.
5) v2 creation will be allowed pass the "v3 Network/Code Maturity" point until we phase it out for real from the code. "HiddenServiceVersion 2" will be how to do it. I've been convinced actually and mostly the main argument that made me tick is "If we don't allow it, some hackish tor or ugly recipe will be recommended on the intertubez."
6) We DO NOT want a v3 kill switch or "do not publish descriptor" parameter in the consensus. We couldn't see a useful use case apart from "Oh damn that HS v3 code is DDoSing relays" which would require a real kill switch for v3 and not just "do not publish".
A correct alternative we think is to actually have consensus params that tweak the protocol such as "Num intro point default" and "Lifetime intropoint default" and such... (there are few of those magic numbers in the code) and in case of catastrophic failure, we can tweak the knob that is causing the problem. I'll open a ticket about it if no arguable objections on this thread.
That's about it!
Please raise concerns on any point here. I personally think it's simple enough and gives us enough room to transition over time.
Thanks to everyone with your feedbacks btw, really good stuff! More prop224 email will be in an INBOX near you in the coming weeks/months as we have more points to discuss.
Cheers! David
On 6 Dec. 2016, at 07:42, David Goulet dgoulet@ev0ke.net wrote:
On 22 Nov (17:36:33), David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor
with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network
has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of
time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
- When v3 is released in a tor stable version, it will NOT be the default
version for new service, v2 will until maturity.
- HiddenServiceVersion is used to control which version of the service you
want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
- ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is
with the tor you have.
- In order to avoid relying on a tor stable version release to switch the
default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change. ...
This is problematic: we validate and create onion services at torrc load, before loading the consensus.
Do we plan to do it afterwards? That seems like it could be really confusing.
T
On 06 Dec (07:53:16), teor wrote:
On 6 Dec. 2016, at 07:42, David Goulet dgoulet@ev0ke.net wrote:
On 22 Nov (17:36:33), David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor
with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network
has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of
time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
- When v3 is released in a tor stable version, it will NOT be the default
version for new service, v2 will until maturity.
- HiddenServiceVersion is used to control which version of the service you
want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
- ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is
with the tor you have.
- In order to avoid relying on a tor stable version release to switch the
default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change. ...
This is problematic: we validate and create onion services at torrc load, before loading the consensus.
Do we plan to do it afterwards? That seems like it could be really confusing.
You need a consensus anyway to pick your intro point or HSDir but yes the engineering part here will be that if you ever configured v2 service and you realize it's v3, we'll rollback all the things.
I agree with you that it could be confusing and will require some fun engineering but I think the advantages of having such a thing outweight the engineering effort. Unfun part will be on the user side as they will think their service is ready to start but then will have a warning after consensus download that it's not.
It also means we need to do key creation _after_ consensus download so we don't expose the wrong address to the user.
Fun... Maybe we can do better, not sure :S
David
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 6 Dec. 2016, at 08:02, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (07:53:16), teor wrote:
On 6 Dec. 2016, at 07:42, David Goulet dgoulet@ev0ke.net wrote:
On 22 Nov (17:36:33), David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor
with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network
has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of
time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
- When v3 is released in a tor stable version, it will NOT be the default
version for new service, v2 will until maturity.
- HiddenServiceVersion is used to control which version of the service you
want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
- ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is
with the tor you have.
- In order to avoid relying on a tor stable version release to switch the
default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change. ...
This is problematic: we validate and create onion services at torrc load, before loading the consensus.
Do we plan to do it afterwards? That seems like it could be really confusing.
You need a consensus anyway to pick your intro point or HSDir but yes the engineering part here will be that if you ever configured v2 service and you realize it's v3, we'll rollback all the things.
I agree with you that it could be confusing and will require some fun engineering but I think the advantages of having such a thing outweight the engineering effort. Unfun part will be on the user side as they will think their service is ready to start but then will have a warning after consensus download that it's not.
It also means we need to do key creation _after_ consensus download so we don't expose the wrong address to the user.
Fun... Maybe we can do better, not sure :S
I'd much prefer a #define for the default, and we can change it in the next release.
That way, we ensure the code gets proper testing in an alpha series.
T
On 06 Dec (09:43:20), teor wrote:
On 6 Dec. 2016, at 08:02, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (07:53:16), teor wrote:
On 6 Dec. 2016, at 07:42, David Goulet dgoulet@ev0ke.net wrote:
On 22 Nov (17:36:33), David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor
with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network
has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of
time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
- When v3 is released in a tor stable version, it will NOT be the default
version for new service, v2 will until maturity.
- HiddenServiceVersion is used to control which version of the service you
want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
- ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is
with the tor you have.
- In order to avoid relying on a tor stable version release to switch the
default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change. ...
This is problematic: we validate and create onion services at torrc load, before loading the consensus.
Do we plan to do it afterwards? That seems like it could be really confusing.
You need a consensus anyway to pick your intro point or HSDir but yes the engineering part here will be that if you ever configured v2 service and you realize it's v3, we'll rollback all the things.
I agree with you that it could be confusing and will require some fun engineering but I think the advantages of having such a thing outweight the engineering effort. Unfun part will be on the user side as they will think their service is ready to start but then will have a warning after consensus download that it's not.
It also means we need to do key creation _after_ consensus download so we don't expose the wrong address to the user.
Fun... Maybe we can do better, not sure :S
I'd much prefer a #define for the default, and we can change it in the next release.
Two downside of this. First, we make the v3 switch in a tor stable release which is 6 months period at the very least extending the time of adoption. Which also as a downside because of Linux distros updating stable package. Second, we lose the ability to rollback.
I guess I could be convinced that it's ok but tbh I _really_ like that we can decide when we want to make the switch instead of relying on stable release.
That way, we ensure the code gets proper testing in an alpha series.
Right, I guess that's an argument against the rollback ability. I don't forsee us to rollback ever tbh. But also it is the whole idea of the period between v3 release and the day we make it a default version through consensus param. We ensure that we have proper testing.
Is your worry mostly about the uglyness of rolling back hidden service if that consensus param ever goes from v3 to v2? If so, I can understand how meh it is...
David
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 6 Dec. 2016, at 09:56, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (09:43:20), teor wrote:
On 6 Dec. 2016, at 08:02, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (07:53:16), teor wrote:
On 6 Dec. 2016, at 07:42, David Goulet dgoulet@ev0ke.net wrote:
On 22 Nov (17:36:33), David Goulet wrote:
Hi everyone!
We are soon at the stage of implementing the service part of proposal 224 (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing here ticket #18054.
In a nutshell, we need to figure out the interface for the torrc file[1]. We currently have some options to configure an hidden service and the question is now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor
with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network
has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of
time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
- When v3 is released in a tor stable version, it will NOT be the default
version for new service, v2 will until maturity.
- HiddenServiceVersion is used to control which version of the service you
want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
- ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is
with the tor you have.
- In order to avoid relying on a tor stable version release to switch the
default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change. ...
This is problematic: we validate and create onion services at torrc load, before loading the consensus.
Do we plan to do it afterwards? That seems like it could be really confusing.
You need a consensus anyway to pick your intro point or HSDir but yes the engineering part here will be that if you ever configured v2 service and you realize it's v3, we'll rollback all the things.
I agree with you that it could be confusing and will require some fun engineering but I think the advantages of having such a thing outweight the engineering effort. Unfun part will be on the user side as they will think their service is ready to start but then will have a warning after consensus download that it's not.
It also means we need to do key creation _after_ consensus download so we don't expose the wrong address to the user.
Fun... Maybe we can do better, not sure :S
I'd much prefer a #define for the default, and we can change it in the next release.
Two downside of this. First, we make the v3 switch in a tor stable release which is 6 months period at the very least extending the time of adoption. Which also as a downside because of Linux distros updating stable package. Second, we lose the ability to rollback.
I guess I could be convinced that it's ok but tbh I _really_ like that we can decide when we want to make the switch instead of relying on stable release.
That way, we ensure the code gets proper testing in an alpha series.
Right, I guess that's an argument against the rollback ability. I don't forsee us to rollback ever tbh. But also it is the whole idea of the period between v3 release and the day we make it a default version through consensus param. We ensure that we have proper testing.
Is your worry mostly about the uglyness of rolling back hidden service if that consensus param ever goes from v3 to v2? If so, I can understand how meh it is...
No, my main concern is that we can't validate the options on launch, if we need a consensus to do it.
And what do we do if the options are valid for v2, but not v3, and the HS downloads a consensus that says "use v3"? Crash, probably, at least the way the code is written at the moment.
Here's a suggested strategy: * at load time, validate the HS options as if v2 is the default, and validate them as if v3 is the default, and fail if either validation fails. * then, act on the HS options as if v2 is the default, and also act as if v3 is the default, and fail if either action fails. (We need to do this because we don't discover some option issues until runtime, such as whether the directory can be created.) * then, when each consensus is downloaded, publish whichever descriptor is the default in the consensus (if the HS config does not specify a specific version).
This means creating some redundant keys, but that's ok.
There is still a minor issue with the hostname file. We could add two new files: * hostname3: the hostname of the v3 hidden service * hostname_published: the hostname of the published hidden service (only set when the HS is actually published)
I think this is better for backwards-compatibility than changing the content of the hostname file.
T
On 06 Dec (10:09:57), teor wrote:
On 6 Dec. 2016, at 09:56, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (09:43:20), teor wrote:
On 6 Dec. 2016, at 08:02, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (07:53:16), teor wrote:
On 6 Dec. 2016, at 07:42, David Goulet dgoulet@ev0ke.net wrote:
On 22 Nov (17:36:33), David Goulet wrote: > Hi everyone! > > We are soon at the stage of implementing the service part of proposal 224 > (next gen. hidden service.). Parent ticket is #20657 but I'll be discussing > here ticket #18054. > > In a nutshell, we need to figure out the interface for the torrc file[1]. We > currently have some options to configure an hidden service and the question is > now how do we proceed on using those for next version?
[snip]
(Please read original email for some initial context.)
So over the last week, we've mashed up all the things that were said in this thread, me and asn discussed it with some arma also on the side!
I think the following is the best of all the non ideal solutions we can come up with. Below is a superb ascii timeline of what we plan to do in terms of transition from v2 to v3:
Our dystopian reality of now. (https://youtu.be/NrmMk1Myrxc it's not a Black Mirror SO3 trailer) v | ~Aug '17 ~Dec '17 Unknown date |-----------------|-----------------|----------------|----------> | ^ ^ ^ v3 Network/Code | tor stable release Maturity | with v3 support No more v2 (0.3.1)
Ok, might not be my best ascii art work but I hope it will do. In short:
- With 0.3.1 (scheduled for August 2017, subject to change), we'll have a tor
with v3 support BUT v2 will still be the default value for *new* HS. (HiddenServiceVersion option)
- For a period of ~4 months we estimate, we'll hope that enough of the network
has upgraded to support v3 (relay and HSDir support are in 030) and that the code as some sort of maturity that we are confident to switch and make all new HS be v3. This is the "v3 Network/Code Maturity" marker. Note that it could easily go to 2018 for this switch to happen.
- When the switch happens, there will be, most likely years, a long period of
time where v2 will be deprecated and warnings given to users but still used. That "Unknown date" will be the time we will release a tor stable version _without_ v2 support at all. It's quite unknown when we'll be able to do it. Chances are that we'll rely on our HS statistics and metrics for that.
That being said, here are the conclusions based on this thread and f2f discussion considering the above "procedure":
- When v3 is released in a tor stable version, it will NOT be the default
version for new service, v2 will until maturity.
- HiddenServiceVersion is used to control which version of the service you
want. Starting from the first time v3 is supported, you'll be able to use it but without any guarantee as we'll be entering the "stabilizing period" but it should be usable.
- ADD_ONION "BEST" will map to whatever default value HiddenServiceVersion is
with the tor you have.
- In order to avoid relying on a tor stable version release to switch the
default version from 2 to 3, we'll use a consensus parameter. Meaning that once we set that param. to v3, HiddenServiceVersion default value will be that value unless explicitely defined in torrc. It will also allow us to rollback to v2 if we see a swarm of badness occuring. To be clear, service already v3 will stay v3 but the default version for creation will simply change. ...
This is problematic: we validate and create onion services at torrc load, before loading the consensus.
Do we plan to do it afterwards? That seems like it could be really confusing.
You need a consensus anyway to pick your intro point or HSDir but yes the engineering part here will be that if you ever configured v2 service and you realize it's v3, we'll rollback all the things.
I agree with you that it could be confusing and will require some fun engineering but I think the advantages of having such a thing outweight the engineering effort. Unfun part will be on the user side as they will think their service is ready to start but then will have a warning after consensus download that it's not.
It also means we need to do key creation _after_ consensus download so we don't expose the wrong address to the user.
Fun... Maybe we can do better, not sure :S
I'd much prefer a #define for the default, and we can change it in the next release.
Two downside of this. First, we make the v3 switch in a tor stable release which is 6 months period at the very least extending the time of adoption. Which also as a downside because of Linux distros updating stable package. Second, we lose the ability to rollback.
I guess I could be convinced that it's ok but tbh I _really_ like that we can decide when we want to make the switch instead of relying on stable release.
That way, we ensure the code gets proper testing in an alpha series.
Right, I guess that's an argument against the rollback ability. I don't forsee us to rollback ever tbh. But also it is the whole idea of the period between v3 release and the day we make it a default version through consensus param. We ensure that we have proper testing.
Is your worry mostly about the uglyness of rolling back hidden service if that consensus param ever goes from v3 to v2? If so, I can understand how meh it is...
No, my main concern is that we can't validate the options on launch, if we need a consensus to do it.
And what do we do if the options are valid for v2, but not v3, and the HS downloads a consensus that says "use v3"? Crash, probably, at least the way the code is written at the moment.
Well, we are talking about new code here for v3. Currently, the code is NOT prepared for that, it would just ignore the param. This is entirely code starting to be used in 031.
Here's a suggested strategy:
- at load time, validate the HS options as if v2 is the default, and validate them as if v3 is the default, and fail if either validation fails.
- then, act on the HS options as if v2 is the default, and also act as if v3 is the default, and fail if either action fails. (We need to do this because we don't discover some option issues until runtime, such as whether the directory can be created.)
- then, when each consensus is downloaded, publish whichever descriptor is the default in the consensus (if the HS config does not specify a specific version).
This is a reasonable way to proceed considering we use a consensus param to know which version of default HS to create. I see this as more of an engineering problem that can be solved.
Which what I would like us to decide on if we think that consensus param controlling the default version is a good idea or not. If we think yes, we can pull it off, if not everything is simpler :).
So just to be clear, I'm behind you on the concern of making sure we validate the options on launch instead of failing at consensus download. There are ways we can address that like you outlined above.
This means creating some redundant keys, but that's ok.
There is still a minor issue with the hostname file. We could add two new files:
- hostname3: the hostname of the v3 hidden service
- hostname_published: the hostname of the published hidden service (only set when the HS is actually published)
Ouf, not sure I would do that :) ... actually this is an entirely new discussion that we discussed a bit also last week. I was planning on making a tor-dev@ thread just for this subject. So let's not go into more details.
Cheers! David
I think this is better for backwards-compatibility than changing the content of the hostname file.
T
-- Tim Wilson-Brown (teor)
teor2345 at gmail dot com PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n xmpp: teor at torproject dot org
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 6 Dec. 2016, at 10:32, David Goulet dgoulet@ev0ke.net wrote:
On 06 Dec (10:09:57), teor wrote:
...
Here's a suggested strategy:
- at load time, validate the HS options as if v2 is the default, and
validate them as if v3 is the default, and fail if either validation fails.
- then, act on the HS options as if v2 is the default, and also act as
if v3 is the default, and fail if either action fails. (We need to do this because we don't discover some option issues until runtime, such as whether the directory can be created.)
- then, when each consensus is downloaded, publish whichever descriptor
is the default in the consensus (if the HS config does not specify a specific version).
This is a reasonable way to proceed considering we use a consensus param to know which version of default HS to create. I see this as more of an engineering problem that can be solved.
Which what I would like us to decide on if we think that consensus param controlling the default version is a good idea or not. If we think yes, we can pull it off, if not everything is simpler :).
So just to be clear, I'm behind you on the concern of making sure we validate the options on launch instead of failing at consensus download. There are ways we can address that like you outlined above.
Yes, I think it is possible, and a better outcome than baking a protocol default into the first release we want to use it in.
As long as you are willing to put in the extra dev and test effort!
T
David Goulet dgoulet@ev0ke.net writes:
On 06 Dec (10:09:57), teor wrote:
On 6 Dec. 2016, at 09:56, David Goulet dgoulet@ev0ke.net wrote:
<snip>
Here's a suggested strategy:
- at load time, validate the HS options as if v2 is the default, and validate them as if v3 is the default, and fail if either validation fails.
- then, act on the HS options as if v2 is the default, and also act as if v3 is the default, and fail if either action fails. (We need to do this because we don't discover some option issues until runtime, such as whether the directory can be created.)
- then, when each consensus is downloaded, publish whichever descriptor is the default in the consensus (if the HS config does not specify a specific version).
This is a reasonable way to proceed considering we use a consensus param to know which version of default HS to create. I see this as more of an engineering problem that can be solved.
Which what I would like us to decide on if we think that consensus param controlling the default version is a good idea or not. If we think yes, we can pull it off, if not everything is simpler :).
So just to be clear, I'm behind you on the concern of making sure we validate the options on launch instead of failing at consensus download. There are ways we can address that like you outlined above.
Hello,
I agree that this is more of an engineering problem that can be solved. I also like the agility that the consensus parameter gives us.
That said when we get close to implementing this feature, IMO we should estimate how long it will take us to implement this consensus parameter logic, and evaluate whether it's worth doing. This feature might be an evening of work, but it could also be two annoying weeks of testing and debugging crazy networking and consensus fetching edge cases, so we should make sure we don't sign up for too much craziness.
The alternative would be that we don't do the consensus parameter, and instead we just change the version parameter in alpha/stable releases when we feel it's ready. This will be a slower and more gradual deployment which is not necessarily a terrible thing when we are talking about huge features like prop224.
On 20 Dec (17:44:31), George Kadianakis wrote:
David Goulet dgoulet@ev0ke.net writes:
On 06 Dec (10:09:57), teor wrote:
On 6 Dec. 2016, at 09:56, David Goulet dgoulet@ev0ke.net wrote:
<snip>
Here's a suggested strategy:
- at load time, validate the HS options as if v2 is the default, and validate them as if v3 is the default, and fail if either validation fails.
- then, act on the HS options as if v2 is the default, and also act as if v3 is the default, and fail if either action fails. (We need to do this because we don't discover some option issues until runtime, such as whether the directory can be created.)
- then, when each consensus is downloaded, publish whichever descriptor is the default in the consensus (if the HS config does not specify a specific version).
This is a reasonable way to proceed considering we use a consensus param to know which version of default HS to create. I see this as more of an engineering problem that can be solved.
Which what I would like us to decide on if we think that consensus param controlling the default version is a good idea or not. If we think yes, we can pull it off, if not everything is simpler :).
So just to be clear, I'm behind you on the concern of making sure we validate the options on launch instead of failing at consensus download. There are ways we can address that like you outlined above.
Hello,
I agree that this is more of an engineering problem that can be solved. I also like the agility that the consensus parameter gives us.
That said when we get close to implementing this feature, IMO we should estimate how long it will take us to implement this consensus parameter logic, and evaluate whether it's worth doing. This feature might be an evening of work, but it could also be two annoying weeks of testing and debugging crazy networking and consensus fetching edge cases, so we should make sure we don't sign up for too much craziness.
The alternative would be that we don't do the consensus parameter, and instead we just change the version parameter in alpha/stable releases when we feel it's ready. This will be a slower and more gradual deployment which is not necessarily a terrible thing when we are talking about huge features like prop224.
Agree. In a matter of few hours, we'll know if this is too much engineering with lots of corner case untested or actually doable easily.
The fallback to use the parameter is a good idea to avoid going overboard if we realize it is kind of crazy.
Cheers! David
-
chelsea komlo -
ckomlo＠thoughtworks.com -
David Goulet -
George Kadianakis -
Jesse V -
Matthew Finkel -
meejah -
s7r -
teor -
Yawning Angel