It's not just about getting the protocol stack right, but also the ancillary software environment; people keep asking me for "V3 support in EOTK" and my stock response is this:

==== BEGIN ==== OnionBalance requires STEM support for V3, before it can be updated (possibly a substantial rewrite will be needed) to support the new format onions. It's not only a matter of "longer addresses" but also a matter of cross-signing the descriptors to support new-style cryptography, so in fact it might be safest to create a new, separate OnionBalance for V3.

So: STEM needs updating and testing for V3, and then OnionBalance needs to support the new STEM library and encryption. Then (for me) EOTK needs to support the new OnionBalance.

I am not expecting a solution to ship until 2019, earliest. ==== END ====

...and that's even without refactoring the other bits of EOTK to address the changes when STEMv3 lands.

OTOH, I have been performance testing simultaneous regular-expression matching of v2/3 addresses, and so far this is the winner:

"\b([a-z2-7]{16}(?:[a-z2-7]{40})?\.onion)\b"

...and it's already in the codebase at https://github.com/alecmuffett/eotk/blob/master/templates.d/nginx.conf.txt#L...

- alec :-)

On 27 April 2018 at 19:48, Damian Johnson atagar@torproject.org wrote:

...

OnionBalance requires STEM support for V3

Hi Alec, would you mind clarifying what you need from Stem? As far as I'm aware Stem supports v3 onion service creation... ... I'm unaware of the ball being in my court on any v3 Onion Service stuff. If there's something I should have on my radar then please let me know!

Hi Damian!

That's awesome, and good to know - I first wrote that text a few months ago (on the basis of David's comments in that ticket) and haven't revised it since, so I am heartened to see progress.

However I am also not the best person to say what else will be needed, because that would probably be Donncha re: the future of OnionBalance for v3.

At the moment in OnionBalance the v2 Introduction Points of the (predetermined) worker onions are passively scraped from the HSDir; the descriptors are parsed, the IPs are blended and re-combined and re-signed under the master onion key (eg: nytimes3xbfgragh) and then published back to the HSDirs, with the result that NewYorkTimes-browsing clients end up communicating with 1 of N possible "worker" daemons, thereby sharing the load.

My understanding - and please jump in, if I am wrong - is that the synthesis of a v3 descriptor which blends the introduction points of several independent v3 "worker" Tor daemons will be a more complex affair than the existing process, because (?) the "worker" tor daemons will somehow have to be more actively involved - apparently they may have to sign the v3 Introduction Points themselves, though I am not sure how that will work for a blended descriptor? Multiple/distinct signatures, perhaps? The last opportunity I had to speak with anyone (re: this) was more than 1 year ago, so I am rusty, and I apologise if I have gotten some details wrong.

So: OnionBalance relies heavily upon Stem ( https://github.com/DonnchaC/onionbalance/tree/develop/onionbalance) and I am not qualified to say what, if any, additional v3 Stem features will be useful or outstanding to support the descriptor-blending that is needed for loadbalanced configurations.

But OnionBalance for v3 will certainly be necessary. :-)

-a

On 28 Apr (09:34:09), teor wrote:

On 28 Apr 2018, at 04:48, Damian Johnson atagar@torproject.org wrote:

...

...

OnionBalance requires STEM support for V3

Hi Alec, would you mind clarifying what you need from Stem? As far as I'm aware Stem supports v3 onion service creation...

https://trac.torproject.org/projects/tor/ticket/25124

See the 'version 3' note at the end of...

https://stem.torproject.org/api/control.html#stem.control.Controller.create_...

I'm unaware of the ball being in my court on any v3 Onion Service stuff. If there's something I should have on my radar then please let me know!

OnionBalance requires Stem support for v3 HSFETCH But Stem requires Tor support for v3 HSFETCH https://trac.torproject.org/projects/tor/ticket/25417

Hmmmm I was told before the v3 control port work by Donnacha that for v3 support, HSFETCH won't be ncessary...

It is the main reason it hasn't been done that is for a lack of use case. The HSFETCH for v3 is much more tricky in terms of engineering and interface thus we decided to implement it on the "need it basis".

If OnionBalance actually needs it for v3, then we should try to get that on the roadmap.

Cheers! David

for v3 support, HSFETCH won't be ncessary...

Noooo...

HSFETCH <v2.onion | v3.onion>

is an absolutely necessary control function now critical to operation of a variety of onionland search / index / status / webhosting / research services, and any other basic commandline checks that have zero wish to be spawning heavy browser / wget / specific application apparatus or wasting resources establishing TCP connections to the services themselves, and for debugging tor client and network itself independant from all such other tools, and helpful for "Why can't I connect" questions from users, etc.

Further, a minimum request and result option semantics of HSFETCH regarding the above were roughly specified previously...

HSFETCH <v2.onion | v3.onion> [fetch_mode] [output_each] [raw_desc] [sync_here] [update_cache] as integrated with vN-DescId and SERVER= elements

General operation - fetch mode: 0 default client resolution method, 1 force from network [3 only, do not recurse to cache], 2 force from cache [4 only, do not recurse to network] - source of the returned descriptor: network, or local cache - network may have different data than cache so [update_cache] or not, default yes - result status of the fetch [a]: ok found, 404 not found, network failure + why: reason - status of the descriptor [a]: ok good, bad + why: crypto failed, expired, parsing data type, truncated, corrupt, etc - decoding and printing out all fields of the returned descriptor, optionally also print the whole descriptor as received, regardless of any bad status, in hex [raw_desc] to further help debugging and other uses For each HSDir by respective HSDir identifier [output_each=true] - above result status of the fetch for each HSDir queried (typically six) - above status of the descriptor from each responding HSDir - above decoding and printing for each responding HSDir [output_each=verbose]

[a] These should be tagged 0 if all sources / HSDirs were ok, or 1 if an ok descriptor was ultimately able to be returned to the fetch but some of the sources / HSDirs had negatives thus pointing to further investigation, or 2 if none were ok (and print the status if all six had same status, or print "various" if they varied).

Since nodes may be performing other tasks in parallel, even over the same onions, and due to various mandatory modes and sources being selected, and network delays, it is very helpful and unambiguous if the output is selectably synchronus to this command and console [sync_here], thus leaving the HS_DESC / HS_DESC_CONTENT event streams doing their thing if so enabled possibly on / for other control connections / monitors. A command instance identifier should be added to be returned to match up with respective event logstream (which would otherwise perhaps be identified as "socks / tunnel / trans / dns / natd / etc" for those sources), but that does not replace utility of [sync_here] for lesser capable users or cases.

Future applications may (and perhaps already do) use HSDir descriptor mechanism as their own datastore via HSPOST, for which similar HSFETCH models would be useful, thus good to consider herein, certainly as an extensible.

There were a tickets and emails somewhere on these concepts so that they could be further and better implemented. Some elements were committed, others remain :)

It is the main reason it hasn't been done that is for a lack of use case. The HSFETCH for v3 is much more tricky in terms of engineering and interface thus we decided to implement it on the "need it basis".

If OnionBalance actually needs it for v3, then we should try to get that on the roadmap.

Jonathan Marquardt mail@parckwart.de writes:

On Wed, Apr 25, 2018 at 04:58:36PM -0400, grarpamp wrote:

...

In onionland, there seems to be little knowledge of v3, thus little worry about v2 in cases where v3 would actually apply to benefit, that's bad.

v3 onion services just seem like a way worse deal to the average user and the unknowledgeable admin. Mainly because the addresses are way too long. I can remember a couple of v2 addresses, but not a single v3 address. So that's just bad advertising from the start.

IMO if you have the ability to memorize v2 addresses by heart, you are already not an average user. Average users just google most things they try to visit.

That said, I do share your concerns, and that's why I mentioned that finding a solution to the onion name issue is a priority before v3 can go mainstream (or even v2).

Before at least Facebook, DuckDuckGo, The New York Times, the Debian Project and even the Tor Project themselves (!) have rolled out their v3 onion services, one shouldn't even think about deprecating HSv2. It's going to be around for many years to come, taking for them just as long to vanish as an SSL version, I think, unfortunately.

Agreed. We are indeed a long way from deprecating HSv2 :)

On Fri, Apr 27, 2018 at 04:03:00PM -0400, grarpamp wrote:

a) If defined as shifting v3 to be "provisioned by default" via docs and function, while *continuing to support v2* functionality on the network, there's no problem, everyone is happy. b) While v2 and v3 do share some capabilities, since v2 and v3 do offer their own exclusive subset of capabilities to users that cannot currently be found in the opposing version, *removing v2 support* is a definite issue.

I think that, before making v3 the default, all features from v2 like HidServAuth should be implemented and should have been around for a couple of Tor versions.

Also, what would happen to an old Tor instance with v2 onion services configured after the upgrade? These onion services should definitely not automatically be switched to v3, as it could break many configurations on systems with automatic software updates. I suggest that, if Tor sees an onion service configured in torrc and if there's no "HiddenServiceVersion 3" in torrc and there are v2 keys in the HiddenServiceDir, it should continue to serve a v2 service.

But leaving a note in the log about v3 services if there's a v2 configured could be a good thing, I think.

v3 is a nice advancement and is needed by lots of users for what it can do for them, just as v2 is.

Totally agreed.

On 28 Apr 2018, at 05:56, nusenu nusenu-lists@riseup.net wrote:

...

And also we will be able to reduce the requirements for becoming an HSDir which will strengthen and make our network more robust.

That said, I think we are unfortunately still far from deprecating v2 onions:

thanks for listing all these relevant ticket IDs, I pasted your email into trac and I made them child tickets of https://trac.torproject.org/projects/tor/ticket/25955

lets try to link all relevant tickets there

Please don't, that confuses our workflow. A lot of these tickets are unrelated to v3 onion service features in Tor.

If you want to link all these tickets somewhere, open another ticket.

T