On Mon, Dec 29, 2014 at 3:33 PM, Tom van der Woerdt info@tvdw.eu wrote:

Sounds good!

I spent some time writing a patch that removes v1 of the link protocol from both the server and client, and so far it seems to work nicely: the code compiles nicely, all test cases pass, and the resulting binary has relayed a few gigabytes of data without any problems.

As I didn't really have a place to put the branch, I uploaded it to Github: https://github.com/TvdW/tor/commits/master

It's a rather large patch, though not as large as the patch that will remove v2 of the protocol. However, before I write that one, can someone please check whether my patch is sane and I'm not violating any standards or policies?

Hi, Tom!

Sorry about the delay; this has been a busy time, between CCC drama and the new year.

A few procedural suggestions:

* Maybe call branches like this something other than "master"? It's a good idea to have a separate topic branch for each patch series you write, so that the upstream project can merge them independently.

* It's a good idea to link to branches like this from the appropriate tickets on the bugtracker, so that we can't lose track of them.

* Every Tuesday on the #tor-dev IRC channel on OFTC, at 1800 UTC, it's patch workshop time, where a bunch of people get together to review one another's patches. Maybe you'd like to stop by? :)

And, some fast notes on the patch itself:

* Actually, it's not that long. "git log -p -b" only says it's 607 lines, which is comparatively easy to review.

* Some of the stuff can't actually get removed yet by the terms of what we've said we're doing. The V2_CIPHER_LIST, for example, is used to detect Tor versions up through 0.2.3.17-beta. We've only mentioned dropping support for 0.2.2 and earlier, right?. Similarly with the tor_tls_session_secret_cb logic.

Now, maybe we _should_ drop support for versions before 0.2.3.17-beta as well. If so, we can rip out even more code. (And that might be a good idea.) What do people on the list think?

I'll try to have a closer look over the weekend. Thanks for the code!

yrs,

Nick Mathewson schreef op 02/01/15 om 15:27:

On Mon, Dec 29, 2014 at 3:33 PM, Tom van der Woerdt info@tvdw.eu wrote:

...

Sounds good!

I spent some time writing a patch that removes v1 of the link protocol from both the server and client, and so far it seems to work nicely: the code compiles nicely, all test cases pass, and the resulting binary has relayed a few gigabytes of data without any problems.

As I didn't really have a place to put the branch, I uploaded it to Github: https://github.com/TvdW/tor/commits/master

It's a rather large patch, though not as large as the patch that will remove v2 of the protocol. However, before I write that one, can someone please check whether my patch is sane and I'm not violating any standards or policies?

Hi, Tom!

Sorry about the delay; this has been a busy time, between CCC drama and the new year.

A few procedural suggestions:

• Maybe call branches like this something other than "master"? It's

a good idea to have a separate topic branch for each patch series you write, so that the upstream project can merge them independently.

• It's a good idea to link to branches like this from the

appropriate tickets on the bugtracker, so that we can't lose track of them.

Thanks, I'll create an account.

• Every Tuesday on the #tor-dev IRC channel on OFTC, at 1800 UTC,

it's patch workshop time, where a bunch of people get together to review one another's patches. Maybe you'd like to stop by? :)

Sounds fun!

And, some fast notes on the patch itself:

• Actually, it's not that long. "git log -p -b" only says it's 607

lines, which is comparatively easy to review.

• Some of the stuff can't actually get removed yet by the terms of

what we've said we're doing. The V2_CIPHER_LIST, for example, is used to detect Tor versions up through 0.2.3.17-beta. We've only mentioned dropping support for 0.2.2 and earlier, right?. Similarly with the tor_tls_session_secret_cb logic.

Now, maybe we _should_ drop support for versions before 0.2.3.17-beta as well. If so, we can rip out even more code. (And that might be a good idea.) What do people on the list think?

I'd definitely like ripping out more legacy support code :-) According to the consensus, 0.2.3.24-rc is the lowest recommended version right now, and I (mis)interpreted that as the lowest version that must still be supported after I rip out code.

On a related note, I've uploaded a branch of the torspec [1] to reword (most of) the handshake part to match what I hope the code will do as soon as I'm done cleaning. Spec changes may need to go through proposals though, should I add it as one?

Also, in case anyone is wondering why a complete stranger is suddenly so interested in ripping parts out of Tor: I attempted to implement a Tor client and was constantly faced with legacy stuff in the spec. Might as well get rid of it if you don't need it - also makes my client implementation easier.

Tom

[1] https://github.com/TvdW/torspec/commits/remove-legacy-protocol-support

On 12 Jan 2015, at 16:25, Philipp Winter phw@nymity.ch wrote:

...

On Sat, Dec 27, 2014 at 03:38:28PM +0100, Tom van der Woerdt wrote: After reading the Tor spec [1] I did some digging and realized that the old handshakes and link protocols (v1 (certs up-front) and v2 (renegotiation)) are not used anymore as of 0.2.3.6-alpha which introduced link proto v3.

Supporting v1 and v2 requires (among other things) supporting SSLv3 which (imho) should be deprecated everywhere.

I was curious about how many relays and clients still want to speak version 1 and 2. I patched one of my guard relays to keep statistics about the content of VERSIONS cells. Here's the result after almost three days. The numbers include relays as well as clients.

Awesome! Looks like we can drop it.

Versions | Amount total | Amount w/o duplicate hosts ---------+---------------+--------------------------- 1 and 2 | 34,648 (9%) | 21,552 (23%)

We debugged this last week on IRC, as 1,2 is an invalid combination according to the specification. After correlating the ip addresses, we concluded that this is GFW scanning and not actual client usage. Are you sure you are deduplicating correctly? That's a lot of hosts.

    3 |  73,202 (18%) | 54,307 (59%)

3 and 4 | 291,807 (73%) | 16,235 (18%)

These two are normal.

    4 |       3  (0%) |      2  (0%)

That's me! I have a client implementation that will only return the highest understood version matching what the server supports, instead of a list of all supported versions.

So yeah, based on that data I conclude that 1 and 2 are no longer used.

Tom

Cheers, Philipp _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

David Fifield schreef op 12/01/15 om 18:46:

On Mon, Jan 12, 2015 at 06:26:14PM +0100, Tom van der Woerdt wrote:

...

...

On 12 Jan 2015, at 16:25, Philipp Winter phw@nymity.ch wrote: Versions | Amount total | Amount w/o duplicate hosts ---------+---------------+--------------------------- 1 and 2 | 34,648 (9%) | 21,552 (23%)

We debugged this last week on IRC, as 1,2 is an invalid combination according to the specification. After correlating the ip addresses, we concluded that this is GFW scanning and not actual client usage.

I'm sure some of the 1+2 is GFW scanning, but probably not all of it. Mainstream tor definitely sends 1+2 when using a v2 handshake.

https://gitweb.torproject.org/tor.git/tree/src/or/connection_or.c?id=b0c3210...

/** Array of recognized link protocol versions. */ static const uint16_t or_protocol_versions[] = { 1, 2, 3, 4 }; /** Number of versions in <b>or_protocol_versions</b>. */ static const int n_or_protocol_versions = (int)( sizeof(or_protocol_versions)/sizeof(uint16_t) );

/** Send a VERSIONS cell on <b>conn</b>, telling the other host about the

• link protocol versions that this Tor can support.
• If <b>v3_plus</b>, this is part of a V3 protocol handshake, so only
• allow protocol version v3 or later. If not <b>v3_plus</b>, this is
• not part of a v3 protocol handshake, so don't allow protocol v3 or
• later.

**/ int connection_or_send_versions(or_connection_t *conn, int v3_plus) { var_cell_t *cell; int i; int n_versions = 0; const int min_version = v3_plus ? 3 : 0; const int max_version = v3_plus ? UINT16_MAX : 2; tor_assert(conn->handshake_state && !conn->handshake_state->sent_versions_at); cell = var_cell_new(n_or_protocol_versions * 2); cell->command = CELL_VERSIONS; for (i = 0; i < n_or_protocol_versions; ++i) { uint16_t v = or_protocol_versions[i]; if (v < min_version || v > max_version) continue; set_uint16(cell->payload+(2*n_versions), htons(v)); ++n_versions; } cell->payload_len = n_versions * 2;

connection_or_write_var_cell_to_buf(cell, conn); conn->handshake_state->sent_versions_at = time(NULL);

var_cell_free(cell); return 0; }

...

Are you sure you are deduplicating correctly? That's a lot of hosts.

Even if it were only GFW probing, GFW rarely uses duplicate IPs, except for a few. Most IPs you will only see once or twice over the course of months.

David Fifield

Wow, nice find. Then, based on the fact that 23% of the network (WTF?) is still running old clients, maybe it's best to wait with dropping the old link versions for now. (OTOH if it takes that long for people to update, dropping it will take years anyway, and removing it from a few relays won't hurt anything)

23% is a lot though - so high that I really doubt it's true. The ratios between handshakes and deduplicated handshakes is also rather strange. Is there anything we can do to the dataset to find out why the amount is so high?

Tom

Philipp Winter schreef op 12/01/15 om 20:14:

On Mon, Jan 12, 2015 at 06:57:01PM +0100, Tom van der Woerdt wrote:

...

23% is a lot though - so high that I really doubt it's true. The ratios between handshakes and deduplicated handshakes is also rather strange. Is there anything we can do to the dataset to find out why the amount is so high?

When looking at the ratio, consider that the majority of relays runs newer versions of Tor [0]. Over these three days, my relay has established hundreds of connections to other relays over and over again. When deduplicating relays' addresses, all these connections get reduced to one which explains why the per-host fraction of version 3 and 4 is much smaller than the per-connection fraction.

Apart from that, I agree that the number of old clients is unexpected. First, I suspected the Sefnit botnet (which might still account for ~50% of Tor "users") but apparently the malware uses Tor v0.2.3.25.

I think the same experiment could be repeated by adding the following to your tor config:

Log [or]info file /path/to/logfile

And then, the negotiated protocol versions can be counted by running, for example:

grep -c 'Negotiated version 2' /path/to/logfile

[0] https://metrics.torproject.org/versions.html

Interestingly, that paints a completely different picture. I added that line to two machines (guard+exit) and after a few minutes :

# cat /var/lib/tor/node*/infolog | grep Negotiated | awk '{ print $8 }' | sort | uniq -dc 40 2 76 3 3811 4

# cat /var/lib/tor/node*/infolog | grep Negotiated | awk '{ print $8 }' | sort | uniq -dc 50 2 122 3 6269 4

I'll let it run a bit longer but these two machines (which are both exits as well - probably relevant) get almost solely v4 handshakes.

Tom

On Thu, Jan 15, 2015 at 02:29:28PM +0100, Philipp Winter wrote:

On Mon, Jan 12, 2015 at 08:24:58PM +0100, Tom van der Woerdt wrote:

...

Interestingly, that paints a completely different picture. I added that line to two machines (guard+exit) and after a few minutes :

# cat /var/lib/tor/node*/infolog | grep Negotiated | awk '{ print $8 }' | sort | uniq -dc 40 2 76 3 3811 4

# cat /var/lib/tor/node*/infolog | grep Negotiated | awk '{ print $8 }' | sort | uniq -dc 50 2 122 3 6269 4

I'll let it run a bit longer but these two machines (which are both exits as well - probably relevant) get almost solely v4 handshakes.

I now did the same for my two relays and I get two different distributions, which I found surprising. That's the relay [0] whose results I showed earlier.

Negotiated version | Per connection | Per host -------------------+----------------+------------- 2 | 12,236 (8%) | 9,292 (21%) 3 | 29,768 (20%) | 23,393 (52%) 4 | 108,884 (72%) | 12,051 (27%)

And here's a relay [1] on the same physical machine with almost the same configuration. The major difference is that this relay is not configured to run a directory service whereas the other one is. The numbers are close to yours, Tom.

Negotiated version | Per connection | Per host -------------------+----------------+------------ 2 | 761 (1%) | 279 (3%) 3 | 4,468 (5%) | 1,301 (14%) 4 | 82,811 (94%) | 7,494 (83%)

[0] https://atlas.torproject.org/#details/9B94CD0B7B8057EAF21BA7F023B7A1C8CA9CE645 [1] https://atlas.torproject.org/#details/CCEF02AA454C0AB0FE1AC68304F6D8C4220C1912

Is there an easy way to tell which connections are from clients and which are from other relays? One of the relays has a higher guard probability; maybe that has something to do with it. Probably there is a different distribution of versions among relays than among client, especially when you account for bandwidth weighting (any relay connecting to you is more likely to be fast than slow, which may mean it's more likely to be newer than older).

David