Tim: Thank you for the detailed email. I've also included tor-dev on CC. Anyone is welcome to comment.
To reply to your concerns:
1. All the documentation we have is basically in email correspondences as well as in the generic simple-quic codebase here: https://github.com/kku1993/simple-quic. The quicksock_t interface is in the above repo under /quicsock. There isn't much documentation because: a) our designs are still under test and large portions are not stable yet, b) we haven't gotten much time to properly document designs and changes to the Tor codebase. But the emails should be somewhat clear on some key design aspects. 2. About providing an alternative for QUIC connections: - Great idea about keeping both type of sockets in connection. I think this will give us a cleaner design. - Exit relay will speak normal TCP. This is part of our design from the start but we haven't gotten around to look into the exit connection code. - Connections to the directory authorities should be normal TCP. - While compatibility with the current system is very important, our main goal is to compare performances of the OR connections with TCP vs. QUIC. So we haven't spent much effort making things beautiful yet. After all, it is possible that QUIC is a bad idea for Tor.. 3. I was looking at chutney the other day but I couldn't find much documentation on how it works. I will play with it more and then I might ask for more help.
If anyone else is expert on the SSL part of the codebase, please provide some insights about the security aspect of the design, specifically: "About SSL, we want to adopt the design where QUIC handles SSL by not using the ssl_buffer in TOR code. Any concerns at all?" (not using read_to_buf_tls() and other similar ssl functions)
Another concern is the assumption on TOR_SOCKET_T in the Tor codebase. In particular, there are many call sites of fcntl, getsockname, setsockopt etc made directly on this object, assuming that it is a UNIX socket. For compatibility, after replacing UNIX socket with quicsock_t, we would use "qs_get_fd" for all those low-level calls. Note that quicsock_t holds a UDP socket internally. Without the testing system ready, we don't know the effect of these calls on a UDP socket instead of the normal TCP socket.
Any other feedback also welcomed! Li.
On Mon, Apr 4, 2016 at 10:43 PM, Tim Wilson-Brown - teor <teor2345@gmail.com
wrote:
Hi Li,
If you're ok with emailing the tor-dev mailing list, you will likely get better answers to some of these questions there. All of these questions could benefit from the wider Tor developer community. When I know I can't fully answer a question, I'll say so below.
Please feel free to forward my response to tor-dev.
On 5 Apr 2016, at 05:38, Xiaofan Li xli2@andrew.cmu.edu wrote:
Hi Tim: I think maybe now is a good time to bother you again. In the following
email, we outlined some current progress on the TOR+QUIC project. This weekend we booted up our first TOR node running QUIC (then of course it failed to contact the directory authorities and hanged). We hope to move on to testing our design while clean up some uncertain "loose-ends" in the integration process, with your help. In particular:
• We decided to implement a QUIC interface similar to the UNIXsocket interface. We have replaced TOR_SOCKET_T with our quicksock_t. Any issues about this design?
I'm not particularly familiar with this part of the tor codebase, you might get better answers from tor-dev. Also, without knowing the definition of quicksock_t, I can't say much more.
Overall, I think your design has two major flaws:
- you need to provide QUIC as an alternative, not a replacement, because
Exits need to speak TCP, and so might Clients and Onion Services
- you've chosen to modify all tor sockets to speak QUIC, rather than
replacing SSL over TCP with QUIC
Why not put your quicksock_t in connection_t (in or.h), and then choose whether to use QUIC or a plain socket based on whether you're using an OR connection (or_connection_t) or not? You'd also need some logic in OR listener connections (listener_connection_t) that creates a QUIC or plain socket from the listener depending on whether it's an OR listener or not.
See "Kinds of Connections" in https://gitweb.torproject.org/user/nickm/torguts.git/tree/02-dataflow.md And https://gitweb.torproject.org/user/nickm/torguts.git/plain/diagrams/02/02-co... And maybe https://gitweb.torproject.org/user/nickm/torguts.git/plain/diagrams/02/02-da...
Here are my concerns in detail:
I think you need a design that enables QUIC over UDP as an alternative to SSL over TCP sockets, like the way Tor's IPv6 support provides IPv6 as an alternative to IPv4. (Of course, IPv6 is not the same level of the protocol stack as QUIC.)
If you completely replace TCP with QUIC, then Exits will also speak QUIC to sites on the wider Internet. (As you describe below.) Also, relays with speak QUIC to the directory authorities when downloading descriptors, rather than using HTTP over TCP. And you'll force the browser and Tor controllers to speak QUIC to the Tor Client. And Tor Onion Services to speak QUIC to the backend servers they connect to. (I think some parts of Tor's DNS might also be affected, but it depends on the level of abstraction they're at.)
This is unlikely to be what you want. And there could also be other impacts on connections that should stay TCP.
• About SSL, we want to adopt the design where QUIC handles SSL bynot using the ssl_buffer in TOR code. Any concerns at all?
You'll need to ask tor-dev for this one, I'm not familiar with this code at all.
• During integration, I found many instances where TOR assumesTOR_SOCKET_T is INT. Why can TOR make this assumption? Is the type TOR_SOCKET_T only created to accommodate windows dev?
I think the assumption was already there when Windows support was added. It would appear that tor_socket_t exists for Windows compatibility.
From the definition of tor_socket_t in compat.h:
/* XXX Actually, this should arguably be SOCKET; we use intptr_t here so that
- any inadvertent checks for the socket being <= 0 or > 0 will probably
- still work. */
• For testing, what's the most correct (and easy enough to do in afew weeks) way to simulate the TOR network? Right now I'm thinking using the microdesc cache to avoid contacting any directory authorities. I will create a cluster of about 5 nodes in server mode, all of which only know the other nodes running QUIC. Do you think this will work?
Tor has a TestingTorNetwork config option, which modifies a significant number of settings to enable local testing. If you define your own authorities, they can provide a consensus to your clients. This could be much more reliable than using the microdesc cache. (Please read the TestingTorNetwork entry in the tor manual for more details.)
You may find this easiest to set up using chutney, which is how we test a full tor network on a local machine. Chutney will configure authorities, relays, and clients for you, then launch the network. The easiest way to set it up is to:
- Clone https://gitweb.torproject.org/chutney.git
- Clone your tor repository next to the chutney clone
- Run tor's ./configure
- Run tor's make test-network
If this doesn't work, you might find it easiest to start with the chutney "basic" template: 5. Run src/test/test-network.sh --flavor basic
If you start by running chutney on tor master, you'll get an idea of how it works. Then you can see what breaks when you try it with your QUIC-based tor.
• Anything else bad about the implementation?I haven't seen the implementation. Have you sent me a link, or did you want me to give advice based on the design issues you describe in the email you forwarded?
As far as I can see, your major issues are getting a Tor test network running (I strongly recommend chutney), and dealing with Exits needing QUIC, TCP, and DNS/UDP. If you want interoperability with non-QUIC browsers and servers, you'll need to work out similar issues with Clients and Onion Services as well.
Tim
Thank you! Li.
---------- Forwarded message ---------- From: Xiaofan Li xli2@andrew.cmu.edu Date: Mon, Apr 4, 2016 at 3:20 PM Subject: [TOR+QUIC] Progress Update To: Mashael AlSabah malsabah@gmail.com Cc: Srinivasan Seshan srini@cs.cmu.edu, Kevin Ku kku@andrew.cmu.edu
Hi all, Before the project meetings, we want to debrief you on the recent
progress of the project so that our discussions can be more efficient. In general, this week we made reasonable progress and now we are in the debugging/testing phase. Particularly, we:
• Designed and implemented the interaction between QUIC and thelibevent system inside TOR.
• Integrating QUIC with TOR causes mismatches ofabstractions (see below). We worked hard to come up with a clean design while not blocking development.
• We have to pass the event_base to QUIC for it toregister an alarm callback that is required by QUIC.
• libevent -> TOR-register callbacks -> QUIC interface ->data -> QUIC processing -> only application data gets passed up to TOR
• Minimum code change in TOR: we still use all thecallback mechanisms (timer + signal + async) in TOR but we modify those callbacks.
• Modified code paths inside TOR to accommodate QUIC. • We turned off Tor's SSL code path in the read/writecallbacks. We should probably also turn off the handshake inside TOR as well.
• TOR has a lot of implementation shortcut that makeassumptions about the TOR_SOCKET_T type. We cleaned this up.
• Some remaining issues about direct low-level calls onthe socket (e.g. fcntl or setsockopt). The effects of these issues are unclear right now.
• Small hacks to make sure things ran. • We hardcoded the certificate and private key used byQUIC.
• QUIC is compiled into a shared library and it seems tobe working well. (Don't know about the SSL things right now.)
• Tweaked the our QUIC interface to allow generality and smoothintegration with TOR.
Some challenging problems we encountered: • Mismatched abstractions: This is obviously the most challenging
implementation problem. We hope our experience can perhaps suggestion some designs for the ongoing QUIC interface design:
• SSL/TLS: Fundamentally, QUIC is providing TLS securityon a lower level than what TOR is expecting and this is causing problems. We should take away the TOR's SSL/TLS code to achieve the benefits of QUIC.
• Other Implementations: Compared to the UNIX socket type,our QUIC "socket" provides different interface and functionalities and it is possible that there are assumptions made by TOR about the socket interface. For example, the fact that QUIC requires an alarm system is destroying the abstraction separation between TOR and QUIC. The alarms are used for path discovery, handshakes, retransmission etc. We believe that those should be internal to the lower level transport and thus hidden by QUIC. We are unable to hide them right now because QUIC does not have the necessary OS support to do so. The performance implication is unclear.
• Testing: We are in the phase of testing. We are investigatingvarious testing methods. The key problem is that we have to basically scrap all the systems that TOR already provides because we speak a different protocol. Particularly, we need to:
• Replace or disable the directory authorities. Disablingis easier because we can hardcode the bootstrapping process and make the OP contact our QUIC node, but this way we might not have the most accurate (and comparable) data on the bootstrapping process.
• We need to set up some number of circuits anddifferentiate normal nodes from exit nodes. Right now the exit nodes will also speak QUIC which is not good.
• We will probably need 3-5 machines at least. (SomeAmazon AWS credit maybe??)
Right now, our simple_quic interface is relatively stable and it is
located at: https://github.com/kku1993/simple-quic . It offers a purely generic interface that does not specifically tie to TOR. Feel free to take a look! We are thinking about giving it to the QUIC people at Google after we polish and document it better.
Kevin, feel free to add anything.
Looking forward to the meetings! Li.