We'll have a meeting tomorrow to talk about redesigning changing the format of client–broker rendezvous messages, in order to support client registration methods other than domain-fronted HTTPS. To save time, this message has a proposed agenda and background information to read before the meeting.
- Answer questions about background, the status quo, and future broker development goals. - Message versioning - Inside or outside JSON? - Backward compatibility with legacy-format messages. - Future need for encrypted/signed messages. - How do alternative rendezvous methods interface with the broker? - Same process or separate processes? - If separate, then how does IPC with the broker work? Convert registrations to localhost HTTP, or create some other local communication protocol?
Recent work on this topic is: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
Background
A client sends a message to the Snowflake broker to request proxy service. See https://www.bamsoftware.com/papers/thesis/#p279 for a high-level description. Concretely, the client sends a domain-fronted HTTP POST request with a body that is a JSON serialization of an SDP offer (https://pkg.go.dev/github.com/pion/webrtc/v3#SessionDescription): https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client... A request looks like this: POST /client HTTP/1.1 Snowflake-NAT-Type: restricted
{"type":"offer","sdp":"v=0\r\n..."} If the broker can match the client with a proxy, the response has status 200 and the body is the JSON serialization of an SDP answer: HTTP/1.1 200 OK
{"type":"answer","sdp":"v=0\r\n..."} If the broker cannot make a match, the response has a status of 503 or 504, and an empty body: HTTP/1.1 503 Service Unavailable If there is a syntax problem with the client's offer, the response has status 400: HTTP/1.1 400 Bad Request
The problem with the current message formats is that they rely of features of HTTP not present in other protocols, and they are not easily extensible. The HTTP bodies are assumed to contain an SDP offer/answer and that only, so any side information has to go somewhere else, such as Snowflake-NAT-Type which is sent in an HTTP header (#34129). The 200/503/400 status codes also are not present in other protocols.
Parts of what is conceptually a single message are scattered across the HTTP body, header, and status code. The goal is to move all the information into a common message format that can also be used by other rendezvous channels. That means *all* the necessary information will be present in the HTTP body, and status codes will always be 200.
Some past design sketches: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... When the client requests /client, the broker returns either a 200 with a response body, an empty 503, or an empty 400. This is awkward when doing rendezvous over non-HTTP channels, or even over AMP cache, which doesn't reliably pass through the server's original status code... It would be easier if all the necessary information in the broker's response were in the HTTP body, because that's easier to port to other channels. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... I propose to remove this ambiguity by including the data payload inside the JSON object that represents an error. Just like in an HTTP response, we have two things—a status code and a message—inside one package.
We have already made a similar change to /proxy messages, for example changing status codes 200 and 504 to the strings "client match" and "no match": https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... https://github.com/cohosh/snowflake/pull/8/files We don't have plans to use anything other than HTTP for proxy–broker messages, so it was less important in that case to remove all reliance on HTTP. We are also generally less concerned about backward compatibility for proxies than we are for clients.
cohosh has drafted a merge request that moves almost all message information into JSON in HTTP bodies: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... The above client request would become: POST /client HTTP/1.1
{"Version":"1.0","Offer":{"type":"offer","sdp":"v=0\r\n..."},"NAT":"restricted"} In the event of a match, the responses would be: HTTP/1.1 200 OK
{"Answer":{"type":"answer","sdp":"v=0\r\n..."}} If no match: HTTP/1.1 200 OK
{"Error":"timed out waiting for answer!"} Parsing errors currently still use an HTTP status code: HTTP/1.1 400 Bad Request
This work is primarily towards the goal of supporting other rendezvous methods: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... But we also want to keep in mind other development goals for the Snowflake broker. We want to encrypt and authenticate broker messages in the future, and the message format should facilitate that, without our having to think of another backcompat scheme in the future: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... And we want to be able to move the broker away from being one big process that manages everything. Ideally, different rendezvous methods run in different processes, with reduced privileges, and can crash and be restarted independently of the main broker. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
Summary of the discussion today:
- The JSON formats from https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... look suitable, but the 400 responses should also be somehow encapsulated as errors inside the message body. Internal refactoring of the broker is not necessary at this point, as long as we're reasonably sure the message formats will work for other rendezvous methods. - We talked about where to put message version numbers. cohosh likes the idea of having the version number at the start of the message, outside the JSON container. We talked about an algorithm like this: - Read up to 64 bytes or the first newline. - If the first byte is '{', then this is a legacy-format message. - If there is no newline in the first 64 bytes, then error. - Interpret everything up to the first newline as a version number. - If the version number is not understood, then error. - Read the remainder of the body (up to some sane limit) - Parse the remainder according to the version number. - Should there be a version number on the broker's response messages too? Or can the client assume that the broker returns a response in a format appropriate for whatever the client's registration message was? We can make that assumption now, because we tightly control the broker, but if there were more broker or ones that we don't directly control, it might be harder to guarantee that. I don't remember what we decided on this point. - We brainstormed ideas for splitting the broker into components (#26092). Because this is a matter internal to the broker that does not affect protocol messages, we don't have to decide it now. We discussed placing the broker behind a TLS terminator like Apache, with the broker being a separate localhost web server, as is done with the ProxyPass setup for BridgeDB. Separate TLS termination would help with a future AMP cache rendezvous, which needs to somehow share TCP port 443 with the broker's HTTPS rendezvous. The most expedient way to do this would be to add /amp routes to the broker's existing HTTP handlers. An intermediate way would be to have the broker act like a TLS terminator for /amp routes, and proxy those requests to a separate localhost HTTP server that handles AMP cache rendezvous. cohosh and meskio may proceed with starting to break the broker into components if they have a good vision of how to do it. - Regardless of how the broker is factored, where should message parsing (and in the future, decryption/encryption) happen? Should the rendezvous receivers pass their messages to the broker matching module verbatim, without interpretation? Or should they parse the incoming messages into a uniform in-memory data structure, and pass that to the broker matching module? With encrypted messages, there is no possibility of parsing/interpretation, unless the rendezvous modules are trusted with the broker's long-term decryption key. - A single DNS query doesn't have enough room to contain a client registration. (Needs about 1500 bytes or 500 bytes if compressed, about 140 bytes are available.) We wondered if some domain-aware compression could shrink the message enough: stripping out fields we know are implied or unnecessary, and reinserting them on receipt. On a quick inspection, the a=fingerprint field is 32 bytes of non-compressible data, and a=ice-ufrag and a=ice-pwd are 16 and 24 bytes of high-entropy data. - Currently, the broker may return both status code 503 (Service Unavailable) and 504 (Gateway Timeout): https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/broker... https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/broker... but the client only knows about 503, handling 504 in a default case: https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client...
anti-censorship-team@lists.torproject.org
-
David Fifield