On Thu, Sep 14, 2023 at 08:22:22PM +1000, Matt Jolly wrote:
I recently inadvertently opened a much larger can of worms than I'd intended when fixing a bug reported downstream where cURL would, when configured with certain DNS backends, fail to resolve .onion addresses.
[...]
Hi Matt,
Thanks for the detailed problem description and references, and I'm sorry to hear about the trouble you're getting when trying to fix it.
I've asked in the Tor Specifications issue (inspired by Silvio's suggestions), and in the cURL issue, but I seem to be getting nowhere and the impacted users are clamouring for a quick band-aid solution, which I feel will work out worse for everyone in the long run:
How can client applications (safely):
1.discover that they're in a Tor-enabled environment 2.resolve onion services only via Tor in that circumstance 3.not leak .onion resolution attempts at all
Right now, not making these requests in the first place is the safest (and correct) thing to do, however inconvenient it may be. Rather than immediately trying to come up with a band-aid approach to this problem, a sane mechanism needs to be implemented to:
1.prevent each application from coming up with their own solution 2.prevent inconsistency in .onion resolution (i.e. no "oh it only leaks if DO_ONION_RESOLUTION is set") 3.provide a standardised mechanism for applications that want to be Tor aware to discover that they're in a Tor-enabled environment.
I'm not particularly attached to that last point, but it's worth discussing.
On a related note: -is the use of a transparent proxy recommended?
I consider it a valid use case. The only concern I'm aware is about proxy leaks. I don't see any reason not to recommend transparent Tor proxies if using a configuration that avoids such leakage.
-is there a sane alternative that involves as minimal configuration as possible for these users?
Can't think of anything right now besides what was already covered in length in the threads you mentioned. Instead, I'll focus in the overall problem.
I'm not sure what the best way forward is here, but I'm hoping that actual Tor developers might have a useful opinion on the matter, or at least be able to point me in the right direction.
The long analysis below does not necessarily represents an "official Tor position". It may be frustrating that there is no "official" position right now, maybe because this still need further discussion, and also because it may be a hard topic.
# Interpreting RFC 7686
Let's start with an hermeneutical exercise, by reading carefully what RFC 7686 says:
- Application Software: Applications (including proxies) that implement the Tor protocol MUST recognize .onion names as special by either accessing them directly or using a proxy (e.g., SOCKS [RFC1928]) to do so. Applications that do not implement the Tor protocol SHOULD generate an error upon the use of .onion and SHOULD NOT perform a DNS lookup.
Applications MUST support .onion either directly or through a proxy. The RFC gives SOCKS as an example, but it could as well have mentioned transparent proxies. So applications could easily allow environment variables, command line flags, configuration parameters or "environment detection" to indicate that they support .onion domains.
I don't see why an upstream shouldn't allow users and operating systems toggle the behavior regarding RFC 7686. Seems like the best default approach so far is to generate an error, but nothing is blocking applications to expose ways users can indicate that their systems acts as proxies for .onion.
I suggest we take a corollary from this RFC portion: applications implementing the Tor protocol are those which either access .onion addresses directly or use a proxy that supports Tor. If the application does not support any way to properly resolve .onion addresses, then it must make sure to refuse any attempts to resolve .onion addresses. Nothing is blocking an application to operate in two modes: rejecting .onion resolution attempts by default, except if it's told to do so.
I agree that "implement the Tor protocol" is rather vague. The "Tor protocol" might mean a lot of stuff, but I'm assuming here the _intention_ behind this statement is this: in order to build a Tor-enabled application, one has to interface with Tor somehow (being a Tor client or using a proxy).
As for curl, it seems to me that upstream wants uniform behavior with all DNS backend libraries. And c-ares is rejecting .onion resolution, so for the sake of uniformity curl is just rejecting it for all backends.
But the RFC explicitly mentions "Applications" and not "Libraries". Shall we assume then that the RFC is referring to curl, but not to c-ares?
Would c-ares maintainers be free do follow whatever they prefer?
Or this is not a concern for c-ares, since it's a library and not an application? In this case they don't need to comply with the RFC, and could just rollback this change, so curl maintainers could be more open to accept toggling the RFC behavior.
We could say that users interact with applications, and these rely on libraries. By the RFC interpretation above, it's in the application level that .onion resolution should be allowed or not. It just don't make any statement about what libraries SHOULD or SHOULD NOT do, or even how a system resolver should behave.
I don't really think curl maintainers should accept a toggle only if c-ares is rolled back, but maybe this could help change their minds.
Alternatively, c-ares could provide some optional initialization parameter indicating it should actually try to resolve an .onion address, if that's feasible.
Also note the use of SHOULD in the RFC. As of BCP 14 / RFC 8174:
SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.
From the discussion so far, there are enough "valid reasons in particular circumstances" when things can be done differently. I haven't followed RFC 7686's process, but maybe the authors and reviewers had carefully chosen the terminology to acknowledge that there might be special cases -- such as transparent proxies -- where the RFC requirements can be relaxed, given that implementors know what they're doing.
Hope this gives enough arguments to argue for implementing a tunable to account for the transparent proxy use case in curl and in other software.
# Standardization
What seems more difficult is to standardize a way that any application from any operating system could follow some setting, or auto-detect it's behind an .onion-enabled proxy.
I don't see an easy way to solve this that would make everyone happy in the short term.
But it would be possible to split proposals by timeframe (short, mid and long term) and scope (specifications, libraries, applications, operating systems).
Short timescales and "band-aid" fixes usually happen within libraries and applications.
Operating system changes are somehow mid-term.
Tor specs ranges from quick fixes to mid or long term.
IETF standards are long term. Could take years to have a RFC amendment, and years more to implementors follow the it. RFC 7686 could have improved wording and be more detailed or nuanced in the possibilities. Right now we don't have any concrete plans or people engaged in amending the RFC, but here and there people are discussing this. Example: https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/85
As for the quick fixes, I don't see nothing beyond advocating patches implementing flags, environment variables, parameters etc for a RFC 7686-compliant tunable. Unfortunately this have to be done in a case-by-case basis, but having RFC 7686 may actually be an advantage when requesting such changes. I imagine that not having the RFC would make way harder to propose anything related.
# Approaches for transproxy auto-discovery
Now let's talk a bit about long-term, idealized fixes.
What follows is just brainstorming.
It could be considered some method for transproxy auto-discovery, be it some widespread standard or only followed by a subset of related applications and operating systems.
An .onion auto-detection feature could be implemented on applications by trying to detect some special-use domain like "proxy.onion" in two ways:
1. By supporting a local hostname similar to "localhost" in many systems. If a name "proxy.onion" resolves locally (say to 127.0.0.1), then an application (or library) could assume it's behind a proxy, be it SOCKS5h or in a system with Tor transparent proxy.
2. By having an .onion-aware DNS resolver return a valid entry for "proxy.onion": * That won't leak the address, but may leak to the DNS that someone is using a given library, which may be harmless if it's widespread.
* The query could be for a TXT record, which could give more space for custom replies, but could be incompatible with the option 1 above.
Upon initialization, an application could do this test to determine whether it should try to resolve .onion addresses.
The Tor client would have to change as well, in order to support the special "proxy.onion" address instead of triggering a malformed .onion address error.
Thanks for your time, Cheers, Matt
Thanks for all your effort and dedication put into this issue :)