tor-dev March 2020

tor-dev@lists.torproject.org

20 participants
18 discussions

Lets give every circuit its own exit IP?
by nusenu 18 Mar '20

18 Mar '20

The unbearable situation with Google's reCAPTCHA motivated this email (but it is not limited to this specific case). This idea came up when seeing a similar functionality in unbound (which has it for a different reason). Assumption: There are systems that block some tor exit IP addresses (most likely the bigger once), but they are not blocked due to the fact that they are tor exits. It just occurred that the IP got flagged because of "automated / malicious" requests and IP reputation systems. What if every circuit had its "own" IP address at the exit relay to avoid causing collateral damage to all users of the exit if one was bad? (until the exit runs out of IPs and starts to recycle previously used IPs again) The goal is to avoid accumulating a bad "reputation" for the single used exit IP address that affects all tor users of that exit. Instead of doing it on the circuit level you could do it based on time. Change the exit IP every 5 minutes (but do _not_ change the exit IPs for _existing_ circuits even if they live longer than 5 minutes). Yes, no one has that many IPv4 addresses but with the increasing availability of IPv6 at exits and destinations, this could be feasible to a certain extend, depending on how many IPv6 addresses the exit operator has. There are exit operators that have entire /48 IPv6 blocks. problems: - will not solve anything since reputation will shift to netblocks as well (How big of a netblock are you willing to block?) - you can tell two tor users easily apart from each other even if they use the same exit (or more generally: you can tell circuits apart). There might be all kinds of bad implications that I'm not thinking off right now. - check.tpo would no longer be feasible - how can do we still provide the list of exit IPs for easy blocking? Exits could signal their used netblock via their descriptor. What if they don't? (that in turn opens new kinds of attacks where an exit claims to be /0 and the target effectively blocks everything) - more state to track and store at the exit -... some random thoughts, nusenu -- https://mastodon.social/@nusenu twitter: @nusenu_

4 6

Auto close old tor GitHub pull requests?
by teor 16 Mar '20

16 Mar '20

Hi all, I'd like to use a GitHub bot to automatically close old tor and torspec pull requests. The Network Team currently uses GitHub to review pull requests (but we use Trac for task tracking). Some GitHub users won't contribute to a project with lots of open pull requests, because they think it's been abandoned. (Or they think we're bad at merging PRs.) But lots of our PRs get left open because we've squashed and merged the PR, and GitHub doesn't recognise the squashed commits, so GitHub doesn't auto-close the PR. I'm proposing closing PRs with no activity for 60 days, but I think anywhere between 30 and 180 days would work. (Most of our backports happen within 60 days.) For more details, see: https://trac.torproject.org/projects/tor/ticket/33629 T -- teor ----------------------------------------------------------------------

1 0

Request to provide code for Padding in Tor
by DIVYA GUPTA 16 Mar '20

16 Mar '20

Hello, We are under-graduate students from Malaviya National Institute of Technology currently pursuing Computer Science & Engineering. We are working on developing a defense for Tor against Website Fingerprinting Attacks. While reading about tor we came across padding which is implemented in Tor. It would be great if you could make available the code for the padding. Regards, Divya.

2 1

Fwd: Upcoming Tor security releases to fix a denial-of-service issue
by Nick Mathewson 16 Mar '20

16 Mar '20

---------- Forwarded message --------- From: Nick Mathewson <nickm(a)torproject.org> Date: Mon, Mar 16, 2020 at 1:25 PM Subject: Upcoming Tor security releases to fix a denial-of-service issue To: <tor-talk(a)lists.torproject.org> Hello! Some time this week, we currently plan to put out a set of security updates for all supported versions of Tor. These releases will fix a pair of denial-of-service bugs: one that we are classifying at "low" severity, and one that we are classifying at "high" severity. Our recommendation will be for everybody, including relays and clients, to upgrade once packages are available for their platforms. Although these vulnerabilities are "only" denial-of-service issues, any denial-of-service attack against Tor could be leveraged by an attacker to aid in a traffic analysis attack. To the best of our knowledge, these vulnerabilities are not being exploited in the wild. Currently supported release series are 0.3.5, 0.4.1, 0.4.2, and 0.4.3 (alpha). If you have not yet upgraded to one of those, the time to do so is soon. For our policy and process for handing security issues, please see: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Securit… best wishes, -- Nick

1 0

SOOC (Same Origin Onion Certificates) discussion tomorrow
by Richard Pospesel 10 Mar '20

10 Mar '20

Hi Everyone, The current UX situation around HTTPS-enabled onionsites is not the best: Tor Browser shows security warning splash screens and icons in scenarios that do not warrant them. On top of that, the only way for your onion site to not show these warnings is by getting an EV cert which is only practical if you have the $$$. alecmuffet's SOOC proposal ( https://github.com/alecmuffett/onion-dv-certificate-proposal/blob/master/te… ) does seem to fix these UX problems for us. Though the proposal is still incomplete, we can make forward progress with a prototype implementation as the certificate specification is fully fleshed out. We'll be talking about this tomorrow during the Sponsor 27 meeting tomorrow on Tuesday 10th May @ 1500UTC in #tor-meeting. Please feel free to come by if you wish to discuss. best, -Richard

2 1

Walking Onions status update: week 1 notes
by Nick Mathewson 08 Mar '20

08 Mar '20

Hi! On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. If you haven't read it, Walking Onions is a set of ideas meant to transition Tor away from its current dependency on a directory system, to improve scaling in the long term, and performance of low-bandwidth clients in the short term. I'm going assume in the rest of my email that you already have a decent idea of how Tor works, and of the Walking Onions proposal. If you need a Walking Onions refresher, see proposal 300 [PROP300] and the whitepaper that Chelsea Komlo, Ian Goldberg, and I have been working on [WHITEPAPER]. The current scope for this work is to try to solve all the open design questions for Walking Onions, and to write a set of specifications we could use to build the system. I'm aiming to have the initial versions of these specs wrapped up in April. Rather than dumping everything onto the mailing list at once, I'm going to try working in the open, sending weekly updates about my work. Design and specification are a lot of fun for me; I hope you'll have fun too reading along. This past week, I started by writing: - An outline of the set of specifications we'd need to have before we could implement walking onions. (OUTLINE0.txt) - A list of areas of uncertainty that we need to solve before or during writing the specs. (UNCERTAINTY.txt) - A list of related problems that we might wind up solving along with the walking onion specs (OPPORTUNITIES.txt) - A rough grouping of initial tasks, which should eventually turn into a schedule. (PLAN.txt) You can find all these documents in a working git repository [GITREPO] that I've been making for this purpose, if you like. I don't intend for this repository to be a permanent reference or to be necessarily useful for anybody but me: everything important about it should eventually wind up in specifications and on this mailing list. == The biggest area of uncertainty was to pick a meta-format for ENDIVEs and SNIPs. I looked at our existing directory meta-format, along with protobufs, messagepack, and a whole bunch of other systems. I also looked into what kind of efficiency we'd get by just using a hand-tooled non-extensible binary format, so that I could see what kind of overhead we were looking at. The issue here is that I think we don't want to be using a text-based metaformat for SNIPs: we need SNIPs to be as small as possible, and our current text-based directory metaformat only compresses well when we're using huge amounts of it at once. But if we're doing a binary format, we probably shouldn't hand-roll it. Trunnel lets us handle our existing formats safely, but those formats aren't so efficient in practice, and they aren't too standardized. If we use a standardized binary format, then we get encoding, decoding, debugging dumps, optimization, and lots of other stuff "for free". After looking at a pretty huge variety of formats, I think that our best bet is something called "CBOR", standardized as RFC 7049 [CBOR]. It is substantially based on MessagePack, but with a number of improvements. Here are some of CBOR's advantages: + It's optimized first for allowing encoding/decoding to be done with small, simple code -- and second, for having a small binary format. IMO it succeeds at both: the metaformat is simple, and there are decoder implementations in less than 1kb of binary size. + You can think of its data model as a "binary json"; it's very flexible. (yes I know about the other binary jsons) + It's standardized, and there are lots of implementations of it, in lots of different languages. + It has a data definition language [CDDL], which we don't actually need to implement, but which we can use to help write our specs and validate our implementation. + It has a textual encoding for debugging, and a one-way mapping to json. Here are some of its disadvantages: - It doesn't have a built-in object mapping like protobufs does. - It allows multiple encodings for a single piece of data. (The CBOR RFC explains how to make a canonical encoding.) - Our diff story gets slightly more complex when we have a binary format. (I'd be happy to talk about other possibilities, but please make sure you know about the bikeshed first. [BIKESHED]) == Doing diffs from one ENDIVE to the next will be helpful if we want to keep relay bandwidth on the small side. We'll need binary diffs for this. Fortunately, I think that shouldn't be too hard to do with our existing diff code: instead of doing line-by-line diffs, we'll just do CBOR-item-by-item diffs, and encode them in a byte-oriented way. I've got a prototype half-written to make sure this is right. The simplicity of CBOR makes this pretty easy. == There are a few ways to authenticate SNIPs. It's looking like right now the most efficient will be merkle trees, with the root of the tree signed by a threshold signature algorithm like BLS. One of the neat things about merkle trees is that if we are transmitting the leaves and the signature on the root, we don't need to actually transmit the intermediate nodes as part of the ENDIVE. So that's cool, and it will make ENDIVE diffs smaller. One of the not-so-neat things is that the merkle tree paths are a bit long. There are ways to make them shorter: you only need to transmit one digest for each layer, plus a bit to say which hash it is. (I learned about these tricks while working on [WHITEPAPER], thanks to Ian and Chelsea.) One risky thing I've been thinking about is whether we can use a shorter-than-256-bits hash for the merkle tree nodes, if we don't need to worry about collision resistance for the digest functions. The SNIPs are generated by authorities, but they do contain a certain amount of adversary-created material. Randomized hashes with an unpredictable key might help, particularly for intermediate nodes on the tree? [XMSS] has some ideas here. (Suggested by Jack Lloyd) == In the coming week, I'm hoping to finish up making my draft schedule, start doing research as needed to handle more open questions, and write an initial spec for the ENDIVE and SNIP formats. [BIKESHED] "Why Should I Care What Color the Bikeshed Is?" http://bikeshed.com/ [CBOR] RFC 7049: "Concise Binary Object Representation (CBOR)" https://tools.ietf.org/html/rfc7049 [CDDL] RFC 8610: "Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures" https://tools.ietf.org/html/rfc8610 [GITREPO] https://github.com/nmathewson/walking-onions-wip [PROP300] "Walking Onions: Scaling and Saving Bandwidth" https://gitweb.torproject.org/torspec.git/plain/proposals/300-walking-onion… [WHITEPAPER] "Walking Onions: Scaling Anonymity Networks while Protecting Users" https://crysp.uwaterloo.ca/software/walkingonions/ [XMSS] RFC 8391: "XMSS: eXtended Merkle Signature Scheme". https://tools.ietf.org/html/rfc8391

1 0

Regarding GSoC 2020
by Aman Rastogi 05 Mar '20

05 Mar '20

Respected Developers, I am Aman, GSoC 2020 aspirant, and i would like to work for Torr. I am interested in "Privacy Friendly Web" and "OONI Explorer Advanced Search". Can I get some guidance, please? Thanking you, and sorry for such an informal mail. -- Aman

2 1

Request for onionbalance v3 pre-alpha testing
by George Kadianakis 03 Mar '20

03 Mar '20

Hello list, we've been developing Onionbalance v3 for the past months, and I'm pretty hyped to say that the project has reached a stability point that could benefit from some initial testing by curious and adventurous developers and users. The project is not yet ready for proper use in actual production environments (more on this later), but my hope is that this testing will reveal bugs that I can't catch in my test environment and receive user feedback that will speed up the overall process and allow a faster and more stable initial release. This email features a pretty complicated pseudo-guide that if it gets followed to completion it will hopefully result in a functional onionbalance setup. As I said the guide is complicated and in ugly text format, so I suggest you put on some calming music (perhaps some Bill Evans or some Com Truise or some Tor) and go slowly. Trying to speed run this will result in disappointment and perhaps even partial or total loss of faith in humanity. == So what's the current status of Onionbalance v3? == The project lives as a fork of Donncha's original Onionbalance project in this repository: https://github.com/asn-d6/onionbalance The current status is that during my testing I have onionbalanced a real v3 service with two backend instances successfully for a few days with no obvious reachability issues. It also works on a private tor network (using Chutney) for hours with no issues. All the above happened on Debian stable and testing. Also, here are some features that are currently missing but will be included in the initial launch: - This new onionbalance will support both v2 and v3 onion services, altho currently v2 support is botched and only v3 will work for the purposes of this testing. If you can quickly revive v2 support patches are welcome! - Any sort of documentation about v3 mode (this post is all the documentation there is for now!) - Missing streamlined setup process (installation scripts, packages or dependency tracking) - Code documentation and unittests missing - There is no daemon mode. You will have to run this in screen for now - Fun bugs included! Furthermore, v2 and v3 support will not have feature parity at the time of the initial launch, and we will have to incrementally build these features: - Cool v2 features like DISTINCT_DESCRIPTORS are not yet implemented so v3 cannot load balance as deeply as v3 yet (patches welcome!) - There is no way to transfer your existing v3 onion service to onionbalance (patches welcome!) - There is no torrc generation for instances as in v2 (patches welcome!) - It will be possible for clients and HSDirs to figure out whether a descriptor comes from an onionservice vs a regular v3 onion. Making them indistinguishable is possible but requires more engineering and will be left as a TODO for now (see #31857) Finally, there is no guarantee that configs generated with this pre-alpha version of Onionbalance will be forward compatible when we actually launch the project. This means that if you setup onionbalance v3 right now, you might need to re-set it up in a month or so when the actual release happens. tl;dr This is a call for experimental testing and it's meant to help the developers and accelerate development and not to be used in anything remotely serious or close to production. The actual release for operators and users is coming in a month or so. == Before we get started... == OK so let's assume that you still want to help out! Given that this is a pre-alpha test I'm gonna assume that you fill the following prerequisites otherwise you are gonna have an adventure: [ ] Comfortable with git [ ] Familiar with v2 onionbalance (at least in terms of terminology and how it works) [ ] Familiar with setting up and configuring (v3) onion services [ ] Familiar with Linux (or in the mood to find weird bugs in other platforms) [ ] Familiar with compiling C projects and chasing their dependencies [ ] Familiar with running Python projects and chasing their dependencies [ ] Patience to push through various import error messages that might appear The above are not mandatory but if you don't check out all the boxes you might encounter various errors and roadblocks that will frustrate. Actually... even if you check out all these boxes you might encounter various errors that might frustrate you. == Some theory == In any case, if you want to learn more about the architecture of Onionbalance, see: https://blog.torproject.org/cooking-onions-finding-onionbalance As part of this guide, you might hear me say the term "frontend" which is what we call master onionbalance onion service instance (whose address clients type on their browser), or the term "backend" which is the load balancing instances that do the actual introduction/rendezvous work for the frontend. So for example, in the visual below, Alice fetches the frontend's descriptor but does the introduction/rendezvous with the backends: ================================================================================ + + + Alice fetches frontend +^^^^^^^^^^^^^^^^^^^^^^^^^^^^+ + + +~~~~~~~~~~+ descriptor +~~~~~~+ HSDir hash ring ) + + | Alice +~~~~~~~~~~~~~~~~~~~+ +vvvvvvvvvvvv+vvvvvvvvvvvvvvv+ + + | | | + + |the client| +-----------+ | + + +~~~+~~~~~~+ |backend | | Upload frontend + + | |instance 1 +-------+ | descriptor + + | +-----------+ | | + + | | +-------+-----------+ + + | +-----------+ +----+ | + + +~~~~~~~~~~~~~+backend | | frontend | + + Alice does |instance 2 +------------+ onionbalance | + + introduction +-----------+ +----+ service | + + with backend | +-------------------+ + + +-----------+ | + + |backend +-------+ + + |instance 3 | + + +-----------+ + + + ================================================================================ OK enough with theory. Let's jump to practice: == Installation instructions == For the purposes of this guide I will assume you have a frontend service and a bunch of backend instances that you want to load balance. ==== Setup frontend service ==== Our first goal is to get onionbalance and Tor running on your frontend service. - Installing Tor Let's start: SSH to your frontend service, and get yourself a Tor binary to use as the control port for onionbalance. You will want a recent version of Tor (above 0.4.2) but other than that there is no strict requirement. Feel free to get it off your package manager or git. Setup a minimal torrc with a control port enabled. As an example: """ SocksPort 0 ControlPort 127.0.0.1:6666 DataDirectory /home/user/frontend_data/ """ Feel free to tweak your torrc as you feel (also enable logging), but for the purposes of this guide I assume that your control port is at 127.0.0.1:6666. - Running onionbalance for the first time Now we need to run onionbalance and generate keys and a config file for it. Here the fun starts: Let's start by git cloning 'https://github.com/asn-d6/onionbalance'. To run onionbalance v3 you are gonna need a bunch of fun Python dependencies including (but perhaps not limited to): - Python3 above 3.7 - Python3 packages for: pyyaml, pycrypto, setproctitle, future - stem version 1.8 or up (so that it includes v3 support) Please collect all the above whichever way you prefer: if you don't have them, things are gonna crash at some point. There is no dependency tracking for v3 onionbalance right now (i.e. requirements.txt) so I don't enforce the above in an elegant way. Now that you are good to go with dependencies, feel free to install onionbalance using the setup.py script, or you can run it from the source directory if you prefer that. Now is the time to create keys and a config file for your frontend service. Let's do that by running the configuration wizard: $ python3 onionbalance-config.py It should ask you if you want v3 or v2 (say v3!), and a bunch of other questions and by the end you should have a config file somewhere (by default at ./config/master/config.yaml) and the corresponding frontend key file (again at ./config/master/*key). The onion address of your frontend service can be found in the bottom of your config file. So if it says: key: wrxdvcaqpuzakbfww5sxs6r2uybczwijzfn2ezy2osaj7iox7kl7nhad.key your onion address is "wrxdvcaqpuzakbfww5sxs6r2uybczwijzfn2ezy2osaj7iox7kl7nhad.onion" For now, note down the onion address of the frontend service and let's move on to the next step! Make sure you got a v3 onion address out of this process, otherwise something went wrong. If you got to this stage without any major turbulance, you will end up with a functional onionbalance in no time. If you had lots of problems so far, brew up some green tea, because there is more fun coming! ==== Setup backend instances ==== OK now with the frontend onion address noted down, let's move to setting up your backend instances: SSH to one of your backend instances and let's setup Tor in there! The catch is that onionbalance v3 backend instances need a very recent version of Tor because of some issues in the v3 protocol (#32709). As a matter of fact, the patch required has not yet been merged to upstream Tor (it will be merged in mid February because we are currently at a code freeze) so you will need to use my personal repository for now (as indicated by the last comment in #32709). So let's git clone 'https://github.com/asn-d6/tor.git' and checkout branch 'ticket32709_044_02'. Build that branch (using the classic './autogen.sh && ./configure && make' combo) and you should have a Tor binary at ./src/app/tor. If this doesn't happen, you are gonna need to install various C dependencies like libssl-dev, libevent-dev, etc. Now you will need a torrc file for your backend instance. As in onionbalance v2, the torrc file needs to setup an onion service (and in this case a v3 one). So far so good but here comes the twist: 1) In the end of your torrc file, inside the HiddenService block please add the following line: 'HiddenServiceOnionBalanceInstance 1'. 2) Now go in your hidden service directory where the hostname and hs_ed25519_public_key files are living (you will need to startup Tor once with the onion service enabled for the directory to get created) you need to create a new file with the name 'ob_config' that has the following line inside: 'MasterOnionAddress wrxdvcaqpuzakbfww5sxs6r2uybczwijzfn2ezy2osaj7iox7kl7nhad.onion' but substitute the onion address with your frontend's onion address. The points (1) and (2) above are extremely important and if you didn't do them correctly, nothing is gonna work. If you want to ensure that you did things correctly, start up Tor, and check that your *info* log file includes the following line: [info] ob_option_parse(): OnionBalance: MasterOnionAddress wrxdvcaqpuzakbfww5sxs6r2uybczwijzfn2ezy2osaj7iox7kl7nhad.onion registered If you don't see that, then something went wrong. Please try again from the beginning of this section till you make it! This is the hardest part of the guide too, so if you can do that you can do anything (fwiw, we are at 75% of the whole procedure right now). After you get that, also make sure that your instances are directly reachable (e.g. using Tor browser). If they are not reachable, then onionbalance won't be able to see them either and things are not gonna work. OK, you are done! Now do the same for the other backend instances and note down the onion addresses of your backend instances because we are gonna need them for the next and final step. ==== Get it running! ==== OK now let's login back to the frontend server! Go to your onionbalance config and add your instance addresses in the right fields. In the end it should look like this (for a setup with 4 backend instances): """ services: - instances: - address: 4a2zwxetkje5lwvahfv75arzrctn6bznkwmosvfyobmyv2fc3idbpwyd.onion name: node1 - address: i5aqar7fgqcsqwalwjltrb3w3xyj23ppgsnuzhhkzlhbt5337aw2joad.onion name: node2 - address: uutpxl2cfkk5qa7z4xtgdxssnhutmoo2y2rw6igh5ez4hpxaz4dap7ad.onion name: node3 - address: 5xeigil3hjxltcs3ti4skerhpc252ci7ejynrtm73v3bcbxlfuulsjqd.onion name: node4 key: wrxdvcaqpuzakbfww5sxs6r2uybczwijzfn2ezy2osaj7iox7kl7nhad.key """ Now let's fire up onionbalance by running the following command: $ python3 onionbalance.py -v info -c config/master/config.yaml -p 6666 If everything went right, onionbalance should start running and after a bit your frontend service should be reachable!!! You are an onionbalance expert! If something did not go right, that's OK too, don't get sad because this was quite complicated. Please check all your logs and make sure you did everything right according to this guide. Keep on hammering at it and you are gonna get it. If nothing seems to work, please get in touch with some details and I can try to help you. == Now what? == Now that you managed to make it work, please monitor your frontend service and make sure that it's reachable all the time. Check your logs for any errors or bugs and let me know if you see any. If you want you can make onionbalance logging calmer by using the '-v warning' switch. (Also if you see this log message in your frontend Tor logs: "Bug: Unrecognized signal 132..." that's normal and it will be fixed with #33104). If you are in the mood for coding feel free to start hacking on onionbalance, but I would suggest you first get in touch over Github because I'm also working at it at the same time and it would be a shame if we do duplicate work. If you find bugs or do any quick bugfixes, please submit them over Github! If you do code review, please let me know through Github of any issues you find. Also, if you are gonna be at FOSDEM this upcoming weekend feel free to get in touch if you have any questions or you want to live test it. If you have any other feedback, I'd be glad to hear it! == Next steps == I'm gonna be working on this for the next month: fixing bugs, writing documentation, writing tests, creating packages, submitting and fixing bug reports in tor and stem etc. Expect more updates on this mailing list, and also a blog post to announce the initial launch of the project. Finally, if there are any errors on this post, I will reply to this thread with an errata! Hope this works for you and you have fun onionbalancing your services! Thanks a lot for the testing and looking forward to your feedback! o/ PS: Major thanks to Donncha for the super clean onionbalance codebase (I did not expect to reuse so much code), the Tor network team for the support, and Damian for all the work on stem v3 support! Also OTF for the $$$ ;)

1 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-dev March 2020