Hi tor-dev,
I care deeply about the ability for people to have anonymity online, and I've always been really interested in exactly how Tor works.
So for a class project last semester I wrote oppy, an onion proxy in 100% Python. I spent most of my winter break cleaning it up and writing documentation, and today I'm releasing oppy as a free software project :)
code: https://github.com/nskinkel/oppy docs: https://nskinkel.github.com/oppy
Here's a brief description of the project, with some questions I have for tor-devs at the end.
@@@ WARNING @@@
*DO NOT* use oppy if you want anonymity (see warnings on the project README and the documentation homepage). It's currently just a prototype and is not meant for normal client usage. ---------
oppy works like a regular Tor client (i.e. any application that can use SOCKS 5 can use oppy), and it can even work with the Tor Browser Bundle if you fiddle with the TorButton settings to use oppy instead of the normal Tor process. See: https://nskinkel.github.io/oppy/usage.html
There are a number of simplifications made, with the major ones primarily centering around circuit management/build logic and how and when network status documents are collected (see: simplifications.md in the repo or https://nskinkel.github.io/oppy/simplifications.html). So oppy doesn't currently work *exactly* the same as a standard Tor OP according to tor-spec.
oppy seems to get reasonable performance (for an OP at least). I haven't done any serious performance tests so this is all anecdotal, but oppy seems to work fast enough for normal usage (e.g. I can generally stream videos without issues).
I hope anyone on this list who is interested will try it out and/or have a look at the code! :)
I have a few questions for tor-devs:
1. Do you think this project is/could be interesting, useful, or potentially beneficial as an addition to the world of Tor software?
2. If so, do you see any major use-cases for oppy? At this point it's still really just a prototype, so if anyone has some really cool use-cases for oppy I could try to design future modifications to support them. Some things I had considered so far:
- a tool for research and/or rapidly prototyping new protocol modifications (e.g. new kinds of handshakes, circuit management behavior, or path selection)
- a tool for testing/fuzzing Tor. With a few modifications, oppy could pretty easily be turned into a somewhat-advanced fuzzer (e.g. it'd be easy to do something like "After building C circuits and sending R relay data cells on circuit X, send N number of malformed cells of type Y to the middle node on circuit X").
- a drop-in OP replacement for specialized applications. oppy is still a pretty long way from this being a viable option, but maybe someday people could use oppy as a regular Tor client for certain use-cases (e.g. if someone wants to ship a full Python Tor stack for a locked down, special-purpose application using Tor).
3. If yes to 1 or 2, would anyone here be interested in hacking on oppy with me? :)
I had a *ton* of fun building oppy and learned a lot about Tor in the process, and I'd love to continue working on it.
I'm currently thinking about what a project roadmap for oppy could look like, but whether or not I continue hacking on oppy to make it a solid piece of software (rather than just a prototype) or just leave it as is as a reference depends on whether or not the Tor development community sees any real uses or future potential for the project.
Thanks for reading!
All the best, Nik
Hi Nik, very nice work! We love seeing alternative tor implementations since it gives us the chance to test if specs are up to snuff (Orchid is the only other one that comes to mind, but that has been inactive since 2013 [1]). Personally I've been curious if we can shift core tor to Python, Ruby, or Go for some functionality like descriptor handling (C isn't exactly renowned for string parsing, and memory management could make things safer), but that's Nick's show and he has good reasons for it to be pure C.
Your codebase mentions that you had trouble with the ExitPolicy's can_exit_to() when you omit an address. Could you please provide an example of a policy you were having trouble with and the expected behavior? From the docs and code it sounds like if you omit an address it should be permissive. [2][3]
- Do you think this project is/could be interesting, useful, or
potentially beneficial as an addition to the world of Tor software?
Certainly! Always happy for new people to get involved. :)
- If so, do you see any major use-cases for oppy?
Interesting ideas. I'm not entirely sure how Oppy dovetails with any current work. First thought was 'I wonder if there's anything good to merge into Stem' and second was 'maybe this could benefit Chutney or other core tor work?'. Nick, thoughts?
- If yes to 1 or 2, would anyone here be interested in hacking on oppy
with me? :)
Personally I try to stay pretty focused on Stem and arm but if this treads into those areas I'd be delighted to work with you. Wish you the best of luck finding people to collaborate with though - most of our projects are one man efforts, and that's unfortunate.
Cheers! -Damian
[1] https://subgraph.com/orchid/index.en.html [2] https://stem.torproject.org/api/exit_policy.html#stem.exit_policy.ExitPolicy... [3] https://gitweb.torproject.org/stem.git/commit/?id=caee7d6c
Hi Damian,
Thanks for the reply!
Your codebase mentions that you had trouble with the ExitPolicy's can_exit_to() when you omit an address. Could you please provide an example of a policy you were having trouble with and the expected behavior? From the docs and code it sounds like if you omit an address it should be permissive. [2][3]
Yes, thanks for the reminder. I meant to do this earlier but apparently forgot. I just added an issue on Trac with a more complete description and code to reproduce: https://trac.torproject.org/projects/tor/ticket/14314
I'm still not actually sure if this is a bug or if I'm just misinterpreting the documentation.
If it is in fact a bug, I can work on putting together a patch when I have some time this weekend if you/someone else hasn't gotten to it yet.
We love seeing alternative tor implementations since it gives us the chance to test if specs are up to snuff (Orchid is the only other one that comes to mind, but that has been inactive since 2013 [1]).
Great! I actually did find a couple things that were either worded in a confusing way (I thought) or mistyped in `tor-spec.txt`, and when I have a few minutes later this week I'll send a patch.
- Do you think this project is/could be interesting, useful, or
potentially beneficial as an addition to the world of Tor software?
Certainly! Always happy for new people to get involved. :)
Cool! I wasn't sure how y'all felt about other implementations, so that's great to hear. I'll keep hacking on oppy then :) There are a few (relatively straight-forward) user-facing issues that make it slightly annoying to use right now, but I think I can fix those pretty easily.
There are a number of more serious core things (e.g. directly affecting anonymity) that need to be fixed/implemented, however, and that leads to...
- If so, do you see any major use-cases for oppy?
Interesting ideas. I'm not entirely sure how Oppy dovetails with any current work. First thought was 'I wonder if there's anything good to merge into Stem'
Now that I think about it, something that would be great to have in Stem would be path selection capabilities. So something like, say, given a list of RelayDescriptors and some constraints that must hold for a path, return some randomly chosen path that satisfies those constraints. A starting point might be just implementing Tor's default path selection algorithm with some basic options the caller can configure, but it would be nice to have the ability to do more novel things too (e.g. build a path using only exits in country X with the 'BadExit' flag, using only nodes in \16 Y for the middle position, etc.). This could be pretty cool from a research/testing standpoint, I think.
One of the next things I need to do for oppy is implement Tor's full path selection algorithm. However, if you think it would be appropriate, it'd be great if some of this could eventually just wind up in Stem (so I could make oppy's code simpler and also so other people could use these capabilities more easily).
How would you feel about adding path selection capabilities to Stem?
Aside from allowing one to do nice things with descriptors, this could also work well with some method analogous to stem.control.Controller.new_circuit() that could do stuff like auto-choose a path for you based on some desired path attributes the caller specifies.
oppy has some (very minimal) path selection functionality that can maybe serve as a starting point, but it has some issues (it was hacked together pretty quickly to hit course deadlines, it doesn't try to be efficient, it returns/uses a Twisted deferreds because of how it gets descriptors, etc.).
I'd be more than happy to work on/help out with this, contribute code/docs, etc., if you think something along these lines might be appropriate for inclusion in Stem. I'll need to do this for oppy anyway, so if you think this might have a place in Stem it'd be nice to consolidate effort in one place :)
Thoughts?
All the best, Nik
PS: Thanks for writing/maintaining Stem! It's a really great library and made writing oppy and working with network status docs and descriptors *much* simpler.
On 01/20/2015 11:21 AM, Damian Johnson wrote:
Hi Nik, very nice work! We love seeing alternative tor implementations since it gives us the chance to test if specs are up to snuff (Orchid is the only other one that comes to mind, but that has been inactive since 2013 [1]). Personally I've been curious if we can shift core tor to Python, Ruby, or Go for some functionality like descriptor handling (C isn't exactly renowned for string parsing, and memory management could make things safer), but that's Nick's show and he has good reasons for it to be pure C.
Your codebase mentions that you had trouble with the ExitPolicy's can_exit_to() when you omit an address. Could you please provide an example of a policy you were having trouble with and the expected behavior? From the docs and code it sounds like if you omit an address it should be permissive. [2][3]
- Do you think this project is/could be interesting, useful, or
potentially beneficial as an addition to the world of Tor software?
Certainly! Always happy for new people to get involved. :)
- If so, do you see any major use-cases for oppy?
Interesting ideas. I'm not entirely sure how Oppy dovetails with any current work. First thought was 'I wonder if there's anything good to merge into Stem' and second was 'maybe this could benefit Chutney or other core tor work?'. Nick, thoughts?
- If yes to 1 or 2, would anyone here be interested in hacking on oppy
with me? :)
Personally I try to stay pretty focused on Stem and arm but if this treads into those areas I'd be delighted to work with you. Wish you the best of luck finding people to collaborate with though - most of our projects are one man efforts, and that's unfortunate.
Cheers! -Damian
[1] https://subgraph.com/orchid/index.en.html [2] https://stem.torproject.org/api/exit_policy.html#stem.exit_policy.ExitPolicy... [3] https://gitweb.torproject.org/stem.git/commit/?id=caee7d6c _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Yes, thanks for the reminder. I meant to do this earlier but apparently forgot. I just added an issue on Trac with a more complete description and code to reproduce: https://trac.torproject.org/projects/tor/ticket/14314
Thanks Nik! Think I know what's going on and agreed that it doesn't match the docs.
I actually did find a couple things that were either worded in a confusing way (I thought) or mistyped in `tor-spec.txt`, and when I have a few minutes later this week I'll send a patch.
Great!
Now that I think about it, something that would be great to have in Stem would be path selection capabilities. So something like, say, given a list of RelayDescriptors and some constraints that must hold for a path, return some randomly chosen path that satisfies those constraints.
I'd be up for that. This isn't a very commonly requested piece of functionality, but it was part of TorCtl (which Stem deprecated in 2012)...
https://gitweb.torproject.org/pytorctl.git/tree/PathSupport.py
A starting point might be just implementing Tor's default path selection algorithm
This part concerns me a tad since it would be duplicating a bit of tor and likely a moving target. Path selection isn't as simple as "pick some random relays", rather it's based on bandwidth authority measurements, stability, and other heuristics. Not to mention guards, families, bad-exit flags, etc.
One of the next things I need to do for oppy is implement Tor's full path selection algorithm. However, if you think it would be appropriate, it'd be great if some of this could eventually just wind up in Stem (so I could make oppy's code simpler and also so other people could use these capabilities more easily).
How would you feel about adding path selection capabilities to Stem?
Gotcha. Honestly I haven't looked into path selection much. If it turns out to be reasonably straight forward and a maintainable thing for Stem to include then great.
Aside from allowing one to do nice things with descriptors, this could also work well with some method analogous to stem.control.Controller.new_circuit() that could do stuff like auto-choose a path for you based on some desired path attributes the caller specifies.
Agreed, that would be neat.
PS: Thanks for writing/maintaining Stem! It's a really great library and made writing oppy and working with network status docs and descriptors *much* simpler.
My pleasure! Glad you found it useful. :)
Cheers! -Damian
On Wed, 21 Jan 2015 10:20:36 -0800 Damian Johnson atagar@torproject.org wrote:
Now that I think about it, something that would be great to have in Stem would be path selection capabilities. So something like, say, given a list of RelayDescriptors and some constraints that must hold for a path, return some randomly chosen path that satisfies those constraints.
I'd be up for that. This isn't a very commonly requested piece of functionality, but it was part of TorCtl (which Stem deprecated in 2012)...
https://gitweb.torproject.org/pytorctl.git/tree/PathSupport.py
There's a mostly complete implementation in torps, that uses stem. The code isn't very Python-like as afaik it's a straightforward conversion of the tor path selection code. The last time I tried this, downloading and processing all the documents took forever and consumed a hilarious amount of RAM, but I belive stem has seen performance improvements since I tried using it for this.
https://github.com/torps/torps
Regards,
There's a mostly complete implementation in torps, that uses stem. The code isn't very Python-like as afaik it's a straightforward conversion of the tor path selection code. The last time I tried this, downloading and processing all the documents took forever and consumed a hilarious amount of RAM, but I belive stem has seen performance improvements since I tried using it for this.
It has. Stem 1.3.0 had a 40% improvement, and in just a few days it'll get even better. I've been working on a feature branch that does lazy loading of descriptors. Very, very close to being done. I'll be sending an announcement with benchmarks when it's merged.
Cheers! -Damian
Yawning Angel,
This is great, thanks for posting. I wasn't aware of torps, and this will save a bunch of work.
Best, Nik
On 01/21/2015 01:08 PM, Yawning Angel wrote:
On Wed, 21 Jan 2015 10:20:36 -0800 Damian Johnson atagar@torproject.org wrote:
Now that I think about it, something that would be great to have in Stem would be path selection capabilities. So something like, say, given a list of RelayDescriptors and some constraints that must hold for a path, return some randomly chosen path that satisfies those constraints.
I'd be up for that. This isn't a very commonly requested piece of functionality, but it was part of TorCtl (which Stem deprecated in 2012)...
https://gitweb.torproject.org/pytorctl.git/tree/PathSupport.py
There's a mostly complete implementation in torps, that uses stem. The code isn't very Python-like as afaik it's a straightforward conversion of the tor path selection code. The last time I tried this, downloading and processing all the documents took forever and consumed a hilarious amount of RAM, but I belive stem has seen performance improvements since I tried using it for this.
https://github.com/torps/torps
Regards,
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
I've barely had time to poke at this much, but it's really neat. I was actually originally going to call "txtorcon" simply "txtor" but figured I'd leave that name in case anyone wrote an actual Tor implementation as a Twisted protocol -- which I guess you've now done :) [...and you're welcome to the name if you want].
Anyway.
For sure "prototyping stuff" was the first thing that sprang to my mind as being useful; I've always had good success using Python to do proof-of-concept things -- and sometimes it's simply fast enough already.
I'd suggest looking at Chutney (a Tor testing framework) with an eye to having your implementation as one option for "a Tor instance" to instantiate. Certainly, if you wanted to go along the protocol-testing/fuzzing lines, that would be a good place to start. https://gitweb.torproject.org/chutney.git/
It would also be beneficial to have some unit-tests.
I'm certainly excited to check it out some more...Thanks for releasing it!
Cheers, meejah
Hi meejah,
Thanks for checking it out!
For sure "prototyping stuff" was the first thing that sprang to my mind as being useful; I've always had good success using Python to do proof-of-concept things -- and sometimes it's simply fast enough already.
Yes...I've actually been somewhat pleasantly surprised by the performance so far, with the caveat that it's mainly been tested with just web browsing (although that does include streaming videos).
I'd suggest looking at Chutney (a Tor testing framework) with an eye to having your implementation as one option for "a Tor instance" to instantiate. Certainly, if you wanted to go along the protocol-testing/fuzzing lines, that would be a good place to start. https://gitweb.torproject.org/chutney.git/
Thank you for posting this. I will definitely take a look at Chutney and consider how oppy can potentially integrate with it. This can hopefully also give me a better idea at how to more rigorously test oppy.
It would also be beneficial to have some unit-tests.
Definitely. After getting some feedback from people on this list, my immediate mental roadmap for this project is starting to look something like:
- Write comprehensive unit/regression tests. - Squash known bugs and stuff that's flushed out in the testing process. - Begin fixing more protocol-level issues that are not in full compliance with tor-spec.
I'm certainly excited to check it out some more...Thanks for releasing it!
Absolutely! Thanks again for having a look.
Best, Nik
On 01/21/2015 12:05 AM, meejah wrote:
I've barely had time to poke at this much, but it's really neat. I was actually originally going to call "txtorcon" simply "txtor" but figured I'd leave that name in case anyone wrote an actual Tor implementation as a Twisted protocol -- which I guess you've now done :) [...and you're welcome to the name if you want].
Anyway.
For sure "prototyping stuff" was the first thing that sprang to my mind as being useful; I've always had good success using Python to do proof-of-concept things -- and sometimes it's simply fast enough already.
I'd suggest looking at Chutney (a Tor testing framework) with an eye to having your implementation as one option for "a Tor instance" to instantiate. Certainly, if you wanted to go along the protocol-testing/fuzzing lines, that would be a good place to start. https://gitweb.torproject.org/chutney.git/
It would also be beneficial to have some unit-tests.
I'm certainly excited to check it out some more...Thanks for releasing it!
Cheers, meejah _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On 1/20/15 1:15 AM, Nik wrote:
Hi tor-dev,
I care deeply about the ability for people to have anonymity online, and I've always been really interested in exactly how Tor works.
So for a class project last semester I wrote oppy, an onion proxy in 100% Python. I spent most of my winter break cleaning it up and writing documentation, and today I'm releasing oppy as a free software project :)
code: https://github.com/nskinkel/oppy docs: https://nskinkel.github.com/oppy
It would be an amazing evolution to see support for Tor Hidden Services.
There are plenty of software (such as GlobaLeaks) that use Tor only: - as a Client (to make outbound connection) - as a Server (to receive outbound connection)
It's a wet dream to have a pure-python Tor implementation integrated within Twisted, that enable to: - connect to the outside world using Tor - receive inbound connection as a Tor Hidden Service
I understand that's quite tricky to reach production-grade, but for software-integration it would be a very valuable approach.
Another approach to achieve better "software integration" is to have Tor as a Library (it has been discussed few times, there are some existing architectural issues within Tor and some concern on how it shall be done).
Hi naif,
Thanks for the input. This is good to know.
The ability to build internal circuits and at least access hidden services as a client is a feature that I definitely consider "must-have" at some point in the future. There are a number of things that have precedence over implementing this, though (e.g. more serious testing, fixing some simplifications made that make using it somewhat annoying, and bringing things like path selection and when to get network status docs up to full spec compliance).
However, when I get to the point of really digging into rend-spec I will definitely keep this in mind and take a serious look at supporting hidden services too.
Thanks for the idea.
Best, Nik
On 01/21/2015 02:42 AM, Fabio Pietrosanti (naif) - lists wrote:
On 1/20/15 1:15 AM, Nik wrote:
Hi tor-dev,
I care deeply about the ability for people to have anonymity online, and I've always been really interested in exactly how Tor works.
So for a class project last semester I wrote oppy, an onion proxy in 100% Python. I spent most of my winter break cleaning it up and writing documentation, and today I'm releasing oppy as a free software project :)
code: https://github.com/nskinkel/oppy docs: https://nskinkel.github.com/oppy
It would be an amazing evolution to see support for Tor Hidden Services.
There are plenty of software (such as GlobaLeaks) that use Tor only:
- as a Client (to make outbound connection)
- as a Server (to receive outbound connection)
It's a wet dream to have a pure-python Tor implementation integrated within Twisted, that enable to:
- connect to the outside world using Tor
- receive inbound connection as a Tor Hidden Service
I understand that's quite tricky to reach production-grade, but for software-integration it would be a very valuable approach.
Another approach to achieve better "software integration" is to have Tor as a Library (it has been discussed few times, there are some existing architectural issues within Tor and some concern on how it shall be done).
-
Damian Johnson -
Fabio Pietrosanti (naif) - lists -
meejah -
Nik -
Yawning Angel