-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I was looking at the Gitian descriptor for the pluggable transports at https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/descriptors/windows/gitian-pluggable-transports.yml , and I noticed that it has an input file called "python.msi". Furthermore, I noticed the following line in https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/versions :
PYTHON_MSI_URL=https://www.python.org/ftp/python/$%7BPYTHON_VER%7D/$%7BPYTHON_ MSI_PACKAGE}
- From this, I conclude that Python is not being built in Gitian, and the download from www.python.org is assumed to be safe / not backdoored. Is this correct?
If I'm correct, is there a reason that Python is not being built in Gitian? Was it attempted and found that Python cannot easily be built for Windows in Gitian? Or was it not attempted and just still on the to-do list? I don't see any relevant ticket on Trac.
Thanks, - -Jeremy Rand
On Sun, Sep 06, 2015 at 11:26:16PM +0000, Jeremy Rand wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I was looking at the Gitian descriptor for the pluggable transports at https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/descriptors/windows/gitian-pluggable-transports.yml , and I noticed that it has an input file called "python.msi". Furthermore, I noticed the following line in https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/versions :
PYTHON_MSI_URL=https://www.python.org/ftp/python/$%7BPYTHON_VER%7D/$%7BPYTHON_ MSI_PACKAGE}
- From this, I conclude that Python is not being built in Gitian, and
the download from www.python.org is assumed to be safe / not backdoored. Is this correct?
If I'm correct, is there a reason that Python is not being built in Gitian? Was it attempted and found that Python cannot easily be built for Windows in Gitian? Or was it not attempted and just still on the to-do list? I don't see any relevant ticket on Trac.
Way way back when pluggable transports were first integrated into Tor Browser, we tried compiling Python and it was too problematic to be worth it. Here is the comment you want to read:
https://trac.torproject.org/projects/tor/ticket/9444#comment:18 https://trac.torproject.org/projects/tor/ticket/9444#comment:20
Those comments are two years old now. Maybe things have changed and it's easier to cross-compile for Windows now. If it's something you have expertise with, it'd be great if you tried it!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 09/07/2015 12:29 AM, David Fifield wrote:
Way way back when pluggable transports were first integrated into Tor Browser, we tried compiling Python and it was too problematic to be worth it. Here is the comment you want to read:
https://trac.torproject.org/projects/tor/ticket/9444#comment:18 https://trac.torproject.org/projects/tor/ticket/9444#comment:20
Those comments are two years old now. Maybe things have changed and it's easier to cross-compile for Windows now. If it's something you have expertise with, it'd be great if you tried it!
Hi David, thanks for the reply. I don't have any particular expertise with this myself (I've done some reproducible build work using Gitian but it was all fairly standard C++ code), but I'm aware that the Bitcoin wallet Armory is attempting to do Gitian builds for Windows, and a lot of their code is Python. I've pointed their reproducible build specialist to this thread. Armory's progress on this isn't very complete yet (they're still working through bugs introduced by cross-compiling Python), but maybe they'll learn something from your notes, and maybe they'll be able to move things forward.
Cheers, - -Jeremy Rand
Another option here, besides getting python to build in gitian is to phase out support for python-based pluggable transports. It's something to consider at least. Which transports are still only available in python?
On Sun, Sep 6, 2015 at 7:26 PM, Jeremy Rand biolizard89@gmail.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I was looking at the Gitian descriptor for the pluggable transports at https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/descriptors/windows/gitian-pluggable-transports.yml , and I noticed that it has an input file called "python.msi". Furthermore, I noticed the following line in https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/versions :
PYTHON_MSI_URL=https://www.python.org/ftp/python/$%7BPYTHON_VER%7D/$%7BPYTHON_ MSI_PACKAGE}
- From this, I conclude that Python is not being built in Gitian, and
the download from www.python.org is assumed to be safe / not backdoored. Is this correct?
If I'm correct, is there a reason that Python is not being built in Gitian? Was it attempted and found that Python cannot easily be built for Windows in Gitian? Or was it not attempted and just still on the to-do list? I don't see any relevant ticket on Trac.
Thanks,
- -Jeremy Rand
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2
iQIcBAEBCAAGBQJV7Mt1AAoJEAHN/EbZ1y06nL8QAM4qhFMupoippBIvI4JwlLZ6 Vf4wNWA2/IY+62DQ4hpkPRPX/vT48lJIJnPXUxQ427ruX/txYs+T8Yw/RyiW6eq8 AWrkj3DZTYE4RtgEnTazA7NEpiv0YFOuRdyKoyjZyR1MAZTWpZ12TS/hkXaksWsx mZX7EJMv9GBNthMLGoJ5cpTdXymhbGgvOGcF1esxZlzEQnusbaMjCNHv+GzLXGBr xEFUzjuIeuvzZxHgmTqTujjMev9kMYqpRVoyg2JbRhZz/LwbfuQssDMlTRVOmpCu FwCw9RbyZyaBzrnoZLbJ9hs6JQz/cKYcv0kOmEncTprc6IVdZDnMj5guT0rwUycn r9x0ZB0Uz7FRPnqIgmeHLn9JmFPuF9IWc6Kl3fILNsYMCF+AyOBAB7QVFgSftXMQ 2+ifWG6OdAZfM0jau1L6phlILJtvc79ClHhp86wIeSQ7k0mEKn9JOzEg+YRXLS4L ZAoTFKSfNrukMtmdUhgb97yl7MA+wsdWCpMybZRpSJQYMiNlL33njG8kYMdbuhX/ uDTFmc1b3IGWhfRdAtaIK9sZzkTYTzFg30Ypr8uYte7McU+QOMHbWZGZ46KL79k7 uZxPt2bcKSzEP9KP28Pkz0Hc0v0qGYD0BrAzeYkYVB9FhLQF5vs1jDfn2YI7VQbI HSs0NTTkxWHkro3tnzeU =jg/n -----END PGP SIGNATURE----- _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 09/09/2015 06:43 PM, Brandon Wiley wrote:
Another option here, besides getting python to build in gitian is to phase out support for python-based pluggable transports. It's something to consider at least. Which transports are still only available in python?
Indeed -- the reason I originally asked this question was to evaluate whether Namecoin (of which I'm a developer) should keep developing code in Python or whether we should port that code to Golang. We really want reproducible builds, and based on the info David and Joseph have shared, Namecoin is probably going to (where feasible) port to Golang (which Tor is successfully building reproducibly with very minimal code). I'm curious whether the Tor devs have come to a similar conclusion.
Cheers, - -Jeremy
I am in favor of standardizing on the Go codebase for pluggable transports that ship with Tor. This is something we talked about at the last developer meeting. The reason I favor this is not for reproducible build reasons, but because maintaining four implementations (C, Python, C++, and Go) is confusing for PT developers. As far as I know, since the last developer meeting all Tor products have been migrating towards shipping the Go PT implementation so that they can get obfs4 support. Last I checked, some of Tor products are also shipping other PT implementations in order to maintain access to transports not available in Go. I imagine that there is some time in the future where there will no longer be any bridges available for the older transports and so bundling clients for them will no longer be necessary. However, I don't know what the current level of use for non-Go transports is. I'd love to know if someone has those stats.
I also don't know how well reproducible builds work with Go, so if someone knows that would be interesting information.
On Wed, Sep 9, 2015 at 2:51 PM, Jeremy Rand biolizard89@gmail.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 09/09/2015 06:43 PM, Brandon Wiley wrote:
Another option here, besides getting python to build in gitian is to phase out support for python-based pluggable transports. It's something to consider at least. Which transports are still only available in python?
Indeed -- the reason I originally asked this question was to evaluate whether Namecoin (of which I'm a developer) should keep developing code in Python or whether we should port that code to Golang. We really want reproducible builds, and based on the info David and Joseph have shared, Namecoin is probably going to (where feasible) port to Golang (which Tor is successfully building reproducibly with very minimal code). I'm curious whether the Tor devs have come to a similar conclusion.
Cheers,
- -Jeremy
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2
iQIcBAEBCAAGBQJV8H+MAAoJEAHN/EbZ1y06utEQAIrpmeB7k6UDqDpmeeEdOrBB cqKsxg28UmNo4c32+sY+ocouYBPAgqKAQwXyMkUI4sMvpPXsbWSjf8UdyXxPdcT0 JfUr/skxEf/9jfHNccHm0UfOymYeNoTsAvPTUfOxbZ2F1ZfQ49gfKzqfcntXqpKa B3nVoQSAz8xTWeIvzJdrdZgMlNDc61Xvv4UznALJItrpGEC33RK8p4la9SwC2iss /v/YtuqI7fPzCwzrDnQOU7ol3j40lRKhHbskTwKFBNmDp9hEKUbgQJbloiNYfXwg cS7Q3fYM+JpPyIgJwMTV1CwTSrcF+rGLhG2gRLeCSMzKF9c8MAVCOCdpNjeKxcFx Rc6RjWw4ClyXLyB24BrKLipd9gY0I6DWI29cEfTVKfcyHN+0qFvcecBjrFsCGWYE RkRDVuDSRzRnIdz/kIhcxpGhezvwl09PRGCuJ8f/oLGk79QGpkqXunv2gvV9AlNM MmIffRoEhvhqvPGf6sVx0xxphP9sk6WiCKb6n8onrQRcPVN4mPG6MjQ+eyxPOsCa Eo+BLb78c4CVzNnhher1HeuBbR4UxZvi66V8NhcPW4pSUZamIrIxVsnwAsV70vBP 2NDfkGgpcaHVr+nr5XvccGJZWsFcYv38GlB9PuwxSNCmJI6pALQic7ArS6VMS7QZ H8jm1cUe+2jSeFMNLiq/ =uyhm -----END PGP SIGNATURE----- _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On Wed, Sep 09, 2015 at 03:33:24PM -0400, Brandon Wiley wrote:
I am in favor of standardizing on the Go codebase for pluggable transports that ship with Tor. This is something we talked about at the last developer meeting. The reason I favor this is not for reproducible build reasons, but because maintaining four implementations (C, Python, C++, and Go) is confusing for PT developers. As far as I know, since the last developer meeting all Tor products have been migrating towards shipping the Go PT implementation so that they can get obfs4 support. Last I checked, some of Tor products are also shipping other PT implementations in order to maintain access to transports not available in Go. I imagine that there is some time in the future where there will no longer be any bridges available for the older transports and so bundling clients for them will no longer be necessary. However, I don't know what the current level of use for non-Go transports is. I'd love to know if someone has those stats.
You can see the usage of each transport here:
https://metrics.torproject.org/userstats-bridge-transport.html?graph=usersta...
obfs2 - Go obfs3 - Go obfs4 - Go meek - Go ScrambleSuit - Go flash proxy - Python FTE - Python
Thanks David, great info! Last time I checked, I think the C implementation was also still shipping with something, I think Orbot for Android. Perhaps this is also for either flash proxy or FTE support, since Python is not the best option on Android.
From the graphs it looks like FTE is still in use, but that flash proxy
seems to no longer be used.
If I recall correctly, the core FTE code is actually written in C and is just using the Python PT implementation with Python-C bindings to the FTE library. So a port of the FTE PT to the Go PT implementation seems possible.
On Wed, Sep 9, 2015 at 4:50 PM, David Fifield david@bamsoftware.com wrote:
On Wed, Sep 09, 2015 at 03:33:24PM -0400, Brandon Wiley wrote:
I am in favor of standardizing on the Go codebase for pluggable
transports that
ship with Tor. This is something we talked about at the last developer
meeting.
The reason I favor this is not for reproducible build reasons, but
because
maintaining four implementations (C, Python, C++, and Go) is confusing
for PT
developers. As far as I know, since the last developer meeting all Tor
products
have been migrating towards shipping the Go PT implementation so that
they can
get obfs4 support. Last I checked, some of Tor products are also
shipping other
PT implementations in order to maintain access to transports not
available in
Go. I imagine that there is some time in the future where there will no
longer
be any bridges available for the older transports and so bundling
clients for
them will no longer be necessary. However, I don't know what the current
level
of use for non-Go transports is. I'd love to know if someone has those
stats.
You can see the usage of each transport here:
https://metrics.torproject.org/userstats-bridge-transport.html?graph=usersta...
obfs2 - Go obfs3 - Go obfs4 - Go meek - Go ScrambleSuit - Go flash proxy - Python FTE - Python _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
On Wed, 9 Sep 2015 17:20:11 -0400 Brandon Wiley brandon@blanu.net wrote:
Thanks David, great info! Last time I checked, I think the C implementation was also still shipping with something, I think Orbot for Android. Perhaps this is also for either flash proxy or FTE support, since Python is not the best option on Android.
Orbot used to use obfsproxy (C), migrated to obfsclient (C++) and the main build uses obfs4proxy (Go) these days. I heard that Android x86 might still need to use the C++ code, but I'm not sure.
From the graphs it looks like FTE is still in use, but that flash proxy seems to no longer be used.
It wasn't really every used due to NAT related issues. WebRTC may be the way forward here but it will be an entirely different codebase.
If I recall correctly, the core FTE code is actually written in C and is just using the Python PT implementation with Python-C bindings to the FTE library. So a port of the FTE PT to the Go PT implementation seems possible.
The core FTE code was originally in C++, but IIRC it's in python now.
FWIW, I don't particularly think that there must be One True PT language[0], I just recommend Go over the other alternatives due to it being both memory safe and easy to build on mobile. If someone writes a new PT in Python, I don't consider it a deal breaker, though it won't be as useful due to the difficulty of mobile support.
All of this sort of ties into the other thread where people are talking about buffing up pyptlib (might make sense), and further deprecating obfsproxy (Python). I don't particularly see a reason to do either things, though the Debian people apparently see obfsproxy as the program and not a library, when it's both.
Regards,
On Thu, Sep 10, 2015 at 12:55 AM, Yawning Angel yawning@schwanenlied.me wrote:
FWIW, I don't particularly think that there must be One True PT language[0], I just recommend Go over the other alternatives due to it being both memory safe and easy to build on mobile. If someone writes a new PT in Python, I don't consider it a deal breaker, though it won't be as useful due to the difficulty of mobile support.
[0]: MUST be able to be built deterministically. SHOULD be memory safe. Past that, people can do what they want. If they ignore the SHOULD clause, the code needs to undergo more thorough auditing before it will be deployed into production.
I'm not advocating that the various PT implementations be abandoned, just that we have a common implementation across products when possible. If I recall correctly, there was a time when TBB, Tails, and Orbot were all shipping different implementations. I think the current state of PT implementation deployment is the following:
TBB: Go, Python Tails: Go Orbot: Go, C++ (on x86)
The benefit of having the Go implementation ship with all products is that PT authors can target one implementation and achieve deployment across all of the products.
As far as reproducibility of builds goes, if a reproducible Python build is a challenge, an alternative is to port FTE to Go and retire flashproxy.
On Thu, 10 Sep 2015 10:20:59 -0400 Brandon Wiley brandon@blanu.net wrote:
I'm not advocating that the various PT implementations be abandoned, just that we have a common implementation across products when possible. If I recall correctly, there was a time when TBB, Tails, and Orbot were all shipping different implementations. I think the current state of PT implementation deployment is the following:
TBB: Go, Python Tails: Go Orbot: Go, C++ (on x86)
That's correct. It's worth noting that the Python component of TBB is almost entirely FTE that hovers around ~200 users. Out of those, I am unsure how many use FTE because it is the only thing that works in their environment.
The benefit of having the Go implementation ship with all products is that PT authors can target one implementation and achieve deployment across all of the products.
Sure. Go would be a fine choice for people, but I'd like it to be explicit that I'm open to more options, even if it means reducing the deployment base if that's what it takes for people to write something (I'd rather see more circumvention methods, than fewer).
As far as reproducibility of builds goes, if a reproducible Python build is a challenge, an alternative is to port FTE to Go and retire flashproxy.
Or port both to Go (flashproxy would be easy)/deprecate both.
Regards,
David Fifield transcribed 1.4K bytes:
On Wed, Sep 09, 2015 at 03:33:24PM -0400, Brandon Wiley wrote:
I am in favor of standardizing on the Go codebase for pluggable transports that ship with Tor. This is something we talked about at the last developer meeting. The reason I favor this is not for reproducible build reasons, but because maintaining four implementations (C, Python, C++, and Go) is confusing for PT developers. As far as I know, since the last developer meeting all Tor products have been migrating towards shipping the Go PT implementation so that they can get obfs4 support. Last I checked, some of Tor products are also shipping other PT implementations in order to maintain access to transports not available in Go. I imagine that there is some time in the future where there will no longer be any bridges available for the older transports and so bundling clients for them will no longer be necessary. However, I don't know what the current level of use for non-Go transports is. I'd love to know if someone has those stats.
You can see the usage of each transport here:
https://metrics.torproject.org/userstats-bridge-transport.html?graph=usersta...
obfs2 - Go obfs3 - Go obfs4 - Go meek - Go ScrambleSuit - Go flash proxy - Python FTE - Python
FWIW, I suspect a majority of the obfs2/3 bridges which do not advertise obfs4 are still running the Python obfsproxy, so roughly 25% of the obfs2/3 bridges. Here's some numbers.
total bridges: 4323 total bridges with PTs: 1867 total obfs4: 1413 (obfs2 or obfs3): 1739 (obfs2 or obfs3) and (not obfs4): 438 (obfs2 or obfs3) and (obfs4): 1301 obfs4 only (no other transports): 112
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 09/09/2015 07:33 PM, Brandon Wiley wrote:
I also don't know how well reproducible builds work with Go, so if someone knows that would be interesting information.
Take a look here:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitia n/descriptors/windows/gitian-pluggable-transports.yml
A number of Go projects are built in that Gitian script. It looks pretty straightforward. (I haven't tested that Gitian script for determinism, but I assume that Tor wouldn't be using it if it weren't deterministic.)
- -Jeremy Rand
-
Brandon Wiley -
David Fifield -
isis -
Jeremy Rand -
Yawning Angel