On Sat, Oct 05, 2013 at 11:08:40AM -0400, Kevin P Dyer wrote:
Sorry I missed the most recent Pluggable Transport bi-weekly meeting.
In regards to the PTTBB build, I plan to make progress towards a streamlined build process.
Platforms I want to get the PTTBB build working on:
- CentOS 5.9 (32-bit)
- CentOS 5.9 (64-bit)
- Windows 7
- OSX 10.8
For each build platform I'm going to:
- Create a VirtualBox image.
- Construct a script that takes the latest TBB release and VirtualBox
image as input, and outputs a PTTBB zip file. The output PTTBB zip file will include all of the "deployed" pluggable transports.
The build process must be deterministic and reproducible.
Thank you for taking an interest in this. Two things: 1. There already exist build scripts and VM instructions, which we use to build the PT TBB. If you want to build a bundle including FTE, your best bet is to start by modifying those scripts. 2. The Tor Project already has a very nice reproducible build system, not yet used for the PT TBB. I want to start using it for the PT bundles in the nearish future.
For the scripts we use to build the PT TBB now, please see https://gitweb.torproject.org/pluggable-transports/bundle.git https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Mak... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... They work much the way you describe: you boot a VM, and run a "make" command. The makefile unzips the vanilla bundle, builds the pluggable transports and copies them into the bundle, then zips it up again.
At the time we started making PT TBBs, Tor's reproducible build system was not finished. I think the new system has great advantages for PT TBB builds, so I want to start building them that way. That is the subject of this ticket: https://trac.torproject.org/projects/tor/ticket/9444
About the reproducible build system, please see this blog post and its linked documentation: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-det... You make a good point about the need for reproducibility. This existing system took six months of work working around nontrivial problems (see the blog post), and it's super nice. It's not something you want to reinvent by yourself.
I was hoping to do this on Amazon Web Services, to avoid the licensing issues. However, AWS does not support OSX.
A huge advantage of the reproducible build system is that it requires neither a Windows license nor an OS X install. Everything is cross-compiled from Ubuntu. From a practical perspective, I care about this feature even more than reproducibility. The need to boot up an actual physical Mac is one reason why PT builds have lagged behind (#9391).
I anticipate this will be a non-trivial effort to get working correctly. So, it would be great if you could help me adjust my plan to minimize headache!
As you see, it's a bit muddled because we are in a transition between two build systems. What I recommend is first doing a proof-of-concept using the bundle-gnulinux.txt instructions (because they are the easiest). Building the base VM image takes about an hour, and then running "make" to build a bundle takes under five minutes. (Technically you don't even need to use a VM; we do so mainly for filesystem hygiene reasons.)
You should create a new ticket "Add FTE to pluggable transports bundle" and Cc me. There you can attach patches or link to the repo you are working in. I understand there are other issues preventing FTE from being included right away, but I would have no problem with there being a branch so that it is easy to build experimental bundles including FTE.
As for the transition to the reproducible build system (#9444), your help would be appreciated with that. As I understand it, what we need to do is create one or more new Gitian "descriptors" for the pluggable transports, then modify gitian-bundle.yml to copy them into the bundle. https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... So far, the most progress I have made on #9444 is to do a reproducible build of the vanilla bundle.
David Fifield