Hi David,
Sorry I missed the most recent Pluggable Transport bi-weekly meeting.
In regards to the PTTBB build, I plan to make progress towards a streamlined build process.
Platforms I want to get the PTTBB build working on: 1) CentOS 5.9 (32-bit) 2) CentOS 5.9 (64-bit) 3) Windows 7 4) OSX 10.8
For each build platform I'm going to: 1) Create a VirtualBox image. 2) Construct a script that takes the latest TBB release and VirtualBox image as input, and outputs a PTTBB zip file. The output PTTBB zip file will include all of the "deployed" pluggable transports.
The build process must be deterministic and reproducible.
I was hoping to do this on Amazon Web Services, to avoid the licensing issues. However, AWS does not support OSX.
By using VirtualBox, it reduces the problem to each developer acquiring their own VirtualBox image for each target platform. Not desirable, but I don't have a better solution.
I anticipate this will be a non-trivial effort to get working correctly. So, it would be great if you could help me adjust my plan to minimize headache!
Thanks!
-Kevin
On Sat, Oct 05, 2013 at 11:08:40AM -0400, Kevin P Dyer wrote:
Sorry I missed the most recent Pluggable Transport bi-weekly meeting.
In regards to the PTTBB build, I plan to make progress towards a streamlined build process.
Platforms I want to get the PTTBB build working on:
- CentOS 5.9 (32-bit)
- CentOS 5.9 (64-bit)
- Windows 7
- OSX 10.8
For each build platform I'm going to:
- Create a VirtualBox image.
- Construct a script that takes the latest TBB release and VirtualBox
image as input, and outputs a PTTBB zip file. The output PTTBB zip file will include all of the "deployed" pluggable transports.
The build process must be deterministic and reproducible.
Thank you for taking an interest in this. Two things: 1. There already exist build scripts and VM instructions, which we use to build the PT TBB. If you want to build a bundle including FTE, your best bet is to start by modifying those scripts. 2. The Tor Project already has a very nice reproducible build system, not yet used for the PT TBB. I want to start using it for the PT bundles in the nearish future.
For the scripts we use to build the PT TBB now, please see https://gitweb.torproject.org/pluggable-transports/bundle.git https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Mak... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... They work much the way you describe: you boot a VM, and run a "make" command. The makefile unzips the vanilla bundle, builds the pluggable transports and copies them into the bundle, then zips it up again.
At the time we started making PT TBBs, Tor's reproducible build system was not finished. I think the new system has great advantages for PT TBB builds, so I want to start building them that way. That is the subject of this ticket: https://trac.torproject.org/projects/tor/ticket/9444
About the reproducible build system, please see this blog post and its linked documentation: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-det... You make a good point about the need for reproducibility. This existing system took six months of work working around nontrivial problems (see the blog post), and it's super nice. It's not something you want to reinvent by yourself.
I was hoping to do this on Amazon Web Services, to avoid the licensing issues. However, AWS does not support OSX.
A huge advantage of the reproducible build system is that it requires neither a Windows license nor an OS X install. Everything is cross-compiled from Ubuntu. From a practical perspective, I care about this feature even more than reproducibility. The need to boot up an actual physical Mac is one reason why PT builds have lagged behind (#9391).
I anticipate this will be a non-trivial effort to get working correctly. So, it would be great if you could help me adjust my plan to minimize headache!
As you see, it's a bit muddled because we are in a transition between two build systems. What I recommend is first doing a proof-of-concept using the bundle-gnulinux.txt instructions (because they are the easiest). Building the base VM image takes about an hour, and then running "make" to build a bundle takes under five minutes. (Technically you don't even need to use a VM; we do so mainly for filesystem hygiene reasons.)
You should create a new ticket "Add FTE to pluggable transports bundle" and Cc me. There you can attach patches or link to the repo you are working in. I understand there are other issues preventing FTE from being included right away, but I would have no problem with there being a branch so that it is easy to build experimental bundles including FTE.
As for the transition to the reproducible build system (#9444), your help would be appreciated with that. As I understand it, what we need to do is create one or more new Gitian "descriptors" for the pluggable transports, then modify gitian-bundle.yml to copy them into the bundle. https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... So far, the most progress I have made on #9444 is to do a reproducible build of the vanilla bundle.
David Fifield
Hi David!
Thanks for the detailed response. My thoughts are inline.
On Sun, Oct 6, 2013 at 3:30 PM, David Fifield david@bamsoftware.com wrote:
On Sat, Oct 05, 2013 at 11:08:40AM -0400, Kevin P Dyer wrote:
Sorry I missed the most recent Pluggable Transport bi-weekly meeting.
In regards to the PTTBB build, I plan to make progress towards a streamlined build process.
Platforms I want to get the PTTBB build working on:
- CentOS 5.9 (32-bit)
- CentOS 5.9 (64-bit)
- Windows 7
- OSX 10.8
For each build platform I'm going to:
- Create a VirtualBox image.
- Construct a script that takes the latest TBB release and VirtualBox
image as input, and outputs a PTTBB zip file. The output PTTBB zip file will include all of the "deployed" pluggable transports.
The build process must be deterministic and reproducible.
Thank you for taking an interest in this. Two things:
- There already exist build scripts and VM instructions, which we use to build the PT TBB. If you want to build a bundle including FTE, your best bet is to start by modifying those scripts.
- The Tor Project already has a very nice reproducible build system, not yet used for the PT TBB. I want to start using it for the PT bundles in the nearish future.
For the scripts we use to build the PT TBB now, please see https://gitweb.torproject.org/pluggable-transports/bundle.git https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Mak... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bun... They work much the way you describe: you boot a VM, and run a "make" command. The makefile unzips the vanilla bundle, builds the pluggable transports and copies them into the bundle, then zips it up again.
This is a good start and goes a long way towards automating the build process. However, I'm looking for "Press a button, make a sandwich, have all builds (for all platforms) ready when I come back."-type of automation. It appears that the current procedure requires booting up four systems (virtual or physical) manually, then manually running the build script in each instance.
I've had success fully-automating the build using vagrant for the linux builds. (Execute "vagrant up," wait a few minutes, then we have i386/amd64 PTTBB .tar.gz files.) I only encountered two issues [1,2]. OSX is next. How may I share this work on https://gitweb.torproject.org/?
At the time we started making PT TBBs, Tor's reproducible build system was not finished. I think the new system has great advantages for PT TBB builds, so I want to start building them that way. That is the subject of this ticket: https://trac.torproject.org/projects/tor/ticket/9444
About the reproducible build system, please see this blog post and its linked documentation: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-det... You make a good point about the need for reproducibility. This existing system took six months of work working around nontrivial problems (see the blog post), and it's super nice. It's not something you want to reinvent by yourself.
I agree. We should definitely work towards using that system long term.
I was hoping to do this on Amazon Web Services, to avoid the licensing issues. However, AWS does not support OSX.
A huge advantage of the reproducible build system is that it requires neither a Windows license nor an OS X install. Everything is cross-compiled from Ubuntu. From a practical perspective, I care about this feature even more than reproducibility. The need to boot up an actual physical Mac is one reason why PT builds have lagged behind (#9391).
Cross-compilation would certainly be a nice feature to have. However, for testing the product of a cross-compiled build, I'll need to have a development environment with each of the target platforms. I'm unwilling to have a bunch of physical machines for testing. Hence, I'm eager for full virtualization, which also helps with full-automation of the build.
I've looked a bit more into the OSX EULA. Even though running OSX as a guest on Windows/Linux seems prevalent, it is against the EULA [4]. However, the EULA, since 10.7, states (c.f., sec. 2.B.iii) that you can spin up virtual instances on Mac hardware. So, I guess it will be up to each of us to figure out our ideal configuration, but the only legal way to do full virtualization requires Mac hardware. Lame.
I anticipate this will be a non-trivial effort to get working correctly. So, it would be great if you could help me adjust my plan to minimize headache!
As you see, it's a bit muddled because we are in a transition between two build systems. What I recommend is first doing a proof-of-concept using the bundle-gnulinux.txt instructions (because they are the easiest). Building the base VM image takes about an hour, and then running "make" to build a bundle takes under five minutes. (Technically you don't even need to use a VM; we do so mainly for filesystem hygiene reasons.)
You should create a new ticket "Add FTE to pluggable transports bundle" and Cc me. There you can attach patches or link to the repo you are working in. I understand there are other issues preventing FTE from being included right away, but I would have no problem with there being a branch so that it is easy to build experimental bundles including FTE.
Done [3]!
As for the transition to the reproducible build system (#9444), your help would be appreciated with that. As I understand it, what we need to do is create one or more new Gitian "descriptors" for the pluggable transports, then modify gitian-bundle.yml to copy them into the bundle. https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/git... So far, the most progress I have made on #9444 is to do a reproducible build of the vanilla bundle.
I'm sure we'll make progress on this as I get up to speed. I added myself to the CC list of #9444.
-Kevin
[1] https://trac.torproject.org/projects/tor/ticket/9939 [2] https://trac.torproject.org/projects/tor/ticket/9940 [3] https://trac.torproject.org/projects/tor/ticket/9941 [4] http://www.apple.com/legal/sla/docs/macosx107.pdf
On Wed, Oct 09, 2013 at 12:02:20PM -0400, Kevin P Dyer wrote:
On Sun, Oct 6, 2013 at 3:30 PM, David Fifield david@bamsoftware.com wrote: This is a good start and goes a long way towards automating the build process. However, I'm looking for "Press a button, make a sandwich, have all builds (for all platforms) ready when I come back."-type of automation. It appears that the current procedure requires booting up four systems (virtual or physical) manually, then manually running the build script in each instance.
I've had success fully-automating the build using vagrant for the linux builds. (Execute "vagrant up," wait a few minutes, then we have i386/amd64 PTTBB .tar.gz files.) I only encountered two issues [1,2]. OSX is next. How may I share this work on https://gitweb.torproject.org/?
My point is that the "push a button" technique is already handled by the Gitian build. (Even though it's more like "make 20 sandwiches.") I just want to caution you against redoing a lot of work that has already been done. Did you know that the Gitian build has a Vagrant configuration? https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/HEAD:/vag... Many of these battles have already been fought.
Also understand my motives: in the future, I don't want the building of PT bundles to fall post-hoc to pluggable transport authors. I want them to be made as a side effect of the vanilla bundle builds, so that nor I nor others not interested in making bundles will have to make them. So while improvements to the bundle.git system are welcome, I eventually want to get rid of it completely.
Pushing a git repo somewhere is a good way to share your changes.
A huge advantage of the reproducible build system is that it requires neither a Windows license nor an OS X install. Everything is cross-compiled from Ubuntu. From a practical perspective, I care about this feature even more than reproducibility. The need to boot up an actual physical Mac is one reason why PT builds have lagged behind (#9391).
Cross-compilation would certainly be a nice feature to have. However, for testing the product of a cross-compiled build, I'll need to have a development environment with each of the target platforms. I'm unwilling to have a bunch of physical machines for testing. Hence, I'm eager for full virtualization, which also helps with full-automation of the build.
I've looked a bit more into the OSX EULA. Even though running OSX as a guest on Windows/Linux seems prevalent, it is against the EULA [4]. However, the EULA, since 10.7, states (c.f., sec. 2.B.iii) that you can spin up virtual instances on Mac hardware. So, I guess it will be up to each of us to figure out our ideal configuration, but the only legal way to do full virtualization requires Mac hardware. Lame.
Virtualizing OS X is problematic. There are many OS X testers, however, on the tor-qa list, to which new bundles are announced before they are published.
David Fifield
On Wed, Oct 9, 2013 at 12:22 PM, David Fifield david@bamsoftware.com wrote:
On Wed, Oct 09, 2013 at 12:02:20PM -0400, Kevin P Dyer wrote:
On Sun, Oct 6, 2013 at 3:30 PM, David Fifield david@bamsoftware.com wrote: This is a good start and goes a long way towards automating the build process. However, I'm looking for "Press a button, make a sandwich, have all builds (for all platforms) ready when I come back."-type of automation. It appears that the current procedure requires booting up four systems (virtual or physical) manually, then manually running the build script in each instance.
I've had success fully-automating the build using vagrant for the linux builds. (Execute "vagrant up," wait a few minutes, then we have i386/amd64 PTTBB .tar.gz files.) I only encountered two issues [1,2]. OSX is next. How may I share this work on https://gitweb.torproject.org/?
My point is that the "push a button" technique is already handled by the Gitian build. (Even though it's more like "make 20 sandwiches.") I just want to caution you against redoing a lot of work that has already been done. Did you know that the Gitian build has a Vagrant configuration? https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/HEAD:/vag... Many of these battles have already been fought.
Also understand my motives: in the future, I don't want the building of PT bundles to fall post-hoc to pluggable transport authors. I want them to be made as a side effect of the vanilla bundle builds, so that nor I nor others not interested in making bundles will have to make them. So while improvements to the bundle.git system are welcome, I eventually want to get rid of it completely.
I conflated two issues: challenges in setting up my personal development environment and challenges in the formal Tor build process. I'm personally motivated to get fully-automated PTTBB builds working to make my development life easier. I'll make less mistakes if I can make a change, then easily test those changes across multiple platforms.
-Kevin
-
David Fifield -
Kevin P Dyer