Goals ~~~~~ It would be useful to have a tool that can be used for building Tor Bundle Browser, and packages for any of the Tor components. Tor Bundle Browser ^^^^^^^^^^^^^^^^^^ In the current build process for the Tor Bundle Browser, gitias is used to : - create and start an Ubuntu VM in KVM or LXC - install build dependencies in the VM - copy the sources files in the VM and start the build The parts that are not done by gitian but by a shellscript based wrapper around gitian: - initial clone of the git repository containing the sources (fetch-inputs.sh) - git tag gpg signature check (verify-tags.sh) - download tarballs from mirrors, and verify gpg signature or sha256sum (fetch-inputs.sh) Improvement that can be done from the current process: - some informations are duplicated in different places, such as git URLs (both in gitian descriptor and in fetch-inputs.sh) - linux, mac and windows gitian descriptors are separate, so part of them are duplicating some informations. It would be better to be able to share the parts that are common on all OSs. - build process is not very modular. It is not possible or simple to build only some components. - the build process can currently only be run from Ubuntu - the current process requires using git tags if gpg signature verification is wanted. It could be useful to be able to use gpg signed commits instead. To solve this, we can have a tool that handle both the tasks currently done by gitian, and the tasks currently done by shellscript wrappers: - store in a single descriptor the build instructions, build dependencies, git URL, tarball URLs, gpg keyring file, sha256sum, etc ... and have all the operations done by the tool. This should remove the need for wrappers. - create a separate descriptor for each software that we build, so that they can be built separately. The tool should be able to manage dependencies between separate builds to create a bundle of different software. - a templating system allows us to have some variables and use them at different places. This allows us to have a common build descriptor for all OSs, and use conditional templating instructions where the process differ between the different OSs. A prototype to show a possible build process is available, with more details below. Packages ^^^^^^^^ In addition to building the Tor Browser Bundle, we also need to build some rpm or Debian packages for different components. Some of them are also included in the Tor Browser Bundle. To be able to do continuous packaging, we need a tool that would allow us to produce an rpm or Debian package from a git commit. In the jenkins/tools.git repository, some shell scripts are used to produce Tor Debian packages from the latest commit. To avoid duplicating build instructions, URLs, gpg keyring and other infos, it would be useful if a single tool was able to produce the Tor Browser Bundle and also individual packages for some of the components. Prototype and current status ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I have created recently a tool that could be used to build Bundle, and Debian and rpm packages : http://burps.boklm.eu/[burps (build & upload reproducible packaging system)]. It currently has the following features: - build software from a git repository, with tag signature verification and/or commit signature verification - download files from an URL, and check gpg signature, or sha256sum - use build result from the build of an other component - build on a remote host, VM or chroot - build rpm or Debian packages, or tarballs - define variables in the descriptors, and use them anywhere using template instructions - override some variables on command line by selecting build targets. It is currently missing the following features from gitian: - creation and start/stop of the Ubuntu build VMs. We can keep the gitian scripts for that, and improve them later. - definition of the build dependencies in the descriptor and installation. This should not be a lot of work to add this feature. Prototype ^^^^^^^^^ A prototype for a new build process for the Tor Browser Bundle using this tool is available at the following URL: https://github.com/boklm/burps-tor It currently includes the following components: libevent, tor, tor-browser, torbutton, tor-launcher, https-everywhere, all built separately and combined in the bundle. It does not produce a functional build yet, but is enough to show what the build process would look like. Installation ^^^^^^^^^^^^ Debian packages for 'burps' should be available soon. Until they are available you can do a manual install: - clone the git repository in some directory - install the following perl modules: YAML::XS, Getopt::Long, Template, IO::CaptureOutput, File::Slurp, String::ShellQuote, Sort::Versions, Data::Dump On Debian this can be done with: ---- $ apt-get install libyaml-libyaml-perl libtemplate-perl \ libio-captureoutput-perl libfile-slurp-perl \ libstring-shellquote-perl libsort-versions-perl \ libdata-dump-perl $ git clone https://github.com/boklm/burps ~/burps ---- In the commands from the following examples, replace 'burps' with the path to the burps executable (for instance '~/burps/burps'). How it works ~~~~~~~~~~~~ To understand how it works, we can look at different files from the prototype and see what they do. Files and directories list ^^^^^^^^^^^^^^^^^^^^^^^^^^ The following files and directories are present in the repository: ---- . burps.conf - The main configuration file git_clones - directory containing git clones keyring libevent.gpg - The gpg keyring for the libevent project tor.gpg - The gpg keyring for the tor project ... - other keyring files out - Output directory for build files projects libevent build - build instructions for libevent config - configuration for libevent tbb build - build instructions for the bundle config - configuration for the bundle tor build - build instructions for tor config - configuration for tor ... - other projects directories ---- burps.conf ^^^^^^^^^^ The 'burps.conf' file is used to indicate the root of the burps repository. It also contains the configuration options shared by all projects. All those options can be overriden by each project configuration. libevent ^^^^^^^^ The file 'projects/libevent/config' contains the configuration for the libevent build: ---- git_url: https://github.com/libevent/libevent.git version: '[% c("abbrev") %]' var: dest_filename: 'libevent-[% c("version") %]-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar' targets: notarget: dev stable: git_hash: release-2.0.21-stable tag_gpg_id: 1 dev: git_hash: master ---- The following options are defined: git_url:: This option contains the URL of the git repository that should be cloned to create the sources tarball. version:: This option contains the string that is used as version number in the sources tarball. In this case, we're saying that we want to use 'abbrev' as version string, which is the abbreviated hash of the commit that has been selected. var/dest_filename:: This option is used in the 'build' file, and defines the name of the output file. The name of this option is arbitrary, so to avoid a conflict with an option used by a future version of burps, we add the 'var' prefix. The reason to use a variable to define this file name is to allow other projects which need this file (in this case, this libevent build is used by tor), to know the name of the file that is produced. In this case, we can see that the filename contains the following informations: abbreviated commit hash, distribution name and release, architecture. targets:: We can see that different targets are defined. Targets allows us to set different options depending on the target selected on command line. In this case, we defined the 'stable' and 'dev' targets. http://burps.boklm.eu/burps_targets.html[This page] has more details about how the targets work. git_hash:: This option is used to select the git commit from which to create the sources tarball. This can be anything that git calls a 'tree-ish' (hashes, tags, branch names). tag_gpg_id:: This option indicates that the gpg signature of the tag pointed by 'git_hash' should be checked, using the keyring file 'libevent.gpg' (as defined by the 'keyring' option in 'burps.conf'). We can only set this option for the stable target, because the dev target does not use a tag. The libevent commits are not currently signed, but if they were we could set the 'commit_gpg_id' option to check signature of the commit in the master branch. The file 'projects/libevent/config' contains the build instructions for libevent: ---- [%- USE date -%] #!/bin/sh set -e [% c('var/reproducible_init') %] tar xvf [% project %]-[% c('version') %].tar.[% c('compress_tar') %] cd [% project %]-[% c('version') %] ./autogen.sh [% c('var/touch_directory', { directory => '.' }) %] mkdir -p inst/libevent ./configure --disable-static --prefix="$(pwd)/inst/libevent" make -j$(nproc) make install cd inst [% c('var/touch_directory', { directory => '.' }) %] [% c('tar', { tar_src => 'libevent', tar_args => '-cf ' _ dest_dir _ '/' _ c('var/dest_filename') }) %] ---- We can see that it uses the following options: var/reproducible_init:: This option is defined in 'burps.conf', and export the environment variables to enable 'libfaketime'. The time used by libfaketime is defined by the option 'timestamp', which by default is the time of the selected git commit. var/touch_directory:: This option sets the modification time of all files in the selected directory, using the time from 'timestamp' option. tar:: This option is defined in burps (you can run 'burps showconf' to see it), and is used to create a deterministic tar file. It is doing something similar to the script gitian/build-helpers/dtar.sh from the current tbb build process. Following are some examples of commands that we can use to build libevent. Creating a sources tarball for the 'stable' target. In this example we can see that the gpg signature of the git tag is checked: ---- $ burps tar libevent --target stable Tag release-2.0.21-stable is signed with key B35BF85BF19489D04E28C33C21194EBB165733EA Created /home/boklm/work/burps-tor/out/libevent-64177777165d.tar.gz ---- We can do the same for the 'devel' target, and we will get the sources from the master branch instead: ---- $ burps tar libevent --target dev Created /home/boklm/work/burps-tor/out/libevent-4c8ebcd3598a.tar.gz ---- If what we want is a build, we use the 'build' command instead of 'tar'. ---- $ ~/work/mkpkg/burps build libevent --target dev Created /home/boklm/tmp/burps-keaW4/libevent-4c8ebcd3598a.tar.gz [some build logs] $ ls -l out/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar -rw-r--r-- 1 boklm boklm 3430400 Jan 2 14:43 out/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar $ tar tvf out/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar drwx------ root/root 0 2013-12-24 21:02 libevent/ drwx------ root/root 0 2013-12-24 21:02 libevent/lib/ lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_pthreads.so -> libevent_pthreads-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_openssl-2.1.so.3 -> libevent_openssl-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_openssl.so -> libevent_openssl-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent-2.1.so.3 -> libevent-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_core.so -> libevent_core-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_core-2.1.so.3 -> libevent_core-2.1.so.3.0.0 -rwx------ root/root 117534 2013-12-24 21:02 libevent/lib/libevent_openssl-2.1.so.3.0.0 -rwx------ root/root 1018 2013-12-24 21:02 libevent/lib/libevent_core.la -rwx------ root/root 1042 2013-12-24 21:02 libevent/lib/libevent_pthreads.la -rwx------ root/root 25459 2013-12-24 21:02 libevent/lib/libevent_pthreads-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_pthreads-2.1.so.3 -> libevent_pthreads-2.1.so.3.0.0 -rwx------ root/root 861887 2013-12-24 21:02 libevent/lib/libevent_core-2.1.so.3.0.0 lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_extra.so -> libevent_extra-2.1.so.3.0.0 -rwx------ root/root 1024 2013-12-24 21:02 libevent/lib/libevent_extra.la -rwx------ root/root 1364542 2013-12-24 21:02 libevent/lib/libevent-2.1.so.3.0.0 [...] ---- We can do the same on the 'stable' target: ---- $ ~/work/mkpkg/burps build libevent --target stable Tag release-2.0.21-stable is signed with key B35BF85BF19489D04E28C33C21194EBB165733EA Created /home/boklm/tmp/burps-0d1Cy/libevent-64177777165d.tar.gz [...] autoreconf: running: aclocal --force -I m4 aclocal: warning: autoconf input should be named 'configure.ac', not 'configure.in' configure.in:15: error: 'AM_CONFIG_HEADER': this macro is obsolete. You should use the 'AC_CONFIG_HEADERS' macro instead. /usr/share/aclocal-1.13/obsolete-err.m4:12: AM_CONFIG_HEADER is expanded from... configure.in:15: the top level autom4te: /usr/bin/m4 failed with exit status: 1 aclocal: error: echo failed with exit status: 1 autoreconf: aclocal failed with exit status: 1 Error: Error running build ---- In this example I am using Mageia, and the version of autoconf we have there doesn't work with this version of libevent, so the build fails. However we can run the build inside a Ubuntu Lucid chroot (more details about running builds in a chroot or VM below): ---- $ burps build libevent --target chroot-ubuntu-lucid --target stable Tag release-2.0.21-stable is signed with key B35BF85BF19489D04E28C33C21194EBB165733EA Created /home/boklm/tmp/burps-6PNXA/libevent-64177777165d.tar.gz [...] $ ls out/libevent-64177777165d-Ubuntu-10.04-x86_64.tar out/libevent-64177777165d-Ubuntu-10.04-x86_64.tar $ tar tvf out/libevent-64177777165d-Ubuntu-10.04-x86_64.tar drwx------ root/root 0 2012-11-18 07:39 libevent/ drwx------ root/root 0 2012-11-18 07:39 libevent/lib/ lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_pthreads.so -> libevent_pthreads-2.0.so.5.1.9 lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_openssl.so -> libevent_openssl-2.0.so.5.1.9 lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_core.so -> libevent_core-2.0.so.5.1.9 -rwx------ root/root 942988 2012-11-18 07:39 libevent/lib/libevent-2.0.so.5.1.9 -rwx------ root/root 398207 2012-11-18 07:39 libevent/lib/libevent_extra-2.0.so.5.1.9 -rwx------ root/root 1032 2012-11-18 07:39 libevent/lib/libevent_core.la -rwx------ root/root 1056 2012-11-18 07:39 libevent/lib/libevent_pthreads.la -rwx------ root/root 569248 2012-11-18 07:39 libevent/lib/libevent_core-2.0.so.5.1.9 -rwx------ root/root 21158 2012-11-18 07:39 libevent/lib/libevent_pthreads-2.0.so.5.1.9 -rwx------ root/root 87749 2012-11-18 07:39 libevent/lib/libevent_openssl-2.0.so.5.1.9 lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_extra.so -> libevent_extra-2.0.so.5.1.9 -rwx------ root/root 1038 2012-11-18 07:39 libevent/lib/libevent_extra.la [...] ---- We can notice that the modification time of all files in the archive is 2012-11-18, which is the time of the commit '2.0.21-stable'. libevent build is then used in the build of tor. We can se it in input_file of the tor config: ---- input_files: - filename: '[% pc("libevent", "var/dest_filename") %]' name: libevent project: libevent pkg_type: build ---- The libevent build file name contains the abbreviated git hash, distribution name, release and architecture, so if any of those change, libevent will be rebuilt automatically. Otherwise, the existing build will be used. Python ^^^^^^ Python is an example of tool that we build using a tarball downloaded from an URL. The config file for python contains this: ---- version: 2.7.5 timestamp: 1 var: dest_filename: 'python-[% c("version") %]-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar.xz' input_files: - name: python filename: 'Python-[% c("version") %].tar.bz2' URL: 'http://www.python.org/ftp/python/[% c("version") %]/[% c("filename") %]' sha256sum: 3b477554864e616a041ee4d7cef9849751770bc7c39adaf78a94ea145c488059 ---- We can see that the filename includes the python version. If we update the version number, then the new tarball will be downloaded automatically. If the version number doesn't change, then the tarball will be downloaded only once, and reused each time. This Python build is used during the tor-browser build, only if we are running a distribution that provides an older version of Python (Ubuntu Lucid). We can see it in the tor-browser config file: ---- git_url: https://git.torproject.org/tor-browser.git version: '[% c("abbrev") %]' var: dest_filename: 'tor-browser-[% c("abbrev") %]-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar' dest_filename_debug: 'tor-browser-[% c("abbrev") %]-debug-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar' targets: dev: git_hash: tor-browser-24.2.0esr-3.5.1-build1 tag_gpg_id: 1 stable: git_hash: tor-browser-24.2.0esr-3.5.1-build1 tag_gpg_id: 1 input_files: - filename: '[% pc("python", "var/dest_filename") %]' name: python project: python pkg_type: build enable: '[% c("build_python") %]' distributions: - lsb_release: id: Ubuntu codename: lucid var: build_python: 1 link_yasm: 1 ---- In this file, we can see that the options 'var/build_python' and 'var/link_yasm' are set to '1' if the selected distribution is Ubuntu with codename 'lucid' (more details about how to set distribution specific options can be found on http://burps.boklm.eu/burps_distributions.html[this page]). The option 'var/build_python' is used in the 'input_files' to enable or disable the python build. It is also used in the tor-browser 'build' file, to decide if a python tarball should be extracted: ---- [% IF c('var/build_python') %] tar xvf [% c('input_files_by_name/python') %] export PATH="$rootdir/python/bin:$PATH" [% END %] ---- Pluggable transports ^^^^^^^^^^^^^^^^^^^^ This prototype does not include the pluggable transports, except 'pyptlib', to show how they can be included optionally. The 'burps.conf' file contains an 'enable_pt' target: ---- targets: include_pt: var: include_pt: 1 ---- The option 'var/include_pt' is then used in tbb include_files config, to optionaly include pyptlib as input: ---- - filename: '[% pc("pyptlib", "var/dest_filename") %]' name: pyptlib project: pyptlib pkg_type: build enable: '[% c("var/include_pt") %]' ---- In the tbb build script, it is used to optionaly extract pyptlib in the Tor directory: ---- [% IF c('var/include_pt') %] pushd "$instdir/Tor" tar xvf "$rootdir"/[% c('input_files_by_name/pyptlib') %] popd [% END %] ---- We can now use the 'include_pt' target to include pyptlib in the bundle: ---- $ burps build tbb --target dev --target include_pt Created /home/boklm/tmp/burps-uLDRA/tbb-x.dev.tar.gz Using file /home/boklm/work/burps-tor/projects/tbb/tor-573ee36eae63-Mageia-3-x86_64.tar Using file /home/boklm/work/burps-tor/out/tor-browser-a451d459fa5e-Mageia-3-x86_64.tar Using file /home/boklm/work/burps-tor/out/tor-launcher-1e34dd95dc7f.xpi Using file /home/boklm/work/burps-tor/out/https-everywhere-4be8003e7a6a.xpi Using file /home/boklm/work/burps-tor/projects/tbb/torbutton-2c825593cbcb.xpi Using file /home/boklm/work/burps-tor/out/pyptlib-97be8ad7c74e.tar [...] $ tar tvf out/tbb-x.dev-Mageia-3-x86_64.tar.xz | grep pypt | wc -l 45 ---- If we're not using the 'include_pt' target, pyptlib is not included: ---- $ ~/work/mkpkg/burps build tbb --target dev Created /home/boklm/tmp/burps-Zvgm4/tbb-x.dev.tar.gz Using file /home/boklm/work/burps-tor/projects/tbb/tor-573ee36eae63-Mageia-3-x86_64.tar Using file /home/boklm/work/burps-tor/out/tor-browser-a451d459fa5e-Mageia-3-x86_64.tar Using file /home/boklm/work/burps-tor/out/tor-launcher-1e34dd95dc7f.xpi Using file /home/boklm/work/burps-tor/out/https-everywhere-4be8003e7a6a.xpi Using file /home/boklm/work/burps-tor/projects/tbb/torbutton-2c825593cbcb.xpi [...] $ tar tvf out/tbb-x.dev-Mageia-3-x86_64.tar.xz | grep pypt | wc -l 0 ---- Build in a chroot or VM ^^^^^^^^^^^^^^^^^^^^^^^ In order to produce reproducible builds, we need to build all software in a fixed environment. To do that, the current build process is using Ubuntu Lucid (for Linux builds) and Ubuntu Precise (for Windows and Mac OS X builds) inside a VM or LXC container. The creation of a VM image or chroot directory is not yet automated in this prototype. However, there is support building in a chroot or in a VM, if you create it manually. http://burps.boklm.eu/burps_remote.html[This page] has details about how to do that. If you want to build a bundle in an Ubuntu lucid chroot, first create the chroot in some directory, using 'debootstrap', and create in this chroot the user that will be used to build the software. You can then add the following configuration in your main configuration file '/etc/burps.conf' (this would also work if it was added to the repository configuration file, but as it contains configuration specific to your local system, it's better to have it in your system's configuration file): ---- targets: chroot-ubuntu-lucid: lsb_release: id: Ubuntu release: 10.04 codename: lucid chroot_path: /home/chroots/ubuntu-lucid chroot_user: build remote: build: exec: '[% c("remote_chroot/exec") %]' ---- We are setting the following options: lsb_release:: we are saying that the distribution on which we are building is Ubuntu lucid. If you are not setting this, the output from the 'lsb_release' command will be used. chroot_path:: The path to the chroot you want to use. This is used by the 'remote_chroot/exec' option. chroot_user:: The user inside the chroot that should be used to build. This is used by the 'remote_chroot/exec' option. remote/build/exec:: This is setting the command that should be used to execute something inside the chroot. We are using the 'remote_chroot' option (you can see the content of this option with 'burps showconf'). This command is using 'sudo chroot', so you should have sudo access to do that. After adding those changes to the configuration, you should be able to build any software inside this chroot by adding the arguments '--target chroot-ubuntu-lucid' to your burps commands. For instance if you want to build a bundle in the dev target, you can run: ---- $ burps build tbb --target chroot-ubuntu-lucid --target dev ---- Debugging tips ^^^^^^^^^^^^^^ When building some project, burps will create a temporary directory, copy all the input files and a build script in that directory, then run the build script. After the build finish (or fails), that temporary directory is removed, which can be annoying if you want to understand why the build failed. To avoid this, you can enable the 'debug' option. Add the following line to '/etc/burps.conf': ---- debug: 1 ---- Or use the '--debug' command line argument. If the debug option is enabled and a build fails, burps will open an interactive shell in the temporary build directory, so you can inspect the build script and other files, and maybe continue the build. When you exit that shell, the temporary directory will be removed. When you are writting burps project descriptors, you can use some template directives, target specific options, distribution specific options, etc ... It can sometime be difficult to understand what will be the value of an option, or the content of a build script. To check the value of an option, you can use the 'showconf' burps command. This option takes a project's name, an option name, and the same options as other build commands. For instance if you want to check the abbreviated git commit in dev and stable targets for libevent, you could run the following commands: ---- $ burps showconf libevent abbrev --target dev 4c8ebcd3598a $ burps showconf libevent abbrev --target stable 64177777165d ---- If you want to compare the build script of tbb with and without the 'include_pt' target: ---- $ burps showconf tbb build --target dev --target include_pt [...] $ burps showconf tbb build --target dev [...] ---- Pubishing builds ^^^^^^^^^^^^^^^^ burps can be used to publish the builds. The 'publish' command will build the selected project, and if the build suceeds run the script from the 'publish' option to publish the build result. The prototype has an example publish script, in the file 'projects/common/publish': ---- #!/bin/bash set -e pubdir='[% c('var/publish_dir', {error_if_undef => 1}) %]/[% project %]/[% c('version') %]' mkdir -p "$pubdir" mv * "$pubdir" rm -f "$pubdir/build" ---- We can see that it uses the option 'var/publish_dir' to select the directory where the files should be put, so we can define it in '/etc/burps.conf' like this: ---- var: publish_dir: /tmp/publish ---- We can now run the following command if we want to publish a build of libevent: ---- $ burps publish libevent --target dev [...] $ find /tmp/publish/libevent/ /tmp/publish/libevent/ /tmp/publish/libevent/4c8ebcd3598a /tmp/publish/libevent/4c8ebcd3598a/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar ---- 'burps build' can be the command that jenkins use to build and publish new builds. This publish script could be extended to compress tarballs, send emails or update the xref:builds-database[builds database]. Testing builds ^^^^^^^^^^^^^^ burps does not have yet a 'test' command, but it could be added, with a script to be run after the build succesfully ended, and before the publish script is run. Using the xref:vm-tools[VM tools] described below, those test scripts could create and start some VM or containers in which the project that was built and useful test helpers are installed, and some tests are run. This would be used by Jenkins to run the tests, but is also available for developers who want to run the tests on their own computer. Summary of improvements ^^^^^^^^^^^^^^^^^^^^^^^ The main improvements in this prototype from the current build process are: - all components are built separately, and include in their output file name the commit hash or version, architecture and OS (if architecture dependant). This allows us to keep previous builds if the commit/version/architecture/OS didn't change. So we can rebuild a bundle very quickly when the browser didn't change. - the gitian replacment has features to download tarballs and verify them with sha256sum or gpg signature, so this can replace the fetch-inputs.sh script. - we can remove the linux/windows/macosx descriptors duplication, and instead use template directives for the parts that differ between those builds (it's still possible to use separate files if they differ completly). - we can define variables based on selected OS. This allows for instance to build python 2.7 when building on Ubuntu Lucid, but avoid building it on other distros which already provide python 2.7. - we can define variables based on "targets" that are set on command line. For instance in the prototype using "--target enable_pt" instructs to include the portable transports (only pyptlib in this prototype) in the bundle. - we can easily switch from building in a VM to building locally - build descriptors can depend on the result of another build descriptor. This remove needs for scripts like mkbundle-*.sh. And I think those improvements should make it easier to rebuild a new bundle automatically when any of the components of the bundle receives a new commit, and then run tests on this bundle. What is currently missing ^^^^^^^^^^^^^^^^^^^^^^^^^ The following features are not yet implemented in this prototype but will be needed: package dependencies installation:: We should be able to list the required build dependencies, so that they are installed automatically (and optionally removed at the end of the build). I have some ideas about how to do that so that it can work on multiple distributions, but have not implemented it yet. VM or chroot creation:: While burps can be used to build inside a chroot, or a VM with ssh, the current prototype does not handle the creation of a VM image or chroot. So we need a tool to automate that part, similar to the 'gitian/make-vms.sh' script. Future work ^^^^^^^^^^^ Here are some ideas for future work to be done on this tool. Integration of tests:: We can make it possible to run some tests from this tool after a build. Those tests can instanciate some VMs to run the tests inside those VMs. Integration with builds database:: See the proposal about a builds database. We can integrate burps so that builds can be automatically uploaded to this database. We can also make it possible to download some input files instead of build them, if the files are available in the build database with enought signatures from trusted people. Reproducible build testing:: We can add an option to build software using a library that randomize readdir(3) and python dict.iterkeys to make non-reproducibility issues more reproducible, as https://lists.torproject.org/pipermail/tor-dev/2013-December/005925.html[discussed on tor-dev].