On Wed, Feb 12, 2020 at 02:40:38PM +0100, Nicolas Vigier wrote:
The above NYTProf output is not from a full build. It's from modifying projects/snowflake/config only and then building snowflake for only one target; still it took almost an hour. perl -d:NYTProf rbm/rbm build snowflake --target testbuild --target torbrowser-linux-x86_64 nytprofhtml
Was it a new build of snowflake where some of the dependencies had to be built, or a build where only snowflake had to be built?
In the first case, I think part of the reason why it takes time is that snowflake has many small dependencies, and each one is built in a separate container, so some time is spent extracting/removing chroots.
If only snowflake and no other dependencies was built, then I don't know why it took almost one hour. In my case, adding a blank line to projects/snowflake/build and rebuilding it takes around 8 minutes.
It was a build of snowflake only, none of the dependencies. In other words, I had already done a complete "make testbuild" (this is the one that took all weekend), then I modified projects/snowflake/config, then I did "rbm build snowflake" as shown above.
Overnight I did a "make testbuild" to build snowflake for the rest of the targets, and it took 9 hours: real 544m23.414s user 482m3.219s sys 73m29.899s I'll put the top-level stdout at the end of this message; I'm pretty sure it shows that only snowflake and tor-browser neede to be rebuilt, and the android targets didn't need to be rebuilt at all because they don't have snowflake as a dependency.
Notice how there are three loops of RBM::process_template before doing an RBM::config_p that calls back into further RBM::process_template.
Yes, there is a lot of recursive calls.
For example the filename in projects/tor-browser/config is defined like this:
filename: 'tor-browser-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %]'
computing this filename will require computing var/build_id, which is defined like this:
so this require recursively computing filenames of all other projects, which requires computing sha256sum of all input files.
Thank you for the explanation. It makes a lot of sense. I like this feature of rbm a lot, that I can feel confident the right things will be rebuilt when I change something.
I suppose Go projects are especially troublesome. In my build over the weekend I was testing a change that added about 10 new transitive dependencies to snowflake: https://lists.torproject.org/pipermail/anti-censorship-team/2020-February/00... https://gitweb.torproject.org/user/dcf/tor-browser-build.git/commit/?h=snowf... In my build last night (the one whose stdout is below), I removed all those dependencies again and added only 1 dependency: https://gitweb.torproject.org/user/dcf/tor-browser-build.git/commit/?h=snowf...
I opened a ticket for this: https://trac.torproject.org/projects/tor/ticket/33283
I think this patch should avoid doing multiple `git checkout` and `git log` on the same projects: https://gitweb.torproject.org/user/boklm/rbm.git/commit/?h=bug_33283&id=...
Thanks, I'll try the patch after I finish packaging the build from last night.
----
Output of last night's "time make testbuild" with only snowflake needing to be rebuilt. I had already manually built snowflake for linux-x86_64. It was commit 5eff0f3ba406eb7d8cd90711edab7de7d7bda4fa in https://git.torproject.org/user/dcf/snowflake.git.
``` # time make testbuild git submodule update --init ./rbm/rbm build release --target testbuild --target torbrowser-all Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-armv7-143b27 Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-x86-b005b1 Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-x86_64-5512f4 Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-android-aarch64-0dec52 Building project tor-browser - tor-browser-9.5a4-linux-x86_64-7ef7ab Using file /root/tor-browser-build/out/container-image/container-image_wheezy-amd64-a32817165f00.tar.gz Using file /root/tor-browser-build/projects/tor-browser/run_scripts Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-linux-x86_64-46c98c Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-linux-x86_64-5408dc Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-b894fe.xpi Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-074f0d.tar.gz Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-linux-x86_64-62b5ad.tar.gz Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-linux-x86_64-fa4d5c.tar.gz Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/start-tor-browser.desktop Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/execdesktop Using file /root/tor-browser-build/projects/tor-browser/gtk3-settings.ini Build log: /root/tor-browser-build/logs/tor-browser-linux-x86_64.log Finished build of project tor-browser - tor-browser-9.5a4-linux-x86_64-7ef7ab Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-linux-x86_64-7ef7ab Building project tor-browser - tor-browser-9.5a4-linux-i686-56bb5f Using file /root/tor-browser-build/out/container-image/container-image_wheezy-amd64-ae74865f5ad7.tar.gz Using file /root/tor-browser-build/projects/tor-browser/run_scripts Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-linux-i686-78f104 Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-linux-i686-8cabbd Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-b894fe.xpi Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-074f0d.tar.gz Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-linux-i686-0d2a0b.tar.gz Building project snowflake - snowflake-5eff0f3ba406-linux-i686-172b69.tar.gz Created /root/tor-browser-build/tmp/rbm-7RbRC/snowflake-5eff0f3ba406.tar.gz Using file /root/tor-browser-build/out/container-image/container-image_wheezy-amd64-f2bcf2cde274.tar.gz Using file /root/tor-browser-build/out/go/go-1.12.13-6f6dc9.tar.gz Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-linux-i686-095a9f.tar.gz Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-linux-i686-43ea9b.tar.gz Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-linux-i686-02e5a3.tar.gz Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-linux-i686-4b046f.tar.gz Build log: /root/tor-browser-build/logs/snowflake-linux-i686.log Finished build of project snowflake - snowflake-5eff0f3ba406-linux-i686-172b69.tar.gz Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-linux-i686-172b69.tar.gz Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/start-tor-browser.desktop Using file /root/tor-browser-build/projects/tor-browser/RelativeLink/execdesktop Using file /root/tor-browser-build/projects/tor-browser/gtk3-settings.ini Build log: /root/tor-browser-build/logs/tor-browser-linux-i686.log Finished build of project tor-browser - tor-browser-9.5a4-linux-i686-56bb5f Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-linux-i686-56bb5f Building project tor-browser - tor-browser-9.5a4-windows-i686-465936 Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-860c78a36523.tar.gz Using file /root/tor-browser-build/projects/tor-browser/run_scripts Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-windows-i686-89cecd Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-i686-b41491 Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-46dd96.xpi Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-f954ac.tar.gz Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-windows-i686-72bc65.tar.gz Building project snowflake - snowflake-5eff0f3ba406-windows-i686-dcf129.tar.gz Created /root/tor-browser-build/tmp/rbm-oAH9V/snowflake-5eff0f3ba406.tar.gz Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-18fd98cf5daf.tar.gz Using file /root/tor-browser-build/out/go/go-1.12.13-4c7c84.tar.gz Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-windows-i686-1a6f4a.tar.gz Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-windows-i686-852b18.tar.gz Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-windows-i686-dacf47.tar.gz Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-windows-i686-01524c.tar.gz Build log: /root/tor-browser-build/logs/snowflake-windows-i686.log Finished build of project snowflake - snowflake-5eff0f3ba406-windows-i686-dcf129.tar.gz Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-windows-i686-dcf129.tar.gz Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi Using file /root/tor-browser-build/out/nsis/nsis-3.04-windows-i686-6d8d58.tar.gz Using file /root/tor-browser-build/out/tbb-windows-installer/tbb-windows-installer-src-0.5-dbf177.tar.gz Using file /root/tor-browser-build/projects/tor-browser/pe_checksum_fix.py Using file /root/tor-browser-build/out/tor-browser/pefile-2017.11.5.tar.gz Build log: /root/tor-browser-build/logs/tor-browser-windows-i686.log Finished build of project tor-browser - tor-browser-9.5a4-windows-i686-465936 Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-windows-i686-465936 Building project tor-browser - tor-browser-9.5a4-windows-x86_64-9f74bd Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-860c78a36523.tar.gz Using file /root/tor-browser-build/projects/tor-browser/run_scripts Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-windows-x86_64-ffc30f Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-x86_64-79026c Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-46dd96.xpi Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-f954ac.tar.gz Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-windows-x86_64-bb2f23.tar.gz Building project snowflake - snowflake-5eff0f3ba406-windows-x86_64-57b6ff.tar.gz Created /root/tor-browser-build/tmp/rbm-eOk0z/snowflake-5eff0f3ba406.tar.gz Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-18fd98cf5daf.tar.gz Using file /root/tor-browser-build/out/go/go-1.12.13-846325.tar.gz Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-windows-x86_64-739ef5.tar.gz Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-windows-x86_64-8eaaca.tar.gz Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-windows-x86_64-629396.tar.gz Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-windows-x86_64-ae697f.tar.gz Build log: /root/tor-browser-build/logs/snowflake-windows-x86_64.log Finished build of project snowflake - snowflake-5eff0f3ba406-windows-x86_64-57b6ff.tar.gz Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-windows-x86_64-57b6ff.tar.gz Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi Using file /root/tor-browser-build/out/nsis/nsis-3.04-windows-x86_64-91a7bd.tar.gz Using file /root/tor-browser-build/out/tbb-windows-installer/tbb-windows-installer-src-0.5-dbf177.tar.gz Using file /root/tor-browser-build/projects/tor-browser/pe_checksum_fix.py Using file /root/tor-browser-build/out/tor-browser/pefile-2017.11.5.tar.gz Build log: /root/tor-browser-build/logs/tor-browser-windows-x86_64.log Finished build of project tor-browser - tor-browser-9.5a4-windows-x86_64-9f74bd Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-windows-x86_64-9f74bd Building project tor-browser - tor-browser-9.5a4-osx-x86_64-db1da8 Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-1d614657ea1b.tar.gz Using file /root/tor-browser-build/projects/tor-browser/run_scripts Using directory /root/tor-browser-build/out/firefox/firefox-78e8d897cd2e-osx-x86_64-56070b Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-osx-x86_64-cd810e Using file /root/tor-browser-build/out/https-everywhere/https-everywhere-2019.11.7-46dd96.xpi Using file /root/tor-browser-build/out/fonts/fonts-720e34851382-5830a6.tar.gz Using file /root/tor-browser-build/out/obfs4/obfs4-0.0.11-osx-x86_64-9c01d4.tar.gz Building project snowflake - snowflake-5eff0f3ba406-osx-x86_64-f9fc60.tar.gz Created /root/tor-browser-build/tmp/rbm-hxgYM/snowflake-5eff0f3ba406.tar.gz Using file /root/tor-browser-build/out/container-image/container-image_stretch-amd64-c1192d34d787.tar.gz Using file /root/tor-browser-build/out/go/go-1.12.13-bbe0e3.tar.gz Using file /root/tor-browser-build/out/uniuri/uniuri-8902c56451e9-osx-x86_64-6d72b5.tar.gz Using file /root/tor-browser-build/out/goptlib/goptlib-v1.1.0-osx-x86_64-ab0a8a.tar.gz Using file /root/tor-browser-build/out/pion-webrtc/pion-webrtc-77c6e3b827e4-osx-x86_64-e18e7a.tar.gz Using file /root/tor-browser-build/out/quic-go/quic-go-907071221cf9-osx-x86_64-a6f2f4.tar.gz Build log: /root/tor-browser-build/logs/snowflake-osx-x86_64.log Finished build of project snowflake - snowflake-5eff0f3ba406-osx-x86_64-f9fc60.tar.gz Using file /root/tor-browser-build/out/snowflake/snowflake-5eff0f3ba406-osx-x86_64-f9fc60.tar.gz Using directory /root/tor-browser-build/projects/tor-browser/Bundle-Data Using file /root/tor-browser-build/out/tor-browser/noscript_security_suite-11.0.11-an+fx.xpi Using file /root/tor-browser-build/out/libdmg-hfsplus/libdmg-hfsplus-dfd5e5cc3dc1-62fa3e.tar.gz Build log: /root/tor-browser-build/logs/tor-browser-osx-x86_64.log Finished build of project tor-browser - tor-browser-9.5a4-osx-x86_64-db1da8 Using directory /root/tor-browser-build/out/tor-browser/tor-browser-9.5a4-osx-x86_64-db1da8 Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-i686-d97207 Using directory /root/tor-browser-build/out/tor/tor-0.4.3.1-alpha-windows-x86_64-18debd Build log: /root/tor-browser-build/logs/release-linux-x86_64.log
real 544m23.414s user 482m3.219s sys 73m29.899s ```