On Mon, 14 Jul 2014, Georg Koppen wrote:
Nicolas Vigier:
Hello,
I am currently investigating how we can run the Mozilla xpcshell tests on our version of firefox, and the other unit tests.
The xpcshell tests are included on the firefox source tree, and require a complete build tree, built without the '--disable-tests' option. So we cannot run those tests on the version that is shipped in the bundle.
To run those tests, I can see different options:
Run the tests as part of the gitian build process and save the output in a file that can be analyzed later.
The bundle build process is already very long, and we probably don't want to make it even longer. Running the xpcshell tests on my laptop (i5-2520M CPU @ 2.50GHz) took 2.5 hours. So this is probably not a good idea.
Save the build tree in a tarball at the end of the build, and make it available with the bundle files. We can then use it to run the tests as part of our test suite.
The main problem with this is that the obj directory at the end of the build is taking 6GB. It's possible to reduce it to a 888MB tar.xz file, but it takes 50 minutes to create it on my laptop.
Build the browser separatly, in the VM that we use for running the tests. We enough test VMs to be able to do that.
This also allow us to run those tests on any commit.
The main problem with this is that we are not running the tests on the same binaries that are shipped in the bundle.
So I'm thinking about doing 3. Does anybody have any other idea or comment about this ?
Assuming all commits make it into nightlies first I think I am a fan of having the tests as part of our regular nightly builds. I am even in favor of having the latter as debug builds by default which would help us debugging test failures more easily, too. But that is an orthogonal issue. Anyway, both could easily be achieved by a special .mozconfig file and some minor patching of our browser descriptors (like "take the nightly .mozconfig iff we are about to build a Tor Browser for versions.nightly")
I have started integrating xpcshell tests into our test suite by doing [3].
I've been running it on two commits, 155c4c79792c997b8d70 (mozilla commit), and c660ca4cb98351bbce6c (tor-browser-24.7.0esr-3.x-1 branch): https://people.torproject.org/~boklm/tmp/tests/r/browser-155c4c79792c997b8d7... https://people.torproject.org/~boklm/tmp/tests/r/browser-c660ca4cb98351bbce6...
My plan now is to to add a diff of test failures between current and previous commit, and launch it on all of our commits, to find which commit is responsible for which test failure.
Then I will be looking at doing [2], to run the tests on our nightly build binaries.