Hello,
I am currently investigating how we can run the Mozilla xpcshell tests on our version of firefox, and the other unit tests.
The xpcshell tests are included on the firefox source tree, and require a complete build tree, built without the '--disable-tests' option. So we cannot run those tests on the version that is shipped in the bundle.
To run those tests, I can see different options:
1. Run the tests as part of the gitian build process and save the output in a file that can be analyzed later.
The bundle build process is already very long, and we probably don't want to make it even longer. Running the xpcshell tests on my laptop (i5-2520M CPU @ 2.50GHz) took 2.5 hours. So this is probably not a good idea.
2. Save the build tree in a tarball at the end of the build, and make it available with the bundle files. We can then use it to run the tests as part of our test suite.
The main problem with this is that the obj directory at the end of the build is taking 6GB. It's possible to reduce it to a 888MB tar.xz file, but it takes 50 minutes to create it on my laptop.
3. Build the browser separatly, in the VM that we use for running the tests. We enough test VMs to be able to do that.
This also allow us to run those tests on any commit.
The main problem with this is that we are not running the tests on the same binaries that are shipped in the bundle.
So I'm thinking about doing 3. Does anybody have any other idea or comment about this ?
Nicolas
After running the xpcshell tests on Firefox 24.6 and Tor Browser, I have seen that:
- 6 tests failed on firefox, and it took 10 minutes to run those tests
- 121 test failed on tor-browser, and it took 2.5 hours to run those tests
My plan now is to automatically run the tests on all of our commits (starting from the last from mozilla), save the results and time spent running each test, and compare with the previous commit, in order to find which commits cause tests to fail.
Nicolas
On 7/11/14 11:02 AM, Nicolas Vigier wrote:
After running the xpcshell tests on Firefox 24.6 and Tor Browser, I have seen that:
6 tests failed on firefox, and it took 10 minutes to run those tests
121 test failed on tor-browser, and it took 2.5 hours to run those tests
My plan now is to automatically run the tests on all of our commits (starting from the last from mozilla), save the results and time spent running each test, and compare with the previous commit, in order to find which commits cause tests to fail.
Do you plan to investigate why more tests fail (and take so long) on tor-browser? Many are probably explainable due to our patches, but it would be good to know for sure. Or maybe we need to divide the investigation task up among the various TBB developers?
On Fri, 11 Jul 2014, Mark Smith wrote:
On 7/11/14 11:02 AM, Nicolas Vigier wrote:
After running the xpcshell tests on Firefox 24.6 and Tor Browser, I have seen that:
6 tests failed on firefox, and it took 10 minutes to run those tests
121 test failed on tor-browser, and it took 2.5 hours to run those tests
My plan now is to automatically run the tests on all of our commits (starting from the last from mozilla), save the results and time spent running each test, and compare with the previous commit, in order to find which commits cause tests to fail.
Do you plan to investigate why more tests fail (and take so long) on tor-browser? Many are probably explainable due to our patches, but it would be good to know for sure. Or maybe we need to divide the investigation task up among the various TBB developers?
What I'm planning to do is to make something that will rebuild and run the tests on each of our commits, and compare results and time spent with results from the previous commit. This should tell us for each test failure (or increase in testing time), which commit introduced it. Once we have those results, it should simplify the investigations, and we can divide the task if needed.
Nicolas Vigier:
Hello,
I am currently investigating how we can run the Mozilla xpcshell tests on our version of firefox, and the other unit tests.
The xpcshell tests are included on the firefox source tree, and require a complete build tree, built without the '--disable-tests' option. So we cannot run those tests on the version that is shipped in the bundle.
To run those tests, I can see different options:
Run the tests as part of the gitian build process and save the output in a file that can be analyzed later.
The bundle build process is already very long, and we probably don't want to make it even longer. Running the xpcshell tests on my laptop (i5-2520M CPU @ 2.50GHz) took 2.5 hours. So this is probably not a good idea.
Save the build tree in a tarball at the end of the build, and make it available with the bundle files. We can then use it to run the tests as part of our test suite.
The main problem with this is that the obj directory at the end of the build is taking 6GB. It's possible to reduce it to a 888MB tar.xz file, but it takes 50 minutes to create it on my laptop.
Build the browser separatly, in the VM that we use for running the tests. We enough test VMs to be able to do that.
This also allow us to run those tests on any commit.
The main problem with this is that we are not running the tests on the same binaries that are shipped in the bundle.
So I'm thinking about doing 3. Does anybody have any other idea or comment about this ?
Assuming all commits make it into nightlies first I think I am a fan of having the tests as part of our regular nightly builds. I am even in favor of having the latter as debug builds by default which would help us debugging test failures more easily, too. But that is an orthogonal issue. Anyway, both could easily be achieved by a special .mozconfig file and some minor patching of our browser descriptors (like "take the nightly .mozconfig iff we are about to build a Tor Browser for versions.nightly")
Georg
On Mon, 14 Jul 2014, Georg Koppen wrote:
Nicolas Vigier:
Hello,
I am currently investigating how we can run the Mozilla xpcshell tests on our version of firefox, and the other unit tests.
The xpcshell tests are included on the firefox source tree, and require a complete build tree, built without the '--disable-tests' option. So we cannot run those tests on the version that is shipped in the bundle.
To run those tests, I can see different options:
Run the tests as part of the gitian build process and save the output in a file that can be analyzed later.
The bundle build process is already very long, and we probably don't want to make it even longer. Running the xpcshell tests on my laptop (i5-2520M CPU @ 2.50GHz) took 2.5 hours. So this is probably not a good idea.
Save the build tree in a tarball at the end of the build, and make it available with the bundle files. We can then use it to run the tests as part of our test suite.
The main problem with this is that the obj directory at the end of the build is taking 6GB. It's possible to reduce it to a 888MB tar.xz file, but it takes 50 minutes to create it on my laptop.
Build the browser separatly, in the VM that we use for running the tests. We enough test VMs to be able to do that.
This also allow us to run those tests on any commit.
The main problem with this is that we are not running the tests on the same binaries that are shipped in the bundle.
So I'm thinking about doing 3. Does anybody have any other idea or comment about this ?
Assuming all commits make it into nightlies first I think I am a fan of having the tests as part of our regular nightly builds. I am even in favor of having the latter as debug builds by default which would help us debugging test failures more easily, too. But that is an orthogonal issue. Anyway, both could easily be achieved by a special .mozconfig file and some minor patching of our browser descriptors (like "take the nightly .mozconfig iff we are about to build a Tor Browser for versions.nightly")
I have started integrating xpcshell tests into our test suite by doing [3].
I've been running it on two commits, 155c4c79792c997b8d70 (mozilla commit), and c660ca4cb98351bbce6c (tor-browser-24.7.0esr-3.x-1 branch): https://people.torproject.org/~boklm/tmp/tests/r/browser-155c4c79792c997b8d7... https://people.torproject.org/~boklm/tmp/tests/r/browser-c660ca4cb98351bbce6...
My plan now is to to add a diff of test failures between current and previous commit, and launch it on all of our commits, to find which commit is responsible for which test failure.
Then I will be looking at doing [2], to run the tests on our nightly build binaries.
On Fri, 18 Jul 2014, Nicolas Vigier wrote:
My plan now is to to add a diff of test failures between current and previous commit, and launch it on all of our commits, to find which commit is responsible for which test failure.
Last night I launched a rebuild of all of our commits in the tor-browser-24.7.0esr-3.x-1 branch. It should take a few days to rebuild and test everything. The first results are available at this URL: http://93.95.228.164/reports/
Each result page should show at the top the differences with the previous commit results.
For instance, it shows that commit 956774564037dcc502c2 introduce a failure in a test in toolkit/components/contentprefs: http://93.95.228.164/reports/r/browser-956774564037dcc502c2/browser-95677456...
I'll make a summary of which commits introduce which test failures once everything has been rebuilt.
-
Georg Koppen -
Mark Smith -
Nicolas Vigier