On Fri, 11 Jul 2014, Mark Smith wrote:
On 7/11/14 11:02 AM, Nicolas Vigier wrote:
After running the xpcshell tests on Firefox 24.6 and Tor Browser, I have seen that:
6 tests failed on firefox, and it took 10 minutes to run those tests
121 test failed on tor-browser, and it took 2.5 hours to run those tests
My plan now is to automatically run the tests on all of our commits (starting from the last from mozilla), save the results and time spent running each test, and compare with the previous commit, in order to find which commits cause tests to fail.
Do you plan to investigate why more tests fail (and take so long) on tor-browser? Many are probably explainable due to our patches, but it would be good to know for sure. Or maybe we need to divide the investigation task up among the various TBB developers?
What I'm planning to do is to make something that will rebuild and run the tests on each of our commits, and compare results and time spent with results from the previous commit. This should tell us for each test failure (or increase in testing time), which commit introduced it. Once we have those results, it should simplify the investigations, and we can divide the task if needed.