Hello,
Starting from now, when a new build for a new release is available, the test suite will be run automatically, and the results sent to this mailing list.
They are also published on this page: https://people.torproject.org/~boklm/tbbtests/
The detection of new builds works like this:
- the most recent signed tag on the master branch on builders/tor-browser-bundle.git is used to find the current version that needs testing
- the build directory of the tbb builders (currently this is 'mikeperry', 'gk', 'linus', 'erinn', 'boklm'. Let me know if someone needs to be added to that list) on people.tpo is scanned to find if the version is available.
- the 5 first letters of the sha256sum of the sha256sums.txt file is used as a build identifier. The same build is not tested twice. When a new build is uploaded by someone, the sha256sums.txt and build identifier change, so the tests are run again.
The tests are run on all the platforms the test suite supports (currently a few Linux distributions, soon Windows and later Mac OS X).
Because of these changes that I'm pushing now, the tests on current 3.6 version will be run again (although that version is already released) and results sent here.
There are sometimes some false positives in the tests:
- in the 'check' test, this error: NoSuchElementException: Message: u'Unable to locate element: {"method":"css selector","selector":"h1.on"}' It might be because check sometimes think we are not using tor when we are.
- sometimes the https-everywhere or https-everywhere-disabled tests fail because of a timeout while loading the http page we expect https-e will redirect to https (currently http://www.mediawiki.org/ is used in the tests).
I'm going to try to remove those false positive errors, but for now they should be ignored.
The test results on new releases are sent to this mailing list, but the tests results on the nightly builds are sent only to Georg, Mike and me (starting today). If anyone else wants to receive them, let me know.
Nicolas