On Wed, Dec 19, 2012 at 2:29 PM, Simon simonhf@gmail.com wrote: [...]
Maybe there is no automated testing for any Tor projects? At least a quick search on the wiki only found [1] which lists possible ways to test (but was created 7 months ago and apparently not updated since and collecting dust) and [2] discussing a manual test procedure for TBB. However, tor-0.2.3.25.tar.gz does reveal some test files but the source code ratio of production code to test code is not inspiring at first glance:
[...]
Be aware that we've also been using 'chutney' and 'experimentor' for integration testing. They supplement coverage a bit, though they need more tests, and each tends to hide certain classes of error.
Tor seems to have good planning compared to most open source projects. So I would be interested in hearing why testing is apparently 'falling between the cracks'. Why isn't there just 10 times more test LOC?
Not because of any hatred or disapproval of tests--just because nobody's written that 100 kloc of testing code yet.
I think that's for two main reasons: * We were in a hurry when we wrote lots of the stuff we wrote. * Large parts of the codebase have been written in a tightly coupled style that needs refactoring before it can be tested without a live Tor network at hand. * Until Tor 0.2.2, our testing framework didn't let us have tests that touched global state, which made our tests in general pretty fragile.
What about implementing a new policy immediately: Any new production LOC committed must be covered by tests, or peer reviewed and democratically excluded?
Goodness knows we need more review and testing.
It doesn't seem fair to reject patches for lacking test coverage when they are patches to code that itself lacks coverage, though. If you write a 5-line patch to connection_ap_rewrite_and_attach, for instance, you probably shouldn't be on the hook for refactoring the whole function to make it testable, though you will be hard-pressed to write any tests for that monster without a complete refactor.
It might be a reasonable goal to try to set a plan for increasing test coverage by a certain percentage with each release.
If you like and you have time, it would be cool to stop by the tickets on trac.torproject.org for milestone "Tor: 0.2.4.x-final" in state "needs_review" and look to see whether you think any of them have code that would be amenable to new tests, or to look through currently untested functions and try to figure out how to make more of them tested and testable.
yrs, -- Nick