Hi there,
I'm done with the first batch of work on the test side. You have the (rebased just now) work here: https://github.com/emanchado/tor/commits/master. The rest of this e-mail summarises what I've done and explains my plan for further work.
The content of the commits -------------------------- I have reviewed the whole test_util.c and made three kinds of changes:
1. Added more tests. Some of them failed, and after checking with other people on IRC the conclusion seems to be that they should pass (ie. they reproduce actual bugs in the code). To keep the test suite from failing, they are inside "#if 0" blocks. So someone should look for "#if 0" inside test_util.c and fix the code that makes those fail. Or maybe I should file bugs for those?
2. In comparison assertions, the general convention seems to be to place the expected value first ("test_eq(0, functioncall(...))" rather than "test_eq(functioncall(...), 0)"). I have modified the assertions not following that convention, so they all look the same.
3. General clean up, small code reorganisations, fix typos and such. Eg, I have turned all the "tt_int_op(a, ==, b)" into "test_eq(a, b)".
I was thinking of blogging about what I saw (esp. related to point 1). I think there are valuable lessons to be learned, which will help other people writing tests (both for Tor and outside of Tor). I'm not sure if there's enough content for a blog post, but if I do it after all, should I post the link here?
Plan for the future -------------------
As I haven't explored coverage yet, I'll start with that for the code in src/common/ and see what I can do. My gut feeling (without having looked at other parts of the code yet!) is that tackling unit tests for code in src/or/ will be hard for me, and it seems better to explain what kind of things I have done in src/common/ so others, more familiar with that code, can fix/improve the unit tests for src/or/*. In other words, it seems much more effective for me to support someone who knows the code in src/or/*, rather than the other way around.
After I'm done with what I think is easy enough for me to do with the unit tests, I'll have a look at Stem (Damian's suggestion) and chutney (Nick's suggestion), and decide what I do next.
On Sat, Feb 11, 2012 at 7:12 AM, Esteban Manchado Velázquez emanchado@demiurgo.org wrote:
Hi there,
I'm done with the first batch of work on the test side. You have the (rebased just now) work here: https://github.com/emanchado/tor/commits/master.
A suggestion: In the future, it's best to do commits on one or more "topic branches", where each branch is for a separate kind of work. That way, it's way easier for upstream to merge some of the commits, hold off on others, and decline others.
As it stands, if you do all your commits in a "master" branch, and I want to take some but not all of them, I have to cherry-pick the individual commits. Worse still, your branch and the upstream branch will then have diverged: if you try to pull the official repository onto your master again, you won't have the actual history of the Tor master branch , but some other thing that only exists on your master branch. This can make stuff yucky fast.
For now, let's leave the current branches as they are. Once we've got the contents of your current master branch reviewed/merged/not-merged, you can reset your master to match tor's, and then do future work in topic branches.
All that said: I like the granularity of your commits! Each one is logically independent and easy to review.
The rest of this e-mail summarises what I've done and explains my plan for further work.
The content of the commits
I have reviewed the whole test_util.c and made three kinds of changes:
- Added more tests. Some of them failed, and after checking with other
people on IRC the conclusion seems to be that they should pass (ie. they reproduce actual bugs in the code). To keep the test suite from failing, they are inside "#if 0" blocks. So someone should look for "#if 0" inside test_util.c and fix the code that makes those fail. Or maybe I should file bugs for those?
Filing bugs is the right move; it looks like you've already started to do this.
- In comparison assertions, the general convention seems to be to place the
expected value first ("test_eq(0, functioncall(...))" rather than "test_eq(functioncall(...), 0)"). I have modified the assertions not following that convention, so they all look the same.
Hm. I don't think we actually had a convention on this one.
- General clean up, small code reorganisations, fix typos and such. Eg, I
have turned all the "tt_int_op(a, ==, b)" into "test_eq(a, b)".
Actually, test_eq was the old way; tt_int_op is the newer way since we switched to tinytest.
Some other comments:
In general, the hardest thing for me to review here is not whether the tests are right, but whether they removed any old tests in revising them. I'll need to have another look through the patch series to be sure.
On commit 5740e0fc1f00fa91be107ee6c4315d114c5ffdc4, the snprintf() calls there should be tor_snprintf().
On commit f40c04a2137724f7b285e8d69ee62e47df1f9049, "iff" is not a typo. It is a standard abbreviation for "if and only if." We use it to say things like "Return true iff X", since otherwise we would need to say "Return true if X; return false otherwise." (If we just said "Return true if X," the function would technically be allowed to _always_ return true.)
I was thinking of blogging about what I saw (esp. related to point 1). I think there are valuable lessons to be learned, which will help other people writing tests (both for Tor and outside of Tor). I'm not sure if there's enough content for a blog post, but if I do it after all, should I post the link here?
Please do!
peace,
On 12 Feb 2012, at 03:28, Nick Mathewson wrote:
- In comparison assertions, the general convention seems to be to place the
expected value first ("test_eq(0, functioncall(...))" rather than "test_eq(functioncall(...), 0)"). I have modified the assertions not following that convention, so they all look the same.
Hm. I don't think we actually had a convention on this one.
It's not so important with tinytest because the output messages don't distinguish between expected and actual. In test frameworks where it does matter (e.g. JUnit) expected usually comes first in my experience. So I think the proposed convention is a good one and could be useful should we wish to change error messages to indicate the test failure message to say what was expected. I have found this feature of JUnit helpful, although by no means critical.
Steven.
On Sun, 12 Feb 2012 04:28:38 +0100, Nick Mathewson nickm@alum.mit.edu wrote:
On Sat, Feb 11, 2012 at 7:12 AM, Esteban Manchado Velázquez emanchado@demiurgo.org wrote:
Hi there,
I'm done with the first batch of work on the test side. You have the (rebased just now) work here: https://github.com/emanchado/tor/commits/master.
A suggestion: In the future, it's best to do commits on one or more "topic branches", where each branch is for a separate kind of work. That way, it's way easier for upstream to merge some of the commits, hold off on others, and decline others.
In this concrete case, how would you separate them? One branch per test function? Which in this case is more or less one branch per commit?
As it stands, if you do all your commits in a "master" branch, and I want to take some but not all of them, I have to cherry-pick the individual commits. Worse still, your branch and the upstream branch will then have diverged: if you try to pull the official repository onto your master again, you won't have the actual history of the Tor master branch , but some other thing that only exists on your master branch. This can make stuff yucky fast.
Yeah, I thought of that too late.
For now, let's leave the current branches as they are. Once we've got the contents of your current master branch reviewed/merged/not-merged, you can reset your master to match tor's, and then do future work in topic branches.
OK.
All that said: I like the granularity of your commits! Each one is logically independent and easy to review.
Yes, that was my idea. Cool that works for you :-)
[...] To keep the test suite from failing, they are inside "#if 0" blocks. So someone should look for "#if 0" inside test_util.c and fix the code that makes those fail. Or maybe I should file bugs for those?
Filing bugs is the right move; it looks like you've already started to do this.
Yes, it seems it was only 3 after all. I didn't know what component to use so I left it blank. If there's any better way to file them (component, Cc or whatever), I'd be happy to learn that and use it in the future :-)
- In comparison assertions, the general convention seems to be to
place the expected value first ("test_eq(0, functioncall(...))" rather than "test_eq(functioncall(...), 0)"). I have modified the assertions not following that convention, so they all look the same.
Hm. I don't think we actually had a convention on this one.
Yeah, "convention" was a strong word. It was just my impression that there were more "expected, actual" than "actual, expected", and I thought forcing a convention would be good. Although, as Steven said, tinytest doesn't actually specify "expected" an "actual" in the output, I find it's less "cognitive load" to debug or make sense of tests if you can assume what's the order of the values in the failure output.
- General clean up, small code reorganisations, fix typos and such.
Eg, I have turned all the "tt_int_op(a, ==, b)" into "test_eq(a, b)".
Actually, test_eq was the old way; tt_int_op is the newer way since we switched to tinytest.
Oh, hehe, oops. I found test_eq much more readable so I thought that was the preferred way.
I can turn everything into tt_* calls and add a comment to test.h stating that test_eq and such are deprecated. Should I do that? In that case, however, I would place the expected at the end, because otherwise the code will look like "tt_int_op(0, ==, strlen(foo))", which I think looks pretty awkward.
Some other comments:
In general, the hardest thing for me to review here is not whether the tests are right, but whether they removed any old tests in revising them. I'll need to have another look through the patch series to be sure.
OK. I *did* remove a couple of assertions, either because I didn't think they added anything, or because they were replaced by (in my view) better assertions. In particular, I removed the hardcoded gzip magic number check (there's already a check_compression_method, see commit c4c1d56d96623a45775ec2544c0c6951fbfa2d9f) and I changed some of the strcasecmpend cases (commit 03876f0a721ced6ffebb0c61134d5b8396d7600e). There are probably others.
On commit 5740e0fc1f00fa91be107ee6c4315d114c5ffdc4, the snprintf() calls there should be tor_snprintf().
Good catch.
On commit f40c04a2137724f7b285e8d69ee62e47df1f9049, "iff" is not a typo. It is a standard abbreviation for "if and only if." We use it to say things like "Return true iff X", since otherwise we would need to say "Return true if X; return false otherwise." (If we just said "Return true if X," the function would technically be allowed to _always_ return true.)
Right, my bad.
I was thinking of blogging about what I saw (esp. related to point 1). I think there are valuable lessons to be learned, which will help other people writing tests (both for Tor and outside of Tor). I'm not sure if there's enough content for a blog post, but if I do it after all, should I post the link here?
Please do!
I went through the commits and I definitely have enough to write about. I'll send the link when I'm done.
-
Esteban Manchado Velázquez -
Nick Mathewson -
Steven Murdoch