-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi all,
Would there be interest in using lcov [0] for test coverage analysis?
Obviously, as lcov is an external pacakge, the existing scripts that only use built in gcov commands must remain.
However, if I were to create a script that generates a pretty, HTML coverage report using lcov & genhtml, would this be something others would use? And how would you like to see this implemented? As a custom make target (e.g., make htmlcoverage + autotools black magic to silently disable/hide the target if we can't find lcov), or as a standalone script (i.e., ./scripts/test/run_lcov.sh).
For what it's worth, I've manually run lcov on tor after enabling coverage through ./configure, and the output seems sane, so I think from a technical standpoint this would work.
[0]: http://ltp.sourceforge.net/coverage/lcov.php
Cheers, daube
On Sun, Jun 15, 2014 at 8:00 AM, David Murray spam@kdmurray.id.au wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi all,
Would there be interest in using lcov [0] for test coverage analysis?
Obviously, as lcov is an external pacakge, the existing scripts that only use built in gcov commands must remain.
However, if I were to create a script that generates a pretty, HTML coverage report using lcov & genhtml, would this be something others would use? And how would you like to see this implemented? As a custom make target (e.g., make htmlcoverage + autotools black magic to silently disable/hide the target if we can't find lcov), or as a standalone script (i.e., ./scripts/test/run_lcov.sh).
So, I think that if I'm reading the documentation right, all that should be necessary would be an additional make target. (I'm assuming that the make target wouldn't call "configure" -- we shouldn't define our make targets that do that, IMO.)
(If I'm reading this properly, lcov uses the same command-line arguments to gcc that gcov does, so it could also depend on the --enable-coverage configure flag.)
I don't think it's necessary to have the target disabled when there is no lcov, though it would be nice if it gave a useful error message instead of "lcov: command not found".
Additionally, one thing that I'd really love to see -- though I don't at all know whether lcov can do this as it stands -- is a semantic diff between two coverage outputs. When writing new unit tests at random, or when checking functions that you know need coverage, it's useful to ask "Which lines are not covered?" But for validating new code and checking patches, the question I want to ask is "Which *new* or *changed* lines are not covered?"
We kinda have a script for this now, but learning to read its outputs is not as easy as I'd like.
cheers,
On Mon, Jun 16, 2014 at 10:55 AM, Nick Mathewson nickm@freehaven.net wrote: [...]
Additionally, one thing that I'd really love to see -- though I don't at all know whether lcov can do this as it stands -- is a semantic diff between two coverage outputs. When writing new unit tests at random, or when checking functions that you know need coverage, it's useful to ask "Which lines are not covered?" But for validating new code and checking patches, the question I want to ask is "Which *new* or *changed* lines are not covered?"
We kinda have a script for this now, but learning to read its outputs is not as easy as I'd like.
It would appear at first glance that you can do something *almost* like this using lcov --diff and genhtml --highlight. Those options together seem to be able to let you know about lines that used to be covered but which are no longer covered. (I'm hoping to learn about likes that changed or started to exist which are uncovered.)
On Mon, Jun 16, 2014 at 4:07 PM, Nick Mathewson nickm@freehaven.net wrote:
On Mon, Jun 16, 2014 at 10:55 AM, Nick Mathewson nickm@freehaven.net wrote: [...]
Additionally, one thing that I'd really love to see -- though I don't at all know whether lcov can do this as it stands -- is a semantic diff between two coverage outputs. When writing new unit tests at random, or when checking functions that you know need coverage, it's useful to ask "Which lines are not covered?" But for validating new code and checking patches, the question I want to ask is "Which *new* or *changed* lines are not covered?"
We kinda have a script for this now, but learning to read its outputs is not as easy as I'd like.
It would appear at first glance that you can do something *almost* like this using lcov --diff and genhtml --highlight. Those options together seem to be able to let you know about lines that used to be covered but which are no longer covered. (I'm hoping to learn about likes that changed or started to exist which are uncovered.)
I asked this question on the ltp list, and got an answer:
http://sourceforge.net/p/ltp/mailman/message/32486080/
This is a kludge too, but we should investigate whether it's better than our current kludge.
-
David Murray -
Nick Mathewson