|Improving unit/automated test coverage||Mike Bland||6/2/14 7:38 AM|
With Ben Laurie's help, I recently contributed ssl/heartbeat_test.c,
which is a unit test that acts as a regression test against the
Heartbleed bug. I'd like to contribute more to the project in the
coming months in terms of helping grow a robust suite of
It seems that the encryption algorithms themselves are relatively
well-tested; in contrast, Heartbleed was an infrastructure bug. It's
in shoring up the test coverage of the infrastructure bits where I can
be of most direct service, but I'm hoping others may see opportunities
to apply similar techniques to more advanced testing issues.
I'd like to make sure there are at least a handful of contributors who
are willing to work closely with me to establish some new policies
around unit testing and code reviews (e.g. no new non-trivial changes
without tests; smaller, well-tested changes vs. monolithic, untested
changes), in addition to the actual writing of tests. There'd be some
tool setup and documentation work as well. The goal would be to help
everyone learn effective unit testing strategies so that, over time,
test coverage and code quality steadily improves. It will be a
lengthy, imperfect process, but one that I believe will ultimately
make a positive difference in the code base if people are willing to
My goal would be to help everyone learn to fish, to use the tired
cliché. I currently have very little knowledge of the OpenSSL code
base or community, and I don't have a ton of time to do all the heavy
lifting by myself; nor do I think that being the lone "testing guy"
would be the best use of my time or in the best interests of OpenSSL.
However, I want to contribute however I can to help the community as a
whole address this one particular issue, and to maximize the impact of
Happy to hear people's thoughts on this. If the uptake is positive, I
can help organize the effort to get things moving soon.
OpenSSL Project http://www.openssl.org
Development Mailing List opens...@openssl.org
Automated List Manager majo...@openssl.org
|Re: Improving unit/automated test coverage||Matt Caswell||6/2/14 7:52 AM|
On 2 June 2014 15:38, Mike Bland <mbl...@acm.org> wrote:I think that this is an exciting and important initiative. I would
love to see a much stronger test suite for OpenSSL.
|RE: Improving unit/automated test coverage||"Mody, Darshan (Darshan)"||6/2/14 8:04 AM|
I would like to volunteer for the same. I can spare some time on the weekends for it. Please do note that even I am new to openssl and it would be good to get to know more on the code by doing unit test.
|Re: Improving unit/automated test coverage||Mike Bland||6/4/14 2:58 PM|
Thanks to a few brave volunteers and the support of the core OpenSSL
team, it looks like we can begin moving on this effort soon. I've
begun to document the current state of things on the wiki:
There's lots to discuss with regard to the "Goals", "Tasks", and
"Discussion Topics" enumerated on the wiki. I'm happy to start a
discussion on this thread, or other openssl-dev threads; at the same
time, I'm wondering if a new openssl-testing group would be more
appropriate. If it's desirable, I could manage it via Google Groups,
rather than create more majordomo work for the core team. Thoughts?
|Re: Improving unit/automated test coverage||Kurt Roeckx||6/4/14 3:29 PM|
On Mon, Jun 02, 2014 at 10:38:05AM -0400, Mike Bland wrote:As far as I know the test covering SSL now try to set up a server
and client with various options and see that they can connect to
each other. It only seems to be testing the happy path. I would
like to see more tests covering the non-happy path. That of
course also goes for all crypto related things.
|Re: Improving unit/automated test coverage||Matt Caswell||6/4/14 3:40 PM|
That is definitely where the high value will be obtained. But that's a
hard problem I think to start with. It might be better to start with
something simpler - at least until the team is established and has
figured out ways of working.
|Re: Improving unit/automated test coverage||Reini Urban||6/5/14 7:04 AM|
On 06/04/2014 04:58 PM, Mike Bland wrote:Mike,
The first which would spring to my mind are valgrind memcheck test
targets and -fsanitize=* rules.
With the current testsuite it is almost impossible to run it via
valgrind and there are no sane sanitize rules yet: (address, undefined
for -O3, thread, memory)
There's only a static debug-ben-debug-64-clang config.
See for contrast polarssl which include those tests and configs.
Not speaking about coverage yet.
Working towards a true Modern Perl.
Slim, functional, unbloated, compile-time optimizable
|Re: Improving unit/automated test coverage||Mike Bland||6/5/14 11:56 AM|
Actually, I was asking for thoughts on whether to create a separate
openssl-testing mailing list, which I'm leaning towards at the moment,
as I plan to get very chatty with the volunteers helping with unit
That said, I've limited experience with valgrind. Are you volunteering
to help add some of this valgrind support?