Improving unit/automated test coverage

[Mike Bland]

Hey folks,

With Ben Laurie's help, I recently contributed ssl/heartbeat_test.c,
which is a unit test that acts as a regression test against the
Heartbleed bug. I'd like to contribute more to the project in the
coming months in terms of helping grow a robust suite of
unit/integration/automated tests.

It seems that the encryption algorithms themselves are relatively
well-tested; in contrast, Heartbleed was an infrastructure bug. It's
in shoring up the test coverage of the infrastructure bits where I can
be of most direct service, but I'm hoping others may see opportunities
to apply similar techniques to more advanced testing issues.

I'd like to make sure there are at least a handful of contributors who
are willing to work closely with me to establish some new policies
around unit testing and code reviews (e.g. no new non-trivial changes
without tests; smaller, well-tested changes vs. monolithic, untested
changes), in addition to the actual writing of tests. There'd be some
tool setup and documentation work as well. The goal would be to help
everyone learn effective unit testing strategies so that, over time,
test coverage and code quality steadily improves. It will be a
lengthy, imperfect process, but one that I believe will ultimately
make a positive difference in the code base if people are willing to
try it.

My goal would be to help everyone learn to fish, to use the tired
cliché. I currently have very little knowledge of the OpenSSL code
base or community, and I don't have a ton of time to do all the heavy
lifting by myself; nor do I think that being the lone "testing guy"
would be the best use of my time or in the best interests of OpenSSL.
However, I want to contribute however I can to help the community as a
whole address this one particular issue, and to maximize the impact of
my contributions.

Happy to hear people's thoughts on this. If the uptake is positive, I
can help organize the effort to get things moving soon.

Thanks,

Mike
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List opens...@openssl.org
Automated List Manager majo...@openssl.org

[Matt Caswell]

On 2 June 2014 15:38, Mike Bland <mbl...@acm.org> wrote:
>
> My goal would be to help everyone learn to fish, to use the tired
> cliché. I currently have very little knowledge of the OpenSSL code
> base or community, and I don't have a ton of time to do all the heavy
> lifting by myself; nor do I think that being the lone "testing guy"
> would be the best use of my time or in the best interests of OpenSSL.
> However, I want to contribute however I can to help the community as a
> whole address this one particular issue, and to maximize the impact of
> my contributions.
>
> Happy to hear people's thoughts on this. If the uptake is positive, I
> can help organize the effort to get things moving soon.

I think that this is an exciting and important initiative. I would
love to see a much stronger test suite for OpenSSL.

Matt

[Mody, Darshan (Darshan)]

Hi Mike,

I would like to volunteer for the same. I can spare some time on the weekends for it. Please do note that even I am new to openssl and it would be good to get to know more on the code by doing unit test.

Thanks
Darshan

[Mike Bland]

Thanks to a few brave volunteers and the support of the core OpenSSL
team, it looks like we can begin moving on this effort soon. I've
begun to document the current state of things on the wiki:

http://wiki.openssl.org/index.php/Unit_Testing

There's lots to discuss with regard to the "Goals", "Tasks", and
"Discussion Topics" enumerated on the wiki. I'm happy to start a
discussion on this thread, or other openssl-dev threads; at the same
time, I'm wondering if a new openssl-testing group would be more
appropriate. If it's desirable, I could manage it via Google Groups,
rather than create more majordomo work for the core team. Thoughts?

Thanks,

Mike

[Kurt Roeckx]

On Mon, Jun 02, 2014 at 10:38:05AM -0400, Mike Bland wrote:
> It seems that the encryption algorithms themselves are relatively
> well-tested; in contrast, Heartbleed was an infrastructure bug. It's
> in shoring up the test coverage of the infrastructure bits where I can
> be of most direct service, but I'm hoping others may see opportunities
> to apply similar techniques to more advanced testing issues.

As far as I know the test covering SSL now try to set up a server
and client with various options and see that they can connect to
each other. It only seems to be testing the happy path. I would
like to see more tests covering the non-happy path. That of
course also goes for all crypto related things.


Kurt

[Matt Caswell]

On 04/06/14 23:29, Kurt Roeckx wrote:
> On Mon, Jun 02, 2014 at 10:38:05AM -0400, Mike Bland wrote:
>> It seems that the encryption algorithms themselves are relatively
>> well-tested; in contrast, Heartbleed was an infrastructure bug. It's
>> in shoring up the test coverage of the infrastructure bits where I can
>> be of most direct service, but I'm hoping others may see opportunities
>> to apply similar techniques to more advanced testing issues.
>
> As far as I know the test covering SSL now try to set up a server
> and client with various options and see that they can connect to
> each other. It only seems to be testing the happy path. I would
> like to see more tests covering the non-happy path. That of
> course also goes for all crypto related things.

That is definitely where the high value will be obtained. But that's a
hard problem I think to start with. It might be better to start with
something simpler - at least until the team is established and has
figured out ways of working.

Matt

[Reini Urban]

On 06/04/2014 04:58 PM, Mike Bland wrote:
> Thanks to a few brave volunteers and the support of the core OpenSSL
> team, it looks like we can begin moving on this effort soon. I've
> begun to document the current state of things on the wiki:
>
> http://wiki.openssl.org/index.php/Unit_Testing
>
> There's lots to discuss with regard to the "Goals", "Tasks", and
> "Discussion Topics" enumerated on the wiki. I'm happy to start a
> discussion on this thread, or other openssl-dev threads; at the same
> time, I'm wondering if a new openssl-testing group would be more
> appropriate. If it's desirable, I could manage it via Google Groups,
> rather than create more majordomo work for the core team. Thoughts?

Mike,

The first which would spring to my mind are valgrind memcheck test
targets and -fsanitize=* rules.
With the current testsuite it is almost impossible to run it via
valgrind and there are no sane sanitize rules yet: (address, undefined
for -O3, thread, memory)
There's only a static debug-ben-debug-64-clang config.

See for contrast polarssl which include those tests and configs.
Not speaking about coverage yet.

--
Reini

Working towards a true Modern Perl.
Slim, functional, unbloated, compile-time optimizable

[Mike Bland]

Hi Reini,

Actually, I was asking for thoughts on whether to create a separate
openssl-testing mailing list, which I'm leaning towards at the moment,
as I plan to get very chatty with the volunteers helping with unit
testing.

That said, I've limited experience with valgrind. Are you volunteering
to help add some of this valgrind support?

Mike