slow tests

[Noa Resare]

So, I had a look at the newly landed code, and I must say that it feels a bit extreme to actually do exhaustive testing of all possible fragmentation variants. 

While code could certainly be designed to exhibit parsing bugs with fragmentation that doesn't get triggered by just testing for the cases "fragment into single byte buffers" and "fragment into all possible combinations of two buffers", I think that a few minutes of 4 cores crunching all the permutations seems limiting.

I propose we move to a much simpler testing strategy for the common case (where you do some other development and don't modify the logic of the framing parser) and save the exhaustive search for when we actually do such changes.

There is a branch: https://github.com/noaresare/netty-zmtp/tree/framentation_test_speedup

that contains such a change, and the ZMTPMessageParserTest runs in .163s on my laptop with that change.

This is, of course, a tradeoff, and one could easily envision doing more fragmentation permutations without going for the exhaustive list. 

What do you think?

/n

--

Spotify Free Software coordinator, strategist and possibly even evangelist. I/O Tribe.

[Daniel Norberg]

I want it to be difficult to miss running the full fragmentation test. We discovered bugs in the parser using it, so I believe it's worth the pain. I've optimized the test to be some 3x faster, which lessens the pain, albeit not completely eliminating it.

The fragmentation test can be disabled during development by e.g. commenting it out, adding an @Ignore annotation or specifying ^(?!.*ParserTest$).*$ as a test class filter in intellij.

--
dano

--
You received this message because you are subscribed to the Google Groups "netty-zmtp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty-zmtp+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Noa Resare]

I see the value of doing an exhaustive test of all fragmentation permutations. However, in practice I would expect a full search for all permutation actually resulting in worse overall test coverage as very slow test speed per input would discourage testing a lot of different exotic test combinations.

The number of tests executed is doubled in 0866ee29bdd614103fbb4ab701d3062263dd61a4 (Test both ZMTP/1.0 and ZMTP/2.0 parser) and it takes now takes 4 minutes and 47 seconds on my pretty kickass quad core i7.

Please note that this is not a strongly held opinion. It's you call, and I can always maintain some local changes to have fewer strategically chosen fragmentation permutations. I wouldn't want to skip all the parser tests however.

/n