Test coverage challenge

[Elin Waring]

http://developer.joomla.org/coverage/

Recently we've gone over 41% on test coverage of lines of code.  There are a bit more than 28,000 lines of code  which means that 2800 more covered lines would get us to 50% which would be amazing considering where we were last July. Not that we are so shallow as to focus on mindless statistics (and they are kind of mindless) ... but   if  28 people each would cover 100 lines we would be there. There is definitely some low hanging fruit out there in the form of tests that  are not so hard to write and classes that have just a few missing pieces especially in some of the older classes. The older classes are where we should be worried about backward compatibility issues as they get updated so it would be really great to get some done.

So that's the challenge, pitch in an help get us to 50%.

Since I've been working on learning how to write better tests I was wondering if there would be any interest in an irc test camp where people could work on tests together one weekend soon.

Elin

[Emerson da Rocha Luiz]

Elin,

I guess that one good way to do it do what Mark Dexter do on JBS. He do webnars and put on Youtube [ https://groups.google.com/forum/#!msg/joomlabugsquad/h3scl1UyXUo/7WkmGIqomy4J ]. But for this case, since this task depends much more on code, better or explain line by line of one already good standard test, or even start from scratch a new now to simple simple test and show a list of other complex tests that have a good standard to follow. So, ok that can be done like webnar, but can also be better just do it as a offline tutorial

Even if some IDEs help a bit on how to create draft of tests, can be a good idea avoid these features if will not show how to install and use on at least 2 IDEs, like Eclipse and Netbeans. But this is a case of a diferent tutorial or just a few links of how to install.

And, if someone will do and make available on Youtube, would be really good with some help of others, and who does not necessarily make the video, made English subtitles. This helps those who do not understand very well spoken English, apparently youtube even allows you to translate the subtitles. Not that this is the main target, but it helps who is deaf, is without earphone or (do not laugh) who is listening to music.

emerson

--

Emerson Rocha Luiz
+55 51 9881-9146 | Skype: fititnt | GTalk: fititnt | Twitter: @fititnt | http://www.fititnt.org

Membro do JUGRS

[garyamort]

Well, I hate IRC, so I'll post here instead.   I'm working on patches for countModules - so I switched gears for the moment.   Here is the process I am using:

gmJoomla\joomla-platform - my github clone of the joomla-platform

my branches: 

-->evalFix: my initial pass on coding a fix for this untested method

-->countModulesUnitTests: a new clean branch for just adding the unit tests

I need to rewrite the evalFix patch, but have put that on hold and am working on the unitTests as a seperate patch/pull request.   Is this the correct process[submit unit tests seperately, then fixes] - or should they be combined?

My pull request:

https://github.com/joomla/joomla-platform/pull/1303 

I've added tests, so this code now has unit test coverage.  These tests are not yet complete because there are many other conditions which need testing -  so I changed the "this test has not been implemented" text mark to "this test has not been completed".

Because there are so MANY conditions to test for to provide an adequate level of coverage, I've only included 17 new tests out of approximately 50 tests which are needed.

Some questions I have if anyone knows the answer, otherwise I'll look them up later:

There are times when we need to determine if the result of a test was boolean or integer.   Ie sometimes it returns false, and sometimes it returns 0.  Is the PHPUnittest method isEqualTo sufficient or do I need to use a different method there?

There is one test where there won't be a return value, but instead an exception will be thrown[if your countModules parameter is 'dummy1 / dummy2' and the dummy2 position is empty, then that is a divide by zero exception] - how do you catch exceptions in unitTests for testing?

Lastly, is the correct process so far?  And will partial implementations of unit tests be accepted?  It's rather tedious and tiresome to write 50+ conditions before knowing if your even doing things correctly, so getting the first 17 in would at least provide motivation and guidance for the remaining 30+. :-)

[Elin Waring]

I tend to write a lot of conditions, but that's partly the kind of tests I work on and that match my skill level (lots of form fields and rules so dealing with user input). But don't think you need to check every possible condition, what you need to do is say did the general piece work?  Don't fall into the pit of unit testing php -- you just have to make sure the joomla parts work.  I'd pay more attention to things like what happens if the property doesn't exist?  

I am trying to do TDD in the sense of having the discipline to write the failing tests for bugs, but I'm not always succeeding, partly because sometimes that means writing a test from scratch. Tests with out code --- beautiful. Code without tests not so much.

You can do Expected Exception.

https://github.com/joomla/joomla-platform/blob/staging/tests/suites/unit/joomla/application/JApplicationWebTest.php#L774 

You might want to make the tests each a separate method so that they all run even if one fails. 

Elin 

[Rouven Weßling]

On 22.06.2012, at 19:56, garyamort wrote:

Lastly, is the correct process so far?  And will partial implementations of unit tests be accepted?  It's rather tedious and tiresome to write 50+ conditions before knowing if your even doing things correctly, so getting the first 17 in would at least provide motivation and guidance for the remaining 30+. :-)

Yes, you're following the correct process :) You don't need to write an assertion for every possible situation (it's great if you do thou ;) ) but we're happy to get forward however we can.

Your patch actually revealed a more fundamental problem so it may take a bit longer than usual to get in, but that's not your fault.

Best regards

Rouven

[garyamort]

In this case I have another pull request, https://github.com/joomla/joomla-platform/pull/1296 to eliminate an eval call altogether but since there are no tests at the moment to check for correct functionality, I need to add the tests and make sure their correct, then I can revisit 1296 to see if it passes.

Almost a chicken/egg situation with almost but not quite new functionality added in 1296 and 1307 - and 1303 containing the tests to ensure the functionality works.   [Not sure if it should be called "new functionality" - since it is in line with the current API documentation, it just removes some of the undocumented functionality in countModules.  For example: countModules('pos1 ==1;eval(base64_decode('arbitraryphpcommandsbase64encoded');1== pos2') will currently execute the php code and  countModules('pos1 && pos2') will work even though it is not documented as an acceptable operator.

[Elin Waring]

Thanks to Diana and Gary we're now up to 41.4% which is real progress --- great work!  Thanks!

Still lots of opportunities to help out.   

Elin