Need help designing some JUnit tests
Rhino
from it but I'm still struggling with composing tests for some
situations. If anyone can help with any of these situations, I'd love to
hear your suggestions.
--------------------------------
Scenario 1 - Moving Targets
What is the best way to test a method whose output is unpredictable
because that output is a moving target? For example, I have a method that
uses Locale.getAvailableLocales() to display all of the Locales on the
current JVM. It works fine but how do I write a JUnit test, given that an
upgrade to the JVM could introduce additional Locales? I'd have the same
kind of problem if a method was returning a list of area codes or cell
phone providers. There must be some standard way of writing a JUnit test
for that but I'm drawing a blank.
--------------------------------
Scenario 2 - Constructors
Given a hypothetical class named Foo where the constructor is:
public Foo () {
//anything from no code at all to umpteen lines
}
is this adequate as a JUnit test?
public void testFoo() {
Foo myFoo = new Foo();
if (!myFoo instanceof Foo) fail ("Failed to instantiate Foo");
}
If not, what would be a better test?
--------------------------------
Scenario 3 - getInstance()
Given a hypothetical class named Fuzz where the constructors and
getInstance() methods are:
private Fuzz() {
// do something
}
private Fuzz(Locale locale) {
// do something
// initialize instance variable for locale
}
public getInstance() {
return Foo();
}
public getInstance(Locale locale) {
return Foo(locale);
}
Is it necessary to write JUnit tests for the constructors, given that
they are private?
Would the following be adequate as JUnit tests for the getInstance()
methods?
public void testGetInstance() {
Fuzz fuzz = Fuzz.getInstance();
if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz");
}
public void testGetInstancelocale() {
Fuzz fuzz = Fuzz.getInstance(new Locale("FR", "fr"));
if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz"):
}
If not, what tests would be better?
--------------------------------
Lastly - and thanks for bearing with me this far - is it normal practice
to write JUnit tests for all methods that are in parent classes of the
class being tested? For example, while writing tests for one of my
classes, the wizard in Eclipse asked if I wanted to produce test methods
to test methods of Object like equals() and wait(). Should I be writing
tests for those methods as well as for those in my own class? I'm
inclined to assume that the methods in Object, for example, have already
been thoroughly tested and further testing by me seems redundant. Even
parent classes that I myself wrote would presumably have had their own
sets of JUnit Tests. Then again, the Eclipse developers presumably put
that functionality in there for a reason so I'm left unclear about
whether I should be trying to write tests for them too.
--
Rhino
Lew
> I'm getting more and more comfortable with JUnit after a long absence
> from it but I'm still struggling with composing tests for some
> situations. If anyone can help with any of these situations, I'd love to
> hear your suggestions.
There may be several "right" answers to your questions. I will take a stab at
some of them.
> --------------------------------
>
> Scenario 1 - Moving Targets
>
> What is the best way to test a method whose output is unpredictable
> because that output is a moving target? For example, I have a method that
When the target "moves", write new tests.
A unit test should test the behaviors of the methods given the domain of
inputs and range of desired outputs or responses. Thus, given your scenario:
> uses Locale.getAvailableLocales() to display all of the Locales on the
> current JVM. It works fine but how do I write a JUnit test, given that an
> upgrade to the JVM could introduce additional Locales? I'd have the same
Your current JVM doesn't have new Locales, so writing for ones that don't
exist is clearly not possible. However, you can use
<http://java.sun.com/javase/6/docs/api/java/util/Locale.html#getAvailableLocales()>
to return an array of all currently available locales, and for-loop through
them in your unit test:
for ( Locale loc : Locale.getAvailableLocales() )
{
assertTrue( "insane", sanityCheck( loc ));
}
> kind of problem if a method was returning a list of area codes or cell
> phone providers. There must be some standard way of writing a JUnit test
> for that but I'm drawing a blank.
>
> --------------------------------
>
> Scenario 2 - Constructors
>
> Given a hypothetical class named Foo where the constructor is:
>
> public Foo () {
> //anything from no code at all to umpteen lines
> }
>
> is this adequate as a JUnit test?
>
> public void testFoo() {
>
> Foo myFoo = new Foo();
> if (!myFoo instanceof Foo) fail ("Failed to instantiate Foo");
> }
>
> If not, what would be a better test?
No, it's ridiculous. It is literally impossible for 'new Foo()' to return an
instance of something that is not an instance of Foo, so checking for that is
wacky. Checking that the constructor does not throw an exception is what you
want.
> --------------------------------
>
> Scenario 3 - getInstance()
>
> Given a hypothetical class named Fuzz where the constructors and
> getInstance() methods are:
>
> private Fuzz() {
> // do something
> }
>
> private Fuzz(Locale locale) {
> // do something
> // initialize instance variable for locale
> }
>
> public getInstance() {
Where is your return value?
This won't even compile.
> return Foo();
> }
>
> public getInstance(Locale locale) {
>
> return Foo(locale);
> }
Where is your return value?
This won't even compile.
> Is it necessary to write JUnit tests for the constructors, given that
> they are private?
Write a test for the method, after you get it to compile.
Normally you don't unit-test private or package-private methods, as they are
not part of the published contract. Unit tests usually test the public contract.
> Would the following be adequate as JUnit tests for the getInstance()
> methods?
>
> public void testGetInstance() {
>
> Fuzz fuzz = Fuzz.getInstance();
> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz");
> }
If 'getInstance()', for which you did not show a return type, has a return
type of 'Fuzz', then testing that the returned value is an instance of that
type is wacky. The compiler guarantees it, so a run-time test is silly. You
want to test what happens at run time, i.e., that the returned value exists,
is not null, and there was no exception.
Actually, it's also valid to test that there is an exception thrown if initial
conditions demand it. For example, testing a constructor that takes an
argument that must not be null, you might pass a null argument and confirm the
'IllegalArgumentException' from the constructor.
It's just plain foolish to test that the type of the result matches what the
compiler already guarantees is the type of the result. Unit tests are for
run-time behaviors.
> public void testGetInstancelocale() {
>
> Fuzz fuzz = Fuzz.getInstance(new Locale("FR", "fr"));
> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz"):
> }
>
> If not, what tests would be better?
> --------------------------------
>
> Lastly - and thanks for bearing with me this far - is it normal practice
> to write JUnit tests for all methods that are in parent classes of the
> class being tested? For example, while writing tests for one of my
Depends.
> classes, the wizard in Eclipse asked if I wanted to produce test methods
> to test methods of Object like equals() and wait(). Should I be writing
Not sure how you'd write a test for 'wait()'. Writing a test for 'equals()'
makes sense IF the class overrides 'equals()', in which case you'd also test
that 'hashCode()' and 'toString()' match, as should 'compareTo()' if the class
implements 'Comparable<YourType>'. It is quite important that those overrides
be consistent.
> tests for those methods as well as for those in my own class? I'm
> inclined to assume that the methods in Object, for example, have already
> been thoroughly tested and further testing by me seems redundant. Even
Depends. Did you override them?
> parent classes that I myself wrote would presumably have had their own
> sets of JUnit Tests. Then again, the Eclipse developers presumably put
> that functionality in there for a reason so I'm left unclear about
> whether I should be trying to write tests for them too.
The reason is that you routinely override methods like 'equals()' (and
'hashCode()', 'toString()' and 'compareTo()' with it).
--
Lew
Eric Sosman
> [...]
> Scenario 2 - Constructors
>
> Given a hypothetical class named Foo where the constructor is:
>
> public Foo () {
> //anything from no code at all to umpteen lines
> }
>
> is this adequate as a JUnit test?
>
> public void testFoo() {
>
> Foo myFoo = new Foo();
> if (!myFoo instanceof Foo) fail ("Failed to instantiate Foo");
> }
This is inadequate because it won't compile. Insert the
missing parentheses and it's not inadequate, but it's pointless:
if the `new' completes without throwing an exception, `myFoo'
*will* be an instance of Foo -- in fact, it will be an instance
of Foo exactly, and not even of a Foo subclass.
For constructors, what you want to test is that they throw
exceptions when they're supposed to (e.g., InvalidArgumentException),
and that the newly-constructed object satisfies all the invariants
it's supposed to. In your no-argument Foo() constructor exceptions
seem unlikely (possible, though: think HeadlessException), but you
might check `myFoo.getHeight() * myFoo.getWidth() == myFoo.getArea()'
or whatever.
> Scenario 3 - getInstance()
>
> Given a hypothetical class named Fuzz where the constructors and
> getInstance() methods are:
>
> private Fuzz() {
> // do something
> }
>
> private Fuzz(Locale locale) {
> // do something
> // initialize instance variable for locale
> }
>
> public getInstance() {
>
> return Foo();
> }
>
> public getInstance(Locale locale) {
>
> return Foo(locale);
> }
>
> Is it necessary to write JUnit tests for the constructors, given that
> they are private?
No, for two reasons. First, since the constructors are private
you can't call them anyhow. Second, the code won't compile.
> Would the following be adequate as JUnit tests for the getInstance()
> methods?
>
> public void testGetInstance() {
>
> Fuzz fuzz = Fuzz.getInstance();
> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz");
> }
Inadequate because (and this is becoming tiresome; I *wish*
people would stop posting garbage and pretending it's code samples!)
the code won't compile.
You should test that the factory method behaves as advertised.
If it can return null under some circumstances, you should check that
it does so when it's supposed to and does not when it isn't. If it
returns non-null, the thing returned will necessarily be of the type
declared for the factory method -- but not necessarily of that "exact"
type, as it might be a subclass or any arbitrary implementation of an
interface type. That may make a difference in what you test.
> public void testGetInstancelocale() {
>
> Fuzz fuzz = Fuzz.getInstance(new Locale("FR", "fr"));
> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz"):
> }
(Broken) won't (record) compile (broken) won't (record) compile ...
> If not, what tests would be better?
Something that compiles would be better. "Hello, world!" for
example, would be better -- not much of a test, I'll grant, but it
would be better.
> Lastly - and thanks for bearing with me this far - is it normal practice
> to write JUnit tests for all methods that are in parent classes of the
> class being tested? For example, while writing tests for one of my
> classes, the wizard in Eclipse asked if I wanted to produce test methods
> to test methods of Object like equals() and wait(). Should I be writing
> tests for those methods as well as for those in my own class? I'm
> inclined to assume that the methods in Object, for example, have already
> been thoroughly tested and further testing by me seems redundant. Even
> parent classes that I myself wrote would presumably have had their own
> sets of JUnit Tests. Then again, the Eclipse developers presumably put
> that functionality in there for a reason so I'm left unclear about
> whether I should be trying to write tests for them too.
I'd say it's unnecessary to test wait() and notify() and other
final methods of Object. More generally, it's probably unnecessary
to test final methods of any superclass.
But equals() is not final, and if the class being tested has its
own equals() you should test it. (Note that it's extremely rare to
inherit equals() from a superclass unless you're inheriting it all
the way from Object undisturbed.) If you were writing tests for
Integer, you might test `new Integer(42).equals(new Integer("42"))',
for example, and `! new Integer(42).equals(new Integer("-42"))'.
--
Eric Sosman
eso...@ieee-dot-org.invalid
Rhino
Actually, my getLocales() method is really just a convenience method that
massages the results of Locale.getAvailableLocales() itself.
Just to be sure I'm using the term "convenience method" correctly, I'm
referring to a method I write that uses existing Java API methods but
that combines several lines of code into one or two. For example, since I
prefer my Locales list to be in alphabetical order, I've written this:
public Map<String, String> getLocales() {
Locale[] listOfLocales = Locale.getAvailableLocales();
Map<String, String> locales = new TreeMap<String, String>();
for (Locale singleLocale : listOfLocales) {
locales.put(singleLocale.toString(), singleLocale.getDisplayName
(locale));
}
return locales;
}
As such, I don't know how to do a JUnit test on it, specifically how to
generate an expected result that can be compared to my actual result. It
seems self-evident that I have to get my expected result in a different
way than I get the actual result, otherwise, I'm not proving anything.
But the same situation applies to things that aren't based on convenience
methods. A list of telephone area codes or the names of all chiropractors
in Ohio or any of a thousand other results from methods are also "moving
targets". I'm really not sure how to test those. I can certainly execute
the methods and prove that they didn't throw an exception but how can I
verify that they are giving full and complete information?
Or is it the case that such a method CAN'T have its accuracy tested in
this way and no such attempt should be made? Is it enough to prove that
the method executes without throwing an exception?
>> --------------------------------
>>
>> Scenario 2 - Constructors
>>
>> Given a hypothetical class named Foo where the constructor is:
>>
>> public Foo () {
>> //anything from no code at all to umpteen lines
>> }
>>
>> is this adequate as a JUnit test?
>>
>> public void testFoo() {
>>
>> Foo myFoo = new Foo();
>> if (!myFoo instanceof Foo) fail ("Failed to instantiate Foo");
>> }
>>
>> If not, what would be a better test?
>
> No, it's ridiculous. It is literally impossible for 'new Foo()' to
> return an instance of something that is not an instance of Foo, so
> checking for that is wacky. Checking that the constructor does not
> throw an exception is what you want.
>
Ah, now it starts to make sense....
>
>> --------------------------------
>>
>> Scenario 3 - getInstance()
>>
>> Given a hypothetical class named Fuzz where the constructors and
>> getInstance() methods are:
>>
>> private Fuzz() {
>> // do something
>> }
>>
>> private Fuzz(Locale locale) {
>> // do something
>> // initialize instance variable for locale
>> }
>>
>> public getInstance() {
>
> Where is your return value?
>
> This won't even compile.
>
Sorry, I just hacked that together to save a minute. I probably should
have copied in a compiled example....
>> return Foo();
>> }
>>
>> public getInstance(Locale locale) {
>>
>> return Foo(locale);
>> }
>
> Where is your return value?
>
> This won't even compile.
>
>> Is it necessary to write JUnit tests for the constructors, given that
>> they are private?
>
> Write a test for the method, after you get it to compile.
>
> Normally you don't unit-test private or package-private methods, as
> they are not part of the published contract. Unit tests usually test
> the public contract.
>
Excellent, that makes sense to me.
>> Would the following be adequate as JUnit tests for the getInstance()
>> methods?
>>
>> public void testGetInstance() {
>>
>> Fuzz fuzz = Fuzz.getInstance();
>> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz");
>> }
>
> If 'getInstance()', for which you did not show a return type, has a
> return type of 'Fuzz', then testing that the returned value is an
> instance of that type is wacky. The compiler guarantees it, so a
> run-time test is silly. You want to test what happens at run time,
> i.e., that the returned value exists, is not null, and there was no
> exception.
>
> Actually, it's also valid to test that there is an exception thrown if
> initial conditions demand it. For example, testing a constructor that
> takes an argument that must not be null, you might pass a null
> argument and confirm the 'IllegalArgumentException' from the
> constructor.
>
That makes perfectly good sense to me. So, if the constructor doesn't
throw any exceptions and is public, you say that I should test that "the
returned values exists and is not null". What's my best way of doing
that? Am I right in assuming that a simple
if (Foo != null)
will cover both of those?
> It's just plain foolish to test that the type of the result matches
> what the compiler already guarantees is the type of the result. Unit
> tests are for run-time behaviors.
>
Fair enough. I couldn't quite see the point of the test either but
thought I'd check it out with people that knew more than me.
>> public void testGetInstancelocale() {
>>
>> Fuzz fuzz = Fuzz.getInstance(new Locale("FR", "fr"));
>> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz"):
>> }
>>
>> If not, what tests would be better?
>> --------------------------------
>>
>> Lastly - and thanks for bearing with me this far - is it normal
>> practice to write JUnit tests for all methods that are in parent
>> classes of the class being tested? For example, while writing tests
>> for one of my
>
> Depends.
>
>> classes, the wizard in Eclipse asked if I wanted to produce test
>> methods to test methods of Object like equals() and wait(). Should I
>> be writing
>
> Not sure how you'd write a test for 'wait()'. Writing a test for
> 'equals()' makes sense IF the class overrides 'equals()', in which
> case you'd also test that 'hashCode()' and 'toString()' match, as
> should 'compareTo()' if the class implements 'Comparable<YourType>'.
> It is quite important that those overrides be consistent.
>
>> tests for those methods as well as for those in my own class? I'm
>> inclined to assume that the methods in Object, for example, have
>> already been thoroughly tested and further testing by me seems
>> redundant. Even
>
> Depends. Did you override them?
>
Nope.
>> parent classes that I myself wrote would presumably have had their
>> own sets of JUnit Tests. Then again, the Eclipse developers
>> presumably put that functionality in there for a reason so I'm left
>> unclear about whether I should be trying to write tests for them too.
>
> The reason is that you routinely override methods like 'equals()' (and
> 'hashCode()', 'toString()' and 'compareTo()' with it).
>
Okay, that makes sense: if I override a method inherited from the parent,
I should test the overridden method to make sure I didn't mess something
up.
--
Rhino
Rhino
@news.eternal-september.org:
> On 5/20/2010 1:48 PM, Rhino wrote:
>> [...]
>> Scenario 2 - Constructors
>>
>> Given a hypothetical class named Foo where the constructor is:
>>
>> public Foo () {
>> //anything from no code at all to umpteen lines
>> }
>>
>> is this adequate as a JUnit test?
>>
>> public void testFoo() {
>>
>> Foo myFoo = new Foo();
>> if (!myFoo instanceof Foo) fail ("Failed to instantiate Foo");
>> }
>
> This is inadequate because it won't compile.
Sorry!! I just dashed that off quickly from memory and I obviously missed
a few bits.... It was really just meant to be illustrate a standard
constructor but perhaps I should have lifted one from an actual class (or
omitted the class code altogether)....
> Insert the
> missing parentheses and it's not inadequate, but it's pointless:
> if the `new' completes without throwing an exception, `myFoo'
> *will* be an instance of Foo -- in fact, it will be an instance
> of Foo exactly, and not even of a Foo subclass.
>
> For constructors, what you want to test is that they throw
> exceptions when they're supposed to (e.g., InvalidArgumentException),
> and that the newly-constructed object satisfies all the invariants
> it's supposed to. In your no-argument Foo() constructor exceptions
> seem unlikely (possible, though: think HeadlessException), but you
> might check `myFoo.getHeight() * myFoo.getWidth() == myFoo.getArea()'
> or whatever.
>
Could you expand on this last paragraph a bit. The part about verifying
that exceptions get thrown at the appropriate time are fine but I'm not
clear on what you mean by the invariants that it is supposed to satisfy.
The last sentence is almost not quite clear. Are you saying to find some
aspect of the work that is done in the constructor and verify that it
took place, so that if it is drawing a GUI component, that the component
exists and has dimensions greater that 0 x 0? What if the constructor
does very very little - maybe just a super() - or even nothing at all? In
those cases, is it sufficient to just do
if (Foo == null) fail("Constructor failed to instantiate the class");
>> Scenario 3 - getInstance()
>>
>> Given a hypothetical class named Fuzz where the constructors and
>> getInstance() methods are:
>>
>> private Fuzz() {
>> // do something
>> }
>>
>> private Fuzz(Locale locale) {
>> // do something
>> // initialize instance variable for locale
>> }
>>
>> public getInstance() {
>>
>> return Foo();
>> }
>>
>> public getInstance(Locale locale) {
>>
>> return Foo(locale);
>> }
>>
>> Is it necessary to write JUnit tests for the constructors, given that
>> they are private?
>
> No, for two reasons. First, since the constructors are private
> you can't call them anyhow. Second, the code won't compile.
>
>> Would the following be adequate as JUnit tests for the getInstance()
>> methods?
>>
>> public void testGetInstance() {
>>
>> Fuzz fuzz = Fuzz.getInstance();
>> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz");
>> }
>
> Inadequate because (and this is becoming tiresome; I *wish*
> people would stop posting garbage and pretending it's code samples!)
> the code won't compile.
>
Sorry!!
> You should test that the factory method behaves as advertised.
> If it can return null under some circumstances, you should check that
> it does so when it's supposed to and does not when it isn't. If it
> returns non-null, the thing returned will necessarily be of the type
> declared for the factory method -- but not necessarily of that "exact"
> type, as it might be a subclass or any arbitrary implementation of an
> interface type. That may make a difference in what you test.
>
Can you elaborate on this a bit? Can you show me a simple example of a
constructor returning a subclass or implementation of an interface?
>> public void testGetInstancelocale() {
>>
>> Fuzz fuzz = Fuzz.getInstance(new Locale("FR", "fr"));
>> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz"):
>> }
>
> (Broken) won't (record) compile (broken) won't (record) compile
Again, sorry....
So, in a nutshell, only test the methods of the parent classes if I
overrode them; otherwise, don't worry about them. That makes sense to me!
Thanks!!
--
Rhino
Eric Sosman
> Eric Sosman<eso...@ieee-dot-org.invalid> wrote:
>> [...]
>> For constructors, what you want to test is that they throw
>> exceptions when they're supposed to (e.g., InvalidArgumentException),
>> and that the newly-constructed object satisfies all the invariants
>> it's supposed to. In your no-argument Foo() constructor exceptions
>> seem unlikely (possible, though: think HeadlessException), but you
>> might check `myFoo.getHeight() * myFoo.getWidth() == myFoo.getArea()'
>> or whatever.
>>
> Could you expand on this last paragraph a bit. The part about verifying
> that exceptions get thrown at the appropriate time are fine but I'm not
> clear on what you mean by the invariants that it is supposed to satisfy.
> The last sentence is almost not quite clear. Are you saying to find some
> aspect of the work that is done in the constructor and verify that it
> took place, so that if it is drawing a GUI component, that the component
> exists and has dimensions greater that 0 x 0? What if the constructor
> does very very little - maybe just a super() - or even nothing at all? In
> those cases, is it sufficient to just do
Read the Javadoc for your Foo class, and consider what
properties a newly-constructed Foo instance should satisfy.
Test that they are satisfied -- that is, test that a newly-
constructed Foo complies with its "contract."
> if (Foo == null) fail("Constructor failed to instantiate the class");
Rhino, if you keep on spewing this sort of codecrap I'm going
to shove that horn of yours firmly up the orifice that spews.
>> You should test that the factory method behaves as advertised.
>> If it can return null under some circumstances, you should check that
>> it does so when it's supposed to and does not when it isn't. If it
>> returns non-null, the thing returned will necessarily be of the type
>> declared for the factory method -- but not necessarily of that "exact"
>> type, as it might be a subclass or any arbitrary implementation of an
>> interface type. That may make a difference in what you test.
>>
> Can you elaborate on this a bit? Can you show me a simple example of a
> constructor returning a subclass or implementation of an interface?
No, because a constructor cannot do such a thing. But you seemed
to be talking about a factory method (although it's hard to be sure
from reading your codecrap), and a factory method -- any method, in
fact -- can return anything compatible with its declared type. Wasn't
it you who had the problem with SpinnerNumberModel recently, where the
getNumber() method sometimes returned a Short, sometimes a Long,
sometimes something else?
>> I'd say it's unnecessary to test wait() and notify() and other
>> final methods of Object. More generally, it's probably unnecessary
>> to test final methods of any superclass.
>>
>> But equals() is not final, and if the class being tested has its
>> own equals() you should test it. (Note that it's extremely rare to
>> inherit equals() from a superclass unless you're inheriting it all
>> the way from Object undisturbed.) If you were writing tests for
>> Integer, you might test `new Integer(42).equals(new Integer("42"))',
>> for example, and `! new Integer(42).equals(new Integer("-42"))'.
>
> So, in a nutshell, only test the methods of the parent classes if I
> overrode them; otherwise, don't worry about them. That makes sense to me!
There's a subtle point there, a conflict between "black box"
and "clear box" testing. The only way you can *know* that a subclass
inherits a non-final method rather than overriding it is to peek
into the subclass' implementation (either by looking at the source
or by using reflection). But what if somebody comes along next week
and decides to override a method you decided not to test, on the
grounds that it was not overridden?
One thing you might do is run some of Super's unit tests on Sub
instances. Another might be to include a "sanity check" test in your
Sub, something that reflects on Sub and verifies that the methods
you've chosen not to test are in fact inherited.
Finally, you've got to realize that unit testing, important as it
is, is not the be-all and end-all of verifying correctness.
--
Eric Sosman
eso...@ieee-dot-org.invalid
Jim Janney
> I'm getting more and more comfortable with JUnit after a long absence
> from it but I'm still struggling with composing tests for some
> situations. If anyone can help with any of these situations, I'd love to
> hear your suggestions.
>
> --------------------------------
>
> Scenario 1 - Moving Targets
>
> What is the best way to test a method whose output is unpredictable
> because that output is a moving target? For example, I have a method that
> uses Locale.getAvailableLocales() to display all of the Locales on the
> current JVM. It works fine but how do I write a JUnit test, given that an
> upgrade to the JVM could introduce additional Locales? I'd have the same
> kind of problem if a method was returning a list of area codes or cell
> phone providers. There must be some standard way of writing a JUnit test
> for that but I'm drawing a blank.
Strictly speaking, unit testing means testing small bits of code in
isolation -- you break a program down into the smallest pieces that
can be tested independently (in Java this generally means a class) and
verify that each one satisfies its public contract. The public
contract is whatever it is that says how the class is supposed to
behave. In some languages (Eiffel and, um, uh, ...) this can be
formally specified in the language itself but more usually it's some
written description or just an idea in the programmer's mind. In Java
it may be in the Javadoc (the JRE classes are pretty good about this).
For example, the Javadoc for Locale.getAvailableLocales() specifies
that the result must contain Locale.US, so you could sensibly test for
this in a unit test. If your application depends on the presence of
other locales you may want to test for those also. Once you move past
this you're no longer doing unit testing but some sort of integration
testing. This is also valuable but the rules are different so it's
useful to remember which kind of testing you're doing.
If you have a class that finds, say, all the zip codes in a given
city, and it gives you a wrong answer, where is the problem likely to
be? In the class or in some database that it's searching? You can
verify that the class searches the data correctly but not that the
data is correct. Or perhaps you can, but that really belongs in some
other test.
--
Jim Janney
Lew
> Actually, my getLocales() method is really just a convenience method that
> massages the results of Locale.getAvailableLocales() itself.
>
> Just to be sure I'm using the term "convenience method" correctly, I'm
> referring to a method I write that uses existing Java API methods but
> that combines several lines of code into one or two. For example, since I
Yep.
> prefer my Locales list to be in alphabetical order, I've written this:
>
> public Map<String, String> getLocales() {
>
> Locale[] listOfLocales = Locale.getAvailableLocales();
>
> Map<String, String> locales = new TreeMap<String, String>();
> for (Locale singleLocale : listOfLocales) {
> locales.put(singleLocale.toString(), singleLocale.getDisplayName
> (locale));
Umm, what is 'locale' in this line? I mean, it's obvious that it's a
'Locale', but what is it?
> }
>
> return locales;
> }
Your 'listOfLocales' variable is, perhaps, not necessary.
That idiom also works if you want to retrieve the Locales themselves based on
name:
Map <String, Locale> locales = new TreeMap <String, Locale> ();
for( Locale loc : Locale.getAvailableLocales() )
{
locales.put( loc.getDisplayName(locale), loc );
}
or something like.
> As such, I don't know how to do a JUnit test on it, specifically how to
> generate an expected result that can be compared to my actual result. It
> seems self-evident that I have to get my expected result in a different
> way than I get the actual result, otherwise, I'm not proving anything.
Unit tests cannot do everything that the class under test does, otherwise the
unit test class would be the class under test. What unit tests do is test the
"happy path" and various corner cases to provide a high level of certainty
that the tested class will correctly handle the infinite variety of stuff
thrown at it in production. In other words, a unit test is really a sanity
check that takes care of the most likely issues. If a unit test could prevent
all possible errors, we'd never need logging.
For your case, you might test that the 'Map' has the same number of entries as
'getLocales()' has elements and that it correctly returns the right values for
some representative keys.
...
> Or is it the case that such a method CAN'T have its accuracy tested in
> this way and no such attempt should be made? Is it enough to prove that
> the method executes without throwing an exception?
You should go farther than that.
>>> --------------------------------
>>>
>>> Scenario 3 - getInstance()
>>>
>>> Given a hypothetical class named Fuzz where the constructors and
>>> getInstance() methods are:
>>>
>>> private Fuzz() {
>>> // do something
>>> }
>>>
>>> private Fuzz(Locale locale) {
>>> // do something
>>> // initialize instance variable for locale
>>> }
>>>
>>> public getInstance() {
Lew wrote:
>> Where is your return value?
>>
>> This won't even compile.
>>
Rhino wrote:
> Sorry, I just hacked that together to save a minute. I probably should
> have copied in a compiled example....
Oopsie. (Giggle)
>>> Would the following be adequate as JUnit tests for the getInstance()
>>> methods?
>>>
>>> public void testGetInstance() {
>>>
>>> Fuzz fuzz = Fuzz.getInstance();
>>> if (!fuzz instanceof Fuzz) fail("Failed to instantiate Fuzz");
>>> }
> ...
> So, if the constructor doesn't
> throw any exceptions and is public, you say that I should test that "the
> returned values exists and is not null".
Non-nullity of the return value should be handled by an 'assert' in the
factory method, and therefore not necessary to test in the unit test.
Existence is already guaranteed by the successful return of the factory method.
> What's my best way of doing that? Am I right in assuming that a simple
>
> if (Foo != null)
Ummm, 'Foo' is a class, right? It better be, and therefore that line will not
compile.
> will cover both of those?
How about 'assertNotNull( fuzz );'?
--
Lew
Lew
>> if (Foo == null) fail("Constructor failed to instantiate the class");
Eric Sosman wrote:
> Rhino, if you keep on spewing this sort of codecrap I'm going
> to shove that horn of yours firmly up the orifice that spews.
Rhino, the source of Eric's irritation is that despite apologizing profusely
multiple times for posting uncompilable code, you did it again anyway.
Apologies don't help much if you don't correct the behavior.
Also, never omit the curly braces in the statement bodies of 'if', 'while',
'for' and 'do...while' statements.
You'd best adhere to the discipline of providing only SSCCEs for a while
<http://sscce.org/>
until you gain more familiarity with the syntax rules for Java.
You won't know that it's an SSCCE unless you actually at least *compile* the
code you post, and really, you should run it first, too.
Doing that will help you. Failing to do that will cause the smartest and most
helpful respondents, such as Eric, not to.
--
Lew
Patricia Shanahan
...
> Actually, my getLocales() method is really just a convenience method that
> massages the results of Locale.getAvailableLocales() itself.
>
> Just to be sure I'm using the term "convenience method" correctly, I'm
> referring to a method I write that uses existing Java API methods but
> that combines several lines of code into one or two. For example, since I
> prefer my Locales list to be in alphabetical order, I've written this:
>
> public Map<String, String> getLocales() {
>
> Locale[] listOfLocales = Locale.getAvailableLocales();
>
> Map<String, String> locales = new TreeMap<String, String>();
> for (Locale singleLocale : listOfLocales) {
> locales.put(singleLocale.toString(), singleLocale.getDisplayName
> (locale));
> }
>
> return locales;
> }
>
> As such, I don't know how to do a JUnit test on it, specifically how to
> generate an expected result that can be compared to my actual result. It
> seems self-evident that I have to get my expected result in a different
> way than I get the actual result, otherwise, I'm not proving anything.
You seem to be assuming that a JUnit test requires an expected result.
Don't forget the assertTrue method, which lets you test arbitrary
conditions.
Patricia
Rhino
> On 5/20/2010 4:47 PM, Rhino wrote:
>> Eric Sosman<eso...@ieee-dot-org.invalid> wrote:
>>> [...]
>>> For constructors, what you want to test is that they throw
>>> exceptions when they're supposed to (e.g.,
>>> InvalidArgumentException), and that the newly-constructed object
>>> satisfies all the invariants it's supposed to. In your no-argument
>>> Foo() constructor exceptions seem unlikely (possible, though: think
>>> HeadlessException), but you might check `myFoo.getHeight() *
>>> myFoo.getWidth() == myFoo.getArea()' or whatever.
>>>
>> Could you expand on this last paragraph a bit. The part about
>> verifying that exceptions get thrown at the appropriate time are fine
>> but I'm not clear on what you mean by the invariants that it is
>> supposed to satisfy. The last sentence is almost not quite clear. Are
>> you saying to find some aspect of the work that is done in the
>> constructor and verify that it took place, so that if it is drawing a
>> GUI component, that the component exists and has dimensions greater
>> that 0 x 0? What if the constructor does very very little - maybe
>> just a super() - or even nothing at all? In those cases, is it
>> sufficient to just do
>
> Read the Javadoc for your Foo class, and consider what
> properties a newly-constructed Foo instance should satisfy.
> Test that they are satisfied -- that is, test that a newly-
> constructed Foo complies with its "contract."
>
Okay, you've given me a broad principle and that is all well and good but
I do a lot better with at least one concrete example. It doesn't have to
be _my_ example either, just something that resembles something I'm
doing.
>> if (Foo == null) fail("Constructor failed to instantiate the
>> class");
>
> Rhino, if you keep on spewing this sort of codecrap I'm going
> to shove that horn of yours firmly up the orifice that spews.
>
I'm really not sure what you mean by "codecrap" here. I assume you're
saying that it won't compile but, in fact, this is a line from an actual
test case with the class name changed to Foo. Here is the full unit test,
WITHOUT the name change, copied and pasted directly from the test case
and I assure you that this does compile:
public void testGetInstance() {
StringUtils stringUtils = StringUtils.getInstance();
if (stringUtils == null) fail("testGetInstance() failed");
}
>>> You should test that the factory method behaves as advertised.
>>> If it can return null under some circumstances, you should check
>>> that it does so when it's supposed to and does not when it isn't.
>>> If it returns non-null, the thing returned will necessarily be of
>>> the type declared for the factory method -- but not necessarily of
>>> that "exact" type, as it might be a subclass or any arbitrary
>>> implementation of an interface type. That may make a difference in
>>> what you test.
>>>
>> Can you elaborate on this a bit? Can you show me a simple example of
>> a constructor returning a subclass or implementation of an interface?
>
> No, because a constructor cannot do such a thing. But you seemed
> to be talking about a factory method (although it's hard to be sure
> from reading your codecrap),
Yes, I do in fact mean a factory method. And here are my constructors and
getInstance() methods, copied and pasted directly from the source:
private StringUtils() {
locale = Locale.getDefault();
localizationUtils = LocalizationUtils.getInstance(locale);
locMsg = localizationUtils.getResources(locale, MSG_PREFIX);
msgFmt.setLocale(locale);
}
private StringUtils(Locale myLocale) {
locale = myLocale;
localizationUtils = LocalizationUtils.getInstance(locale);
locMsg = localizationUtils.getResources(locale, MSG_PREFIX);
msgFmt.setLocale(locale);
}
public static StringUtils getInstance() {
return new StringUtils();
}
public static StringUtils getInstance(Locale myLocale) {
return new StringUtils(myLocale);
}
> and a factory method -- any method, in
> fact -- can return anything compatible with its declared type. Wasn't
> it you who had the problem with SpinnerNumberModel recently, where the
> getNumber() method sometimes returned a Short, sometimes a Long,
> sometimes something else?
>
Yes, that's right. Oh, so _that's_ what you meant in your previous reply!
Again, when someone answers a question only with a generality, it's never
as clear for me as when I also hear a concrete "for instance'. I'm really
not being obtuse although I imagine it must seem like it....
>>> I'd say it's unnecessary to test wait() and notify() and other
>>> final methods of Object. More generally, it's probably unnecessary
>>> to test final methods of any superclass.
>>>
>>> But equals() is not final, and if the class being tested has
>>> its
>>> own equals() you should test it. (Note that it's extremely rare to
>>> inherit equals() from a superclass unless you're inheriting it all
>>> the way from Object undisturbed.) If you were writing tests for
>>> Integer, you might test `new Integer(42).equals(new Integer("42"))',
>>> for example, and `! new Integer(42).equals(new Integer("-42"))'.
>>
>> So, in a nutshell, only test the methods of the parent classes if I
>> overrode them; otherwise, don't worry about them. That makes sense to
>> me!
>
> There's a subtle point there, a conflict between "black box"
> and "clear box" testing. The only way you can *know* that a subclass
> inherits a non-final method rather than overriding it is to peek
> into the subclass' implementation (either by looking at the source
> or by using reflection). But what if somebody comes along next week
> and decides to override a method you decided not to test, on the
> grounds that it was not overridden?
>
My theory is rather weak so I'm not really up on the meanings of "black
box" and "clear box" testing, let alone the subtle differences between
them. Also, I'm not writing the code for anyone but myself right now,
except that I _would_ like to get one project's code looking as
professional as possible so that I could present it to a prospective
client or employer as a portfolio of what I can do. (And then imitate
that in future projects as well as gradually retrofit other existing
projects with what I've learned). With that in mind, would a reasonable
employer/client likely find it acceptable that I just tested the methods
I wrote and overrode myself in my classes or are they going to see me as
the biggest idiot since the development of COBOL if I fail to observe
these subtleties?
> One thing you might do is run some of Super's unit tests on Sub
> instances. Another might be to include a "sanity check" test in your
> Sub, something that reflects on Sub and verifies that the methods
> you've chosen not to test are in fact inherited.
>
> Finally, you've got to realize that unit testing, important as it
> is, is not the be-all and end-all of verifying correctness.
>
I have no problem with that at all. If there are certain things I don't
need to cover in unit testing, that's perfectly fine. If you or anyone
else reading this could point me a good summary of what is and is not a
concern in unit testing, that would be very helpful. Again, I have
effectively NO formal training and what much of the on-the-job stuff I
have was learned many years ago. That means that my memory of the theory
is very incomplete at this point. In short, I don't know what the
prevailing theory is on exactly what should be covered by unit testing,
acceptance testing, regression testing, et. al. I'm not going to worry
about anything beyond unit testing for the moment but if anyone can point
me to a general - and hopefully fairly brief and example-laden -
discussion of the prevailing theories of testing, that would be very
helpful.
--
Rhino
Rhino
> Rhino wrote:
>>> if (Foo == null) fail("Constructor failed to instantiate the
>>> class");
>
> Eric Sosman wrote:
>> Rhino, if you keep on spewing this sort of codecrap I'm going
>> to shove that horn of yours firmly up the orifice that spews.
>
> Rhino, the source of Eric's irritation is that despite apologizing
> profusely multiple times for posting uncompilable code, you did it
> again anyway.
>
> Apologies don't help much if you don't correct the behavior.
>
For what it's worth, I never imagined that anyone was actually going to
compile that code. I just provided that (sort-of) code for illustrative
purposes. I was just trying to show a very trivial constructor and/or
getInstance() so I could understand what testing was needed. Basically,
the (real) code looked so trivial that it wasn't obvious what needed to
be tested.
> Also, never omit the curly braces in the statement bodies of 'if',
> 'while', 'for' and 'do...while' statements.
>
Actually, I rarely do. Again, the "code" that I posted was just to
illustrate my point.
> You'd best adhere to the discipline of providing only SSCCEs for a
> while <http://sscce.org/>
> until you gain more familiarity with the syntax rules for Java.
>
> You won't know that it's an SSCCE unless you actually at least
> *compile* the code you post, and really, you should run it first, too.
>
But I will endeavour to use only real compiled code from here on.
> Doing that will help you. Failing to do that will cause the smartest
> and most helpful respondents, such as Eric, not to.
>
I'm not coming here to piss people off and I'm sorry that I have done so.
I've long been criticized for being verbose so I thought I'd keep the
length of my post down a bit by just summarizing the code rather than
copying it verbatim. That, unfortunately, backfired because it left my
meaning unclear so I will try to use "real" code exclusively from now on.
--
Rhino
Lew
Rhino wrote:
>>> if (Foo == null) fail("Constructor failed to instantiate the
>>> class");
Eric Sosman wrote:
>> Rhino, if you keep on spewing this sort of codecrap I'm going
>> to shove that horn of yours firmly up the orifice that spews.
Rhino wrote:
> I'm really not sure what you mean by "codecrap" here. I assume you're
He's referring to the fact that the code you showed will not compile.
> saying that it won't compile but, in fact, this is a line from an actual
> test case with the class name changed to Foo. Here is the full unit test,
Nope. What you should have shown is the *instance* name changed to 'foo'.
> WITHOUT the name change, copied and pasted directly from the test case
> and I assure you that this does compile:
>
> public void testGetInstance() {
>
> StringUtils stringUtils = StringUtils.getInstance();
> if (stringUtils == null) fail("testGetInstance() failed");
> }
Rhino, Rhino, Rhino.
In the code about which Eric complained, you compared 'Foo', a *class*, to
'null'. That WILL NOT COMPILE!
In the code you assure us does compile you compare an instance to 'null'. Of
course that compiles.
Apples and oranges.
You cannot compare a class to 'null', only an instance. Classes and instances
are different things. What works for one does not necessarily work for the other.
Also, that "actual" code you show neglects the curly braces around the
statement body of the 'if'.
--
Lew
Eric Sosman
> Lew<no...@lewscanon.com> wrote in news:ht4ghr$cbl$1...@news.albasani.net:
>
>> Rhino wrote:
>>>> if (Foo == null) fail("Constructor failed to instantiate the
>>>> class");
>>
>> Eric Sosman wrote:
>>> Rhino, if you keep on spewing this sort of codecrap I'm going
>>> to shove that horn of yours firmly up the orifice that spews.
>>
>> Rhino, the source of Eric's irritation is that despite apologizing
>> profusely multiple times for posting uncompilable code, you did it
>> again anyway.
>>
>> Apologies don't help much if you don't correct the behavior.
>>
> For what it's worth, I never imagined that anyone was actually going to
> compile that code. I just provided that (sort-of) code for illustrative
Nobody was probable to compile the code, not. But a fragment
in full load of the errors is not the "illustrative," it is only
offuscatorio. In order to extract the meaning from a beautiful
insignificant piece of the code, the reading of the person must
make the changes and corrections to arrive to something images that
you could have in mind and are every probability that will supply
some different thing from what has meant and that its ulterior
observations therefore will be not at all useful you.
(In other words: "Illustrate" your thoughts with garbled
utterances, and people may form opinions about your thinking.)
--
Eric Sosman
eso...@ieee-dot-org.invalid
Arved Sandstrom
> Eric Sosman <eso...@ieee-dot-org.invalid> wrote in
> news:ht49d0$i91$1...@news.eternal-september.org:
[ SNIP ]
>> There's a subtle point there, a conflict between "black box"
>> and "clear box" testing. The only way you can *know* that a subclass
>> inherits a non-final method rather than overriding it is to peek
>> into the subclass' implementation (either by looking at the source
>> or by using reflection). But what if somebody comes along next week
>> and decides to override a method you decided not to test, on the
>> grounds that it was not overridden?
>>
> My theory is rather weak so I'm not really up on the meanings of "black
> box" and "clear box" testing, let alone the subtle differences between
> them. Also, I'm not writing the code for anyone but myself right now,
> except that I _would_ like to get one project's code looking as
> professional as possible so that I could present it to a prospective
> client or employer as a portfolio of what I can do. (And then imitate
> that in future projects as well as gradually retrofit other existing
> projects with what I've learned). With that in mind, would a reasonable
> employer/client likely find it acceptable that I just tested the methods
> I wrote and overrode myself in my classes or are they going to see me as
> the biggest idiot since the development of COBOL if I fail to observe
> these subtleties?
No word of a lie, just the fact that you're writing tests already gets
you some serious points with employers or clients. And the huge majority
of people that matter will be perfectly happy with you writing tests for
a class that focus on methods that are lexically in the class. That's
what I do myself.
You perhaps misunderstood Eric - he didn't mean that there is a subtle
difference between black box and white box testing. There isn't - those
two techniques are quite different. He simply called the specific point
he was making subtle.
You don't need to be a testing god - people make FT careers out of
understanding testing. But you should know what black box and white box
testing are, for example. The Wikipedia page on Software Testing is not
a bad start, IMO. Just bear in mind that their terminology and
definitions are debatable on some specifics, but this doesn't detract
from the overall discussion (in particular the first few sentences under
Non-Functional Testing are whacked, try to gloss over them).
>> One thing you might do is run some of Super's unit tests on Sub
>> instances. Another might be to include a "sanity check" test in your
>> Sub, something that reflects on Sub and verifies that the methods
>> you've chosen not to test are in fact inherited.
>
>> Finally, you've got to realize that unit testing, important as it
>> is, is not the be-all and end-all of verifying correctness.
>>
> I have no problem with that at all. If there are certain things I don't
> need to cover in unit testing, that's perfectly fine. If you or anyone
> else reading this could point me a good summary of what is and is not a
> concern in unit testing, that would be very helpful. Again, I have
> effectively NO formal training and what much of the on-the-job stuff I
> have was learned many years ago. That means that my memory of the theory
> is very incomplete at this point. In short, I don't know what the
> prevailing theory is on exactly what should be covered by unit testing,
> acceptance testing, regression testing, et. al. I'm not going to worry
> about anything beyond unit testing for the moment but if anyone can point
> me to a general - and hopefully fairly brief and example-laden -
> discussion of the prevailing theories of testing, that would be very
> helpful.
You'll end up doing a fair bit of reading and playing with code to get a
good handle on testing overall, but it won't take all that long to get a
good grip on the basics.
I will make a few personal recommendations, which shouldn't be taken as
complete. This is based on going on ten years of J2EE work, so depending
on what kind of coding you're doing YMMV...some.
1. Code reviews/inspections and static code analysis with tools _are_ a
form of testing. Keep them in mind.
2. Integration tests are as important as unit tests. In a J2EE web
application these are indispensable - you may have hundreds or thousands
of unit tests all pass but still have integration tests not pass. You'll
hear a lot of people, myself included, refer to a specific class of
integration tests as functional tests - we mean "application functions
from the user perspective" when we use "functional" in that regard.
Examples of integration/functional tests in a J2EE web app range all the
way from ensuring that good things happen in the logic when you click
that "Transmogrify" button all the way to doing a complete pass through
a use case and making sure that your "Issue Fishing License" logic works
in Angler Administrator 2010.
Myself I use Selenium IDE/RC in conjunction with the JUnit framework to
write these kinds of tests.
3. Code coverage - Huge, IMO. How can you know that your unit tests or
integration tests (or even human-tester-driven acceptance tests) are
doing any good unless you know how much of the code is actually being
exercised? Code coverage is very simple to do, and for starters you
can't go wrong investigating Emma or Cobertura. These simply instrument
the Java bytecode, such that when the bytecode is executed (by any
mechanism) coverage counts by line/branch/method/class/package are
written to HTML or XML reports.
4. Carefully consider the issue of test data - test SQL scripts, mock
data, in-memory databases (see http://www.mikebosch.com/?p=8) etc.
5. Your tests are themselves defective. The sad truth is that if the
core source code you just wrote is riddled with defects, then so
probably are your tests. Main take-away here is, be aware that just
because all your unit tests pass, some not insignificant percentage of
those results are wrong.
As a side note, this is where the higher-level layer of integration
tests also helps - it can assist in identifying flawed unit tests.
HTH,
AHS
Rhino
> Rhino wrote:
>> Actually, my getLocales() method is really just a convenience method
>> that massages the results of Locale.getAvailableLocales() itself.
>>
>> Just to be sure I'm using the term "convenience method" correctly,
>> I'm referring to a method I write that uses existing Java API methods
>> but that combines several lines of code into one or two. For example,
>> since I
>
> Yep.
>
Okay, good. At least I know now that I've correctly understood what
_that_ term means. One down and a gazillion to go.... ;-)
>> prefer my Locales list to be in alphabetical order, I've written
>> this:
>>
>> public Map<String, String> getLocales() {
>>
>> Locale[] listOfLocales = Locale.getAvailableLocales();
>>
>> Map<String, String> locales = new TreeMap<String, String>();
>> for (Locale singleLocale : listOfLocales) {
>> locales.put(singleLocale.toString(), singleLocale.getDisplayName
>> (locale));
>
> Umm, what is 'locale' in this line? I mean, it's obvious that it's a
> 'Locale', but what is it?
>
That 'locale' is an instance variable that the caller can set via a
getInstance(locale) method to set the locale for the class. I'm just
trying to make my code "multilingual" so that a user who is operating in
French or German or whatever will get the output in their language.
>> }
>>
>> return locales;
>> }
>
> Your 'listOfLocales' variable is, perhaps, not necessary.
>
> That idiom also works if you want to retrieve the Locales themselves
> based on name:
>
> Map <String, Locale> locales = new TreeMap <String, Locale> ();
>
> for( Locale loc : Locale.getAvailableLocales() )
> {
> locales.put( loc.getDisplayName(locale), loc );
> }
>
> or something like.
>
I like it :-) I had originally tried to build the map using the locale as
the key but ran up against the fact that Locale is not Comparable so I
did it the way you see in this email. But making the DisplayName the key
and the Locale as the value works fine and is concise to boot so I'll do
it your way.
>> As such, I don't know how to do a JUnit test on it, specifically how
>> to generate an expected result that can be compared to my actual
>> result. It seems self-evident that I have to get my expected result
>> in a different way than I get the actual result, otherwise, I'm not
>> proving anything.
>
> Unit tests cannot do everything that the class under test does,
> otherwise the unit test class would be the class under test. What
> unit tests do is test the "happy path" and various corner cases to
> provide a high level of certainty that the tested class will correctly
> handle the infinite variety of stuff thrown at it in production. In
> other words, a unit test is really a sanity check that takes care of
> the most likely issues. If a unit test could prevent all possible
> errors, we'd never need logging.
>
Fair enough.
> For your case, you might test that the 'Map' has the same number of
> entries as 'getLocales()' has elements and that it correctly returns
> the right values for some representative keys.
>
Okay, that's reasonable. Someone else suggested that I verify that the
en_US is there too, since that's part of the contract from
Locale.getAvailableLocales(). That should be enough for unit testing
then, right?
I'm not very familiar with "assert". I'll look into it on my own and add
that code.
>> What's my best way of doing that? Am I right in assuming that a
>> simple
>>
>> if (Foo != null)
>
> Ummm, 'Foo' is a class, right? It better be, and therefore that line
> will not compile.
>
>> will cover both of those?
>
> How about 'assertNotNull( fuzz );'?
>
That would go in the test case (as opposed to the class) and wouldn't be
redundant with the 'assert' you mentioned that should go in the class
itself?
--
Rhino
Rhino
news:WvSdnRf94rW73GvW...@earthlink.com:
I'm not really familiar with assertTrue - I'm just getting back into Java
and JUnit after a gap of a few years and I was never all that fluent with
JUnit even before the gap. I'll look at the JUnit docs and see what I can
learn there about the purpose and best uses of assertTrue....
Thanks for the suggestion, I'm sure it will be helpful once I understand it
better ;-)
--
Rhino
Patricia Shanahan
...
> 5. Your tests are themselves defective. The sad truth is that if the
> core source code you just wrote is riddled with defects, then so
> probably are your tests. Main take-away here is, be aware that just
> because all your unit tests pass, some not insignificant percentage of
> those results are wrong.
Although it is not 100% effective, there is some value in writing, and
running, the test before implementing the feature. Start testing a
method when it is just an empty stub. At that point, a test that passes
is either useless or has a bug in it.
Patricia
Patricia Shanahan
>> You seem to be assuming that a JUnit test requires an expected result.
>> Don't forget the assertTrue method, which lets you test arbitrary
>> conditions.
>>
> I'm not really familiar with assertTrue - I'm just getting back into Java
> and JUnit after a gap of a few years and I was never all that fluent with
> JUnit even before the gap. I'll look at the JUnit docs and see what I can
> learn there about the purpose and best uses of assertTrue....
I'll modify my suggestion from "Don't forget" to "Learn about".
>
> Thanks for the suggestion, I'm sure it will be helpful once I understand it
> better ;-)
Here's a possibly relevant example, a method that tests an
Iterator<String> for non-empty, strictly increasing, no null elements.
/**
* Pass a non-empty String iterator with only non-null elements
* in strictly
* increasing order.
*/
private void testIteratorBehavior(Iterator<String> it) {
assertTrue(it.hasNext());
String oldElement = it.next();
assertNotNull(oldElement);
while (it.hasNext()) {
String newElement = it.next();
assertNotNull(newElement);
assertTrue(newElement.compareTo(oldElement) > 0);
oldElement = newElement;
}
}
Using this sort of technique you can test a structure for conforming to
some known rules without using an expected value.
Patricia
Jim Janney
Speaking as someone who occasionally sits in on job interviews, I'm
delighted when we find a candidate who's written any tests at all.
Most of them haven't. We don't expect people to know more than we do
(it would be great if they did) and we all learned as we went along
and wish we understood it better.
>
>> One thing you might do is run some of Super's unit tests on Sub
>> instances. Another might be to include a "sanity check" test in your
>> Sub, something that reflects on Sub and verifies that the methods
>> you've chosen not to test are in fact inherited.
>>
>
>> Finally, you've got to realize that unit testing, important as it
>> is, is not the be-all and end-all of verifying correctness.
>>
> I have no problem with that at all. If there are certain things I don't
> need to cover in unit testing, that's perfectly fine. If you or anyone
> else reading this could point me a good summary of what is and is not a
> concern in unit testing, that would be very helpful. Again, I have
> effectively NO formal training and what much of the on-the-job stuff I
> have was learned many years ago. That means that my memory of the theory
> is very incomplete at this point. In short, I don't know what the
> prevailing theory is on exactly what should be covered by unit testing,
> acceptance testing, regression testing, et. al. I'm not going to worry
> about anything beyond unit testing for the moment but if anyone can point
> me to a general - and hopefully fairly brief and example-laden -
> discussion of the prevailing theories of testing, that would be very
> helpful.
In the specific case of a utility class, what I look for is clear
Javadoc. I need to be able to decide, without wasting a lot of time,
whether a method does what I need or not, and how it handles corner
cases. For an example of how I like to see it done, go to
http://commons.apache.org/lang/api-release/org/apache/commons/lang/StringUtils.html
and look at, say, the definition for
center(java.lang.String, int)
Notice that once you have a good definition of how the method should
behave, writing a unit test for it is straightforward.
Notice also that testing a utility class is very different from
testing a program or an application framework.
--
Jim Janney
Tom Anderson
> 3. Code coverage - Huge, IMO. How can you know that your unit tests or
> integration tests (or even human-tester-driven acceptance tests) are
> doing any good unless you know how much of the code is actually being
> exercised? Code coverage is very simple to do, and for starters you
> can't go wrong investigating Emma or Cobertura. These simply instrument
> the Java bytecode, such that when the bytecode is executed (by any
> mechanism) coverage counts by line/branch/method/class/package are
> written to HTML or XML reports.
Out of interest, do those play well with other things which modify
bytecode, like JDO/JPA enhancers?
tom
--
09F911029D74E35BD84156C5635688C0 -- AACS Licensing Administrator
Tom Anderson
The true extremos work to this rule - you must always start with a failing
test, then work to make it pass.
In fact, the way we (the programmers where i work) try to work is to write
the test first, before even writing the stub of the method, then treat the
fact that the test doesn't compile as a failure, and work in from there.
Rhino
> Rhino wrote:
>> Patricia Shanahan <pa...@acm.org> wrote in
> ...
>>> You seem to be assuming that a JUnit test requires an expected
>>> result. Don't forget the assertTrue method, which lets you test
>>> arbitrary conditions.
>>>
>> I'm not really familiar with assertTrue - I'm just getting back into
>> Java and JUnit after a gap of a few years and I was never all that
>> fluent with JUnit even before the gap. I'll look at the JUnit docs
>> and see what I can learn there about the purpose and best uses of
>> assertTrue....
>
> I'll modify my suggestion from "Don't forget" to "Learn about".
>
I've already found a basic article on JUnit that talked about the
different sorts of assertXXX methods and looked at the JUnit API to get a
handle on that. It's much clearer now, thanks for the suggestion.
>>
>> Thanks for the suggestion, I'm sure it will be helpful once I
>> understand it better ;-)
>
> Here's a possibly relevant example, a method that tests an
> Iterator<String> for non-empty, strictly increasing, no null elements.
>
> /**
> * Pass a non-empty String iterator with only non-null elements
> * in strictly
> * increasing order.
> */
> private void testIteratorBehavior(Iterator<String> it) {
> assertTrue(it.hasNext());
> String oldElement = it.next();
> assertNotNull(oldElement);
> while (it.hasNext()) {
> String newElement = it.next();
> assertNotNull(newElement);
> assertTrue(newElement.compareTo(oldElement) > 0);
> oldElement = newElement;
> }
> }
>
> Using this sort of technique you can test a structure for conforming
> to some known rules without using an expected value.
>
Thanks you for the interesting suggestion. I hadn't thought of testing to
insure that the sequence was correct or that nulls were excluded and
might have struggled a bit trying to think of ways to ensure that.
Am I right in assuming that it is reasonable to bypass the sequence
testing in the case of my sorted list of Locales, given that I am using a
TreeMap which already guarantees the order? I assume you are just
including these tests to cover situations where I am generating a sorted
list out of thin air (without the benefit of the TreeXXX Collection
classes) so that I can be confident my code worked correctly?
I think I will include a test for nulls in the Set though, just in
case....
--
Rhino
Patricia Shanahan
This principle is also very important for bug fixing. Any bug represents
a defect in the test suite. Fix the test suite first, and verify that
the new tests fail, before any attempt to fix the underlying bug.
Patricia
Rhino
news:jlvJn.4710$z%6.465@edtnps83:
I have to say it is a bit frightening that there are big brownie points
for doing any testing at all: you'd think that would be absolutely
essential to any serious employer or client. I was just hoping to be
doing a reasonable amount of testing to look like I know what I'm doing.
No testing at all didn't seem like an option in any universe I want to
live in :-)
> You perhaps misunderstood Eric - he didn't mean that there is a subtle
> difference between black box and white box testing. There isn't -
> those two techniques are quite different. He simply called the
> specific point he was making subtle.
>
You're right, I did misunderstand him in exactly the way you said.
> You don't need to be a testing god - people make FT careers out of
> understanding testing. But you should know what black box and white
> box testing are, for example. The Wikipedia page on Software Testing
> is not a bad start, IMO. Just bear in mind that their terminology and
> definitions are debatable on some specifics, but this doesn't detract
> from the overall discussion (in particular the first few sentences
> under Non-Functional Testing are whacked, try to gloss over them).
>
Good points. I'll review the Wikipedia definitions and try to remember
the highlights.
I see that you've given some good concrete suggestions for tools to use
in upcoming paragraphs. Are any of these static code analysis tools? If
not, can you suggest some good ones? I'd love to have some feedback on my
code via code reviews too but that's a little difficult to do. It's not
realistic to full (non-trivial) classes in the newsgroup so I just pick
and choose especially concerning problems and post snippets reflecting
them. I'm sure that you folks are all too busy to do detailed
walkthroughs of dozens of classes with hundreds of lines of codes in many
of them ;-) I must see if there is a local Java user group in this area
and see if I can develop a few buddies for doing code walkthroughs....
Agreed! However careful and thorough you think you are, you're bound to
have made a few unreasonable assumptions, been unaware of various gotchas
in the Java API, etc. etc.
> Main take-away here is, be aware that just
> because all your unit tests pass, some not insignificant percentage of
> those results are wrong.
>
Sad but true....
> As a side note, this is where the higher-level layer of integration
> tests also helps - it can assist in identifying flawed unit tests.
>
Agreed. Those other levels of testing - and the other people we involve
in our testing - make sure that everything anyone on the team can think
of gets covered.
Thanks for all the great suggestions, Arved!
I've been holding off on learning about the other testing tools until I
had a bit more time but maybe I really should get at least basic
familiarity with the key ones NOW. Then I can include those test results
in my "portfolio" and show employers/clients that I have a serious
professional attitude, even if my chops are not entirely as fluent as I
would like yet....
--
Rhino
Rhino
Fair enough. I saw an article just the other day that used a buzzword I've
already forgotten which proposed that this was the right way to develop
code: write test cases BEFORE you even start the code and then retest
frequently as the code evolves. I can see some merit in that, even if I've
forgotten the buzzword already ;-)
--
Rhino
Tom Anderson
> Patricia Shanahan <pa...@acm.org> wrote in
> news:NtWdndKRZMMkC2vW...@earthlink.com:
>
>> Arved Sandstrom wrote:
>> ...
>>> 5. Your tests are themselves defective. The sad truth is that if the
>>> core source code you just wrote is riddled with defects, then so
>>> probably are your tests. Main take-away here is, be aware that just
>>> because all your unit tests pass, some not insignificant percentage of
>>> those results are wrong.
>>
>> Although it is not 100% effective, there is some value in writing, and
>> running, the test before implementing the feature. Start testing a
>> method when it is just an empty stub. At that point, a test that passes
>> is either useless or has a bug in it.
>
> Fair enough. I saw an article just the other day that used a buzzword
> I've already forgotten which proposed that this was the right way to
> develop code: write test cases BEFORE you even start the code and then
> retest frequently as the code evolves. I can see some merit in that,
> even if I've forgotten the buzzword already ;-)
Probably 'test-driven development', aka TDD.
And it's no more of a buzzword than 'wearing a seat-belt'. It's just a
bloody good idea!
tom
--
Don't trust the laws of men. Trust the laws of mathematics.
Rhino
> delighted when we find a candidate who's written any tests at all.
> Most of them haven't. We don't expect people to know more than we do
> (it would be great if they did) and we all learned as we went along
> and wish we understood it better.
>
anyone could apply for a job as anything more than a trainee programmer
(one who didn't know anything about programming at all and was expecting
to be trained by the employer) without having some familiarity with
testing and, hopefully being able to demonstrate the familiarity. If
that's the norm, then I feel better about my chances of getting a job
programming in Java already :-)
you, this gives me a good standard to emulate in ALL of my Javadoc
comments!
> Notice that once you have a good definition of how the method should
> behave, writing a unit test for it is straightforward.
>
Yss and no. I don't have a lot of trouble with what to test in routine
methods but, as you can see from the first part of the thread, the proper
testing for some methods, like Constructors and getInstance() methods
isn't quite that obvious to me. But it's becoming a lot more clearer as a
result of this thread :-)
> Notice also that testing a utility class is very different from
> testing a program or an application framework.
>
I hear you. I've got a variety of different projects - servlets, midlets,
GUI-centric programs, as well as more standard Java classes - and I've
got a running question in the back of my mind about how to test some of
them. But I'll leave questions of that kind for another day. I can't
learn everything I don't know in a single day or a single newsgroup
thread ;-)
--
Rhino
Rhino
@news.eternal-september.org:
I get your point now and it is very reasonable. I'll try to do better in
future!
Thanks for your patience and sorry for the confusion.
--
Rhino
Patricia Shanahan
> Am I right in assuming that it is reasonable to bypass the sequence
> testing in the case of my sorted list of Locales, given that I am using a
> TreeMap which already guarantees the order? I assume you are just
> including these tests to cover situations where I am generating a sorted
> list out of thin air (without the benefit of the TreeXXX Collection
> classes) so that I can be confident my code worked correctly?
I would test it.
Suppose at some future time a programmer changes the Map implementation
e.g. from TreeMap to HashMap. That might be done in an attempt to
improve performance, or because of a future decision to standardize on
HashMap unless there is a specific reason to use a different
implementation. The future programmer could be yourself after you have
had time to forget why you used TreeMap.
If you test for alphabetical order, the method's unit test will fail
with HashMap. If not, you could get a clean unit test, but have all the
fun of debugging system level problems.
Patricia
Tom Anderson
> Rhino wrote:
>
>> Am I right in assuming that it is reasonable to bypass the sequence
>> testing in the case of my sorted list of Locales, given that I am using
>> a TreeMap which already guarantees the order? I assume you are just
>> including these tests to cover situations where I am generating a
>> sorted list out of thin air (without the benefit of the TreeXXX
>> Collection classes) so that I can be confident my code worked
>> correctly?
>
> I would test it.
>
> Suppose at some future time a programmer changes the Map implementation
> e.g. from TreeMap to HashMap. That might be done in an attempt to
> improve performance, or because of a future decision to standardize on
> HashMap unless there is a specific reason to use a different
> implementation. The future programmer could be yourself after you have
> had time to forget why you used TreeMap.
Perhaps the method should be defined to return a SortedMap. If the test
says:
SortedMap<String, Locale> locales = LocaleHelper.getLocales(); // can't remember the actual name, sorry
Any attempt to return a non-Sorted Map, including redefining the method to
return a plain Map will be stopped in its tracks by the compiler.
Of course, it doesn't stop someone returning a TreeMap with a custom
comparator that does something bananas. Java's type system has its limits.
tom
--
If goods don't cross borders, troops will. -- Fr
Rhino
>>> throw any exceptions and is public, you say that I should test that
>>> "the returned values exists and is not null".
>>
>> Non-nullity of the return value should be handled by an 'assert' in
>> the factory method, and therefore not necessary to test in the unit
>> test. Existence is already guaranteed by the successful return of the
>> factory method.
>>
> I'm not very familiar with "assert". I'll look into it on my own and
> add that code.
>
supposed to be used in public methods. Since it is my getInstance() method
that invokes the private constructor, the getInstance() is where I'd put
the 'assert' right? After all, my constructor is completely empty.
If I follow the rule about not using 'assert' in public methods, I'm not
sure how to reconcile you advice with the rule in the assertions article
(http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html)....
>>> What's my best way of doing that? Am I right in assuming that a
>>> simple
>>>
>>> if (Foo != null)
>>
>> Ummm, 'Foo' is a class, right? It better be, and therefore that line
>> will not compile.
>>
>>> will cover both of those?
>>
>> How about 'assertNotNull( fuzz );'?
>>
>
> That would go in the test case (as opposed to the class) and wouldn't
> be redundant with the 'assert' you mentioned that should go in the
> class itself?
>
Is the assertNotNull in the test case a reasonable substitute for doing the
assert in the getInstance()?
--
Rhino
Lew
> Followup question. I just read up on assert and it is apparently not
> supposed to be used in public methods. Since it is my getInstance() method
Wrong.
> that invokes the private constructor, the getInstance() is where I'd put
> the 'assert' right? After all, my constructor is completely empty.
> If I follow the rule about not using 'assert' in public methods, I'm not
That's not a rule. You are not reading the advice carefully.
You're not supposed to use 'assert' to validate arguments to a public method.
There are plenty of use cases for 'assert' in a public method.
> sure how to reconcile you advice with the rule in the assertions article
> (http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html)....
Read the article again, more carefully.
The purpose of 'assert' is to prove that invariants hold. It's actually not
so far different from the JUnit 'assertXxx()' methods, where 'assert' proves
within the code itself that algorithmic invariants hold, and JUnit's
assertions prove that test invariants hold in a test class outside the code
itself. Both throw 'Error's when violated.
> Is the assertNotNull in the test case a reasonable substitute for doing the
> assert in the getInstance()?
Not really.
They don't really do the same thing, exactly.
--
Lew
Arne Vajhøj
>> Speaking as someone who occasionally sits in on job interviews, I'm
>> delighted when we find a candidate who's written any tests at all.
>> Most of them haven't. We don't expect people to know more than we do
>> (it would be great if they did) and we all learned as we went along
>> and wish we understood it better.
>>
> As I said to Arved elsewhere in the thread, I find it frightening that
> anyone could apply for a job as anything more than a trainee programmer
> (one who didn't know anything about programming at all and was expecting
> to be trained by the employer) without having some familiarity with
> testing and, hopefully being able to demonstrate the familiarity. If
> that's the norm, then I feel better about my chances of getting a job
> programming in Java already :-)
You would be surprised how many total clueless (wannabee) programmers
there are.
Arne
Arne Vajhøj
> Patricia Shanahan<pa...@acm.org> wrote in
> news:NtWdndKRZMMkC2vW...@earthlink.com:
>> Arved Sandstrom wrote:
>> ...
>>> 5. Your tests are themselves defective. The sad truth is that if the
>>> core source code you just wrote is riddled with defects, then so
>>> probably are your tests. Main take-away here is, be aware that just
>>> because all your unit tests pass, some not insignificant percentage of
>>> those results are wrong.
>>
>> Although it is not 100% effective, there is some value in writing, and
>> running, the test before implementing the feature. Start testing a
>> method when it is just an empty stub. At that point, a test that passes
>> is either useless or has a bug in it.
>
> already forgotten which proposed that this was the right way to develop
> code: write test cases BEFORE you even start the code and then retest
> frequently as the code evolves. I can see some merit in that, even if I've
> forgotten the buzzword already ;-)
TDD
http://www.agiledata.org/essays/tdd.html
Arne
Arne Vajhøj
[lots of stuff that I completely agree with]
> 3. Code coverage - Huge, IMO. How can you know that your unit tests or
> integration tests (or even human-tester-driven acceptance tests) are
> doing any good unless you know how much of the code is actually being
> exercised? Code coverage is very simple to do, and for starters you
> can't go wrong investigating Emma or Cobertura. These simply instrument
> the Java bytecode, such that when the bytecode is executed (by any
> mechanism) coverage counts by line/branch/method/class/package are
> written to HTML or XML reports.
Note that unit tests to achieve 100% code coverage is
not sufficient - one need to test every path through
the code (CCN).
Arne
Tom Anderson
> What is the best way to test a method whose output is unpredictable
> because that output is a moving target? For example, I have a method
> that uses Locale.getAvailableLocales() to display all of the Locales on
> the current JVM. It works fine but how do I write a JUnit test, given
> that an upgrade to the JVM could introduce additional Locales?
I'm a bit late to the party here, and i lost track of the answers, but i'd
think about writing a test where the expectation was driven by the set of
locales available in the JVM:
@Test
public void getLocalesReturnsAllLocales() {
Collection<Locale> locales = RhinoUtils.getLocales();
Locale[] referenceLocales = Locale.getAvailableLocales();
assertEquals(referenceLocales.length, locales.length);
for (Locale locale: referenceLocales) {
assertTrue(locales.contains(locale));
}
}
To be honest, though, if all your method is doing is wrapping
Locale.getAvailableLocales, i wouldn't write it in the first place. The
code you don't write is the easiest code to test!
> Lastly - and thanks for bearing with me this far - is it normal practice
> to write JUnit tests for all methods that are in parent classes of the
> class being tested? For example, while writing tests for one of my
> classes, the wizard in Eclipse asked if I wanted to produce test methods
> to test methods of Object like equals() and wait(). Should I be writing
> tests for those methods as well as for those in my own class?
There's no point at all writing tests for wait and notify, because those
are final methods defined in Object, so nothing you do can change whether
they work properly or not.
If you haven't overridden equals from Object, then for a similar reason,
there's no need to test it. If you have overridden it, then of course you
need to test it.
The one time i can think of where you might want to test a non-overridden
method is if it depends on a method you have overridden (or an abstract
method which you've implemented). For example:
public class CollectionOfThings<E> {
public abstract E[] getAllThings();
public E getFirstThingInOrder() {
SortedMap<String, E> sortedThings = new TreeMap<String, E>();
for (E thing: getAllThings()) sortedThings.put(thing.toString(), thing);
return sortedThings.values().iterator().next();
}
}
public class CollectionOfLocales extends CollectionOfThings<Locale> {
public Locale[] getAllThings() {
return Locale.getAvailableLocales();
}
}
In that case, when testing CollectionOfLocales, you might want to test
getFirstThingInOrder. Like:
@Test
firstLocaleInOrderIsRight() {
assertEquals(
new Locale("ar", "AE"),
new CollectionOfLocales().getFirstThingInOrder());
}
The point would really be that the contract of CollectionOfLocales with
its subclasses requires that the things they provide have toString methods
which return strings which preserve the ordering of the things. You need
to test that this term of the contract is fulfilled by each subclass.
Whilst this case may seem trivial, note that a CollectionOfAges subclass
which supplied Integers would not meet the requirement, because the
ordering of Integers' toStrings do not reflect their numerical order. Such
a class would be intrinsically broken, and this unit test would catch
that. Fixing it would probably involve changing the contract between
CollectionOfThings and its subclasses, and the implementation of
getFirstThingInOrder.
Oh, also note that the code i wrote above for Locales is wrong. I only
discovered this when i actually compiled and ran it (or something a bit
like it, at least - not the exact code above). This illustrates the
importance of tests!
tom
--
They entered the Vortex and the dreams became reality
Tom Anderson
> But the same situation applies to things that aren't based on
> convenience methods. A list of telephone area codes or the names of all
> chiropractors in Ohio or any of a thousand other results from methods
> are also "moving targets". I'm really not sure how to test those.
These methods get their answers from some source of data, right?
So the way you test them is with test data. Rather than plugging the code
into the real data, you make a controlled set of test data, plug them into
that, and test that they behave in the way you would expect given the test
data. You then assume that they'll behave the right way when plugged into
real data.
Naturally, this means structuring your code so that it can easily be
plugged into different sources of data. This is probably a good thing
anyway.
Patricia Shanahan
This is the other side of the principle I was advocating in another
subthread, making an output method use a PrintStream that is either a
method parameter or provided when its object is constructed, rather than
writing to e.g. System.out.
I feel that test driven design, by encouraging separation of the details
of data input and output processing from the choice of data source or
sink, leads to better structured, more flexible code.
Patricia
Lew
> @Test
> public void getLocalesReturnsAllLocales() {
> Collection<Locale> locales = RhinoUtils.getLocales();
> Locale[] referenceLocales = Locale.getAvailableLocales();
> assertEquals(referenceLocales.length, locales.length);
locales.size()
> for (Locale locale: referenceLocales) {
> assertTrue(locales.contains(locale));
> }
> }
--
Lew
Tom Anderson
> Tom Anderson wrote:
>> @Test
>> public void getLocalesReturnsAllLocales() {
>> Collection<Locale> locales = RhinoUtils.getLocales();
>> Locale[] referenceLocales = Locale.getAvailableLocales();
>> assertEquals(referenceLocales.length, locales.length);
>
> locales.size()
Apologies. I knew i was running a risk by posting uncompiled code in this
particular thread, but i really thought i'd got away with it this time.
tom
--
The world belongs to the mathematics and engineering. The world is as
it is. -- Luis Filipe Silva vs Babelfish
Arved Sandstrom
That's a very good question. I've used both Emma and Cobertura with
EclipseLink, so you'd have EclipseLink dynamic weaving (change tracking,
lazy fetching) after the code coverage instrumentation; only in the
entity classes though.
Only my gut feeling, but if I were using anything that did more
elaborate bytecode manipulation than that then I figure I'd want to make
sure that the code coverage instrumentation was last.
AHS
Arved Sandstrom
>> 1. Code reviews/inspections and static code analysis with tools _are_
>> a form of testing. Keep them in mind.
>>
> I see that you've given some good concrete suggestions for tools to use
> in upcoming paragraphs. Are any of these static code analysis tools? If
> not, can you suggest some good ones? I'd love to have some feedback on my
> code via code reviews too but that's a little difficult to do. It's not
> realistic to full (non-trivial) classes in the newsgroup so I just pick
> and choose especially concerning problems and post snippets reflecting
> them. I'm sure that you folks are all too busy to do detailed
> walkthroughs of dozens of classes with hundreds of lines of codes in many
> of them ;-) I must see if there is a local Java user group in this area
> and see if I can develop a few buddies for doing code walkthroughs....
>
[ SNIP ]
FindBugs, if you're not using it already, is a good static code
analyzer. In my experience, just one decent tool like that, plus the
requisite excellent books like Bloch's "Effective Java" and willingness
to spend time reading those books to understand what the analyzer tells
you, is all you really need.
It's also useful to complement FindBugs with Checkstyle, which latter is
a good part of your toolkit for other purposes as well.
AHS
Arved Sandstrom
This discussion brings to mind something I'm faced with at work, for one
client. Although several of the applications I work on have sizeable
numbers of unit tests, they have been allowed to obsolesce, and there
have always been big gaps in coverage. We have a push on to add tests
and fix the ones that are already there. We are also adding functional
tests.
Now, here's the rub - in many cases it's known that the underlying code
is defective - human testers have recorded defects for specific use
cases. So when fixing or adding tests as part of this work, we have to
write no small number of tests in such a way that they will fail,
because if they don't fail then the tests are wrong. :-)
AHS
Arved Sandstrom
>> Speaking as someone who occasionally sits in on job interviews, I'm
>> delighted when we find a candidate who's written any tests at all.
>> Most of them haven't. We don't expect people to know more than we do
>> (it would be great if they did) and we all learned as we went along
>> and wish we understood it better.
>>
> As I said to Arved elsewhere in the thread, I find it frightening that
> anyone could apply for a job as anything more than a trainee programmer
> (one who didn't know anything about programming at all and was expecting
> to be trained by the employer) without having some familiarity with
> testing and, hopefully being able to demonstrate the familiarity. If
> that's the norm, then I feel better about my chances of getting a job
> programming in Java already :-)
>
One good thing about TDD, which has been mentioned elsewhere in this
thread (and in other threads), is that you can slide tests in as part of
the implementation, with no non-technical managers any the wiser. The
major problem with writing tests after the fact, even in reasonably
enlightened shops, is that at least three-quarters of all software
projects don't hit deadlines, for reasons that have very little to do
with the coding. In these circumstances anything that was planned for
the last phases, and appears to be non-critical, gets dispensed with.
Whereas if you start with tests, and refine them as your code gets
written, you barely impact the project schedule in the real world, but
end up with much better code, a complete set of tests, and the ability
to better react to change.
I don't have much sympathy for programmers who have never written tests,
but I do understand programmers who have no *work experience* writing
tests. The latter is quite common, because a majority of software teams
are disinterested in writing tests, or are not permitted to "waste time"
writing tests, or everyone is agreeable but they commit the mistake I
describe above. The former situation tells me that the programmer in
question doesn't do much professional reading, and can't be bothered to
spend some of their own time trying out a test framework...which would
take, like, a single evening.
AHS
Tom Anderson
> Patricia Shanahan wrote:
>> Tom Anderson wrote:
>>> On Fri, 21 May 2010, Patricia Shanahan wrote:
>>>
>>>> Arved Sandstrom wrote:
>>>> ...
>>>>> 5. Your tests are themselves defective. The sad truth is that if the
>>>>> core source code you just wrote is riddled with defects, then so
>>>>> probably are your tests. Main take-away here is, be aware that just
>>>>> because all your unit tests pass, some not insignificant percentage of
>>>>> those results are wrong.
>>>>
>>>> Although it is not 100% effective, there is some value in writing, and
>>>> running, the test before implementing the feature. Start testing a method
>>>> when it is just an empty stub. At that point, a test that passes is
>>>> either useless or has a bug in it.
>>>
>>> The true extremos work to this rule - you must always start with a failing
>>> test, then work to make it pass.
>>>
>>> In fact, the way we (the programmers where i work) try to work is to write
>>> the test first, before even writing the stub of the method, then treat the
>>> fact that the test doesn't compile as a failure, and work in from there.
>>
>> This principle is also very important for bug fixing. Any bug represents
>> a defect in the test suite. Fix the test suite first, and verify that
>> the new tests fail, before any attempt to fix the underlying bug.
>
> This discussion brings to mind something I'm faced with at work, for one
> client. Although several of the applications I work on have sizeable numbers
> of unit tests, they have been allowed to obsolesce, and there have always
> been big gaps in coverage. We have a push on to add tests and fix the ones
> that are already there. We are also adding functional tests.
>
> Now, here's the rub - in many cases it's known that the underlying code is
> defective - human testers have recorded defects for specific use cases. So
> when fixing or adding tests as part of this work, we have to write no small
> number of tests in such a way that they will fail, because if they don't fail
> then the tests are wrong. :-)
Well, at least that makes it easy to know you've got the test right!
Norbert Seriously, folks, this is an old point:
http://c2.com/cgi-bin/wiki?CaptureBugsWithTests
tom
--
Tech - No Babble
Tom Anderson
> Rhino wrote:
>>> Speaking as someone who occasionally sits in on job interviews, I'm
>>> delighted when we find a candidate who's written any tests at all.
>>> Most of them haven't. We don't expect people to know more than we do
>>> (it would be great if they did) and we all learned as we went along
>>> and wish we understood it better.
>>
>> As I said to Arved elsewhere in the thread, I find it frightening that
>> anyone could apply for a job as anything more than a trainee programmer
>> (one who didn't know anything about programming at all and was expecting to
>> be trained by the employer) without having some familiarity with testing
>> and, hopefully being able to demonstrate the familiarity. If that's the
>> norm, then I feel better about my chances of getting a job programming in
>> Java already :-)
>
> One good thing about TDD, which has been mentioned elsewhere in this thread
> (and in other threads), is that you can slide tests in as part of the
> implementation, with no non-technical managers any the wiser.
A crafty move in this situation is to selectively write tests for the
tricky bits. They'll help you get them right, and keep them right, with
less effort, and that serves as a useful evangelical point. Once people
see the utility of tests for some bits of code, they're on the road to
seeing the utility of them for the whole codebase.
tom
--
Vive la chimie, en particulier, et la connaissance en general. --
Herve This
Arne Vajhøj
Have you tried PMD?
If yes how does that stack up against those two?
Arne
Arne Vajhøj
I have no doubt that:
success(TDD) > avg(success(all methodologies))
but I am not quite as convinced that:
success(TDD) > avg(success(methodologies with strong focus on unit tests))
Unit tests is definitely good.
But the middle D and its implications for the design process is
more debatable I think.
Arne
Arne Vajhøj
> Tom Anderson wrote:
>>> Arved Sandstrom wrote:
>>> ...
>>>> 5. Your tests are themselves defective. The sad truth is that if the
>>>> core source code you just wrote is riddled with defects, then so
>>>> probably are your tests. Main take-away here is, be aware that just
>>>> because all your unit tests pass, some not insignificant percentage
>>>> of those results are wrong.
>>>
>>> Although it is not 100% effective, there is some value in writing,
>>> and running, the test before implementing the feature. Start testing
>>> a method when it is just an empty stub. At that point, a test that
>>> passes is either useless or has a bug in it.
>>
>> failing test, then work to make it pass.
>>
>> In fact, the way we (the programmers where i work) try to work is to
>> write the test first, before even writing the stub of the method, then
>> treat the fact that the test doesn't compile as a failure, and work in
>> from there.
>
> This principle is also very important for bug fixing. Any bug represents
> a defect in the test suite. Fix the test suite first, and verify that
> the new tests fail, before any attempt to fix the underlying bug.
Furthermore there is the phenomenon that if a bug was "natural" to
make then there is an increased chance that some maintenance
developer will reintroduce the bug later.
We want to catch that.
Reintroducing bugs does not make bosses and customers happy.
Arne
Tom Anderson
You may be right. I tend to think:
TDD == methodologies with strong focus on unit tests
Although that's probably just my biases. What are other methodologies that
have a strong focus on unit tests? Are they test-first or test-after? What
is a test-first method that's not TDD?
tom
--
Once you notice that something doesn't seem to have all the necessary
parts to enable its functions, it is going to mildly bug you until you
figure it out. -- John Rowland
Arved Sandstrom
I'm not able to give a strong opinion on PMD, Arne. I tried it to the
extent of downloading it, running it on the command line with some
options (selecting rulesets), and briefly inspecting the reports. The
context was that I'd already routinely been using FindBugs on the same
codebase.
My impression was that it was a competent static analyzer, but seeing as
how I already use the other two, that the extra effort of going through
yet another set of reports outweighed the gains of PMD catching a few
things that FindBugs might not, for example. Lots of overlap, IOW.
AHS
Patricia Shanahan
The way I see it, TDD implies that the design grows entirely from failed
tests. That leads to a situation in which one starts with a very simple
design, one that is known not to meet all the first release
requirements, and gradually adds complexity to the design, refactoring
as needed.
That works extremely well for small projects and projects with rapidly
changing requirements.
As the number of programmers grows, the cost of frequent large scale
design changes also grows. It also becomes impractical for every
programmer to understand all the code well enough to work on it directly.
For large projects with well defined requirements it may be better to
think through a large scale design that partitions the code into modules
that can be built relatively independently by sub-projects. Changes to
the interfaces between the top level modules need to be infrequent. That
requires thinking about the top levels of the final design before the
tests have even been written.
Even within a framework of a large scale design that was worked out
before any tests where written, one can still write tests before the
code they test.
Personally, I like to write unit tests and Javadoc comments at the same
time. Thinking about the tests helps me think about questions I should
be answering in the comments. What should this method do if that
parameter is null? Write a comment documenting the behavior and a test
enforcing it.
Patricia
Arne Vajhøj
To me it is a 3 level thing:
level 1 = unit tests everything
level 2 = write the unit tests before the code
level 3 = let the unit tests drive the design
I like up to level 1.5 (unit test everything, either write the
unit tests first *or* get another developer to write the
unit tests).
Arne
Patricia Shanahan
I think it depends on the nature of the project. The last non-trivial
piece of code I wrote was a simulation for my Ph.D. research. That
program existed to be changed. The best runs were those that got me
thinking about new questions, causing further changes to the program.
For that, I was at about level 2.9 - a little thinking about design
before I had a failing test, but not much.
I don't think that would work so well for a project with a few hundred
programmers.
My ideal, which I have never achieved, would be to have two sets of unit
tests, one set written by another developer and one set written by the
same person as the code under test.
Patricia
Tom Anderson
> Arne Vajh?j wrote:
>> On 26-05-2010 04:37, Tom Anderson wrote:
>>
>>> You may be right. I tend to think:
>>>
>>> TDD == methodologies with strong focus on unit tests
>>>
>>> Although that's probably just my biases. What are other methodologies
>>> that have a strong focus on unit tests? Are they test-first or
>>> test-after? What is a test-first method that's not TDD?
>>
>> To me it is a 3 level thing:
>>
>> level 1 = unit tests everything
>> level 2 = write the unit tests before the code
>> level 3 = let the unit tests drive the design
>>
>> I like up to level 1.5 (unit test everything, either write the
>> unit tests first *or* get another developer to write the
>> unit tests).
Ah, i think of test-driven design as meaning the same thing as test-driven
development - they're just different names for the same thing.
Test-driven design as a term is a misnomer anyway. You are doing design
bit-by-bit as you go along, but it's not driven by the tests. It can't be
- if we understand 'design' to mean something like 'architecture' or
'design at a level higher than source code', then it's the design which
drives the tests, because the design is what gives you "i need a class
which embodies such-and-such an idea, and exposes operations this-and-that
to its clients". Having design *driven by* tests is an impossibility.
I suppose test-driven development could be done with a big up-front
design. There's nothing contradictory there. Indeed, even with incremental
design, at every stage you have a picture of what the whole system will
look like when it's finished, even if you're only working on one little
bit of it right now. The only difference between that and up-front design
is that with up-front design, you think about it in more detail before you
start, and you give yourself less flexibility to change it once you do.
That's why i don't see a real distinction between them: even with a big
upfront design, not giving yourself flexibility is a mistake, so with that
out of the way, the only difference is how much thinking you do before you
start. That's clearly a continuum, so there can be no hard and fast
distinction between the two kinds of TDD.
> I think it depends on the nature of the project. The last non-trivial
> piece of code I wrote was a simulation for my Ph.D. research. That
> program existed to be changed. The best runs were those that got me
> thinking about new questions, causing further changes to the program.
> For that, I was at about level 2.9 - a little thinking about design
> before I had a failing test, but not much.
>
> I don't think that would work so well for a project with a few hundred
> programmers.
I'm not sure there are any methodologies at all that work well for a
project with a few hundred programmers, but that's another story.
> My ideal, which I have never achieved, would be to have two sets of unit
> tests, one set written by another developer and one set written by the
> same person as the code under test.
That would be interesting. A while ago, we were working with some
contractors who were not great programmers, and who were especially
hopeless at writing tests (they'd write them, because they knew we wanted
them, but they would often not really test anything). I wondered about
managing the interaction between our teams by having us write the tests,
and having them write the code to make the tests pass. That would have
forced them to write better code, and given us a lot more confidence in
what they were doing.
tom
--
Osteoclasts = monsters from the DEEP -- Andrew