How would you test the main() method?

[Rusi Filipov]

I am just playing around with a small program to solve the Towers of Hanoi problem. I am close to 100% test coverage, but since I don't use a runtime container or dependency injection, the Main class with its main() method is still a hard problem. 

In short, the program reads the input from the command line args, solves the problem, and prints some ASCII art on stdout while working. Let me illustrate:

public class Main {

  private final Solver solver;

  public Main(Solver solver) {

    this.solver = solver;

  }

  public void execute(int towerHeight) {

    solver.solveFor(towerHeight);

  }

  public static void main(String[] args) {

    Parser parser = new Parser(args)

    Main main = new Main(new Solver(new Observer(new Printer())))

    main.execute(parser.towerHeight())

  }

}

Main is the entry point of the application and I do all the object wiring with concrete classes in the main() method. Now while I can replace almost every object with test doubles, I cannot do this with the main() method.

I tried to use a self-shunt test and assert a kind of like this:

public class MainTest extends Main {

  private int actualHeight = 0;

  @Override

  public void execute(int towerHeight) {

    actualHeight = towerHeight)

  }

  @Test

  public void mainEntryPoint() throws Exception {

    Main.main(new String[]{ "3" });

    assertThat(actualHeight, is(3));

  }

}

This way I try to use the test class as a test double (introduced by Uncle Bob in Episode 23) and to assert that the object wiring in the main() method is done right and that execution follows correctly until the execute() method is entered.

But as perhaps some of you may have noticed, the self-shunt object is not instantiated, but another instance of Main is used. This is so, because main() is static and at the same time it is the entry point of the application.

Any ideas how to cover main() with a test?

PS: I am trying to avoid power-mocking techniques provided by frameworks like JMockit, where you can override everyhing, I know how to achieve it with power-mocking.

[Andreas Schaefer]

I'd separate concerns and unit test the parser individually. object wiring and execution path verification would be a system test (integration test) with the real non mocked Main class .. of course it therefor had to return an outcome in some way.

[Rusi Filipov]

Actually, the parser is unit-tested, I have one unit tests for all the classes that collaborate. But I want to test the Main class including the main() method too, and I want to do it with a unit test. 

Because if I go for a system test, then all I can sense is the program output (some fancy ASCII art) which is printed to stdout. You can imagine how brittle it will be to assert on that.

At least believe it should be possible to test main() only by using unit tests. The goal is to achieve 100% unit test coverage...

This is the current unit test which is based on JMockit.

@RunWith(JMockit.class)

public class MainTest {

  @Test

  public void mainEntryPoint(@Mocked final Solver solver) throws Exception {

    Main.main(new String[]{ "3" });

    new Verifications() {

      {

        int actualHeight;

        solver.solveFor(actualHeight = withCapture());

        assertThat(actualHeight, is(3));

      }

    };

  }

}

I believe there must be a better way....still searching.

On Sat, Jan 18, 2014 at 9:32 PM, Andreas Schaefer <schaeferi...@gmail.com> wrote:

I'd separate concerns and unit test the parser individually. object wiring and execution path verification would be a system test (integration test) with the real non mocked Main class .. of course it therefor had to return an outcome in some way.

--
The only way to go fast is to go well.
---
You received this message because you are subscribed to a topic in the Google Groups "Clean Code Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clean-code-discussion/vvca9BGHwes/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clean-code-discu...@googlegroups.com.
To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.

[David Hunt]

100% unit test coverage is not possible. But you can get close! What if main() only contained a single line:

new App().Run();

Would it still deserve a unit test?

Dave

You received this message because you are subscribed to the Google Groups "Clean Code Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discu...@googlegroups.com.

[Rusi Filipov]

Indeed minimizing the work done in main() is a good strategy to get away of the static entry point as soon as possible. But that only shifts the problem to the constructor of App(). I suppose this constructor would create other objects and wire them. How would you cover this constructor? Just like in main(), there is no way to inject anything from the test, so that you can sense the effects...

Btw, as I showed in my previous post, the main method can also be covered with an unit test if using a power mocking framework. But this is something I want to avoid because it has other pitfalls and caveats. It doesn't force you to design for testability and it is easy to make a rigid design even more rigid by covering the code with power mocking tests. 

So I still believe that even 100% is a hard goal, it is not always impossible, and there might be situations where it can be achieved.. Example - if the application is deployed in a container, main() is defined in the container's libraries, so you don't have to test this static entry point. But let's stick to the problem - we have main() and want to unit-test it. Any ideas?

[Andreas Schaefer]

I'd say more important than 100% (unit) test coverage is the developers confidence in the codebase and that the system works. some things just don't need to be under test in a harmonic, good dev team.
I know that doesn't solve the academics of your problem, but you could have a system test using a true mock of Solver that verifies the outcome on the lowest level .. this way avoiding the need to parse ASCII or whatever the eventual output format would be.

[Caio Fernando Bertoldi Paes de Andrade]

Main doesn't have any proper logic in it, so it doesn't deserve unit tests.

Its responsibility is to wire things up, so it deserves system tests testing if the whole application works as expected. System tests are very statist tests that go through the UI, so they will be brittle, and you want them to be brittle in order to give you the confidence that the entire system works together. Just don't make too many system tests, otherwise it'll be difficult to maintain them.

Cheers,

Caio

P.S.: Don't forget to write integration tests that will give you clues about which modules are behaving wrongly when your system tests fail :)

—
Sent from Mailbox for iPhone

On Sun, Jan 19, 2014 at 10:57 AM, Andreas Schaefer <schaeferi...@gmail.com> wrote:

I'd say more important than 100% (unit) test coverage is the developers confidence in the codebase and that the system works. some things just don't need to be under test in a harmonic, good dev team.
I know that doesn't solve the academics of your problem, but you could have a system test using a true mock of Solver that verifies the outcome on the lowest level .. this way avoiding the need to parse ASCII or whatever the eventual output format would be.

[Rusi Filipov]

@Andreas, I agree with your argument concerning team harmony and real projects. If this were the case I wouldn't insist of reaching 100% because the closer you get, the harder it gets. But this is not my intent here. I am just curious to find a good solution for this "hard-to-test" problem. It is a cata-like excersise, with the aim to experiment and try out things that should not be experimented in production code. But what do you mean with true mock of Solver? Solver is an object created by Main, so how can it be mocked out in a system test? The two boundaries of this program are the main method (for passing the user input via the args[]) and its the standard output, so that the user can view the result of the execution. So I imagine, a system test would invoke the program like "java Main 123", and we would need to capture and assert on the standard output - not very elegant.

@Caio, as long as understand Uncle Bob's Clean Architecture, Main is the part that is located on the outer side of the Boundary. The part that creates and wires the main objects and collaborators of the Application. It is also responsible for selecting the concrete implementations of the wired objects. This job of wiring must be done by someone, and if it is not Main, it will be another class close to Main. So it is still a kind of logic, not business logic, but configuration logic. And since Main is also part of the codebase, it should be tested too IMHO, preferrably with a unit test.

[Andreas Schaefer]

Rusi,

But what do you mean with true mock of Solver?

A true mock as defined by Uncle Bob is a spying test double (btw: that is also a stub and that is also a dummy but that is no fake [because in contrast to all other test doubles a fake has behaviour based on runtime variables]) that knows by itself what should have happened and respectively that your test can simply ask for validity (in contrast to a simpler spy with which your test class has to manually/externally read out the spied information and interpret it as valid/expected or not by itself).

For completeness: A dummy returns values as close to zero or null as possible. A stub just returns fixed results to drive your production code along a specific path.

I just wanted to use the correct term here because many people use 'mock' and actually mean the broader _category_ 'test double'.

But if it's a true mock or just a spy doesn't really matter in that case anyway :/

 

But what do you mean with true mock of Solver? Solver is an object created by Main, so how can it be mocked out in a system test? The two boundaries of this program are the main method (for passing the user input via the args[]) and its the standard output, so that the user can view the result of the execution

you're right, I was wrong in using the term "system" test when there is just the one specific "production" Main system / main partition. In that case I should have used "integration" test which doesn't force you to use 100% production code only.

(as a side note: when your SolverSpy/SolverMock inherits from the production Solver, then you have all the production code involved .. plus some test code)

as an alternative: you could have antoher main partition (like explained by Uncle Bob in this thread: https://groups.google.com/forum/?hl=en#!topic/clean-code-discussion/aTZhnLkOk1g/4PF9PBmlxQsJ), e.g. "testMain", that uses a SolverMock in the wiring part. Or you extract the wiring away from Main (as already suggested), wire the SolverMock up in there and thus could use the production Main class. combining these two methods you could potentially reach 100% test coverage.

 

And since Main is also part of the codebase, it should be tested too IMHO, preferrably with a unit test

For being unit tested it (an the component it depends on as well) had to be designed in such a way that you can "mock out" all of it's dependencies so that it can be tested in complete isolation. If it's not completely isolated, then it's not a unit test anymore but some kind of integration test.

[Paolo Laurenti]

On Sun, Jan 19, 2014 at 6:13 PM, Andreas Schaefer <schaeferi...@gmail.com> wrote:

[...]

 

And since Main is also part of the codebase, it should be tested too IMHO, preferrably with a unit test

For being unit tested it (an the component it depends on as well) had to be designed in such a way that you can "mock out" all of it's dependencies so that it can be tested in complete isolation. If it's not completely isolated, then it's not a unit test anymore but some kind of integration test.

@Andreas: I interpreted Unit Test like you do but during the last weeks I changed my mind. I saw the talk of Ian Cooper "TDD, where did all go wrong" and after that I started to read "Test Driven Development Explained" by Kent Beck. Ian Cooper said that Kent Beck was misunderstood when he introduced TDD: the word "Unit" is not related to the SUT but to the test. Unit tests should verify a single behavior in isolation from others test (i.e: a unit test shouldn't use a DB because a test execution that modifies some data, could interfere with the execution of other tests that use the same data. Yes, you could rebuild every time the DB but this is a trick, not the solution of the problem). 

Therefore you can test many classes with one single test. These classes should collaborate at runtime with their objects, in order to implement the behavior that the system should provide.

Thinking that "Unit" is referred to the SUT makes the tests coupled to implementations details (you need to mock/stub/fake/etc every dependencies that your SUTs need), and every time you will change a system behavior, several tests will be red with no reason...

@Rusi: Sorry, I haven't an idea about how to unit test your Main. I think that its responsibilities should be only objects creations/wiring. These responsibilities have to deal with a lot of implementation details (you have to instantiate the concrete objects that you high level policy objects depend on) so I think that Main and all the "Declarative Layer" should be tested by some (a few) system tests and not by Unit test (they focus only on behaviors).

 My 2 cents...

Paolo Laurenti Software Developer
lauren...@gmail.com | http://paololaurenti.wordpress.com

Contact me: Skype/paololaurenti

My blog: Fast TDD: Sublime Text 2 – Guard – RSpec Configurations

Get a signature like this: Click here!

[Rusi Filipov]

@Palo, I think Ian Cooper means component tests in his talk. He shifts the granularity of the term "unit" from class to component. So if a component consists of 20 collaborating classes, Ian suggests not testing the classes in the component individually, but testing the component's behaviors. These tests would use only the component's API, and see the component as a black box. I think such component tests are not better or worse than unit tests, they are just different. The tests are less coupled to the implementation and they test multiple collaborating classes with one shot. But on the other hand, the tests can't put strong focus on the individual class, making it harder to test edge and error cases if the only entry point they use is the component's Facade. 

@Andreas, thank you for the suggestions and references. Yet I am still curious how the main part of the application can be 100% unit-tested, taking into account that there is a main() method involved. Currently, I don't see a good way to deploy test doubles in the main part, if only using conventional injection technques or subclassing and overriding. Maybe you could post a code example?

[Andreas Schaefer]

Paolo,

On Monday, January 20, 2014 2:30:01 PM UTC+1, Paolo Laurenti wrote:

@Andreas: I interpreted Unit Test like you do but during the last weeks I changed my mind. I saw the talk of Ian Cooper "TDD, where did all go wrong" and after that I started to read "Test Driven Development Explained" by Kent Beck.

Thanks for sharing this resources. I still have to take a look at that yet.

 

Ian Cooper said that Kent Beck was misunderstood when he introduced TDD: the word "Unit" is not related to the SUT but to the test. Unit tests should verify a single behavior in isolation from others test (i.e: a unit test shouldn't use a DB because a test execution that modifies some data, could interfere with the execution of other tests that use the same data. Yes, you could rebuild every time the DB but this is a trick, not the solution of the problem). 

Therefore you can test many classes with one single test. These classes should collaborate at runtime with their objects, in order to implement the behavior that the system should provide.

Thinking that "Unit" is referred to the SUT makes the tests coupled to implementations details (you need to mock/stub/fake/etc every dependencies that your SUTs need), and every time you will change a system behavior, several tests will be red with no reason...

not specifically regarding the term 'unit' for now, from what I know so far I'd say that you briefly touched on the topic of .. statist vs. mockist / Classical TDD (Kent Beck) vs. "London" school of TDD / observing outcome vs. observing behaviour. I'm not providing further explanations here, as Uncle Bob in his webcasts goes into great detail on this.

 

@Rusi: Sorry, I haven't an idea about how to unit test your Main. I think that its responsibilities should be only objects creations/wiring. These responsibilities have to deal with a lot of implementation details (you have to instantiate the concrete objects that you high level policy objects depend on) so I think that Main and all the "Declarative Layer" should be tested by some (a few) system tests and not by Unit test ...

+1

 

... (they focus only on behaviors).

 they focus on state as well. see my comment above :)

[Paolo Laurenti]

On Tue, Jan 21, 2014 at 12:37 PM, Rusi Filipov <rfil...@gmail.com> wrote:

@Palo, I think Ian Cooper means component tests in his talk. He shifts the granularity of the term "unit" from class to component. So if a component consists of 20 collaborating classes, Ian suggests not testing the classes in the component individually, but testing the component's behaviors. These tests would use only the component's API, and see the component as a black box. I think such component tests are not better or worse than unit tests, they are just different. The tests are less coupled to the implementation and they test multiple collaborating classes with one shot.

I didn't know the "component test" concept and the description that you provided is exactly what I have understood about "Unit test". May be I am missing something, I don't know.

 

But on the other hand, the tests can't put strong focus on the individual class, making it harder to test edge and error cases if the only entry point they use is the component's Facade. 

From my experience I found that the majority of problems I faced with TDD, are tests with a "strong focus on individual class".
When I wrote these kind of tests I found myself in trouble when I have to change the behaviors of the system that they implement. One key concept of the speech of Ian Cooper is that you can write test coupled to implementation details in order to explore an a solution of a problem. The difference is that when the exploration is finished and you are good with your solution, it would be a good idea to delete these tests. I know it's difficult to accept to delete some tests we wrote, but, in my opinion, tests that are coupled to implementation details, after the exploratory phase where they were useful, can only create problems. 

[Andreas Schaefer]

Rusi,

On Tuesday, January 21, 2014 12:37:20 PM UTC+1, Rusi Filipov wrote:

@Andreas, thank you for the suggestions and references. Yet I am still curious how the main part of the application can be 100% unit-tested, taking into account that there is a main() method involved. Currently, I don't see a good way to deploy test doubles in the main part, if only using conventional injection technques or subclassing and overriding. Maybe you could post a code example?

disregarding the discussion if the main partition should be unit tested instead of integration/system testetd ... I'll try to give you an example: .... and oops, I forgot you can't override a static method. Thus ...

you extract the wiring away from Main (as already suggested), wire the SolverMock up in there and thus could use the production Main class

... can not be done. But that again makes it even more obvious (imho) that the _main partition_ is a system which for testing purposes should be addressed with a system test.

To test the wiring up logic you maybe could extract that part into a poor mans DI in a non static part or which can be configured via a config file. but that only shifts the problem and it gets obvious that there's no difference if you have to edit the wiring up code or if you edit a config file.

sorry that I can't come up with something better. It's just in my guts that there are more important issues out there that have to be addressed ;) just keep the lines of code in the main partition as small as possible, extract 'till you drop, have some system tests .. and you should be fine.

[Andreas Schaefer]

On Tuesday, January 21, 2014 4:49:34 PM UTC+1, Paolo Laurenti wrote:

On Tue, Jan 21, 2014 at 12:37 PM, Rusi Filipov <rfil...@gmail.com> wrote:

But on the other hand, the tests can't put strong focus on the individual class, making it harder to test edge and error cases if the only entry point they use is the component's Facade. 

From my experience I found that the majority of problems I faced with TDD, are tests with a "strong focus on individual class".
When I wrote these kind of tests I found myself in trouble when I have to change the behaviors of the system that they implement. One key concept of the speech of Ian Cooper is that you can write test coupled to implementation details in order to explore an a solution of a problem. The difference is that when the exploration is finished and you are good with your solution, it would be a good idea to delete these tests. I know it's difficult to accept to delete some tests we wrote, but, in my opinion, tests that are coupled to implementation details, after the exploratory phase where they were useful, can only create problems. 

:) what you're describing is the "The Uncertainty Principle of TDD and the never ending battle between the Mockists and the Statists" (http://cleancoders.com/codecast/clean-code-episode-23-p2/show) 

[Paolo Laurenti]

Thank you for the notification. I've not reached episode 23 yet. I have to watch episode 19, I'm following them sequentially. :-)

[Uncle Bob]

There are a number of ways to test the main method.  Most involve some kind of hack or another.  For example, if you add a command line argument like -v which outputs the result of the parser, but does not execute; then you can override stdout to provide a string, and simply invoke main with the appropriate arguments.  When main returns, you look at the stdout string.  

As others have said, you can make main so trivial that it doesn't need a test by delegating all it's functions to some non-static class that can be tested.  For example:

class Main {

  public static void main(String[] args) {

    Main main = new Main();

    main.run(args);

  }

  void run(String[] args) {

    ...

  }

}