Capture stdout/stderr?
Kohsuke Kawaguchi
I noticed that TestNG does not capture the output to System.out and
System.err unlike JUnit. Those output are often very convenient when
you are looking at test failures.
I can work around the problem to some extent by doing
System.setOut/setErr myself, but:
1. it doesn't work well if I run tests in parallel
2. it's tedious
Is there any reason why TestNG can't do that for us?
In JUnit, http://parallel-junit.dev.java.net/ can capture output done
by multiple threads.
Cédric Beust ♔
A couple of reaons:
- If you want to do some serious logging, you should probably not be using System.out or System.err but a logging package (java.util or log4j).
- For "not serious logging", I just don't like it when frameworks silently redirect
System.out or System.err because if I'm not aware, I waste time trying to figure out why my code is not being called while it actually is but the traces are silently being redirected into an obscure file that I need to hunt down.
When I write "System.out", I really mean it: "print this on the standard output" :-)
--
Cedric
Cédric
Kohsuke Kawaguchi
The thing is, it's not that my test code is trying to write to
System.out.
For example, some of my tests call into Javac, which produces messages
on System.out. Or my tests call the schema compiler of JAXB, which does
something similar.
Or I might be launching a web container like Tomcat through Cargo, and
then Tomcat writes a whole bunch of stuff into System.out.
Authors of those respective libraries is not wrong in writing output to
System.out. They do mean to "print this on the standard output". But
when I integrate those libraries as dependencies into a test, I want
the output to be captured into my test execution log.
So given this reality, what is my option?
Cédric Beust ♔
First you run that code and the output goes to a.txt and then you run their tests and because JUnit redirects them as well, the output suddenly goes to b.txt.
This is why ultimately, the decision of redirecting System.out and System.err should go to the end user, that is: you. And by the way, based on what you are saying, even if TestNG captured out/err, you would still have the problem of rogue output when not running TestNG...
Now, to your question: why not implement your own suggestion and set System.out and System.err before starting your system?
--
Cedric
Kohsuke Kawaguchi
> As you can see, the problem is still the same: what if you use two
> frameworks and each of them has their own idea how System.out should be
> captured?
>
> First you run that code and the output goes to a.txt and then you run their
> tests and because JUnit redirects them as well, the output suddenly goes to
> b.txt.
>
> This is why ultimately, the decision of redirecting System.out and
> System.err should go to the end user, that is: you. And by the way, based
> on what you are saying, even if TestNG captured out/err, you would still
> have the problem of rogue output when not running TestNG...
I'm afraid I'm not quite following when you say I'd "still have the
problem of rogue output". If I run it outside a test framework, all the
output goes System.out and that's it. I don't see anything wrong with
that.
> Now, to your question: why not implement your own suggestion and set
> System.out and System.err before starting your system?
Yeah, if I can do it without changing TestNG, I think I'm OK with it.
But there are a few tricky aspects.
First, when tests are to be run in parallel, we can't have each test
individually set and restore System.out and System.err. They need to be
installed once before parallel execution starts, then have each thread
co-operatively use it, and finally restored back to the original value
when the parallel execution ends. I don't know what the place to do
this in TestNG.
Hmm... maybe I can do that before and after I call into
org.testng.TestNG.run()? from my custom PrintWriter I might be able to
send them to Reporter.log().
(BTW I'm looking at the Reporter class and I don't think this is
thread-safe. For example, setCurrentTestResult has just one field,
instead of ThreadLocal.)
Second, err... I think that might be the only issue.
Thank you very much for all your help. (I'd be very grateful if you
could take a look at another e-mail I posted yesterday, too)
Cédric Beust ♔
Hmm... maybe I can do that before and after I call into
org.testng.TestNG.run()? from my custom PrintWriter I might be able to
send them to Reporter.log().
How about putting this code in a @Configuration(beforeSuite = true) method?
It will be run once before any test starts, and you can restore the state in an afterSuite method...
If you need something more flexible, you can create your own runner as shown in the documentation (create a new TestNG() object, etc...).
(BTW I'm looking at the Reporter class and I don't think this is
thread-safe. For example, setCurrentTestResult has just one field,
instead of ThreadLocal.)
True, thanks.
Second, err... I think that might be the only issue.
Thank you very much for all your help. (I'd be very grateful if you
could take a look at another e-mail I posted yesterday, too)
Yes, I haven't answered yet because I am thinking about it :-)
Cédric
Kohsuke Kawaguchi
this should work.
Kohsuke Kawaguchi
fixing the concurrency issue in Reporter, would you be interested in
that?
Or is it fixed already :-) :-) ?
Cédric Beust ♔
--
Cedric
--
Cédric
Kohsuke Kawaguchi
Kohsuke Kawaguchi
Would you like me to send it to this list, or create a branch and
commit it there for your review? You guidance will be appreciated.
Cédric Beust ♔
Thanks!
--
Cedric
--
Cédric