diagram of tests dependencies

[Tomek Kaczanowski]

Cedric, All,

I was thinking about adding a new feature to TestNG. What I would like
to have is a dot (graphviz) diagram of test dependencies. ...not sure
if that is of any use at all, but I feel like it might be fun. :)

I don't know TestNG codebase very well (if at all), so I would like to
ask you for hints on how to proceed.

I guess, that after I feed TestNG with tests, e.g. like this:
TestNG testng = new TestNG();
testng.setTestClasses/Suites(...)
then TestNG holds some internal structure which describes dependencies of tests.

Now the question is: "how can I access it?"

What I figured out is that it is possible by implementing IReporter
interface. This seems to be pretty easy, but there is another issue to
solve. Reporters come into action after tests are finished. I would be
more happy to be able to generate it without running any tests. How
can I skip all tests ? Should I write an annotation transformer that
would make all tests "enabled=false"?

--
Cheers,
Tomek Kaczanowski

[Cédric Beust ♔]

Hi Tomek,

Actually, TestNG already supports this :-)

In the file GraphThreadPoolExecutor.java, set DOT_FILES to `true` and TestNG will generate a .dot file that you can feed to GraphViz (it will tell you the exact path). This implementation will actually generate several pictures that represent the evolution of your method graph as TestNG executes them. It's pretty neat :-)

I added this when I was writing the article explaining how I implemented concurrent dependent methods (part 1, part 2). In particular, part 2 talks a bit about my GraphViz implementation (which is very simple, as you saw).

Let me know if you get it to work, and feel free to share your graph if it's not confidential :-)

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to testn...@googlegroups.com.
To unsubscribe from this group, send email to testng-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
Cédric

[Tomek Kaczanowski]

Hi Cedric,

> Actually, TestNG already supports this :-)

...that is really annoying! ;)

> In the file GraphThreadPoolExecutor.java, set DOT_FILES to `true` and TestNG
> will generate a .dot file that you can feed to GraphViz (it will tell you
> the exact path). This implementation will actually generate several pictures
> that represent the evolution of your method graph as TestNG executes them.

I will have a look at this. Thank you !

--
Regards / Pozdrawiam
Tomek Kaczanowski

[Tomek Kaczanowski]

Hi Cedric,

I tried to use it but with limited success. I run ant build of testng
and got a lot of dot files in /tmp/TestNG-whatever folder. They are
not exactly what I would like to have. It seems like no real test
methods names are used, but some fancy f1, f2 labels.

What I would like is to have clusters of groups and dependencies of
groups and methods. For example something like this:

digraph G {

subgraph cluster_0 {
style=filled;
color=lightgrey;
node [style=filled,color=white];
testA;
testB;
testC;
label = "smoke tests";
}

subgraph cluster_1 {
node [style=filled];
testD;
testE -> testF;
label = "real fun tests";
color=blue
}
testD -> testB;

}

which generates the following diagram: http://tinyurl.com/5s6bmss

(this is just an example of the general idea, would need to work more
on this so the stuff is readable in case of thousands of tests etc.0

Would it be possible with GraphThreadPoolExecutor or should I try to
implement it using the IReporter approach described earlier?

--
Cheers,
Tomek Kaczanowski

[Cédric Beust ♔]

Tomek,

You can modify this class to generate whatever graph works for you, you have access to the entire suite information and the current implementation shows you how to use it.

-- 

Cédric

--
Cheers,
Tomek Kaczanowski

--
You received this message because you are subscribed to the Google Groups "testng-dev" group.
To post to this group, send email to testn...@googlegroups.com.
To unsubscribe from this group, send email to testng-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
Cédric

[Tomek Kaczanowski]

Hi Cedric,

> You can modify this class to generate whatever graph works for you, you have
> access to the entire suite information and the current implementation shows
> you how to use it.

So one option is to fork/hack TestNG and change
GraphThreadPoolExecutor to whatever suits me. Ok, good to know, thanks
for this tip.

But that is not really user-friendly, is it? I mean, it would be much
easier to have it pluggable, which means without changing TestNG. That
is why I ask you so stubbornly about implementing IReporter :)

--
Regards / Pozdrawiam
Tomek Kaczanowski

> --

[Cédric Beust ♔]

Tomek,

Yes, if you want to make your code available to other TestNG users, an IReporter is the correct way to do it. It should be pretty straightforward since the IReporter gets access to pretty much everything that happened during the test run...  Keep us posted!

-- 

Cédric

[Tomek Kaczanowski]

Cedric, All,

> Keep us posted!
you can follow the development on GitHub
https://github.com/tomekkaczanowski/testng-test-dependencies-reporter

--
Cheers,
Tomek Kaczanowski

>
> --

> > >> testng-dev+...@googlegroups.com<testng-dev%2Bunsu...@googlegroups.com>
> > .

> > >> For more options, visit this group at
> > >>http://groups.google.com/group/testng-dev?hl=en.
>
> > > --
> > > Cédric
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "testng-dev" group.
> > > To post to this group, send email to testn...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > > testng-dev+...@googlegroups.com<testng-dev%2Bunsu...@googlegroups.com>
> > .

> > > For more options, visit this group at
> > >http://groups.google.com/group/testng-dev?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-dev" group.
> > To post to this group, send email to testn...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > testng-dev+...@googlegroups.com<testng-dev%2Bunsu...@googlegroups.com>
> > .

[Tomek Kaczanowski]

Cedric, All,

I hacked a first version and described it on my blog http://is.gd/mFIh3J

The code is on github http://is.gd/Lzjs41

Hopefully someone will find it useful. :)

--
Cheers,
Tomek Kaczanowski

On 23 Sty, 16:58, Tomek Kaczanowski <kaczanowski.to...@gmail.com>
wrote:

> Cedric, All,
>
> > Keep us posted!
>

> you can follow the development on GitHubhttps://github.com/tomekkaczanowski/testng-test-dependencies-reporter

> > > >> testng-dev+...@googlegroups.com<testng-dev%2Bunsubscribe@googlegrou­ps.com>

> > > .
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/testng-dev?hl=en.
>
> > > > --
> > > > Cédric
>
> > > > --
> > > > You received this message because you are subscribed to the Google Groups
> > > > "testng-dev" group.
> > > > To post to this group, send email to testn...@googlegroups.com.
> > > > To unsubscribe from this group, send email to

> > > > testng-dev+...@googlegroups.com<testng-dev%2Bunsubscribe@googlegrou­ps.com>

> > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/testng-dev?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "testng-dev" group.
> > > To post to this group, send email to testn...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > > testng-dev+...@googlegroups.com<testng-dev%2Bunsubscribe@googlegrou­ps.com>

[Cédric Beust ♔]

Hi Tomek,

This is very cool! I encourage you to share it on the testng-users mailing-list as well.

One quick comment: you don't need to run the tests in order to find out their dependencies, you can do this with simple introspection since the dependency logic is captured in annotations.

-- 

Cédric

To unsubscribe from this group, send email to testng-dev+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-dev?hl=en.

--
Cédric

[Cédric Beust ♔]

FYI, I added a link to your blog post from the documentation:

http://testng.org/doc/misc.html

-- 

Cédric

2011/2/14 Cédric Beust ♔ <ced...@beust.com>

--
Cédric

[Tomek Kaczanowski]

> I encourage you to share it on the testng-users
> mailing-list as well.

done

> One quick comment: you don't need to run the tests in order to find out
> their dependencies, you can do this with simple introspection since the
> dependency logic is captured in annotations.

Yes, if I am only interested in seeing the dependencies then running
tests is a waste of time.
What do you mean "with simple introspection"? You mean I should use
standard Java mechanism, or is there something in the API of TestNG
that might help me to discover test dependencies without running the
tests?

[Tomek Kaczanowski]

> FYI, I added a link to your blog post from the documentation:
> http://testng.org/doc/misc.html

Thanks for making me famous! ;)

[Cédric Beust ♔]

On Tue, Feb 15, 2011 at 11:30 AM, Tomek Kaczanowski <kaczanow...@gmail.com> wrote:

What do you mean "with simple introspection"? You mean I should use

standard Java mechanism, or is there something in the API of TestNG
that might help me to discover test dependencies without running the
tests?

The dependencies are captured in annotations, so you could collect all the test classes from the testng.xml and then look for all the @Test(dependsOnMethods/dependsOnGroups) annotations and build your graph from there. You would also need to calculate the groups, though, but overall, I think it should fit in ~100 lines of code to calculate the entire graph.

--
Cédric

[Tomek Kaczanowski]

Hi Cedric,

one more question (and probably not the last one).

If a configuration method (e.g. @BeforeClass) fails, then tests, that
should be run after it, are skipped, right?

Now, I would like to learn about why a particular test was skipped.
That is, I want to know because of which failed set up method it was
skipped.

Is it possible? AFAIK configuration methods are not listed by
getMethodsDependedUpon method of ITestNGMethod.

--
Regards / Pozdrawiam
Tomek Kaczanowski

2011/2/15 Cédric Beust ♔ <ced...@beust.com>:

[Cédric Beust ♔]

Hi Tomek,

On Tue, Mar 8, 2011 at 1:15 PM, Tomek Kaczanowski <kaczanow...@gmail.com> wrote:

Hi Cedric,

one more question (and probably not the last one).

If a configuration method (e.g. @BeforeClass) fails, then tests, that
should be run after it, are skipped, right?

Correct. Which methods get skipped depend on which configuration method failed:

@BeforeMethod -> all test methods that follow

@BeforeClass -> the entire class

@BeforeTest -> all the classes left in the <test> tag

etc... you get the idea.

Now, I would like to learn about why a particular test was skipped.
That is, I want to know because of which failed set up method it was
skipped.

Is it possible? AFAIK configuration methods are not listed by
getMethodsDependedUpon method of ITestNGMethod.

Correct. No, there is not direct way of knowing what caused a skip, but it shouldn't be hard to guess: I think it's pretty rare when a configuration method fails, so you will most likely only have one per test run. Guessing it's the culprit will probably be right most of the time.

--
Cédric

[Mengyu Han]

Hi Tomek,

I find it very useful to have a graph of dependency representation, but we have been using ant instead of Maven for a long while .
Is there a way to use your testNG reporter by ant build.xml?

Thank you very much!