Hudson plugin (Re: xUnit style reports?)

[Pekka Klärck]

2009/12/3 regis_desgroppes <air.des...@laposte.net>:
> Is there a way to generate xUnit style reports?

No there isn't. But you should be able to transform the XML output
file into xUnit compatible XML reports using XSLT or some other
mechanism.

> That's to integrate test results into our continuous integration
> engine (Hudson).

Incidentally we've been recently working with a Hudson plugin
ourselves. It's been mainly prototyping so far but in a retrospect we
should have mentioned already this effort earlier. Kari just got the
proto into usable state yesterday but I'm not exactly sure where the
code is or how to take it into use. We are planning to take Hudson
into real use when we start RF 2.2 development but getting feedback,
enhancement ideas, and patches from others even earlier would be
great. Kari, could you please give an update what's the current
situation with the plugin?

Notice also that the simplest way to integrate RF with any CI system
is just running the tests as an external command. RF returns the
number of failed critical tests as a return code (explained thoroughly
in the User Guide) and most systems understand that a non-zero RC
means problems. A special plugin that can also show links to
logs/reports, get some statistics from the XML output, and otherwise
make life easier is obviously much better.

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Kari Husa]

2009/12/3 Pekka Klärck <pekka....@gmail.com>:
> 2009/12/3 regis_desgroppes <air.des...@laposte.net>:

>> That's to integrate test results into our continuous integration
>> engine (Hudson).
>
> Incidentally we've been recently working with a Hudson plugin
> ourselves. It's been mainly prototyping so far but in a retrospect we
> should have mentioned already this effort earlier. Kari just got the
> proto into usable state yesterday but I'm not exactly sure where the
> code is or how to take it into use. We are planning to take Hudson
> into real use when we start RF 2.2 development but getting feedback,
> enhancement ideas, and patches from others even earlier would be
> great. Kari, could you please give an update what's the current
> situation with the plugin?

Hi!

The current prototype version of the Robot Framework Hudson plugin
just enables linking to the reports that are generated by the separate
build step (e.g. Execute shell).

If you are interested to try this version out, you can checout the
source from the Java Tools [1] Subversion repository at [2]. The
project can be built and packaged using maven, by excecuting: mvn
package. This wil generate installable plugin package
(robotframework.hpi) under project's target directory. This can be
istalled into Hudson. The plugin can be run in the development mode by
excecuting: mvn hpi:run, which builds the projects and starts Hudson
with the plugin installed. This is useful when developing or just
browsing the plugin.

When Hudson is up and running, the installed plugin can be enabled in
the jobs configuration page by checking the Robot Framework checkbox
and optionally specifying the relative path to the Robot reports
directory which is relative to the project/workspace directory.

Now that we got RIDE 0.20 out, we ought to have time to write better
usage instructions and plan also to create 0.1 release package. It
would be great if you had time to look at the current version, give
comments, enhancement instructions, or even contribute some code.

[1] http://code.google.com/p/robotframework-javatools/
[2] https://robotframework-javatools.googlecode.com/svn/hudson-plugin/trunk

k

[regis_desgroppes]

Hi Kari,

On Dec 4, 1:54 pm, Kari Husa <kari.h...@gmail.com> wrote:
...

> Now that we got RIDE 0.20 out, we ought to have time to write better
> usage instructions and plan also to create 0.1 release package. It
> would be great if you had time to look at the current version, give
> comments, enhancement instructions, or even contribute some code.
>

Thank you very, very much: I've just checkout the trunk, then built
and installed the .hpi according to your instructions. Everything is
working so far, and your plugin current state seems very promising, up
to the integration into the job menu, wow!
A few ideas, on the fly:
1. allow multiple paths when looking up for results, similar to
"Archive the artifacts" or "Publish JUnit test result report". That
is, by allowing wildcards like **/AcceptanceTestResult.
2. in the Test Execution Log, replace the path to the Source
(referring to its former location on the build workspace) by a
browsable URL.

Régis

[Pekka Klärck]

2009/12/4 regis_desgroppes <regis.de...@acrolinx.com>:

> On Dec 4, 1:54 pm, Kari Husa <kari.h...@gmail.com> wrote:
>> Now that we got RIDE 0.20 out, we ought to have time to write better
>> usage instructions and plan also to create 0.1 release package. It
>> would be great if you had time to look at the current version, give
>> comments, enhancement instructions, or even contribute some code.
>>
>
> Thank you very, very much: I've just checkout the trunk, then built
> and installed the .hpi according to your instructions. Everything is
> working so far, and your plugin current state seems very promising, up
> to the integration into the job menu, wow!

Great to hear! Kari is not online currently but I texted your first
impressions to him and he was so delighted that he said he'll have a
whiskey to celebrate. I think that's well deserved because although
the current implementation is remarkably simple, he needed to fight
with Hudson's plugin API a lot to get there.

> A few ideas, on the fly:
> 1. allow multiple paths when looking up for results, similar to
> "Archive the artifacts" or "Publish JUnit test result report". That
> is, by allowing wildcards like **/AcceptanceTestResult.
> 2. in the Test Execution Log, replace the path to the Source
> (referring to its former location on the build workspace) by a
> browsable URL.

Sounds reasonable. Could you please submit these to the issue tracker
at [1]? There's no HudsonPlugin component yet but it'll be added as
soon as we get the project really started. If you have Hudson plugin
development knowledge you could even try creating a patch yourself. If
you are interested to collaborate more we can easily give you commit
access (or at least move also this version control from Subversion to
Mercurial to make development w/o commit rights easier).

[1] http://code.google.com/p/robotframework-javatools/issues/list

[Pekka Klärck]

[Ccing the reply to rf-users to keep the discussion public.]

2009/12/4 Regis Desgroppes <regis.de...@acrolinx.com>:

> Pekka Klärck wrote:
>>
>> If you have Hudson plugin
>> development knowledge you could even try creating a patch yourself. If
>> you are interested to collaborate more we can easily give you commit
>> access (or at least move also this version control from Subversion to
>> Mercurial to make development w/o commit rights easier).
>

> I've attempted to work on the Hudson core in the past, but my former company
> didn't let me enough time to carry out something committable. Hudson is a
> framework as such, so another world to learn, even for plugins. And it's in
> Java, while I have more fun with Python and C++ (what a strange mix, isn't
> it?). So I prefer send patches first. If it turns out that they are too
> frequent, then I'll request a commit access.

Yeah, I've only looked at Kari working with Hudson (my Java isn't that
strong either) and it looks somewhat complicated. Some things we
though will be simple (like linking to external results generated by a
build-step) turned out the be really complicated or at least so poorly
documented that the only way to find the right way was by trial and
error. Anyway, if you got anything committable done please submit a
patch, but enhancement request from real users are highly valuable
too.

> Otherwise, I've submitted issue #434 for easy_install'ing under Ubuntu 9.xx
> for which I've attached a patch. I'm wondering if this hacky workaround
> could also fix issue #236 by the same time?

I noticed the issue but didn't realize there was a patch earlier. It
looks interesting - stackframe inspection is always cool - and we'll
definitely incorporate it into the mainline in RF 2.2 unless we decide
to redesign the whole installation and how the framework is started.
This and other similar installation issues are related on how the
pybot/jybot/rebot start-up scripts are manipulated during the
installation. If we could somehow got rid of that altogether these
problems ought to be resolved too. Changing the installation procedure
might be somewhat complicated as long as we support Jython 2.2, which
hopefully is not longer than RF 2.2, as it doesn't have distutils
itself.

Does anyone here have interest on helping with the redesign of
installation and start-up? In that case I could share my high-level
ideas and also explain the problems in greater detail.

[Thomas Jaspers]

Just take a quick look at this, but I was wondering how integrating
Robot Framework with Hudson could be any easier than
what it is already. Maybe I have a simplified setup by installing
Hudson and Robot Framework on the same server, but in that
case I simply run Robot via some shell-script, which I anyway have and
then I collect the generated Reports, etc. as Artefacts
with Hudson and they show then up in the Hudson Workspace of that
project. From there I can then simply click them and open
the Reports.

If tests are stored to Subversion I create a second Hudson project to
check the sources out and compile them using Hudson.
Then I know where to find everything and adept my Robot-Job on Hudson
accordingly. Lately I was also generating some
javadoc there for my keywords that could then be browsed directly via
Hudson.

I am somehow sure I am missing a point here as you guys for sure knoiw
what you are doing here ;), but I really like
integration of Robot Framework and Hudson already a lot ... it is just
great :-).

On 7 Dez., 00:58, Pekka Klärck <pekka.kla...@gmail.com> wrote:
> [Ccing the reply to rf-users to keep the discussion public.]
>

> 2009/12/4 Regis Desgroppes <regis.desgrop...@acrolinx.com>:

[Pekka Klärck]

2009/12/10 Thomas Jaspers <jas...@codecentric.de>:

> Just take a quick look at this, but I was wondering how integrating
> Robot Framework with Hudson could be any easier than
> what it is already. Maybe I have a simplified setup by installing
> Hudson and Robot Framework on the same server, but in that
> case I simply run Robot via some shell-script, which I anyway have and
> then I collect the generated Reports, etc. as Artefacts
> with Hudson and they show then up in the Hudson Workspace of that
> project. From there I can then simply click them and open
> the Reports.

If I've gotten it right, the only thing the plugin currently provides
in addition to what you are already doing is that the links to reports
and logs work also from old builds and not only from the workspace.
I've never really used Hudson myself so I might be missing something
too. Kari as the author or Régis as the first user might be able to
comment more.

Cheers,
.peke

[Thomas Jaspers]

Well, actually with "my" approach old report files are stored as well,
as Hudson is automatically
archiving the named artefacts (named by a file pattern). In the
workspace it is then showing the
report files from the last successful build and when clicking on a
specific build it is showing
the reports for that build :).

Maybe I can write some blog entry on this next week when things
hopefully get a bit
more calm @work.

Cheers
- Thomas

On 10 Dez., 23:05, Pekka Klärck <pekka.kla...@gmail.com> wrote:
> 2009/12/10 Thomas Jaspers <jasp...@codecentric.de>:

[regis_desgroppes]

Hi,
I did exactly the same as Thomas before discovering the Hudson plugin
for RF.
The Hudson plugin for RF adds a specific link to job page that allows
to directly jump to the reports. That's cool.
However, compared to xUnit test reports, there are 2 lacks:
1. it's possible to specify one directory only for test results.
Wildcards are not yet supported, see issue:
http://code.google.com/p/robotframework-javatools/issues/detail?id=28
2. Hudson is not aware of the test results as such, so it cannot
compute trends as it does for other test frameworks (JUnit, Python
unittest, Fit, Boost.Test). I will start working on either a XSL-
transformation or another built-in serializer. I'm tempted to favor
the second approach since I really appreciated the similar effort that
the Python nose guys made, see http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/xunit.html
Cheers,
- Rej

[Pekka Klärck]

2009/12/11 regis_desgroppes <regis.de...@acrolinx.com>:

>
> 2. Hudson is not aware of the test results as such, so it cannot
> compute trends as it does for other test frameworks (JUnit, Python
> unittest, Fit, Boost.Test). I will start working on either a XSL-
> transformation or another built-in serializer. I'm tempted to favor
> the second approach since I really appreciated the similar effort that
> the Python nose guys made, see http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/xunit.html

Being able to create xUnit compatible XML output would be really nice
for Hudson integration and other purposes. I'd still consider creating
a separate tool that can create these outputs from the normal XML
output because that's easier than adding a new serializer and doesn't
require new RF release. If you are sure the new serializer is better I
promise to help integrating it into the core, though.