Output message to console

[Finn Neuik]

Hi there - I'm trying to output a message to the console when our tests are running.

We already have our Robot tests integrated with our CI server (TeamCity) by transforming the report XML which gives us pass/fail and counts of each but I'd like to extend this by reporting which tests are running, how long each takes, etc whilst the tests are executing.I can do this by outputing messages to the console by putting the following in the test setup and teardown respectively:

##teamcity[testStarted name='${TEST_NAME}' captureStandardOutput='true']
##teamcity[testFinished name='${TEST_NAME}']

I've tried using the Log keyword with the level set to WARN, but then these messages will appear as errors in the log. Is there any way of sending raw output to the console without this happening (I don't really even need these in the log).

Thanks
Finn

[Pekka Klärck]

2010/9/16 Finn Neuik <fin...@gmail.com>:

There are at least two ways to do this:

1) Implement a keyword that writes messages to stderr stream. These
messages are echoed to the console as well as written to the log.

2) Implement a listener that prints this information. Outputs are not
captured when listeners are executed so whatever they print to stdout
or stderr will end up to the console directly. Listeners also have
information about how long a test took readily available. For more
information about listeners see:
http://robotframework.googlecode.com/svn/tags/robotframework-2.5.3/doc/userguide/RobotFrameworkUserGuide.html#using-listener-interface

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Pekka Klärck]

Hi Finn,

Great that you got this working. If I remember correctly, others have
also integrated Robot Framework with TeamCity. Is there enough generic
in the integration that a separate TeamCity plugin could be created?

Thanks also for the kind words. Always nice to hear positive feedback.

Cheers,
.peke, a Finn too =)

2010/9/16 Finn Neuik <fin...@gmail.com>:
> Yep, thanks. Following Bryan's advice I put together a listener which only
> took a couple of minutes (that part of the docs must have been in a blind
> spot for me). TeamCity correctly reads in the tests (I now get details of
> test timing and the names of the tests) - but only reports them through the
> UI once the nant task I'm using to run the tests completes so it hasn't
> given me exactly what I was after. I'll enquire on the TeamCity forums to
> see what I'm missing (maybe I'm missing a NAnt option or perhaps I need to
> report the suits as well) as the Robot side seems fine now.
>
> Thanks for all the work you guys have put in - I've been using RF for over a
> year now and I'm a big fan!
>
> Cheers
> Finn

[Pekka Klärck]

[Replying also to the list.]

2010/9/17 Finn Neuik <fin...@gmail.com>:
> I think a listener would be pretty generic - currently we're using XSLT to
> transform the test output into the results which TeamCity requires but
> looking at the listeners we can do everything using those instead. This
> would also have the advantage that I can remove the pause I need before
> doing the transform whilst waiting for the report to finishing spooling.

Sounds good.

> I've come across a couple of issues - first of all the test results are only
> getting reported at the end of the test run rather than in realtime. I think
> is probably just something I'm missing in what TeamCity requires so should
> be easy enough to solve.

Listeners allow real time updates so this sounds like an issue with
TeamCity integration. Hope you can resolve it.

> The other is that some of our test names have characters which cannot be
> output to the windows console using the straight print statement due to code
> page issues, so I'll need to transform the output I guess.

Do you necessarily need to go through the console? Could you use files
instead? Or does TeamCity have an API you could you directly from the
listener? That would probably be the best solution.

> I'll try and solve these issues next week and share the results then.

Cool! If you come up with something generic, are you interested in
writing a wiki page about TeamCity integration? If there's some
generic code, starting a separate project might be even better option.

> Cheers
> Finn - the world can't have enough Finns I think!

Must agree with that! =)

[Pekka Klärck]

---------- Forwarded message ----------
From: Finn Neuik <fin...@gmail.com>
Date: 2010/9/23
Subject: Re: Output message to console
To: Pekka Klärck <pe...@iki.fi>

Well I got it all working earlier this week - the problem seems to
have been the "[exec]" text which NAnt prefixes console output with.
As soon as I swapped to MsBuild everything worked first time. I'll try
and get something written up soon and get it over to you.

On 18 September 2010 12:46, Pekka Klärck <pe...@iki.fi> wrote:
>
> [Replying also to the list.]
>
> 2010/9/17 Finn Neuik <fin...@gmail.com>:
> > I think a listener would be pretty generic - currently we're using XSLT to
> > transform the test output into the results which TeamCity requires but
> > looking at the listeners we can do everything using those instead. This
> > would also have the advantage that I can remove the pause I need before
> > doing the transform whilst waiting for the report to finishing spooling.
>
> Sounds good.
>
> > I've come across a couple of issues - first of all the test results are only
> > getting reported at the end of the test run rather than in realtime. I think
> > is probably just something I'm missing in what TeamCity requires so should
> > be easy enough to solve.
>
> Listeners allow real time updates so this sounds like an issue with
> TeamCity integration. Hope you can resolve it.
>
> > The other is that some of our test names have characters which cannot be
> > output to the windows console using the straight print statement due to code
> > page issues, so I'll need to transform the output I guess.
>
> Do you necessarily need to go through the console? Could you use files
> instead? Or does TeamCity have an API you could you directly from the
> listener? That would probably be the best solution.
>

Calling an API would be much better, but as far as I can tell to
communicate with teamcity you need to either write a custom runner
(which would be great, but I don't have the time for that at the
moment, especially as I now have the result I need), via an xml file
(which is how we were integrating until this week, but this only gives
results at the very end of the run) or via the console - which gives
realtime details about what's happening as the tests are running.

[DMW]

Hello gentlemen,

I would just like to mention we are looking at options for TeamCity /
RF integration as well, and if anything exists that could be shared,
we would be most interested and thankful :)

Regards,

Dave Woods

On Sep 22, 5:45 pm, Pekka Klärck <p...@iki.fi> wrote:
> ---------- Forwarded message ----------
> From: Finn Neuik <fin...@gmail.com>
> Date: 2010/9/23
> Subject: Re: Output message to console
> To: Pekka Klärck <p...@iki.fi>
>
> Well I got it all working earlier this week - the problem seems to
> have been the "[exec]" text which NAnt prefixes console output with.
> As soon as I swapped to MsBuild everything worked first time. I'll try
> and get something written up soon and get it over to you.
>

> On 18 September 2010 12:46, Pekka Klärck <p...@iki.fi> wrote:
>
> > [Replying also to the list.]
>
> > 2010/9/17 Finn Neuik <fin...@gmail.com>:
> > > I think a listener would be pretty generic - currently we're using XSLT to

> > > transform the test output into the results whichTeamCityrequires but

> > > looking at the listeners we can do everything using those instead. This
> > > would also have the advantage that I can remove the pause I need before
> > > doing the transform whilst waiting for the report to finishing spooling.
>
> > Sounds good.
>
> > > I've come across a couple of issues - first of all the test results are only
> > > getting reported at the end of the test run rather than in realtime. I think

> > > is probably just something I'm missing in whatTeamCityrequires so should

> > > be easy enough to solve.
>
> > Listeners allow real time updates so this sounds like an issue with

> >TeamCityintegration. Hope you can resolve it.

>
> > > The other is that some of our test names have characters which cannot be
> > > output to the windows console using the straight print statement due to code
> > > page issues, so I'll need to transform the output I guess.
>
> > Do you necessarily need to go through the console? Could you use files

> > instead? Or doesTeamCityhave an API you could you directly from the

> > listener? That would probably be the best solution.
>
> Calling an API would be much better, but as far as I can tell to

> communicate withteamcityyou need to either write a custom runner

> (which would be great, but I don't have the time for that at the
> moment, especially as I now have the result I need), via an xml file
> (which is how we were integrating until this week, but this only gives
> results at the very end of the run) or via the console - which gives
> realtime details about what's happening as the tests are running.
>
> > > I'll try and solve these issues next week and share the results then.
>
> > Cool! If you come up with something generic, are you interested in

> > writing a wiki page aboutTeamCityintegration? If there's some