Re: [easytesting] Re: Timing Problems with optionPane.requireMessage()

[Eric Kolotyluk]

So I noticed the default timeout was 100 ms. I increased the timeout to 1 second, same problem. I increased the timeout to 10 seconds, same problem. I suspect something else is at play here.

Any more ideas of something to try?

Cheers, Eric

----- Original Message -----
From: "Alex Ruiz" <alex.r...@gmail.com>
To: easyt...@googlegroups.com
Sent: Friday, June 5, 2009 10:58:14 AM GMT -08:00 US/Canada Pacific
Subject: [easytesting] Re: Timing Problems with optionPane.requireMessage()

Hi Eric,

It seems to be a timing issue. If you are using the latest release (1.2a2), there is an overloaded verstion of 'optionPane' in ContainerFixture that takes a timeout. Please give it a try and let me know if that helped.

Cheers,
-Alex

On Thu, Jun 4, 2009 at 3:31 PM, Eric Kolotyluk <er...@sfu.ca> wrote:

I have a test that calls OptionPane.requireMessage(). On my desktop the test always works fine. But on our build machine sometimes the test fails because

[junit] Testcase: isNameEmptyErrorMessageDisplayed took 1.093 sec

    [junit]     Caused an ERROR

    [junit] Timed out waiting for option pane to be found using matcher org.fest

.swing.core.TypeMatcher[type=javax.swing.JOptionPane, requireShowing=true]

    [junit] org.fest.swing.exception.WaitTimedOutError: Timed out waiting for op

tion pane to be found using matcher org.fest.swing.core.TypeMatcher[type=javax.s

wing.JOptionPane, requireShowing=true]

    [junit]     at org.fest.swing.timing.Pause.pause(Pause.java:74)

    [junit]     at org.fest.swing.timing.Pause.pause(Pause.java:58)

    [junit]     at org.fest.swing.fixture.ContainerFixture.optionPane(ContainerF

ixture.java:264)

    [junit]     at org.fest.swing.fixture.ContainerFixture.optionPane(ContainerF

ixture.java:256)

    [junit]     at com.kodak.kps.notificationManager.RecipientEditorUnitTests.is

NameEmptyErrorMessageDisplayed(Unknown Source)

    [junit]

    [junit] Test com.kodak.kps.notificationManager.RecipientEditorUnitTests FAIL

ED

We run the build under TeamCity. It is interesting that if the build runs as a Windows service this test always fails, but when run as a normal user login it sometimes fails.

Any advice would be appreciated.

Cheers, Eric

[Eric Kolotyluk]

So we've learned a bit more, but still have no solution.

We have noticed that there seem to be focus problems when the unit test is running - in particular if the user clicks the mouse in different windows (presumably stealing focus) we can get the test to timeout the same way as it fails when running under a TeamCity build agent. We suspect there is some sort of focus problems when running under the build agent, but we can't imagine what. We've sent e-mail to the JetBrains TeamCity list to see if anyone else there knows.

My question to the FEST community is has any one else has similar problems when running FEST tests under build agents?

What is equally strange is that our logs show the test passed on two occasions when running under TeamCity - and we don't know why. We have also noticed that the tests take a lot longer when running under TeamCity (> 10 seconds) but are fast when run live (~ 1 second). If I increase the timeout on the failing test, the test runs longer (> 20 seconds) and times out anyway.

Below is the source of the test that is failing under TeamCity

Cheers, Eric

    @Test
    public void isVisible()    // This test always succeeds
    {
        dialogFixture.requireVisible();
    }

    @Test
    public void isNameEmptyErrorMessageDisplayed() // Fails most of the time under TeamCity
    {
        dialogFixture.button("okButton").click();
        JOptionPaneFixture optionPane = dialogFixture.optionPane();
        //JOptionPaneFixture optionPane = dialogFixture.optionPane(Timeout.timeout(10000));
        optionPane.requireErrorMessage();
        optionPane.requireTitle(Messages.getString("title.DataEntryError"));
        optionPane.requireMessage(Messages.getString("NotificationManager.message.TheRecipientNameFieldIsBlank"));

[Peter Murray]

What user is the TeamCity buildagent running under? Local System or a user account? What version of Windows are you using?

To get all FEST Swing tests to pass under TeamCity for us we needed to run the build agent service under a user account on Windows (and not a domain user account). We have only ever successfully managed to get  TeamCity buildagents running as a service and passing FEST Swing tests on Windows XP, later versions of windows force services to use another desktop, not the actual users desktop, due to security concerns and this causes focus problems (among other things).

Another important thing is to disable the screensaver and automatic locking of the PC when left unattended as this has caused us problems in the past.

As an aside, it might be worth taking a screenshot from within your tests upon failure and adding the screenshot(s) to the build artifacts in TeamCity as these are invaluable when trying to understand failures from the TeamCity build agents (most of the time things are not as they seem, or you think they should be).

 - Peter

[OlivierD]

HI Erick,

What version are you using?
We had some similar problems under Hudson using xvnc.
We found a fix, which is now in fest-swing-1.2a2.

Hope this help,

Thanks,

Olivier

On Jun 9, 10:55 am, Peter Murray <plm...@googlemail.com> wrote:
> What user is the TeamCity buildagent running under? Local System or a user
> account? What version of Windows are you using?
>
> To get all FEST Swing tests to pass under TeamCity for us we needed to run
> the build agent service under a user account on Windows (and not a domain
> user account). We have only ever successfully managed to get  TeamCity
> buildagents running as a service and passing FEST Swing tests on Windows XP,
> later versions of windows force services to use another desktop, not the
> actual users desktop, due to security concerns and this causes focus
> problems (among other things).
>
> Another important thing is to disable the screensaver and automatic locking
> of the PC when left unattended as this has caused us problems in the past.
>
> As an aside, it might be worth taking a screenshot from within your tests
> upon failure and adding the screenshot(s) to the build artifacts in TeamCity
> as these are invaluable when trying to understand failures from the TeamCity
> build agents (most of the time things are not as they seem, or you think
> they should be).
>
>  - Peter
>

[Eric Kolotyluk]

The build agent is running under a user account.

We are using Windows XP SP3.

We are using FEST-Swing 1.2a2

I'll have to figure out how to take a screenshot on failure - I hope it's not hard to set up.

Cheers, Eric

[OlivierD]

Hi Eric,

Taking screenshot on failure, is normally done automatically.
Taking a screenshot when you want, for example just before clicking on
a button is easy.

We have this small api, which allow us to create screenshot. Just
change the myPath part for your project. The parameter you give is the
image name. In case you want to take several screen shot during tests.

public static void doScreenShot(String imgName) {
File currentFolder = new File("myPath");
ScreenshotTaker screenshotTaker = new ScreenshotTaker();

File imageFolder = new File(currentFolder.getAbsolutePath(),
"failed-tests");
if (!imageFolder.exists()) {
imageFolder.mkdirs();
}
File tmpFile = new File(imageFolder.getAbsolutePath(),
imgName);
if (tmpFile.exists()) {
tmpFile.delete();
}
screenshotTaker.saveDesktopAsPng(tmpFile.getAbsolutePath());
}


Hope this help,

Olivier

On Jun 9, 4:06 pm, Eric Kolotyluk <e...@sfu.ca> wrote:
> The build agent is running under a user account.
> We are using Windows XP SP3.
> We are using FEST-Swing 1.2a2
> I'll have to figure out how to take a screenshot on failure - I hope it's not hard to set up.
> Cheers, Eric
> Peter Murray wrote:What user is the TeamCity buildagent running under? Local System or a user account? What version of Windows are you using?
> To get all FEST Swing tests to pass under TeamCity for us we needed to run the build agent service under a user account on Windows (and not a domain user account). We have only ever successfully managed to get  TeamCity buildagents running as a service and passing FEST Swing tests on Windows XP, later versions of windows force services to use another desktop, not the actual users desktop, due to security concerns and this causes focus problems (among other things).
> Another important thing is to disable the screensaver and automatic locking of the PC when left unattended as this has caused us problems in the past.
> As an aside, it might be worth taking a screenshot from within your tests upon failure and adding the screenshot(s) to the build artifacts in TeamCity as these are invaluable when trying to understand failures from the TeamCity build agents (most of the time things are not as they seem, or you think they should be).

>  - PeterOn Mon, Jun 8, 2009 at 7:38 PM, Eric Kolotyluk<er...@sfu.ca>wrote:So we've learned a bit more, but still have no solution.

> We have noticed that there seem to be focus problems when the unit test is running - in particular if the user clicks the mouse in different windows (presumably stealing focus) we can get the test to timeout the same way as it fails when running under a TeamCity build agent. We suspect there is some sort of focus problems when running under the build agent, but we can't imagine what. We've sent e-mail to the JetBrains TeamCity list to see if anyone else there knows.
> My question to the FEST community is has any one else has similar problems when running FEST tests under build agents?
> What is equally strange is that our logs show the test passed on two occasions when running under TeamCity - and we don't know why. We have also noticed that the tests take a lot longer when running under TeamCity (> 10 seconds) but are fast when run live (~ 1 second). If I increase the timeout on the failing test, the test runs longer (> 20 seconds) and times out anyway.
> Below is the source of the test that is failing under TeamCity
> Cheers, Eric
>     @Test
>     public void isVisible()    // This test always succeeds
>     {
>         dialogFixture.requireVisible();
>     }
>     @Test
>     public void isNameEmptyErrorMessageDisplayed() // Fails most of the time under TeamCity
>     {
>         dialogFixture.button("okButton").click();
>         JOptionPaneFixture optionPane = dialogFixture.optionPane();
>         //JOptionPaneFixture optionPane = dialogFixture.optionPane(Timeout.timeout(10000));
>         optionPane.requireErrorMessage();
>         optionPane.requireTitle(Messages.getString("title.DataEntryError"));
>         optionPane.requireMessage(Messages.getString("NotificationManager.message.TheRecipientNameFieldIsBlank"));
>     }
> ----- Original Message -----
> From: "Eric Kolotyluk" <er...@sfu.ca>
> To:easyt...@googlegroups.com
>
> Sent: Monday, June 8, 2009 8:05:06 AM GMT -08:00 US/Canada Pacific

> Subject: [easytesting] Re: Timing Problems with optionPane.requireMessage()So I noticed the default timeout was 100 ms. I increased the timeout to 1 second, same problem. I increased the timeout to 10 seconds, same problem. I suspect something else is at play here.

> Any more ideas of something to try?
> Cheers, Eric
> ----- Original Message -----
> From: "Alex Ruiz" <alex.r...@gmail.com>
> To:easyt...@googlegroups.com
> Sent: Friday, June 5, 2009 10:58:14 AM GMT -08:00 US/Canada Pacific
> Subject: [easytesting] Re: Timing Problems with optionPane.requireMessage()
> Hi Eric,
> It seems to be a timing issue. If you are using the latest release (1.2a2), there is an overloaded verstion of 'optionPane' in ContainerFixture that takes a timeout. Please give it a try and let me know if that helped.
> Cheers,

> -AlexOn Thu, Jun 4, 2009 at 3:31 PM, Eric Kolotyluk<er...@sfu.ca>wrote:I have a test that calls OptionPane.requireMessage(). On my desktop the test always works fine. But on our build machine sometimes the test fails because

>
> [junit] Testcase: isNameEmptyErrorMessageDisplayed took 1.093 sec
>
>     [junit]     Caused an ERROR
>
>     [junit] Timed out waiting for option pane to be found using matcher org.fest
>
> .swing.core.TypeMatcher[type=javax.swing.JOptionPane, requireShowing=true]
>
>     [junit] org.fest.swing.exception.WaitTimedOutError: Timed out waiting for op
>
> tion pane to be found using matcher org.fest.swing.core.TypeMatcher[type=javax.s
>
> wing.JOptionPane, requireShowing=true]
>
>     [junit]     at org.fest.swing.timing.Pause.pause(Pause.java:74)
>
>     [junit]     at org.fest.swing.timing.Pause.pause(Pause.java:58)
>
>     [junit]     at org.fest.swing.fixture.ContainerFixture.optionPane(ContainerF
>
> ixture.java:264)
>
>     [junit]     at org.fest.swing.fixture.ContainerFixture.optionPane(ContainerF
>
> ixture.java:256)
>
>     [junit]     atcom.kodak.kps.notificationManager.RecipientEditorUnitTests.is
>

> ...
>
> read more »

[Eric Kolotyluk]

If there is a screenshot automatically on failure, where do I find it?

[OlivierD]

Actually it's one option in my ant file

<formatter
classname="org.fest.swing.junit.ant.ScreenshotOnFailureResultFormatter"
extension=".xml" />

I think that's all all is needed.

Olivier

> > - PeterOn Mon, Jun 8, 2009 at 7:38 PM, Eric Kolotyluk<e...@sfu.ca>wrote:So we've learned a bit more, but still have no solution.

> > -AlexOn Thu, Jun 4, 2009 at 3:31 PM, Eric Kolotyluk<e...@sfu.ca>wrote:I have a test that calls OptionPane.requireMessage(). On my desktop the test always works fine. But on our build machine sometimes the test fails because

[Eric Kolotyluk]

I've been reading

http://docs.codehaus.org/display/FEST/Taking+Screenshots+of+JUnit+Test+Failures

but it seems to be more complicated - I'm still struggling to actually get a screen shot.

[OlivierD]

If you are using Ant it's quite simple.
How are you running your test today? I actually copy what is in that
document and change with my value.


Olivier

On Jun 9, 8:40 pm, Eric Kolotyluk <e...@sfu.ca> wrote:
> I've been reading
>

> http://docs.codehaus.org/display/FEST/Taking+Screenshots+of+JUnit+Tes...

[Eric Kolotyluk]

Yes, I am using Ant. Below is currently what I'm using. I can see a bunch of XML files generated, but nothing that looks like a screen shot, and nothing in the html directory.

Cheers, Eric

    <target name="test" depends="compile" description="Runs all the JUnit test suites">
        <delete dir="test/report"/>
        <mkdir dir="test/report"/>
        <mkdir dir="test/report/html"/>
       
        <taskdef resource="festjunittasks">
              <classpath>
                  <fileset dir="test/lib">
                      <include name="**/*.jar"/>
                  </fileset>
              </classpath>
        </taskdef>

        <junit printsummary="off" showoutput="off" failureproperty="test.failed">
              <classpath>
                  <pathelement location="build/bin"/>
                  <pathelement location="resources"/>
                  <path refid="build.base.classpath"/>
                  <fileset dir="test/lib">
                      <include name="**/*.jar"/>
                  </fileset>
              </classpath>
            <formatter type="plain" usefile="false"/>

            <formatter classname="org.fest.swing.junit.ant.ScreenshotOnFailureResultFormatter" extension=".xml" />

            <batchtest todir="test/report">
                <fileset dir="build/bin" includes="**/*UnitTests.class"/>
            </batchtest>
        </junit>

        <festreport todir="test/report">
              <classpath>
                  <fileset dir="test/lib">
                      <include name="**/*.jar"/>
                  </fileset>
              </classpath>
            <fileset todir="test/report">
              <include name="TEST-*.xml" />
            </fileset>
            <report format="frames" todir="test/report/html" />
        </festreport>

        <fail if="test.failed" message="Unit Test Failure!"/>
    </target>

[OlivierD]

My junit line is
<junit forkmode="perBatch" printsummary="yes" haltonfailure="no"
haltonerror="no" dir="${tests.running.dir}" maxmemory="512M">
I only had one formater
my batchtest
<batchtest fork="yes" todir="test/report">


Olivier

[Eric Kolotyluk]

So do you have a festreport task defined too?

Looking at my ant run, it does not indicate that mine is even running. Earlier junit was complaining that festreport was undefined because I did not have my taskdef set up properly. Now it's stopped complaining, but does not seem to indicate it is running.

I've changed my junit and batchtest to use fork, but that does not seem to make a difference.

[Eric Kolotyluk]

OK, never mind that previous part - I got my screen-shot working! - the fileset element uses dir and not todir.

Now that I have my screen-shot - the UI of my application does not show in the screen-shot. The screen-shot was taken either too early or too late - I can't tell. Is there any way to control the synchronization?

[Eric Kolotyluk]

OK, I ran my tests under the TeamCity build agent and the screen shot is a blank screen.

Is this normal?

Is there any screen when running an automated build under a build agent?

[OlivierD]

I don't know anything about TeamCity. Do you script end "normally", or
does it crash the JVM?
Try to use the previous API I gave you to take a screenshot, just to
check if you can see anything.

Olivier

> ...
>
> read more »

[Peter Murray]

You say a blank screen, do you mean black? If so that is an indication that you are running the buildagent as a service under the local system account.

It is not possible to run all FEST Swing tests under the local system account. Believe me I have tried.

Try starting the buildagent service using a local user account (an account you have previously logged on with) and rerun the tests. When the buildagent is running with a local user account you can actually watch the FEST tests running by logging is as the user and starting a TeamCity build.

- Peter

Sent from my BlackBerry® wireless device

---

From: Eric Kolotyluk
Date: Tue, 9 Jun 2009 15:19:27 -0700 (PDT)
To: <easyt...@googlegroups.com>

[Eric Kolotyluk]

OK, we finally figured out the problem.

We were starting our build agent in Windows with cmd /c which causes the parent of the build agent to terminate, consequently orphaning the build agent process.

When we start our build agent with cmd /k, it keeps the parent command window around so that the build agent still has a parent ulitimately rooted under explorer.exe and has an attached desktop.

Phew - that was a subtle problem to figure out. Thanks Peter - your comments were useful in helping us get to the root of the problem.

[Eric Kolotyluk]

This never seems to end...

We discovered another problem, if you allow Remote Desktop and someone logs in to the build machine, Remote Desktop clears the desktop when the session is ended - screwing up the environment for FEST tests in the build. We think we can solve this by disabling Remote Desktop and using VNC for remote login, since VNC will leave you logged in even after you end your session.

I'm wondering if we should document these experiences somewhere in the FEST documentation so other people don't have to relive these issues. Maybe a special section somewhere on running FEST in build environments. Would the wiki be the best place for this?

[Alex Ruiz]

Hi Eric,

I was thinking exactly the same! it would be very useful to document your experiences (as well as Peter's and Olivier's) in the project's Wiki. I'd truly appreciate if any of you could document this. I think our options are:

1. Send me a document and I'll add the contents to the Wiki
2. Create an account at http://docs.codehaus.org , and send me an e-mail so I can give you 'write' permissions

Please pick the one that is the easiest for you :)

Many thanks!
-Alex

[Eric Kolotyluk]

I think the most important lessons I learned from this are:

During automated builds

1) Make sure there is a desktop running for the build agent
2) On windows, disable Remote Desktop because it can kill the desktop the build agent is using - use VNC instead because it won't log you out (if you configure it that way)
3) On windows, use a local user account for the build agent, not a service account
4) On windows, if you launch the build agent with the 'cmd' command use the '/k' option not the '/c' option - this prevents the command window from closing and orphaning the build agent

Cheers, Eric

[Alex Ruiz]

Thanks Eric!

I'll be adding your tips to the wiki shortly :)

Regards,
-Alex