Why are these tests failing occasionally?

[Chris Merrill]

While investigating some of my tests that fail only when running headless, I created the attached test class that fails inconsistently (headless or not). There are 20 tests, but it is really 2 tests, each wrapped in 10 identical @Test methods. In a typical run, one or two tests will fail. Sometimes more, sometimes none. The failures produce 2 different ConcurrentModificationExceptions. I've attached each stacktrace. Each comes from deep inside JavaFX when my test calls FxRobot.clickOn().

My questions are:
1) can you reproduce this?
2) is there something wrong with my test?

TestFX 4.0.4-alpha
Java jdk1.8.0_111

TIA!
Chris


java.lang.RuntimeException: java.util.ConcurrentModificationException
    at org.testfx.service.query.impl.CallableBoundsPointQuery.fetchCallableBounds(CallableBoundsPointQuery.java:65)
    at org.testfx.service.query.impl.CallableBoundsPointQuery.query(CallableBoundsPointQuery.java:49)
    at org.testfx.robot.impl.MoveRobotImpl.moveTo(MoveRobotImpl.java:72)
    at org.testfx.robot.impl.ClickRobotImpl.clickOn(ClickRobotImpl.java:70)
    at org.testfx.api.FxRobot.clickOn(FxRobot.java:684)
    at org.testfx.api.FxRobot.clickOn(FxRobot.java:750)
    at HeadlessFailureControlClick.tabKeyTest(HeadlessFailureControlClick.java:141)
    at HeadlessFailureControlClick.tabKey4(HeadlessFailureControlClick.java:106)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
Caused by: java.util.ConcurrentModificationException
    at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:386)
    at java.util.AbstractList$Itr.next(AbstractList.java:355)
    at com.sun.javafx.scene.shape.PathUtils.configShape(PathUtils.java:45)
    at javafx.scene.shape.Path.impl_configShape(Path.java:251)
    at javafx.scene.shape.Path.impl_configShape(Path.java:89)
    at javafx.scene.shape.Shape.impl_computeGeomBounds(Shape.java:864)
    at javafx.scene.Node.updateGeomBounds(Node.java:3579)
    at javafx.scene.Node.getGeomBounds(Node.java:3532)
    at javafx.scene.Node.getLocalBounds(Node.java:3480)
    at javafx.scene.Node.updateTxBounds(Node.java:3643)
    at javafx.scene.Node.getTransformedBounds(Node.java:3426)
    at javafx.scene.Parent.createCachedBounds(Parent.java:1664)
    at javafx.scene.Parent.recomputeBounds(Parent.java:1539)
    at javafx.scene.Parent.impl_computeGeomBounds(Parent.java:1388)
    at javafx.scene.layout.Region.impl_computeGeomBounds(Region.java:3078)
    at javafx.scene.Node.updateGeomBounds(Node.java:3579)
    at javafx.scene.Node.getGeomBounds(Node.java:3532)
    at javafx.scene.Node.computeLocalBounds(Node.java:3595)
    at javafx.scene.Node.updateLocalBounds(Node.java:3625)
    at javafx.scene.Node.getLocalBounds(Node.java:3486)
    at javafx.scene.Node.updateTxBounds(Node.java:3643)
    at javafx.scene.Node.getTransformedBounds(Node.java:3426)
    at javafx.scene.Parent.getChildTransformedBounds(Parent.java:1732)
    at javafx.scene.Parent.recomputeBounds(Parent.java:1524)
    at javafx.scene.Parent.impl_computeGeomBounds(Parent.java:1388)
    at javafx.scene.layout.Region.impl_computeGeomBounds(Region.java:3078)
    at javafx.scene.Node.updateGeomBounds(Node.java:3579)
    at javafx.scene.Node.getGeomBounds(Node.java:3532)
    at javafx.scene.Node.getLocalBounds(Node.java:3480)
    at javafx.scene.Node$MiscProperties$2.computeBounds(Node.java:6474)
    at javafx.scene.Node$LazyBoundsProperty.get(Node.java:9308)
    at javafx.scene.Node$LazyBoundsProperty.get(Node.java:9278)
    at javafx.scene.Node.getBoundsInLocal(Node.java:3158)
    at org.testfx.service.locator.impl.BoundsLocatorImpl.boundsInSceneFor(BoundsLocatorImpl.java:38)
    at org.testfx.service.locator.impl.BoundsLocatorImpl.boundsOnScreenFor(BoundsLocatorImpl.java:60)
    at org.testfx.service.locator.impl.PointLocatorImpl.lambda$callableBoundsFor$0(PointLocatorImpl.java:89)
    at org.testfx.service.query.impl.CallableBoundsPointQuery.fetchCallableBounds(CallableBoundsPointQuery.java:62)
    ... 31 more



java.util.ConcurrentModificationException
    at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:386)
    at java.util.AbstractList$Itr.next(AbstractList.java:355)
    at com.sun.javafx.scene.shape.PathUtils.configShape(PathUtils.java:45)
    at javafx.scene.shape.Path.impl_configShape(Path.java:251)
    at javafx.scene.shape.Path.impl_configShape(Path.java:89)
    at javafx.scene.shape.Shape.impl_computeGeomBounds(Shape.java:864)
    at javafx.scene.Node.updateGeomBounds(Node.java:3579)
    at javafx.scene.Node.getGeomBounds(Node.java:3532)
    at javafx.scene.Node.getLocalBounds(Node.java:3480)
    at javafx.scene.Node.updateTxBounds(Node.java:3643)
    at javafx.scene.Node.getTransformedBounds(Node.java:3426)
    at javafx.scene.Parent.createCachedBounds(Parent.java:1664)
    at javafx.scene.Parent.updateCachedBounds(Parent.java:1554)
    at javafx.scene.Parent.recomputeBounds(Parent.java:1535)
    at javafx.scene.Parent.impl_computeGeomBounds(Parent.java:1388)
    at javafx.scene.layout.Region.impl_computeGeomBounds(Region.java:3078)
    at javafx.scene.Node.updateGeomBounds(Node.java:3579)
    at javafx.scene.Node.getGeomBounds(Node.java:3532)
    at javafx.scene.Node.computeLocalBounds(Node.java:3595)
    at javafx.scene.Node.updateLocalBounds(Node.java:3625)
    at javafx.scene.Node.getLocalBounds(Node.java:3486)
    at javafx.scene.Node.updateTxBounds(Node.java:3643)
    at javafx.scene.Node.getTransformedBounds(Node.java:3426)
    at javafx.scene.Parent.getChildTransformedBounds(Parent.java:1732)
    at javafx.scene.Parent.recomputeBounds(Parent.java:1524)
    at javafx.scene.Parent.impl_computeGeomBounds(Parent.java:1388)
    at javafx.scene.layout.Region.impl_computeGeomBounds(Region.java:3078)
    at javafx.scene.Node.updateGeomBounds(Node.java:3579)
    at javafx.scene.Node.getGeomBounds(Node.java:3532)
    at javafx.scene.Node.getLocalBounds(Node.java:3480)
    at javafx.scene.Node$MiscProperties$2.computeBounds(Node.java:6474)
    at javafx.scene.Node$LazyBoundsProperty.get(Node.java:9308)
    at javafx.scene.Node$LazyBoundsProperty.get(Node.java:9278)
    at javafx.scene.Node.getBoundsInLocal(Node.java:3158)
    at org.testfx.util.NodeQueryUtils.isNodeWithinSceneBounds(NodeQueryUtils.java:215)
    at org.testfx.util.NodeQueryUtils.isNodeVisible(NodeQueryUtils.java:207)
    at org.testfx.util.NodeQueryUtils.lambda$isVisible$37(NodeQueryUtils.java:116)
    at com.google.common.collect.Iterators$7.computeNext(Iterators.java:652)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:314)
    at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:300)
    at com.google.common.collect.FluentIterable.toSet(FluentIterable.java:396)
    at org.testfx.service.query.impl.NodeQueryImpl.match(NodeQueryImpl.java:113)
    at org.testfx.api.FxRobot.queryVisibleNode(FxRobot.java:1171)
    at org.testfx.api.FxRobot.pointOfVisibleNode(FxRobot.java:1138)
    at org.testfx.api.FxRobot.clickOn(FxRobot.java:750)
    at HeadlessFailureControlClick.controlATest(HeadlessFailureControlClick.java:75)
    at HeadlessFailureControlClick.controlA0(HeadlessFailureControlClick.java:16)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)

[Chris Merrill]

Sorry...forgot to attach the code!  SporadicFailures.java is attached.

[Benjamin Gudehus]

Hi Chris,

so you say SporadicFailures.java fails (inconsistently) on headless (Glass robot) and non-headless (AWT robot)?

--Benjamin

On Sun, Feb 5, 2017 at 7:42 PM, Chris Merrill <ch...@ide4selenium.com> wrote:

Sorry...forgot to attach the code!  SporadicFailures.java is attached.

--
You received this message because you are subscribed to the Google Groups "TestFX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testfx-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Merrill]

Correct.

On Sunday, February 5, 2017 at 1:47:25 PM UTC-5, Benjamin Gudehus wrote:

Hi Chris,

so you say SporadicFailures.java fails (inconsistently) on headless (Glass robot) and non-headless (AWT robot)?

--Benjamin

On Sun, Feb 5, 2017 at 7:42 PM, Chris Merrill <ch...@ide4selenium.com> wrote:

Sorry...forgot to attach the code!  SporadicFailures.java is attached.

--
You received this message because you are subscribed to the Google Groups "TestFX" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testfx-discus...@googlegroups.com.

[Chris Merrill]

Sorry Benjamin, my description of what is going on with these tests may have been a bit misleading.

The tests I wrote should ALWAYS pass. One test simply types some text into a field and then verifies that the field contains the text. The other clicks on one field, verifies it is focused, then sends a TAB key and verifies that the focus has changed to a second field.

When run by themselves, the tests always pass. But when run in quick succession (by duplicating the tests in the test class), they fail sporadically. As an example, the first test is just these 2 lines:
        clickOn("#field1").push(KeyCode.CONTROL, KeyCode.A).write("new text").push(KeyCode.TAB);
        Assert.assertEquals("new text", _field1.getText());

When the test is repeated 10 times in quick succession in the JUnit test suite, it occasionally fails. The exception comes from within the clickOn() TestFx method, with a
java.util.ConcurrentModificationException
    at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:386)

I originally thought this had something to do with event handlers on the fields. But commenting out the code that adds the listeners to the text fields does not remove the problem (it does seem to make it less frequent). So I am baffled at how the clickOn() method can generate a ConcurrentModificationException when nothing else should be happening in the UI.

I've simplified the test to the point that I can no longer manufacture a theory on how my test code could be at fault. I've attached a simplified version (SporadicFailures2.java). It fails less often in this version - I frequently need to run the test suite 4-5 times before it will fail. Sometimes 9 or 10 times. But eventually, one test will fail.

Can anyone help me understand how this simple test could fail in this inconsistent manner? What am I doing wrong?

Thanks!
Chris

On Sunday, February 5, 2017 at 1:47:25 PM UTC-5, Benjamin Gudehus wrote:

Hi Chris,

so you say SporadicFailures.java fails (inconsistently) on headless (Glass robot) and non-headless (AWT robot)?

--Benjamin

On Sun, Feb 5, 2017 at 7:42 PM, Chris Merrill <ch...@ide4selenium.com> wrote:

Sorry...forgot to attach the code!  SporadicFailures.java is attached.

--
You received this message because you are subscribed to the Google Groups "TestFX" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testfx-discus...@googlegroups.com.

[Benjamin Gudehus]

>Can anyone help me understand how this simple test could fail in this inconsistent manner? What am I doing wrong?

I'll take a look into your example.

java.util.ConcurrentModificationException: `clickOn()` uses `Plattform.runLater()` and semaphores to "synchronize" the JavaFX thread, maybe this causes the problems. I even got this exception without using TestFX (i.e. me directly interacting with a JavaFX application).

To unsubscribe from this group and stop receiving emails from it, send an email to testfx-discuss+unsubscribe@googlegroups.com.

[Chris Merrill]

Well that is interesting...I have not encountered that outside of TestFX. It is rather disturbing, actually.

Thanks for taking a look!