TestFX and thread synchronization

[Oddbjørn Kvalsund]

Hi,

I am working on a typical client/server application that executes calls to remote web services using JavaFX tasks (javafx.concurrent.Task) running on threads from a thread pool (java.util.concurrent.ThreadPoolExecutor). Using TestFX to test this application works well when both client and server are responsive, but as soon as there is heavy load on the machine running the tests, race conditions occur and tests start failing. There appears to be two main problems:

1) Given a sufficiently high system load the primary Stage is not visible or in focus as soon as the tests start running. This leads to keyboard events synthesized with GuiTest.push(KeyCode ...) being delivered to whatever window had focus when the test started, typically IntelliJ IDEA or Cygwin in my case. I have tried to wait for Stage.isShowing() and Stage.isFocused() before starting the main test logic, but these methods return true long before the window is visible and focused on my screen. This is possibly caused by the Stage being placed on top by the window manager, followed by FxToolkit.showStage() calling stage.toBack() and stage.toFront(), but I haven't investigated this in depth.

2) There is no way to determine that all callbacks submitted as result of a JavaFX event have been run and that the application has settled in an idle state. The event handler may submit a task to the thread pool and this task may submit a Runnable to Platform.runLater() etc. Ideally I would like to synchronize using a method such as WaitForAsyncUtils.waitForFxEvents() that atomically takes into account both Runnables submitted to Platform.runLater() and tasks submitted to my thread pool. I have a semi-working hack for this using PowerMock and Mockito to intercept calls to Platform.runLater() and update a global counter of pending runnables/tasks, but this seems inelegant.

How do you guys deal with issues like these?

Regards,

Oddbjørn Kvalsund

[Benjamin Gudehus]

Hi Oddbjørn,

here my first thoughts:

1)
Input events delivered to wrong non-JavaFX windows, I had this problem a lot. This is no problem in headless mode, which of course doesn't solve the timing problem.

I think javafx.stage.Window.impl_getWindows() returns a list of shown Stages.

2)

We could also try to read the number of pending Runnables from com.sun.javafx.application.PlatformImpl.pendingRunnables.

>How do you guys deal with issues like these?

My tests interact in general with a single window. So, I write very few integration tests. If I have a external data source, I'll just wait for e.g. the table to have more than zero items, or timeout.

--Benjamin

[Oddbjørn Kvalsund]

Hi,

Thanks for your feedback, Benjamin. To clarify: when running GUI tests we mock the web services, so these are not integration tests as such.

1) Removing the call to Stage.toBack() in StageSetupImpl.bringStageToFront() (I'm on TestFX 3.2.1) helps to remedy this problem. Why is this method call there in the first place? Is this a symptom of Stage.toFront() not working properly? I've used HeavyLoad (http://www.jam-software.com/heavyload/) to provoke these kinds of load related focus problems on my local computer.

2) I've tried accessing PlatformImpl.pendingRunnables, but considering that tasks running on threads from the thread pool may enqueue new Runnables to the JavaFX application thread or even trigger new server calls, I can't rely on checking pendingRunnables and the size of active threads in the thread pool individually. I need to evalute the sum of the two as one atomic unit. To further complicate matters, Glass appears to let the underlying window system handle user events such as mouse and keyboard events (ref. https://docs.oracle.com/javase/8/javafx/get-started-tutorial/jfx-architecture.htm), so even though it looks as all events have been processed (from a Java/JavaFX context), the underlying window system may have more user events queued.

It seems that fully deterministic test runs with JavaFX and TestFX is rather tricky or am I missing something obvious?

Regards, Oddbjørn

--

Oddbjørn Kvalsund