Test failures are reproducible now

[Kyo Lee]

​Hi folks,

During the investigation on the current test failure issues, one element that was overlooked was that the actual failing PR branch wasn't used to build the UI when trying to reproduce the failures manually.

When going through the list of the failures in the current pending pull requests, it was discovered that those were the actual failures in the UI code branches -- comments were left on the pull request pages to address those failures.

After addressing the Linux firefox issue mentioned on the Selenium Doc, it appears that the occurrence of the random failures has been seemingly reduced -- based on the observation from the past couple of days' test runs. We cannot say for sure that it was the main cause of the random failures, so we will still need to keep an eye for such failures for next several days.

Another observation made was that the issue with using 'select' command in the selenium test. In the tests, 'select' command is used to select an option from a dropdown menu, specified by ID of the select element and the value of the option, However this may become an issue when the chosen widget is initiated on the select element. Rather than simulating the actual click, the 'select' command would bypass the interaction with the chosen widget and directly select the option value, which is not an allowed behavior when using the browser. So, instead of using the 'select' command, the tests will need to click the option element directly on the widget in order to properly simulate the user behavior on the browser.

In any case, at this point, the PR monitor tests are producing considerably reliable results, which mean that the failures can be reproduced when checked manually, thus the test failures on the PR need to be examined prior to re-triggering the test, and please keep the 'develop' branch to be in the passing condition since such assumption is critical when making assessment on the cause of the test failures.

Thank you all!

Kyo

​