Messages on form load are skewing Selenium tests
Scott
It came to my attention this morning that there is a rather serious
problem with the Ubiquity processor when used in conjunction with the
automated test suite. The problem stems from the fact that Ubiquity
evaluates constraints and bindings on page load. This has the
unfortunate consequence of triggering messages which are never
dismissed by the test suite drivers, which in turn leads to erroneous
behaviour later on. This is perhaps most easily explained through
example: test 6.1.6.a is designed to test the constraint feature. It
should fail because the xpath expression uses string comparison
instead of integer comparison to evaluate the constraint. It is
supposed to compare the "to" input to the "from" input and make sure
that "to" is always greater than "from". This test was inexplicably
passing in Selenium but failing during manual execution. The issue in
Selenium is due to the test initially beginning with "from" equal to
10 and "to" completely empty. When the form is loaded the constraint
is evaluated, and since "" < "10" an xforms-invalid message appears.
While manually you'd dismiss this and move onto the actual test,
Selenium has no such logic. Instead the dialog, although modal in
nature is allowed to persist, and when Selenium triggers the event
which should cause xforms-invalid to display, due to the string
compare bug xforms-valid appears instead. The verifyText test which
follows detects the xforms-invalid which was issued on load, as it
matches, and ignores the xforms-valid message which should have cause
the test to fail. When the form is supplied some initial value for
"to" such that xforms-valid is issued on load, Selenium fails the test
because the xforms-invalid message never appears.
It occurs to me that left unchecked this problem has the possibility
of causing a myriad of false positives and negatives in any test which
uses messages to communicate the test status. Having talked to John
about this already, he informs me that this is in fact a flaw in the
processor, as it should not be issuing these events on load. There are
a number of ways this could be resolved in the test suite, but
ultimately this needs to be dealt with in the processor itself. As I
understand it, this issue is to be discussed tomorrow during the
morning teleconference.
Scott
Mark Birbeck
I guess it's true that the messages should not be displayed on load,
but there is a more general problem that I've flagged up before, which
is the W3C Test Suite's over reliance on xf:message in the tests.
(I prefer not to talk simply of the 'automated test suite' since there
are other tests that use Selenium, that are separate to the W3C's Test
Suite.)
To test without using xf:message is easy; in a test like this the
'xforms-invalid' handler can just set some value in the instance data,
and then an output control can display it. Selenium would then simply
check that the correct value is being displayed.
Another technique (which we used in formsPlayer tests) was to
increment the value, or to concatenate a string onto the value. This
would allow us to spot problems like the one you are referring to,
where we get too many invalid events.
Anyway, as you say we can discuss this tomorrow; I'm certainly not
saying that UXF is behaving correctly, only that using xf:message in
tests has proved to be way more trouble than it's worth.
Regards,
Mark
Mark Birbeck, webBackplane
mark.b...@webBackplane.com
http://webBackplane.com/mark-birbeck
webBackplane is a trading name of Backplane Ltd. (company number
05972288, registered office: 2nd Floor, 69/85 Tabernacle Street,
London, EC2A 4RR)