Ways to make 'Wait Until Keyword Succeeds' handle DataError from resolving variables

[qsorix]

Hi all,

In our tests we came up with a cool (we think) API to access DB and write assertions on the content. The problem is, it turned out not to work nicely with "Run Keyword"-type keywords, because of our uses of variables. I think a simple example will explain it clearly, but first some background.

Our SUT is a server application, that is supposed to create some stuff in a relational database. In tests, we usually trigger SUT by sending it some messages, then wait until the stuff gets to the database, and check if it is what it should be.

Initially, we used code like this:

${r}    Query    SELECT color.red FROM objects LEFT JOIN color ... WHERE ...
Should Be Equal    ${r[0][0]}    255

But since data gets to the database after some delay, we had to wrap that in a separate keyword, and run using WUKS. After some time it became tedious to write queries and assertions on the result, as it was hard to maintain, to read, and to write. We have a lot of these in our tests.

What I'm trying to do, is abstract our database behind a simple, read-only ORM, so we can write the above like this:

Should Be Equal    ${obj.color.red}    255

That's quite easy for our DB. The problem is, WUKS and alike, do not like DataError that is thrown from robot framework's _get_extended_var method, e.g. if obj.color has no red property (because obj.color is still NULL in DB or sth). As a result, WUKS stops working as expected and the whole solution becomes useless.

Here's a code sample. The test:

*** Settings ***
Library    MyOrm

*** Test Cases ***
Using Orm
    ${obj}    Get Object
    Wait Until Keyword Succeeds    3 second    1 seconds    Should Be Equal    ${obj.color.red}    255

And the library:

class Color():
    def __init__(self, r, g, b):
        self.red = r
        self.green = g
        self.blue = b

class MyObject():
    @property
    def color(self):
        # The value of this property is extracted from DB, but it may
        # not be available already. Some asynchronous actions of the SUT
        # makes it available, that's why I use WUKS to test it.
        #
        # If you set processing_finished to True, the WUKS works, because the
        # error is a failing keyword 'Should Be Equal'. The message is:
        #   Timeout 3 seconds exceeded. The last error was: 253 != 255
        #
        # If you set processing_finished to False, then resolving of
        # ${obj.color.red} throws DataError, and that is not handled by WUKS.
        # The error in that case is:
        #   Resolving variable '${obj.color.red}' failed: Object has no color
        #   yet

        processing_finished = True

        if processing_finished:
            return Color(253, 0, 0)
        else:
            raise Exception("Object has no color yet")

def get_object():
    return MyObject()

I can work around this, by returning some Null-Object, that would pretend to have any attributes and return itself as a value, and then raise an exception while being compared, but that's hackish and requires careful coding. I'd prefer to avoid such solutions. Another possible work around is monkey-patching robot.variables.Variables class, but that's even more fragile.

Is there a way to create such property-based ORM and use it with WUKS? Can I tweak robot framework somehow to work like I expect? If not, can you propose other solutions that would make my life with DB easier? I was thinking about adding my own keywords to resolve these ${obj.color.red} things, but then it's less convenient to use in other contexts. Perhaps there is a way to add my own resolving function? E.g. Instead of ${obj} I would write $[[obj]] or sth?

Thank you
Regards,
--
Tomasz Rydzynki

[Kevin O.]

Have you considered using polling/synchronization to solve this?

If you wrote a keyword Wait Until Row Count Is X, then you would not have to worry about the DataError.

Wait Until Row Count Is X    1    SELECT COUNT(*) FROM ....

${record}=    # perform query that gets actual data

One of the dangers of using WUKS is it ignores all errors, not just the ones you expect, and if an unexpected error eventually stops coming before the timeout, you will probably never notice.

[qsorix]

W dniu czwartek, 9 stycznia 2014 04:36:42 UTC+1 użytkownik Kevin O. napisał:

Have you considered using polling/synchronization to solve this?

If you wrote a keyword Wait Until Row Count Is X, then you would not have to worry about the DataError.

Wait Until Row Count Is X    1    SELECT COUNT(*) FROM ....

${record}=    # perform query that gets actual data

This would work, by getting rid of WUKS, so it's certainly an option.

On the other hand, the query is now explicit in the test, plus it requires more lines, so even if I use ORM after the "Wait Until Row Count Is X", the query already defeats the purpose of using ORM. And what I'm mainly trying to accomplish is changing the code like this, that is now everywhere in our tests:

WUKS     Some Keyword Wrapping Queries    ${parent_id}    ${class_id}

...

Some Keyword Wrapping Queries
    [Arguments]    ${parent_id}    ${class_id}
    ${parent_surname}=    Query    SELECT surname FROM people WHERE id=${parent_id}
    ${child_surname}=    Query    SELECT surname FROM classes, people ...
    Should Be Equal    ${parent_surname[0][0]}    ${child_surname[0][0]}

To something, IMHO, simpler:

WUKS    Should Be Equal    ${parent.surname}    ${class.students[0].surname}

but, as I wrote in the first post, it doesn't quite work the same way.

(yeah, I use WUKS in both examples, but I got your point about its drawbacks).

One of the dangers of using WUKS is it ignores all errors, not just the ones you expect, and if an unexpected error eventually stops coming before the timeout, you will probably never notice.

Good point. I'll keep that in mind in the future... I wonder If we're experiencing this problem already :)

Thanks

[Pekka Klärck]

2014/1/7 qsorix <qso...@gmail.com>:

>
> The problem is, WUKS and alike, do not like
> DataError that is thrown from robot framework's _get_extended_var method,
> e.g. if obj.color has no red property (because obj.color is still NULL in DB
> or sth). As a result, WUKS stops working as expected and the whole solution
> becomes useless.

One possible workaround is that you use `Variable Should Exist` inside
WUKS. It fails with a normal AssertionError that WUKS and other
similar keywords do not pass through. After you know the variable
exists, you can then use `Should Be Equal` to verify its value.

Verifying the condition using your own custom keyword, like Kevin
suggested, is a very good solution too. It obviously requires a bit
more initial work, but being able to fine tune it to your exact needs
is often worth the effort. Another benefit is that it avoids using
`--RemoteKeywords WUKS` that is needed if outputs grow large due to
using WUKS a lot.

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org