Dictionary of Dictionaries parsing help (Passing hash of hashes over remote perl library)

[Shawn Speer]

Hello,

  I am passing a multilayer hash over the remote perl library connection and am having some issues parsing data from it.  I am using the built in collections library to perform operations against the returned dictionary and It appears that only the outer keys are being read when performing these related functions.

11:06:25.421     INFO     Dictionary size is 2 and it contains following items:

 content: {'data': {'read_only': 0, 'description': '', 'name': 'test234'}, 'model': 'data', 'id': 92, 'uri': '/api/2.9/data/92'}

 status: 201

As you can see above, the content key is the outer key of the multilayer hash.  It reads like this:

 $VAR1 = {
     'content' => {
         'data' => {
             'name' => 'test234',
             'description' => undef,
             'read_only' => 0
             },
         'model' => 'data',
         'id' => 92,
         'uri' => '/api/2.9/data/92'
         }
     };

'content' is the outer key with a nested hash within it (data) and 3 additional keys (model,id and uri).  If I try the following against the 'status' key, it works fine:

${status}  Get From Dictionary  ${ret}  status

Returned data:

20:19:57.424     INFO  201

Running against the 'content' key returns:

{'data': {'read_only': 0, 'description': '', 'name': 'test234'}, 'model': 'data', 'id': 92, 'uri': '/api/2.9/data/92'}

However, I can't seem to pull any of the values from the nested data key.  I tried assigning the key to a variable (like the above example) and using the 'Should Contain' keyword, but that failed.  Any idea on what I need to do in order to lookup the values form the nested key(s)?

Thanks,
Shawn
 

[David]

Hmm...I'll have to investigate your scenario. I tested the server against scalar, array, and hashes, though primarily against the XML-RPC spec rather than full integration with RF. I don't recall if I went as far as to test a nested hash, array of hash, hash of array, etc. There might be a limitation in how the XML-RPC library encodes the nested hash. Bear in mind remote libraries (except for Python and perhaps Java, which are more native) are generally not meant to be used with complex data structures, only simple ones. I'm sure they can tackle nested structures but only to some degree due to limitations of communicating over XML-RPC.

What is your expected usage of the nested hash in an RF test? Can you provide some sample keyword commands in RF test in how you utilize the returned hash for me to use in the investigation?

Also, it's not quite clear what you are doing, are you passing the nested hash as a keyword argument to Perl remote library to execute with or is the Perl library returning a nested hash back to RF test after executing the keyword?

If it's the latter case of returning a nested hash from Perl library, you can also use the Perl self test code approach (follow the debug sample in exampleremoteserver.pl) to help debug the issue. If the self test approach can allow you to query the returned nested hash as you desire (locally through Perl) then at least the library with the Perl server is functioning correctly but that either the encoding to XML-RPC response is messed up (when run under server mode) or that RF is interpreting it incorrectly upon receipt. If it fails in Perl server locally as well in debug mode, then my server implementation is not returning the hash correctly.

[Kevin O.]

Shawn,

Nested data structures in a scripted language can certainly be confusing. The log messages make it look like the data is coming back correctly, but I do not have any experience with the Perl server. In case it has nothing to do with the fact you are using a remote library, here is a demonstration on accessing nested dictionaries. Using Evaluate here to recreate the object you are receiving from the remote keyword.

Nested Dictionary

    ${ret}=    Evaluate    {'content':{'data': {'read_only': 0, 'description': '', 'name': 'test234'}, 'model': 'data', 'id': 92, 'uri': '/api/2.9/data/92'},'status':201}

    Log Dictionary    ${ret}

    ${content dict}=    Set Variable    ${ret['content']}

    Log Dictionary    ${content dict}

    ${data dict}=    Set Variable    ${content dict['data']}

    Log Dictionary    ${data dict}

    Should Be Equal    ${data dict['read_only']}    ${ret['content']['data']['read_only']}    # really nested !

[David]

Thanks for the insights Kevin.

Looking at the original post & Kevin's example, I looked over the Collections library. I'd venture to say use of Get From Dictionary is the wrong keyword to use in this situation since it appears designed to return a scalar value. I did not see the appropriate keyword variant for something like Get Sub Dictionary From Dictionary, however, there is a Dictionary Should Contain Sub Dictionary keyword (but that is only useful if you already have the sub dictionary to compare with).

The closest keyword you may use from that library to do what you originally intended might be Get Dictionary Items, but that seems to look more like a list than a dictionary, at least in how it's presented in the documentation (version 2.8.4). You might be better off accessing the dictionary data as Kevin presented.

[Shawn Speer]

Thanks Kevin and David for the info.  I believe that we should be able to accomplish what we need using Kevin's example, but I will need to do some testing to be certain. 

For some background, our 'basic' libraries are essentially wrappers to our products external Perl API.  The returned output from the api call is a hash of hashes that contain the http status and content, among other things.  We've been able to fully automate the creation of these basic libraries using json data from our api documentation and add them to our RF remote perl libraries.  The goal is for the test case to parse the returned hash of hash for whatever is needed for the particular test case.   I scrubbed the data prior to posting, but lets say you are creating a new user account in one test case.  Within the returned content key will be the 'userid' that results from the create function (it could be in a nested hash).  We'll want to make sure that the returned data key 'userid' is not null and use the returned value in subsequent test cases.

btw, the remote perl library has been of great benefit to our solution.  Much thanks David! 


Thanks,
Shawn

[Kevin O.]

Get From Dictionary returns the value mapped to the key regardless of what it is. The example uses a scalar, but it is just an example.

If ${foo} a dict-like object with key bar,

    ${value}=    Get From Dictionary    ${foo}    bar

Does the exact same (ignoring unicode vs. str) as:

    ${value}=    Set Variable    ${foo['bar']}

I used Set Variable out of habit.

If the key does not exist, then Get From Dictionary throws a nicer error.

Get From Dictionary does not even care if it is actually a dict, only that it is dict-like.

    ${list}=    Create List    7    8    9

    ${index one}=    Get From Dictionary    ${list}    ${1}

20140507 13:17:55.682 :  INFO : ${list} = [u'7', u'8', u'9']

20140507 13:17:55.683 :  INFO : ${index one} = 8

This may seem strange but I think it is correct and in line with Python and duck-typing.

[Shawn Speer]

Thanks again Kevin.  I did some testing on this today and as your example points out, the 'Set Variable' keyword is able to read the nested dictionary.  I used the following:

    # ${ret} was actually set as the returned hash of hashes value from the remote perl libary keyword

    ${ret}=    Evaluate    {'content':{'data': {'read_only': 0, 'description': '', 'name': 'test234'}, 'model': 'data', 'id': 92, 'uri': '/api/2.9/data/92'},'status':201}

    ${name}=    Set Variable    ${ret['content']['data']['name']}

    Log Dictionary    ${name}

    Should Be Equal    ${name}  test234

I was also able to pull from any layer of the nested hash.  For example ${ret['content'][uri']}.  'Get From Dictionary' doesn't play so nice, but I'm sure I just need to finagle my syntax.  I tried this offhand:

${name}=    Get From Dictionary   ${ret}  ${ret['content']['data']['name']}

Which resulted in

FAIL Dictionary does not contain key 'test234''

Looks like it can read the value, so again I am pretty sure that I just need to figure out the correct syntax for the dictionary call.  The 'Set Variable' keyword works like a champ with the dictionary of dictionaries though. 

Thanks again,
Shawn

[David]

Perhaps Get From Dictionary isn't designed to pull out values from a "nested" key/dictionary, so you must start from each layer, iterating through each level as needed? Kevin's example with a list isn't really nested. I think the key has to be in the top layer of the dictionary whether it's a string key or an indexed number.

e.g.

${content}=    Get From Dictionary   ${ret}  content

${data}=    Get From Dictionary   ${content}  data

${name}=    Get From Dictionary   ${data}  name

vs

${content}=    Get From Dictionary   ${ret}  content

# now access rest of the stuff like ${content['data']['name']} without assigning to any more variables, using this directly with Should Be Equal

vs

${content}=    Get From Dictionary   ${ret}  content

${name}=    Set Variable   ${content['data']['name']}

[Kevin O.]

${ret['content']['data']['name']} refers to the value of the mapping with the key 'name' in the nested dict.

You are going one level too deep.

    ${ret}=    Evaluate    {'content':{'data': {'read_only': 0, 'description': '', 'name': 'test234'}, 'model': 'data', 'id': 92, 'uri': '/api/2.9/data/92'},'status':201}

    ${data}=    Set Variable    ${ret['content']['data']}

    Log Dictionary    ${data}

    ${name}=    Get From Dictionary    ${data}    name

    Should Be Equal    ${name}    test234

If you want to get a value in nested dict without creating intermediate variables using Get From Dictionary, you can do this:

    ${name2}=    Get From Dictionary    ${ret['content']['data']}    name

    Should Be Equal    ${name}    ${name2}

Like I said before, it can be a bit confusing, but you will get the hang of it.

[David]

Good example Kevin, I forgot to account for case of varying the dictionary being accessed by Get From Dictionary and only focused on the incorrect usage of the key parameter.