[Remote]OperatingSystem Library

[binithb]

Hi there,

This is my first post in the dev group, I must say the work you guys are doing is great, the world will be a better place to live now, :) well ,it wont stop global warming, but i think a small part of that is true.

Following is the scenario I am in , its related to being able to call operating system calls on remote hosts without the need to do a lot of things to set things up.

First, existing options which comes to mind/recommended to me were the ssh library and remote interface.
    Of these , 
    1)disadvantage with ssh is that with this model we wont be able to use all the features of RF reporting (I wont get complete details of what happens after the ssh command gets invoked)
    2)remote interface requires some effort from the user and it has another major disadvantage that I will need to maintain two different copies of user level     keywords, one for localhost (with OS library) and the other for other hosts (using remote library)


More details on this user group thread
https://groups.google.com/forum/?fromgroups=#!topic/robotframework-users/OyMoCq89a3w

Now , things could be much simple and easier for me if the following feature was there.

As a user of RF who has to execute tests all over LAN, ideally I would like an operating system library in which all keywords optionally takes an IP argument, and when I run this the particular OperatingSystem call it should get executed in the host with the IP mentioned as argument.

This way ,
    1)The interface is simplified ,no need to use remote library.
    2)since I am only using one library, I just need to maintain one set of User keywords, for local as well as other hosts.


I have made an attempt to implement this in my new RF clone

http://code.google.com/r/binithb-rf-databaseinterface/

A new RemoteOperatingSystem Library is made which has all the methods of OperatingSystem Library.

http://code.google.com/r/binithb-rf-databaseinterface/source/browse/src/robot/libraries/RemoteOperatingSystem.py?spec=svn8ad9fa70508e3e28112c8d0fea8efbce62365435&r=8ad9fa70508e3e28112c8d0fea8efbce62365435

All the method signatures are the same as the existing OperatingSystem library , but all taking two additional, (optional) parameters, namely RF remote server IP and RF remote server port,
Inside each method implementation ,instead of the original flow of the keyword,  an object of the Remote class is created , followed by calling the same method on the RF remote server running on the remote host.
(Note : Ideally the remote interface should not get used if the IP is localhost , but in its current form , when using this library even for localhost the RF remoteserver needs to be running (on localhost itself)  , this needs to be corrected I guess)

Also if the remote invocation has to work out of the box, I will also need a version of the RF remote server which inherits the operating system calls.I have added it as part of the default installables, along with pybot, jybot etc

http://code.google.com/r/binithb-rf-databaseinterface/source/browse/src/bin/robotremoteOSserver.py


The basic functionality is working now, and for me now its much easier to write and maintain user level keywords which get executed in different hosts.

I have not mentioned anything about the clone in the user group as of now, it may cause confusion among normal users.
I thought I will take feedback from the dev group and act based on that.

Requesting you guys/girls to comment on the following.

1) To begin with, is my thinking right or am I grossly wrong with my understanding of current possibilities. Is my requirement valid/justified ?
2) If there is scope for this line of functionality, should this library be maintained as an external one or is it worth considering it to add the remote capability to the existing, default OperatingSystem library (This will need approval and time from core team, of course)
3)Comments about implementation details, how it can get better

(Side Note : The original purpose of the clone was to add some other features in RF, the main one being extracting RF test data and test results to a Database. This work is still raw at the moment , I will come back and post when I have some shape on the Database and other features, please ignore the Database related changes (dbbot) in the clone for now )

Eagerly looking forward for responses,


-binith

[Kevin Ormbrek]

You did not responded to my last post? Here is a crude but working implementation of what I was talking about. UberOperatingSystemLibrary wraps a local instance of OperatingSystem and multiple remotely served OperatingSystem instances. Added keywords add_operating_system_host and switch_operating_system_host are used to change the context of execution.

+ No writing multiple versions of user keywords

+ No copying and modification of RF code

+ No passing around extra arguments relating to which system to execute on

- Uses libdoc code, which could change in the future

- Not much error handling

- Needs more documentation

- No keyword to shut down remote servers (would be easy to implement)

Hope this helps,

Kevin

from robot.libdocpkg import LibraryDocumentation

from robot.libraries.OperatingSystem import OperatingSystem

from robot.libraries.Remote import Remote

from robot.api import logger

from robot.utils import ConnectionCache

from robot.utils import normalize

# use libdoc code to get keyword documentation for a library

osdoc = LibraryDocumentation('OperatingSystem')

del LibraryDocumentation # messes RF up if you don't do this...don't know why

# keywords from libdoc have normalized names, put back to original form

OS_KEYWORDS = dict([(kw.name.replace(' ', '_').lower(), kw) for kw in osdoc.keywords])

class UberOperatingSystemLibrary(object):

    ROBOT_LIBRARY_SCOPE = 'GLOBAL'

    def __init__(self):

        self.oslib = OperatingSystem()

        self.cache = ConnectionCache(no_current_msg="No current remote library")

        self.local = True

    def get_keyword_names(self):

        names = OS_KEYWORDS.keys()

        names.append('add_operating_system_host')

        names.append('switch_operating_system_host')

        return names

    def get_keyword_arguments(self, name):

        if name == 'add_operating_system_host':

            return ['alias', 'uri']

        elif name == 'switch_operating_system_host':

            return ['alias']

        return OS_KEYWORDS[name].args

    def get_keyword_documentation(self, name):

        if name in OS_KEYWORDS:

            return OS_KEYWORDS[name].doc

        else:

            return "TODO"

    def run_keyword(self, name, args):

        if name == 'add_operating_system_host':

            return self.add_operating_system_host(*args)

        if name == 'switch_operating_system_host':

            return self.switch_operating_system_host(*args)

        if self.local:

            func = getattr(self.oslib, name)

            return func(*args)

        else:

            rkw = getattr(self.cache.current, 'run_keyword')

            return rkw(name, args)

    def add_operating_system_host(self, alias, uri):

        remote = Remote(uri)

        logger.info("Adding remote OperatingSystem library with alias '%s' served from %s" % (alias, uri))

        self.local = False

        return self.cache.register(remote, alias)

    def switch_operating_system_host(self, alias):

        if normalize(alias) == 'localhost':

            self.local = True

        else:

            self.cache.switch(alias)

            self.local = False

        logger.info("Switching OperatingSystem host to %s" % alias)

[Kevin Ormbrek]

Oops, forgot to post example test...

*** Settings ***

Library           UberOperatingSystemLibrary.py

*** Test Cases ***

UberTest

    Set Environment Variable    host    LOCAL

    Add Operating System Host    remote 1    http://192.168.120.107:8270/

    Set Environment Variable    host    SYSTEM A

    Switch Operating System Host    localhost

    ${host}=    Get Environment Variable    host

    Should Be Equal As Strings    ${host}    LOCAL

    Switch Operating System Host    remote 1

    ${host}=    Get Environment Variable    host

    Should Be Equal As Strings    ${host}    SYSTEM A

[binithb]

Hi Kevin,

I m really sorry I didnt post back on that thread.
Thanks for following up, and helping with code. Its really nice.
One question - If I use UberOperayingSystem in RIDE, will I get the existing keyword documentation of Operating System library.

The reason why I didnt follow up on the original thread was,at that stage I myself was not sure which approach to follow, I definitely considered your suggestion, along with that , I had got a few more inputs about how it should be done.

I thought I will do it the way I thought best suited my current requirement, and discuss about it once its done, rather than doing a lot of discussions in the beginning.

One of the primary requirement I had was that a lot of keywords will have to be executed in different hosts, in succession.
Because of this, I didnt want to do the switch operating system host part as a different step in the test case, I thought the test case will be more compact if the operating system host is specified as part of the particular operating system library keyword.

Indeed, if I do it my way, from an optimal performance point of view, its not going to be good. There will be multiple tcp connections required to be created and closed for each keyword execution. But I thought this is a worthy trade off for (what I think is)easier test case management.

+ No writing multiple versions of user keywords

[ binith:- I think both yours as well as my approach achieve this requirement]

+ No copying and modification of RF code

[ binith:- I reused Operating System library code because I wanted to continue to have operating system library keyword documentaton available for RIDE users,I admit this is not DRY, most probably it can be done better, suggestions are welcome]

+ No passing around extra arguments relating to which system to execute on

[ binith:- If the remote host information is not passed as an extra argument, this remote host info will be required to be changed in some other way (like doing the switch_operating_ system_host as in UberOperatingSystem library ) , and this constituting an extra statement in the RF test case. From a pure user experience/simplicity point of view, my judgement says that passing as extra argument to the familiar OperatingSystem library call is more suitable. Please comment on this. ]


I was hoping I will get more responses here than the RF user group, but it now seems the RF user group is much more active :)

thanks,
-binith

[Kevin Ormbrek]

On Saturday, April 13, 2013 12:36:08 PM UTC-5, binithb wrote:

Hi Kevin,

I m really sorry I didnt post back on that thread.
Thanks for following up, and helping with code. Its really nice.
One question - If I use UberOperayingSystem in RIDE, will I get the existing keyword documentation of Operating System library.

 UberOperatingSystem implements both get_keyword_arguments and get_keyword_documentation, returning the same as OperatingSystem, so yes.

The reason why I didnt follow up on the original thread was,at that stage I myself was not sure which approach to follow, I definitely considered your suggestion, along with that , I had got a few more inputs about how it should be done.

I thought I will do it the way I thought best suited my current requirement, and discuss about it once its done, rather than doing a lot of discussions in the beginning.

One of the primary requirement I had was that a lot of keywords will have to be executed in different hosts, in succession.
Because of this, I didnt want to do the switch operating system host part as a different step in the test case, I thought the test case will be more compact if the operating system host is specified as part of the particular operating system library keyword.

I would think that you would have user keywords for all your operations in which you could switch hosts at the beginning. This would keep the test cases clean and readable.

Something like this:

Get Log Path of Product ${product} On System ${system}

    Switch Operating System Host    ${system}

    Get Product Log Path    ${product}

Indeed, if I do it my way, from an optimal performance point of view, its not going to be good. There will be multiple tcp connections required to be created and closed for each keyword execution. But I thought this is a worthy trade off for (what I think is)easier test case management.

Performance in automation is frequently not important.  Readability/maintainability are far more important IMHO. This is even true for a lot of application code as well.

+ No writing multiple versions of user keywords
[ binith:- I think both yours as well as my approach achieve this requirement]

+ No copying and modification of RF code
[ binith:- I reused Operating System library code because I wanted to continue to have operating system library keyword documentaton available for RIDE users,I admit this is not DRY, most probably it can be done better, suggestions are welcome]

There are ways to programatically add arguments to existing methods and wrap them, but I have no experience in that. The decorator package would probably be useful there. Copying and pasting large sections of code makes my stomach churn, but sometimes I forget - its just test code.

+ No passing around extra arguments relating to which system to execute on
[ binith:- If the remote host information is not passed as an extra argument, this remote host info will be required to be changed in some other way (like doing the switch_operating_ system_host as in UberOperatingSystem library ) , and this constituting an extra statement in the RF test case. From a pure user experience/simplicity point of view, my judgement says that passing as extra argument to the familiar OperatingSystem library call is more suitable. Please comment on this. ]

Like I said earlier, it gets messy with varargs. Using context changing keywords should cause an extra line in a user keyword or library method, but not in the test cases if you are using test cases with DSL/user keywords.

I was hoping I will get more responses here than the RF user group, but it now seems the RF user group is much more active :)

I too am surprised on how quiet it has been. Maybe people are on vacation. 

thanks,
-binith

In the end it is your decision to make and you should do whatever you think is best.

[Pekka Klärck]

2013/4/7 binithb <bin...@gmail.com>:

>
> This is my first post in the dev group, I must say the work you guys are
> doing is great, the world will be a better place to live now, :) well ,it
> wont stop global warming, but i think a small part of that is true.

Big thanks for kind words! Thinking that Robot Framework, and all the
libraries and tools in the ecosystem, makes the world a better place
made me smile. =)

> Following is the scenario I am in , its related to being able to call
> operating system calls on remote hosts without the need to do a lot of
> things to set things up.
>
> First, existing options which comes to mind/recommended to me were the ssh
> library and remote interface.
> Of these ,
> 1)disadvantage with ssh is that with this model we wont be able to use
> all the features of RF reporting (I wont get complete details of what
> happens after the ssh command gets invoked)
> 2)remote interface requires some effort from the user and it has another
> major disadvantage that I will need to maintain two different copies of user
> level keywords, one for localhost (with OS library) and the other for
> other hosts (using remote library)

You are right about the pros and cons of SSH and Remote approach. It
sounds to me that the latter would work better for you in general.

> More details on this user group thread
> https://groups.google.com/forum/?fromgroups=#!topic/robotframework-users/OyMoCq89a3w

Unfortunately I haven't yet read this thread (I simply haven't had
much time to spent on these groups). Has anyone proposed importing
libraries using `WITH NAME` there? You could do something like this:

*** Settings ***
Library OperatingSystem WITH NAME local
Library Remote http://1.2.3.4 WITH NAME 1.2.3.4

*** Test Cases ***
Example
Copy File local /tmp/f1.txt /tmp/f2.txt
Copy File 1.2.3.4 /tmp/f1.txt /tmp/f2.txt

*** Keywords ***
Copy File
[Arguments] ${library} @{args}
Run Keyword ${library}.Copy File @{args}

Note that not recommending the above as a general solution. It should
work, though, and might work well in your context.

> Now , things could be much simple and easier for me if the following feature
> was there.
>
> As a user of RF who has to execute tests all over LAN, ideally I would like
> an operating system library in which all keywords optionally takes an IP
> argument, and when I run this the particular OperatingSystem call it should
> get executed in the host with the IP mentioned as argument.
>
> This way ,
> 1)The interface is simplified ,no need to use remote library.
> 2)since I am only using one library, I just need to maintain one set of
> User keywords, for local as well as other hosts.

The above example should also provide this functionality. Having a
separate library would have a benefit that it would have a more
meaningful name than Remote and no WITH NAME imports would be needed.

> I have made an attempt to implement this in my new RF clone
>
> http://code.google.com/r/binithb-rf-databaseinterface/
>
> A new RemoteOperatingSystem Library is made which has all the methods of
> OperatingSystem Library.
>
> http://code.google.com/r/binithb-rf-databaseinterface/source/browse/src/robot/libraries/RemoteOperatingSystem.py?spec=svn8ad9fa70508e3e28112c8d0fea8efbce62365435&r=8ad9fa70508e3e28112c8d0fea8efbce62365435
>
> All the method signatures are the same as the existing OperatingSystem
> library , but all taking two additional, (optional) parameters, namely RF
> remote server IP and RF remote server port,

I can see that passing the address of the machine where to run the
operation to every call is useful if you are running running
operations on different machines all the time. When running many
commands on a same machine that gets pretty annoying, though, and
using the Switch Operating System Host approach proposed by Kevin in a
reply to this mail would work much better in that case. Ultimately
choices like this depend on the use case.

> Requesting you guys/girls to comment on the following.
>
> 1) To begin with, is my thinking right or am I grossly wrong with my
> understanding of current possibilities. Is my requirement valid/justified ?

As I already commented above, you are mostly right with the
possibilities and limitations of Remote interface.

> 2) If there is scope for this line of functionality, should this library be
> maintained as an external one or is it worth considering it to add the
> remote capability to the existing, default OperatingSystem library (This
> will need approval and time from core team, of course)

I think this library is better hosted as an external library. Using it
anyway requires installing remote servers with OperatingSystem
functionality on the remote machines. Installing the library itself on
the machine where Robot is running wouldn't make the overall
installation process any more complicated.

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org