Testing C++ application with Robot framework

[Bo Je]

Hello all, I have used robot extensively with selenium. As a new task we are evaluating Robot as a potential option to test our c++ server. To build functional scenario directly with the application.

Does any one have any success using Robot with similar situation? 

Thanks in advance!

BJ

[David]

Please elaborate on how you want or plan to test the c++ server - by command line, by APIs (REST/COM/native c++?), by automation hooks added to the server (e.g. scripting/API back door).

[Bo Je]

Hey David, run from command line to select and execute test cases.

[David]

Bo, actually I meant how will you be interfacing to the c++ server to test it. that will determine how you would use RF with it, and how your test cases will be written.

[Chris Newman]

I'm using Robot/python to test a large C/C++ Unix server application. It's much better than the previous pile of home-brew perl. I've used four different techniques to integrate C with Robot, here's the rundown from my experience:

1. Communicating over sockets works very well. Python is a great language to write TCP test clients. If your product has a TCP-based interface, using Robot/Python to test that will be a pleasure. Just be sure to use python ssl.SSLContext() if you want to test SSL/TLS properly.

2. Running CLIs is somewhat problematic because Robot inherits issues from Python. When running a CLI in a test framework, you really want all three of: 1. ability to provide input to CLI from Robot, 2. ability to accept arbitrary sized output data from CLI, 3. ability to have timeout that aborts the CLI. The Robot Process module does not do all three. This can't be done without using Python 3's subprocess communicate() method or writing that functionality in python 2 (I did the latter because we can't upgrade our infrastructure to Python 3 yet). Anyway, this is the worst problem I've encountered using Robot/Python and was solvable.

3. Python's ctypes module works well for integrating C APIs directly into Robot (as long as you keep your API types clean and simple). So if your server has shared libraries, it's not hard to turn them into Robot keywords. However, be aware that your python binary architecture must match your shared library architecture (e.g., 32-bit vs. 64-bit). If you don't have sufficient control over your python binaries, this integration technique won't work.

4. A final option for integrating C code into Robot is to use the remote library interface. It took me a few days to write an XML RPC server daemon in C so I could do this. It's a bit cumbersome to add new keywords to the C daemon, but the interface runs much faster than a CLI and the ability to gather C debugging and route it to the robot log works nicely.

- Chris
(speaking for myself only)

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Pekka Klärck]

2017-01-10 2:59 GMT+02:00 Chris Newman <chris....@oracle.com>:
> I'm using Robot/python to test a large C/C++ Unix server application. It's
> much better than the previous pile of home-brew perl. I've used four
> different techniques to integrate C with Robot, here's the rundown from my
> experience:
>
> 1. Communicating over sockets works very well. Python is a great language to
> write TCP test clients. If your product has a TCP-based interface, using
> Robot/Python to test that will be a pleasure. Just be sure to use python
> ssl.SSLContext() if you want to test SSL/TLS properly.

Yes, this kind of testing ought to be pretty easy. It shouldn't be too
much more complicated if you needed to use something like a serial
port either.

> 2. Running CLIs is somewhat problematic because Robot inherits issues from
> Python. When running a CLI in a test framework, you really want all three
> of: 1. ability to provide input to CLI from Robot, 2. ability to accept
> arbitrary sized output data from CLI, 3. ability to have timeout that aborts
> the CLI. The Robot Process module does not do all three. This can't be done
> without using Python 3's subprocess communicate() method or writing that
> functionality in python 2 (I did the latter because we can't upgrade our
> infrastructure to Python 3 yet). Anyway, this is the worst problem I've
> encountered using Robot/Python and was solvable.

Python 2 also has Popen.communicate. Is it somehow improved in Python
3? Anyway, this functionality isn't exposed via the Process library
currently. Incidentally there's also "How to send input to command
line programs" email thread currently about the same topic. Do you
have ideas how the Process library could be enhanced in this regard?

> 3. Python's ctypes module works well for integrating C APIs directly into
> Robot (as long as you keep your API types clean and simple). So if your
> server has shared libraries, it's not hard to turn them into Robot keywords.
> However, be aware that your python binary architecture must match your
> shared library architecture (e.g., 32-bit vs. 64-bit). If you don't have
> sufficient control over your python binaries, this integration technique
> won't work.

ctypes works nicely at least in simple cases. For those interested,
there's also a demo project available:
https://bitbucket.org/robotframework/cdemo

In addition to ctypes, there's also CFFI module that allows calling C
from Python. I've never tested it myself nor do I know what possible
benefits it has over ctypes. For more details see
http://cffi.readthedocs.io/en/latest/

> 4. A final option for integrating C code into Robot is to use the remote
> library interface. It took me a few days to write an XML RPC server daemon
> in C so I could do this. It's a bit cumbersome to add new keywords to the C
> daemon, but the interface runs much faster than a CLI and the ability to
> gather C debugging and route it to the robot log works nicely.

Very cool! Any chance you could make your C remote server publicly
available? Would be a great addition into this list:
https://github.com/robotframework/RemoteInterface#available-remote-servers

Cheers,
.peke
--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Chris Newman]

On 10 Jan 2017, at 14:32, Pekka Klärck wrote:

> 2017-01-10 2:59 GMT+02:00 Chris Newman <chris....@oracle.com>:
>> 2. Running CLIs is somewhat problematic because Robot inherits issues
>> from
>> Python. When running a CLI in a test framework, you really want all
>> three
>> of: 1. ability to provide input to CLI from Robot, 2. ability to
>> accept
>> arbitrary sized output data from CLI, 3. ability to have timeout that
>> aborts
>> the CLI. The Robot Process module does not do all three. This can't
>> be done
>> without using Python 3's subprocess communicate() method or writing
>> that
>> functionality in python 2 (I did the latter because we can't upgrade
>> our
>> infrastructure to Python 3 yet). Anyway, this is the worst problem
>> I've
>> encountered using Robot/Python and was solvable.
>
> Python 2 also has Popen.communicate. Is it somehow improved in Python
> 3?

Yes, a timeout argument was added in python 3.

> Anyway, this functionality isn't exposed via the Process library
> currently. Incidentally there's also "How to send input to command
> line programs" email thread currently about the same topic. Do you
> have ideas how the Process library could be enhanced in this regard?

Yes. For Python 3, just expose keywords that use the communicate method.
For Python 2, you'd have to basically copy the implementation of the
communicate method from Python 3 (which is OS-dependent and not simple).
Unfortunately, I don't have cycles to do that in a clean way that would
be suitable as a pull request for Robot.

>> 4. A final option for integrating C code into Robot is to use the
>> remote
>> library interface. It took me a few days to write an XML RPC server
>> daemon
>> in C so I could do this. It's a bit cumbersome to add new keywords to
>> the C
>> daemon, but the interface runs much faster than a CLI and the ability
>> to
>> gather C debugging and route it to the robot log works nicely.
>
> Very cool! Any chance you could make your C remote server publicly
> available? Would be a great addition into this list:
> https://github.com/robotframework/RemoteInterface#available-remote-servers

I used a several proprietary product libraries to get it done that
quickly. It would take time to make it suitable for open source release
that I probably don't have. But I will put some thought into the idea.

- Chris