Cucumber Vs RobotFramework

[Avihay Eyal]

Hi, I what are the pros and cons of each of these framework when compared to one another?

I'm not sure which one is better to adopt.

Some background - I want to use the testing framework for financial applications(mainly trading in all sort of exchanges) testing.

[Pankaj Nakhat]

Its like comparing Orange and Apples. Robot framework is an abstraction for Selenium. However, Cucumber is not tied to a a tool. You can perhaps achieve similar things with Cucumber, but I would suggest strongly against using Cucumber just as Selenium framework.

Regards,
Pankaj Nakhat
www.pankajnakhat.com
t : @pnakhat

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 
You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Avihay Eyal]

Why do you assume Robot Framework is tied to Selenium?

Indeed, it has a Selenium library, but it's just one of many libraries, supporting different operations.

[Stephen Abrams]

Hi Avihay,

I asked this same question a little over a year ago on this list when we were at your stage. Before I respond, a bit about my group: We are primarily a Java shop but the group wanted our QA engineers to be able to use Python to automate tests (Python is sometimes recommended for this, since it can be a relatively easy language for non-developers to use quickly and effectively). We needed to write automation tests to exercise our multi-platform ad network - several applications across the network must work in concert to deliver functionality. At the time, the company was new to the idea of B.D.D. (though not T.D.D.) and the Cucumber-JVM project (https://github.com/cucumber/cucumber-jvm) was not quite ready for prime time. We chose Robot framework for end to end tests but have used Cucumber-JVM for the separate projects. Here are my thoughts after using both in different contexts:

1. Robot Framework seems like it is catered towards testers who have no programming experience. At face value, there are many good libraries (SSHLibrary, Selenium2Library, Database, etc.) that can be utilized to run integration tests without knowing much about programming, as I'm sure you've seen in the examples. 

2. Robot Framework outputs decent report documentation, though chasing down failures can sometimes be tricky. Its suite and test level setup and teardown methods are fairly straight-forward.

3. As a developer (and I imagine any QA engineer with decent programming skills), I find writing complex tests to be a chore in Robot. I'm not a fan of its text files, find IDE support to be lacking (w/ the exception of the RIDE framework, which crashed for me enough to abandon it - perhaps it has improved). "For" loops and conditionals are awkward, and finally writing keyword libraries in Robot as text files (and not Python/Java) makes them difficult to use with Python and Java keyword libraries w/o knowing a lot about the framework. For these reasons, I have often questioned the value of Robot Framework (which ends up being a pseudo programming language as soon as any degree of complexity is introduced) over something like nose (https://nose.readthedocs.org/en/latest/), where I can write simple Python, which will also generate decent reports and allows layered test setups. For this reason, we started bringing most or all of our keywords into Python code exclusively, but then lost the capability to write Gherkin-like Given/When/Then tests (http://blog.codecentric.de/en/2009/11/givenwhenthen-and-example-tables-using-the-robot-framework/) w/o using .txt files. In Cucumber, this is just an annotation over a Python function. Also, Robot's support for the Gherkin-like syntax is extremely limited; I often hit brick walls. Cucumber's use of regular expressions to translate between a natural language like English and any programming language is key.

4. Cucumber is intended for Acceptance testing only. While the line between high level business acceptance testing and lower level integration testing can be blurred, figuring out how to express lower level integration tests as Cucumber tests is challenging and time-consuming (so sometimes worthwhile, for the same reasons T.D.D. helps developers, by forcing you to think through what you're actually building). Cucumber shines at allowing feature discovery through communication and automated testing of those discovered features into the future. Your tests become documentation at the highest business value. I would argue that it is not the right choice for highly data-driven/bounds testing, as the more you test at this level, the less readable and interesting its resulting documentation will be for your stakeholders and new engineers coming on to a project.

I am working on our automation 2.0 vision at our company, and am hoping to write Python domain classes to encapsulate our business domain (and sub-domains, such as infrastructure), which we can then use in either Robot or nose for our integration tests, and Cucumber-JVM for our higher level acceptance tests. 

Hope this helps. You may also find the https://groups.google.com/forum/#!forum/robotframework-users mailing list helpful for this same question. Any way you go, there will be challenges - multi-system testing automation is not a simple problem to solve (though it is a fun one!)

Steve

[Avihay Eyal]

Hi Steve, thanks for your elaborated reply.

SO you're saying that you've experienced both Cucumber & RF, and in your automation vision, you think
that you'll use RF, or nose?

Why not Cucumber? I mean, you did point out that:

1. Writing complex tests in can be a chore in RF.
2. Over time, the RF keyword-language becomes complicated, and losses it's "simplicity argument"

And, regarding nose - don't you lose the ability of non-technical people to write tests, once your tests are written in python solely?

Thanks.