*sigh* My Company went with FitNesse rather than RobotFramework+Selenium2Library

[WillDev Dev]

Where I work, we ended up in a competition between using FitNesse and RobotFramework. I personally strongly prefer RobotFramework (if I'm going to write my tests in java, I'd rather just write them all in java, rather than in the awkward half-java half-fitness-specific-language that FitNesse uses).

I just wanted to share the two reasons led to this happening -

1. I presented RobotFramework as an easier to use tool, that even testers themselves wouldn't have a problem writing tests in. But the installation process was horrible.

I presented RobotFramework as an easy to use windows tool, but this impression was completely undercut when I tried to install the framework on a testers machine. It was a complete nightmare. The Selenium2Library windows installer exe doesn't work. Or at least, it doesn't include all the libraries that it requires to actually run - I don't know why.

So I had to use pip to install everything. That's when it started looking bad. Now I'm trying to sell "an easy to use windows tool" - which requires non-technical people to install using the command line.

But far worse than that, we're on a corporate network behind a proxy - as it's been pretty much everywhere I've ever worked. And pip is not happy with proxies. It installed ok on my windows 7 machine, but on the testers windows xp machine it required them entering their username and password - they really weren't happy that I might be "stealing" their corporate username and password (which I wasn't of course, but why did they have to enter it?). Then it still didn't work anyways - it would install some stuff, but with others it would continue saying it couldn't get a connection. Finally, after an hour of trying to get it to work, the only way I got it installed was asking them to take their machine home and install it over their local network from home. 

After a couple hours of trying to install via a command line, when every other piece of software a tester uses is just "run the install file", all of my credibility that RobotFramework was an easier tool was shot.

2. There's no central server for running tests and viewing test results

This is just how the tool is. Had #1 not been a problem, I might have been able to write and contribute a central server to the project here myself. Just wanted to mention it - they really wanted a place they could have long running test suites running that wasn't their local test machine.

3. I list this last because it's the least important, but figuring out how to get RobotFramework installed was a huge pain as well.

At first it looked easy - there was a big install page with instructions. It took a full 2 days to get it installed.

a. What I needed was a "here's how to quickly get RobotFramework running on Windows" page. Instead, the installation instructions ramble on a bit about the multiple different ways to get it installed which was confusing.

b. The install page casually mentions -

"Starting from Robot Framework 2.5, Python 2.5 is the minimum supported Python version. Earlier versions support also Python 2.3 and 2.4. Robot Framework is currently not compatible with Python 3.x versions."

But it was so casual (and appeared to be a "here's the older version of python you can use" which I didn't matter to me), I didn't notice it, and it took me a while to figure out that this was the problem. It seems like something like this would be bolded or something.

c. As I mentioned earlier, the Selenium2Library windows install file doesn't actually include everything you need to run Selenium2Library.

d. pip is a command line tool, which doesn't work that well with proxies, which makes it very difficult to use.

Bottom Line

I've written my own projects before, and I understand if this gets ignored, as sometimes I don't feel like working on things that are important to other people. But I wanted to share (and perhaps have a chance to vent my frustrations) at how RobotFramework's nasty install process undermines it's ability to be presented as a test framework that's easy enough for testers to use that they can write tests in it themselves, rather than relying on programmers to do it.

How it could be made better -

1. There could be 1 single windows installer (that would work with Windows 7) to get everything installed. Alternatively, it would be great if everything could be run from 1 directory without needed to install anything - just download it and run it.

2. Assuming that's to difficult, it would still be a lot easier if there were only 3 things that needed to be installed:

a. Python - be clear that it needs 2.7, not 3.x

b. RobotFramework (this already works)

c. Selenium2Library - the windows installer would need to actually include all the files it needs, not just some of them.

I still might end up posting, as I'm somewhat interested in writing a java project that would run a central server for running your tests on (and returning results). But unfortunately, it won't be part of my main project - 50% of which is just because of the complex and confusing install process from behind a proxy.

[Kevin O.]

Thank you for sharing. I can relate a lot to what you are saying here.

The day-to-day usage matters more than the installation, but first impressions count -- a lot.

Our solution to the installation part involves installing Python, adjusting system path, installing Oracle client, adjusting path, then running a Python script.

The script installs dependencies using pip and a folder containing 23 pip packages that are mirrored from source control. This bypasses the whole proxy issue.

I wish transparent proxies were more prevalent!

Some people put the entire framework in source control (not just installation files) and others use tools such as buildout. Buildout did not work for us as Database Library and cx_Oracle did not install with it.

Managing dependencies in the Python world is so much harder than with Java/Maven.

As far as Selenium2Library is concerned, I'm not sure what to say. Selenium does not install all the necessary files for Selenium - this was a design choice by the Selenium team.

We run our tests through Jenkins (a centralized server) and use the Robot Framework plugin to track execution history. For larger projects, tests are distributed to Jenkins slaves and a short script calling rebot combines the results.

I would think that one of the consequences of using FitNesse is only people who can code in Java can contribute significantly to the automation effort. With Robot, people who are not quite as technical can contribute as well. The idea of non-technical people being able to contribute significantly to automation is a myth in my eyes.

Kevin

[Mukesh T]

hello,

=> on installation/setup part : yes, i could see the pain in first time setup of RF with various libraries (some installations through windows installer, some with pip...). 

this was when i tried to setup the whole framework on my office workstation. however, on your point "And pip is not happy with proxies", i would say 'pip' has the option of mentioning the proxy too, and this is what i used after i failed to install with simple 'pip install packagename' : 

pip install --proxy="user:password@server:port" packagename

in my opinion, as the tool installation is not going to be a regular activity (it's just a 1 time setup), even if it's a bit complicated, it shouldn't be a strong criteria for deciding the automation tool :). also, i see some experts on this forum who talk about the best practices for installing/deploying the test framwework/libraries (one such thread - https://groups.google.com/forum/#!topic/robotframework-users/0HcqIW9vS2E).

best,
Mukesh

[Taylor, Martin]

I disagree with the opinion about “tool installation is not going to be a regular activity”. You’re ALWAYS going to be adding new users who have to go through a 1^st time installation, and RF and RIDE update often enough that ALL users will have to deal with this update/installation pain on a regular basis.

 

When we adopted RF and RIDE as the standard test automation tools for our company, my manager INSISTED on a “one-click” install capability. We’ve never quite reached this lofty goal, but our solution has been to build our own install and update process. We keep ALL of RF and RIDE source code in our own SCM system and update it there from newly released source code on a periodic basis. BeyondCompare is my favourite tool for doing this. When we create a release of our >30 custom Test Libraries we include the latest RF and RIDE. We make sure that EVERY Test Library has a Python installer, whether its written in Python or not. We also wrote our own “InstallUpdate” tool that knows how to find the latest version of each RF Test Library (and RIDE) from our network share where test automation build artifacts are stored. This way most of what we need is installed from behind the corporate firewall. We do use PIP for a few subcomponent installs, specifically for the Selenium2Library. However we wrap this in our own custom Python setup.py program that knows very well how to negotiate its way through our corporate firewall and get what it needs via PIP. This custom wrapper hast to be quite flexible, since we must deal with installations not only from our main headquarters site but also from remote “embassy” contractor sites with very restricted intranet access rights.

 

So, to summarize, I’d say that RF and RIDE are THE BEST TEST AUTOMATION TOOLS AVAILABLE, and are worth the extra effort to build custom installers for them. Consider this part of the “cost of open source”… TANSTAAFL.

 

Cheers,

Martin

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at http://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/groups/opt_out.

[WillDev Dev]

"pip install --proxy="user:password@server:port" packagename"

I found that while trying to get it to install, but it didn't work. Or more accurately, for some reason I didn't need it on Windows 7, but on Windows Xp it just plain didn't work.

"in my opinion, as the tool installation is not going to be a regular activity (it's just a 1 time setup), even if it's a bit complicated, it shouldn't be a strong criteria for deciding the automation tool :)."

Everyone has a limited time to perform an evaluation of a tool. When 90% of that time is spent trying to get the tool installed - the tool ends up getting evaluated based on it's crummy installation process.

RobotFramework itself could use clearer documentation, in my experience, but the bigger problem was that Selenium2Library which for who-knows-what reason decided to create a windows installer but doesn't include other libraries that it needs in order to work. In order to get it installed, you have to use pip - a command line tool that doesn't work well with a proxy.

It's like trying to sell a "very reliable" car, but the drivers side lock is busted shut, the windshield is cracked, and there's rust on the hood - who the heck is going to believe you?

I could live with running 7 or 8 install files to get it installed, but the apparent inability to install RobotFramework+Selenium2Library from work is a dealbreaker. And gives the strong impression to anyone evaluating it that it's some sort of command line hacker tool.

[WillDev Dev]

"I disagree with the opinion about “tool installation is not going to be a regular activity”. You’re ALWAYS going to be adding new users who have to go through a 1st time installation, and RF and RIDE update often enough that ALL users will have to deal with this update/installation pain on a regular basis."

Yeah, your story shows how far people go to have an easy install process.

"Consider this part of the “cost of open source”… TANSTAAFL."

Yeah, just wanted to share. 

[Mukesh T]

thanks Martin for sharing your views & best practices on RF installation. hope to see such practices documented somewhere which will be useful for the RF community. 

@WillDev Dev: not sure how this pip install with proxy option worked for me without issues (on win xp) :)

best regards,

Mukesh

On Friday, December 20, 2013 12:24:41 AM UTC+5:30, WillDev Dev wrote:

[WillDev Dev]

@WillDev Dev: not sure how this pip install with proxy option worked for me without issues (on win xp) :)

Lol, I don't know either - I've read that there's a Microsoft proxy that open source software has trouble getting through, maybe that's what the IT department is using for WinXP but they switched to something different for our Win7 installs. Our Win7 machines clearly do the proxy differently - for us, you just need the proxy url for win7 (no username or password needed), whereas on winxp it asks for a username and password.

[Lars Nordin]

On Thursday, December 19, 2013 1:54:41 PM UTC-5, WillDev Dev wrote:

2. There's no central server for running tests and viewing test results

This is just how the tool is. Had #1 not been a problem, I might have been able to write and contribute a central server to the project here myself. Just wanted to mention it - they really wanted a place they could have long running test suites running that wasn't their local test machine.

I've used Fitnesse and Robot Framework, and some comments:

Having a central server to run tests doesn't apply to either project - this is how you decide to use the framework.

I've used RF and Jenkins as the central test server and that works really well - especially considering the features of the RF Jenkins plug-in (nice graphs and reports). Using Jenkins gets you the added benefits of being able to do Continuous Integration, once you have the automated deploy worked out.

I've never integrated Jenkins and Fitnesse, but there is a Jenkins plugin for Fitnesse. Now, you could just setup a dedicated system for running Fitnesse tests but you wouldn't get any of the reports and build-deploy-test workflow (CI) that you could get with Jenkins.

[WillDev Dev]

Thanks for the response, but having a central server to run tests does in fact apply differently between the different projects. FitNesse has it built in, RobotFramework requires setting up Jenkins and getting it working with the jenkins plugin (which is pretty annoying, but not a dealbreaker). When the testers were doing their own research, FitNesse was often described as having the advantage of having a central server.

FitNesse made it easy to run a single test. The Jenkins plugin did not.

Neither one was good about scheduling automated tests to run in the future.

FitNesse made it easy to edit the data for the test on the central server. RobotFramework did not - at all. Personally, I would have been happy to argue that this wasn't needed, but something like testers being unable to run tests because the password expired for the test user was a problem.

The "central server" thing was a huge thing though. Like I said, between that and the horrid install process (trying to get pip to get around a proxy, command line commands, etc) - these were just the things that made the difference for the people I was working with.

[Markus Bernhardt]

Hi,

almost exactly a year ago I managed to get FitNesse discarded and RobotFramework introduced at BMW.digital. Some thoughts:

Reasons to discard FitNesse:

• FitNesse tests cannot be versioned with git or svn.
• FitNesse alone cannot drive Selenium. You need to:
• Either develop modules in Java that use Selenium. Therefore you need Testers with Java skills, which you can hardly find. We tried that for a year an it didn't work for us.
• Or use something like Xebium. Buggy and unstable like hell. We tried that for another year an it didn't work for us.
• FitNesse needs an server. You cannot simply execute the tests on a workstation without installing tons of things.
• FitNesse server has no access control.
• FitNesse server has no real scheduling ability.
• FitNesse server cannot control which tests may be executed in parallel and which not.
• FitNesse has no IDE
• It's hard to reuse test components

Therefore we were looking for alternatives and found RobotFramework. Our biggest problem was also to get it installed. We decided to use Jenkins as scheduler, but on that servers it was not possible to get Python installed. So we had to stick to Jython and we realized there is no Selenium2Library for Jython. That's the reason why I ported the Python library to Jython last year. By now we execute all tests with Maven on the central servers and the workstations.

Things to improve for RobotFramework IMHO:

• Install process
• Make RIDE executable by Jython
• Intelligent parallel test execution.
• Make it configurable how much tests are run at the same time.
• Analyze the last runs. Group the tests to the least overall time.
• Make it configuravle which tests are to be executed atomic or not parallel to some other marked tests
• Thread safe libraries. Needed for parallel tests
• Full support of loops with better syntax. 
• Support for blocks of keywords. Something like:
• Run Keyword If     ...    {
          Do A
          Do B
  }

• Get rid of Python. Switch to Java 8-)) 

Only my 2 cent,

Markus

[Guy Kisel]

For parallel test execution, check out https://wiki.jenkins-ci.org/display/JENKINS/Parallel+Test+Executor+Plugin

[Markus Bernhardt]

Hi Guy,

For parallel test execution, check out https://wiki.jenkins-ci.org/display/JENKINS/Parallel+Test+Executor+Plugin

I know that plugin and it's not what I need.

Let's say you have 1000 test cases. 

Testcases can have different runtimes:

• 20 % test cases run in 1 - 5 seconds. 
• 50 % test cases run in 5 - 60 seconds.
• 25 % test cases run in 1 - 5 minutes. 
• 4 % test cases run in 5 - 15 minutes.
• 1 % test cases run in 15 - 120 minutes.

Testcases can have dependencies:

• While the upload from system A to B is tested no changes in A and B are allowed
• While test 4711 runs, test 2342 must not run
• ...

I would like to simply tell Robot to use N test executors. To make that work Robot would have to:

• Build atomic groups of tests by the dependencies
• Order the groups by the projected runtime. At the first run the runtime of every test case is set to the same duration. All further test executions use the duration of the last test execution. Longer running groups come first.
• Start as many executors as requested
• When a executor has nothing to do give him the next group from our list to execute.
• When all tests are finished build a combines test report

I know I can do lots of that things by hand, but to have that built into Robot would rock.

Cheers,

Markus

[Sarah McK]

Seems like an integration with TLB (Test Load Balancer) may be in order -- has anyone used it?

http://test-load-balancer.github.io/