Automation with Selenium Plugins: flow control, include command, data driven

[java_non_ninja]

Wonder if anyone can tell me, the three plugins above, can we use these in automation, ie currently we have batch files, triggered by ant, that run selenium test suites, as well as run some sql and other things - can we incorporate those plugins into that?

If not, can they be incorporated into webdriver?

[Mike Riley]

Selenium project provides a single plug-in for Firefox, the IDE.

So I have no idea what these "three plugins" are that you are talking about.

Mike

[Jim Evans]

Mike, Selenium IDE has a plugin system for itself, adding additional features not provided by the bare bones Selenium IDE. The plugins are produced and maintained by third-parties, mostly individuals, and as such are of varying quality. I'm on record elsewhere concerning my opinion of the utility of Selenium IDE, so I think I'll leave it there.

[java_non_ninja]

Jim I think my bosses are coming round finally to this way of thinking and are thinking of hiring a Selenium guy, ie one who knows webdriver, proper test automation etc. My pleading with them was to allow me more time to learn it, so I guess they listened to me but not in the way I hoped. Depends if we hire a new guy and he keeps, and is allowed to keep, his knowledge hidden from the rest of the team, or if he shares, or is made to share.

It baffles me how you cannot see the utility of IDE though?! Our company has used various testing systems, usually determined by the budget, as well as the whims and fancies, of whichever customer. But for ourselves we use junit for unit testing and selenium IDE for something approaching automated system testing. We have set this up for some clients who have paid us respectable amounts to do so. If we had used a different tool, the cost would have ate into our earnings, if we'd used webdriver, well we couldn't have delivered. Don't you ever meet non technical BAs and testers (or non webdriver savvy programmers) who just want something that just records them clicking through a test? Using ant, bat files, sql scripts, additional java modules, we can still build a functioning, and selling/paying, test suite from it and using IDE means the actual tests can be recorded by BAs and just popped in to the structure.

Webdriver would eliminate this, surely whilst that would have some utility gains, it would have the utility loss I just mentioned too? For my company I think we should retain the ability to let BAs and testers contribute tests to our autosuite, though I hope we developers can use webdriver for ours in future.

[David Lai]

Personally I see Sel IDE's main value is useful for throw away tests.  This is good for attaching to bug reports, or putting into some sort of canary project to run as part of a regression run for  2 or 3 sprints (or until the tests fails again for known reasons).  

Problem is IDE is it's rather limited in what it can do from a programming prospective.  If you want to do any sort of concurrency or parallellization of tests, you're pretty much dead in the water.  From a CI scale perspective, IDE is not a good fit.  Devs like to get feedback the moment their code breaks.  If your test suite cannot be parallelized, it becomes very difficult to incorporate it into a Dev/CI workflow.  Let's say a dev checks in code, and the code starts building, deploying, and tests running.  If the test takes 4 hours to run, then other devs cannot check in their code until the tests finish running, which is makes a full CI workflow with automated tests much harder to implement.

Another issue is it's lack of flexibility.  I think there's a camp of people who think of automation as a replacement for testers, where they run in the backend of release cycle in the testing phase, and another camp who sees automated tests as a compliment to unit tests which runs in the integration phase and not only on CI systems, but also running on dev boxes for devs to test their functionality while they are developing.  For running tests on the front end of the dev cycle, you need something that's highly customizable and configurable which an API like webdriver or RC can provide.  This allows you to say, create custom test runners and build scripts that can support say something like automatic branching support.  This is also why unit testing frameworks as a top level test executers have been pretty popular.  They're easy to customize to fit any dev work flow.

[Jim Evans]

I've never said that record/playback tools had no utility. Or if I did, I misspoke. I do believe that they have extremely limited utility, and that I've never seen one that produces elegant, maintainable code.

Look, automated testing is *development*. It is nearly impossible to create a utility that is record-and-playback, but sophisticated enough to handle the complex scenarios that arise out of all but the very simplest of automated tests. Want to reuse code in another test? I hope your record/playback tool understands subroutines and/or functions. Need to handle multiple possible outcomes for a given user action? Congratulations, your tool now needs to understand branching and Boolean logic. Have to perform the same steps multiple times in a test? Now you need looping constructs.

Efficient code requires those pieces, and once you've grown those things into your record/playback tool, you're programming, whether you call it that it not. And designing a language is *hard*. Knowing that, isn't it better to rely on a language that has years, and in some cases decades, of practical design and real-world use behind it?

I'm not just speaking theoretically, by the way. At a previous employer, I helped design and build a keyword-based automation framework that had "usable by non-programmers like business analysts and testers with no development experience" as one of its stated goals. By the time we finished, we had developed an entire language, complete with our own IDE and debugger. After doing the cost-benefit analysis in hindsight, it would've been cheaper and easier to train other users how to program, and use the development tools available to everyone. Plus, we would've had the support of our development team in running our tests.

Of course, it's probably that I'm just stupid and that the problem really is much easier than I make it out to be. I'm not the smartest guy out there, by any measure. Incidentally, this is why you should take what I say with a grain of salt, because I really am an idiot. But I'm pretty sure that not *all* of the code I contributed to the half-million lines of code making up that framework are crappy, and I *know* I had people smarter than me working on it too.

Bottom line, for me, is that record/playback tools almost always produce inelegant and unmaintainable code. No, rerecording the script doesn't count as maintainable to me. It's entirely possible that the perfect record/playback tool exists, but I've never seen it, and I do try to keep up with the state of the art in my industry. As for elegance, if one isn't striving for elegant solutions to the problems we face as software professionals, I'd submit that one is in the wrong profession.

I'm glad what you're using works for you and your company. I'm self-aware enough to be sure the approach your company takes would never work for me. But again, I'm an idiot, so you'll probably see me taking swipes at record/playback tools again at some point. Feel free to ignore me; most do, and are happier for it.

[java_non_ninja]

Jim obviously I am not suggesting you are an idiot. Clearly when it comes to automated tests, as well as much else no doubt, compared to the people on this board if anyone is the idiot its me. I am the idiot who is trying to educate himself in this regard. I have to persuade myself, so that I can persuade my bosses, that its worth using webdriver rather than what we do at present. Obviously we work in a world of limited time and resources and the question is which will save us time and money in the long run as well as giving us the most robust and flexible tests. As things stand we could write or rewrite a load of IDE tests tomorrow, whereas webdriver is going to incur an upfront cost either of training or hiring.

Its difficult to persuade my bosses so far when I say "apparently IDE can't do this or that" they say "it can, we are doing it" though what they mean is it can in conjunction with the complicated support structure of ant, java and sql tests we have IDE tests running within.

If you are saying that in the long run it would save time if we either trained up our BAs to do webdriver, or got them just to spec us the tests required and we would then build them in webdriver, well I could see that might be possible. We recently did have to rewrite, restructure a lot of our previous tests, and it was a pain, I certainly can believe there are better ways.

As I say though, it does seem my bosses have come round to some extent in that they are looking to hire someone with test automation, including webdriver skills. Perhaps he will be able to convince them this is what to do, or maybe hiring him is a sign they've already been convinced.

"It is nearly impossible to create a utility that is record-and-playback, but sophisticated enough to handle the complex scenarios that arise out of all but the very simplest of automated tests. Want to reuse code in another test? I hope your record/playback tool understands subroutines and/or functions. Need to handle multiple possible outcomes for a given user action? Congratulations, your tool now needs to understand branching and Boolean logic. Have to perform the same steps multiple times in a test? Now you need looping constructs."

Just to answer this, first of all, I have persuaded my bosses that if anything comes up that IDE cannot handle, then we will use Webdriver for those tests at least. So far we haven't hit one, but I'm sure we will. (And part of me hopes we will, more on that in a minute.)
We do reuse code. For example we have a login procedure. We have a page where we add contact details. We have a group of pages where we add payment/banking information. The latter two are done at different stages of different scenarios. We have all these bits as separate IDE files, and then we use batch files to call them in sequence as needed, lots or reuse, we use batch files to perform the same steps multiple times, we don't use it for decision making at present as so far we know what we want to happen in each case and if it doesn't its a fail, but I could see where this is useful, again we can do it in our batch files or in our slq/pl/sql/java files that we run in the batch files.

I guess it could be the case that instead of all this infrastructure, we could use for example page objects in webdriver for login, for contact details, for banking, and do all the other bits I've described in webdriver rather than in bat files? Certainly I could see how this would be more elegant, I'm just left with persuading my bosses to sink the costs to let us learn this.

The other thing I wondered, and they wonder, is whether this is just programmers wanting to do the coolest thing rather than the cheapest, fastest, most reliable things. We suffered from that in the past as two hot shot developers we had would learn something new and then come in and rewrite/restructure huge swathes of our code, which left the rest of us not understanding it, which needed to be retested (but they didn't bother), and I know my bosses think part of this is me wanting to learn webdriver because it would be cooler, more interesting than sticking ide files in batches, and they have a point.

[java_non_ninja]

David I see what you are saying, at present we only run it as a regression test on our build each night so it doesn't cause us too many issues.
If we were trying to run these tests before hotfixing into live or something I could see what you say coming into sharp focus.
At present we tend to just run local tests on such things, whack it in and run the regression at night - certainly see room for improvement there, but again its about what the customer demands and is willing to pay for.

[Jim Evans]

"I guess it could be the case that instead of all this infrastructure, we could use for example page objects in webdriver for login, for contact details, for banking, and do all the other bits I've described in webdriver rather than in bat files?"

This is my point exactly. Let me ask you this: Is it more efficient for a developer to write code for a task in one language or two, all other things being equal? You've created a system whereby you're coding steps in one language, and using shell code (batch files) for "functions" and "loops". Surely you can see the inefficiencies in that as a developer, right? Additionally, you're lucky your organization has decided that you're only going to test on Windows. If you ever decide you need testing on mobile platforms, or, say, Safari 6, your batch files will have to be reworked into something cross-platform.

"The other thing I wondered, and they wonder, is whether this is just programmers wanting to do the coolest thing rather than the cheapest, fastest, most reliable things. [...] I know my bosses think part of this is me wanting to learn webdriver because it would be cooler, more interesting than sticking ide files in batches, and they have a point."

You've already seen how hard refactoring is under your current framework. If your own experience doesn't convince you or your bosses, nothing I say will do so. It's not about "wanting to learn [what's] cooler," it's about being efficient and productive. It comes down to where you want to sink the cost. Using WebDriver (or any code-based tool) is going to require the cost up front in training and infrastructure. Using Selenium IDE (or any record/playback-based tool) is going to require the cost later when the tests need to be maintained. It's been my experience that the overall cost of the former is lower than the cost of the latter, often dramatically so.

[java_non_ninja]

"This is my point exactly. Let me ask you this: Is it more efficient for a developer to write code for a task in one language or two, all other things being equal? You've created a system whereby you're coding steps in one language, and using shell code (batch files) for "functions" and "loops". Surely you can see the inefficiencies in that as a developer, right?"

I agree that there is an inefficiency there, ie it could be faster to write it all in one language. Though I think we still see it as an inefficiency if a developer has to write tests for BAs that the BAs could have written themselves.

Actually thats a very important point that I forgot to mention, I thought that I could have the best of both worlds! Whe I found the "export test case as... webdriver etc" function in IDE, I thought we could get the BAs to write the tests, hit that button, and then run them in webdriver, with the capability of being able to do lots of other processing in java, and ant. However on the system we tried this on, it was an absolute nightmare getting the webdriver tests to run, when we abandoned it I had only got two pages in to the system. This system is a very old one, created with a very old code generator which does what I suppose are weird and ancient things in jsp/html etc.

Since then I moved to our most modern system for a different customer, written properly by us in java - and the export function seems to work fine, very little tweaking involved. I am in the process of setting up a few tests in this in my spare time, hopefully I can demonstrate this as a possible solution, ie the BAs could still write and record tests, then we could convert them and run them in a webdriver format? I think I recall from discussions many moons ago that this is also regarded as sub optimal by Selenium people? Perhaps though its a way of getting webdrivers foot in the door.

"Additionally, you're lucky your organization has decided that you're only going to test on Windows. If you ever decide you need testing on mobile platforms, or, say, Safari 6, your batch files will have to be reworked into something cross-platform. "

Now that is a truly excellent excellent point. We are in an industry, and a process, whereby change involves sometimes us dragging a customer kicking and screaming into the 21st century, or sometimes the customer or the market dragging us, step by step. We haven't had to make a system run on mobile yet but we surely surely will. I can't believe I didn't think of this already to be honest, thank you, this will be added to my case for webdriver and put to my bosses.

Your other paragraph, yes well it almost always works out better to sink the costs and think/work a bit harder at the beginning of a project than later and I can believe this decision is no different. It is also the case that many companies and bosses make the opposite decision though I would say ours are normally good in that regard. I doubt if I will persuade them or the BAs to drop the idea of BAs recording tests without developer support though (even if of course in reality they need loads of support) but if I can get them on at least to a twin track where we have webdriver running as well, or where I convert their tests to webdriver, hopefully webdriver will prove itself and push out the other approach.

I have moved a further distance toward webdriver in light of this conversation so thank you, lets see if I can get my bosses to move a bit too.

[Mike Riley]

You are right Jim.  It has been so long since I used it that I had forgotten!

I used the ones that produced WebDriver and WebDriver with TestNG as I recall, back when I was learning how to use Selenium.

Mike

[webstarter]

If anyone still reads this, I was deleting old emails and saw this thread.

I am pleased to report, either because of my advice, or because of my struggles with this, that my company hired a proper webdriver developer/tester soon afterwards, and we have a fully functioning webdriver suite.

I do more java programming, but I do have some webdriver tests running on my stuff too, either that he has written for me, or where he has given me bits that I can plug in and reuse for my testing.

In hindsight, the advice given was spot on, webdriver is far superior to IDE, to trying to write our own batch processes for testing and so forth.

On Tuesday, 23 April 2013 11:42:54 UTC+1, Jim Evans wrote: