Acceptance (UI) Testing Frameworks KIF vs Jasmine vs ???

[Luke Tupper]

Hi Everyone

A project I am working on has automated acceptance tests written using Jasmine & Jenkins. I have had a brief look at KIF (https://github.com/square/KIF) and it seems to fill in some of the holes that are already apparent in Jasmine. So far, with only a high level look at KIF I see the following adv/disadv:

• All written in Obj-C, although I have been writing Obj-C for nearly 10 years now so I might be biased

• Able to call internal functions to emulate functions. The square example 'injects' a credit card being swiped. 

• Hopefully you will be able to 'dig deeper' than the standard accessibility api allows. For example we have run into issues with Jasmine running being able to only see a concatenated list of label text on button and table view cells. I am sure there is a way to dig into the cells/buttons but it isn't obvious. 

The downside I see is that it requires a bit of work to get it installed, and the KIF framework is installed while testing which could be causing side effects in the app, although the unit tests should be catching anything like this. 

The only 'wrinkle' that our project has is that we are aiming to write one set of acceptance tests and then write two sets of test scripts, one for the iPhone and another for the iPad as the functionality is the same but the steps to get the information is different. For example after login on the iPad we present account information and then a list of current actions to the user, whereas the iPhone the account information is on one tab and the current actions are on another. 

So what I want to know is should we ditch Jasmine and go with KIF, or is there something else out there that is a heap better?

Thanks

Luke

[Jasper Blues]

I haven't actually tried any of these, but for functional-style tests, could you use one of the dedicated functional testing frameworks: 

* iCuke

* LessPainful.com (https://github.com/calabash/calabash-ios)

* http://www.telerik.com/automated-testing-tools/   (this is the company that makes KendoUI)

Jasper

--
You received this message because you are subscribed to the Google Groups "Australian Cocoaheads" group.
To post to this group, send email to cocoah...@googlegroups.com.
To unsubscribe from this group, send email to cocoaheadsau...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cocoaheadsau?hl=en.

[Stewart Gleadow]

Shameless plug: I wrote a comparison of some of the automated UI testing tools a few months back, I think most of the points are still valid. http://sgleadow.github.com/blog/2011/10/26/which-automated-ios-testing-tool-to-use/, my post was based on a talk by the author of Frank, Pete, video is here http://pivotallabs.com/talks/147-frank-cucumber-tests-for-native-ios-apps

I wouldn't touch iCuke, it's a dead project as far as I can tell. Calabash is basically a clone of Frank, to avoid some GPL-ness. It may merge back into the Frank project, and doesn't have the community support Frank has.

I quite like KIF, but I'm not sure how much Square are supporting it. I spent a day with the developers of KIF 2 weeks ago, and they seem to be waiting to see what Apple releases in June around automation before committing more resources to making it better. With so many ex-Apple people at Square, I got the impression they knew something I didn't! Square have some nifty extra tools they use internally that haven't been open sources yet, but would make KIF a whole lot better.

KIF does the job, and is suitable if your team is comfortable with using Objective C for high level tests, and your app doesn't have complex interactions with a backend system. Frank has by far the most active development and community around it, but does add complexity to your test environment. Frank is better when your app does have complex interaction to the outside world, and when your team is comfortable using Cucumber and Ruby already. UIAutomation sucks, but has good device support, and is always going to be up to date with the latest and greatest. Zucchini makes UIAutomation nicer if you don't mind needing to know CoffeeScript, Javascript and Ruby to write your tests.

Luke - the idea of having a common set of test scenarios, with different implementations for iPhone and iPad is something Frank/Cucumber would give you almost for free, just by having separate step definitions. I know projects doing this with common high level tests across iOS, Android and mobile web, with the step definitions implemented differently for each platform (using Frank, Native Driver and Web Driver respectively).

- Stew

[Jasper Blues]

Oops - I was getting mixed-up between Frank and iCuke. . . . and of course should've mentioned UIAutomation too. . .

Speaking of UIAutomation, what are the main limitations, besides not having an easy way to kick of CI/cmd-line builds? (Is there a work-around to this yet?)

[Luke Tupper]

It relies heavily on UIAccessibility, which is good because there are a lot of apps that wouldn't normally be accessible but are because of testing, but the accessibility API does limit what you can see into. For example a UITableViewCell or UIButton that have custom UILabels as subviews, the accessibility will concatenate the text of all the accessibility labels. Eg if you have a table cell with two labels, product and price you will see a accessibility label for the cell of 'Alarm Clock, $12.95', but as far as we have look into it (not very far) you can not check the values of the individual labels. 

[Aidan Steele]

Is this still an issue? I don't seem to have any issues running Zucchini scenarios from the command line. Supposedly it also supports CI as well.

On Mon, May 21, 2012 at 11:13 AM, Jasper Blues <jasper....@gmail.com> wrote:

[Stewart Gleadow]

UIAutomation works from the command line, using the 'instruments' tool. Look at the Zucchini source code to see how you might kick off builds using this.

UIAutomation is a way of running a single Javascript file, in a cut down environment using Apple's own half-baked #import syntax (I think, I'm not a JS guy). It gives success/failure by logging to an output file, no real assertion framework. You can use jasmine with it that makes it a bit more bearable.

The only useful way to manage proper test suites with it is to have a separate tool that compiles the Javascript to feed into UIAutomation. This is what Zucchini does, but goes the extra step of being able to write your tests using CoffeeScript page models and something that's almost business readable.

The command line integration for UIAutomation is a real afterthought, and a pain to deal with. If you want to run from CI, you have to use a third part hack to install apps on the device (like fruitstrap) because of a long-term bug in the instruments command line not actually installing the app you want to test before running the tests. It has the same limitations as KIF for being able to effectively stub out external services.

UIAutomation is useful for writing little scripts to interact with the UI while you have a bunch of other Instruments running, to monitor performance. It's not great for BDD at all.

Bwoken takes a similar approach to Zucchini for making UIAutomation bearable (without the coffeescript, page models and screenshot comparison), but is a bit lighter https://github.com/bendyworks/bwoken

- Stew

[Mark Aufflick]

FWIW I'm liking KIF, but have only written a few tests with it so far.

I haven't used Zucchini, but I have done quite a bit with Apple's UIAutomation and I have to say IMO anything based on the UIAutomation is going to be a world of hurt.

--
Mark Aufflick
 http://mark.aufflick.com/about/contact
 http://pumptheory.com/about

[Derek Clarkson]

Hi, I have a new framework called Simon which is embedded in your app. This gives it the ability to access the internals of the app. It's Objective C based. It's still new, but you may find it interesting. http://drekka.github.com/Simon

ciao

Derek

[Sadat Rahman]

On 21/05/2012, at 9:49 PM, Derek Clarkson wrote:

Hi, I have a new framework called Simon which is embedded in your app. This gives it the ability to access the internals of the app. It's Objective C based. It's still new, but you may find it interesting. http://drekka.github.com/Simon

Getting a 404 on that link. Correct link: http://drekka.github.com/simon/

Simon looks like a nice cross between Frank & KIF.

[Derek Clarkson]

Thanks for the comment. 

I'm really interested to hear from people about what they think. I'm currently thinking about how I can run it from CI. The biggest questions being around techniques for restarting an app after a failure. 

ciao

Derek

[George McKibbin]

This is kind of an interesting project: http://sikuli.org/ It's free
too. It's an image based programming UI testing tool.

[Steven Holloway]

A friend of mine who does a lot of cross platform work is using sikuili….

I have not had a chance to play but the demo he gave me in sydney last month was pretty impressive.

He reckons he gets some test reuse across platforms, however I didn't get enough detail.

Steven Holloway

Software Engineer, Music Producer, Dj

Melbourne Australia

[Sadat Rahman]

On 30/05/2012, at 8:19 PM, Steven Holloway wrote:

> He reckons he gets some test reuse across platforms, however I didn't get enough detail.

SIKULI doesn't allow you to run your UI acceptance tests on a device, which is a big minus in my book. People buy real devices not iPhone Simulators.