UIAutomation—Anybody having success with the tool?

[Billy Gray]

Hi all,

I came across a video from WWDC 2010, "Session 306 – Automating User Interface Testing With Instruments," and I was intrigued. Over the past few days I've been trying to give UIAutomation a real go with one of our apps. From what is shown in the video, it looks like a fantastic tool, and it would do me a lot of good. If I could get it working correctly. I came across three problems:

1) The JS object representing UILabel objects has a bug where .value() just returns .name(). Not a big deal, but surprising. Apparently this has been a problem for some time now? 

2) String comparisons don't seem to work. It looks like the good old == operator should do the trick, but sometimes it fails, despite the strings matching. Sometimes it doesn't! I also tried using localeCompare() instead, but that suffers from the same instability, it's very strange.

3) And then the application target dies during the run. I know, I know, sounds like my app is crashing in Release build and I just didn't know it—but I can't reproduce the crash in Release builds, it only happens during the automation instrument's run, right when my script runs .tap() on the login button of my app's main view. I thought maybe blocking on the main thread or something was causing it to get canned, tried different timeouts, seems unrelated. It goes down immediately. In the system log I end up seeing a crash/backtrace that appears to be from some component of Instruments.

This is all a long-winded way of asking: is anybody making serious and frequent use of this tool to automate some of their user interface testing? It looks very much like it's not ready for prime time, and not being maintained, and that I should stop looking into this now and get back to work on other things, but I figured I'd ask around. I took a look at the Developer Tools forum and it doesn't seem like anyone is having much success with it. 

Thanks for your input!

Cheers,

Billy Gray

[Jim Matthews]

One data point:

I used UIAutomation a fair amount in 2011 for stress-testing an application, e.g. to simulate a long, data-intensive user session to look for low memory handling problems. It was helpful in the sense that once we could get our tests to run for hours without a crash we felt more confident in our code. But we struggled to get to that point, with lots of Instruments crashes. We couldn't do what we most wanted to do, which is run UIAutomation scripts while monitoring memory use with other instruments -- that would always crash for us. And we couldn't make automated tests part of our build process; I don't think there was support for it, and even if there had been Instruments was too flakey for that to have been useful.
--
Jim Matthews
Fetch Softworks
http://fetchsoftworks.com