Status on CI with Frank

[Krukow]

Hello,
I am interested in having a complete continuous integration
environment where I can use hudson/jenkins to run a script that

* checks out sources using mercurial/git/svn

* builds my project for both iPhoneSimulator.platform and
iPhoneOS.platform

* run unit tests

* installs build in iPhoneSimulator

* runs each Frank cucumber feature on simulator using each of the
available iOS'es (now I have : 3.2 ... 4.3.2)

* installs build on a physical device (iPhone/iPad) connected via USB

* runs each Frank cucumber feature on the physical device

* reports all results (including any screenshots and/or videos taken)

So far I can find blog posts that can do the first three steps. And
people here have been talking about running Frank on ci servers - so
some of the latters steps must also be possible. Can anyone report on
their experiences?

This is ambitious, I know :) but I am looking for people that can help
in automating those steps (if it is possible).

Thanks,
/Karl

[Pete Hodgson]

Hi Karl,

We have most of what you describe set up in our CI, which consists of Jenkins plus some rake tasks.

Installing an arbitrary build on a simulator is done by the sim_launcher gem as a freebie when launching that build. You can also specify which OS to run in the simulator as a parameter when using sim launcher.

We record video using the basic screen-recording functionality that's built in to the quicktime player. To be honest it's been a little bit flaky (the build machine has to have a physical monitor connected, for example). Eventually I would like to move to using a vnc-based recorder (probably using Castro, or a ruby equivalent), but it hasn't been a big enough pain to motivate getting that figured out. Our rake scripts are set up to output everything from a test run (movies, cucumber html reports, etc) to an artifact directory which Jenkins archives and serves up for reference. One thing to be aware of is that if you're working on a large product then you will generate a lot of video. Our full regression suite takes about 50 minutes, and is running a lot of the time because it's kicked off whenever someone checks in. So, lots of video is generated.

When diagnosing build issues we found it useful to have a way to correlate the cucumber scenario that was failing with the video of the test run, so I created a custom cucumber formatter which appends a timestamp when describing each scenario. I released that as the timestamped-scenarios gem. It's very easy to use; see https://github.com/moredip/timestamped-scenarios for details.

We've not done much on our team with getting testing on a physical device going. One reason for that is that I think it will be hard to automate. 

- Any tests that involve rotation obviously wouldn't work.

- Same with memory warnings, although Stew Gleadow mentioned to me that he had an idea for firing those programmatically.

- restarting the app between tests might be hard to do. Automating installing it might be even harder, unless I can find a friendly person at Apple to give me some pointers.

- I *have* gotten tests to run on a physical device over wifi (using Bonjour to find the device), but the setup seemed a bit flakey. I heard similar stories from Stew, who has a colleague that has tried the same.

I think that there is a place for running automated tests on the device but I suspect that most of the time it's going to work out better to run them manually, rather than in CI. All that said, I would be very very happy to be proven wrong :)

Finally, if you're running your CI box in your team room and want to *really* irritate your team I highly recommend configuring the cuke-puke gem as part of your CI build - http://vimeo.com/19607048 ;)

Cheers,

Pete 

[Krukow]

On Jun 21, 4:41 pm, Pete Hodgson <phodg...@thoughtworks.com> wrote:
> Hi Karl,
>
> We have most of what you describe set up in our CI, which consists of
> Jenkins plus some rake tasks.
>

[snip.]

Hi Pete,
Thanks for the detailed write up. It sounds like you have a great
setup running in your project.

First is is possible for you to share your rake files/build-scripts or
are they confidential? They seem kind of advanced, i.e., running video
and artefacts putting them in a specified destination.

Second, it seems that the only thing "missing" is the physical device
part (missing is in quotes since you don't actually miss it :)

I think there are some cases where you would like to run tests on a
device. For example, running on an old iPod Touch with iOS 3.1.3 is
something that has given our project some problems that aren't caught
in simulator, or e.g. performance issues on iPhone 3G. I do recognize
the difficulties you point out regarding, e.g., app install and
restart.

I'll do some research and post back if I find something useful. I'd
like to hear from others that are running CI successfully.

Btw, 1) how do you actually discover the device using Bonjour and how
is it "flaky"?

Kind Regards,
- Karl

[Pete Hodgson]

re: discovering the device via bonjour, check out my 'bonjour' branch in the main github repo.

The bonjour discovery itself wasn't flaky, but driving the app on the physical device itself via wifi seemed to be. Sometimes a command just wouldn't go through to the app. That said, I didn't spend much time at all looking into the cause.

Cheers,
Pete
--
Typed on a teeny tiny keyboard

[Stewart Gleadow]

I spent some time with the Xcode engineers in the labs at WWDC, and there is currently no way of installing and running the app on the device from the command line, and no plans to... so running on the device cannot be part of CI at the moment.

It may be that we can piggy back off Instruments at some stage to install and run on the device. The Apple UIAutomation tools allow for simulating rotation (makes the device think it's landscape, even when it's not) but they're currently designed without any consideration for CI or real automated testing unfortunately.

I'll see if I can get those simulated memory warning steps in as soon as possible... it will just be merging in the approach mentioned on iDevRecipes

http://idevrecipes.com/2011/05/04/debugging-magic-auto-simulate-memory-warnings/

- Stew

[Pete Hodgson]

On Tue, Jun 21, 2011 at 3:31 PM, Stewart Gleadow <sgle...@gmail.com> wrote:

I spent some time with the Xcode engineers in the labs at WWDC, and there is currently no way of installing and running the app on the device from the command line, and no plans to... so running on the device cannot be part of CI at the moment.

 

I'm not sure the outlook is that bleak. There is also currently no *official* way of installing and running the app in the simulator from the command line - we just use an undocumented API to do so. Maybe the same kind of thing exists for installing/launching on a device. XCode has to be able to do it somehow, right?

[Derek Longmuir]

Yes, I don't see why we couldn't figure out some way to applescript it
if we had to - might be fragile though.

Derek.

[Mark Warren]

For Xcode 3, I think it would be pretty simple. Use xcodebuild on the command line to build for iphoneos, then using AS, launch the project in Xcode, then issue the "debug" command. If a device is attached it should copy the pre-built executable to the device and then launch it.

For Xcode 4, ah, well, not so much luck unfortunately. Applescript support is somewhat limited in Xcode 4. I know the "debug" command isn't present as yet (Xcode 4.0.2). Also, if you pre-compile using the command line and open the project it does not automatically pick up the target that was built (I've got a project with five targets). Using System Events scripting you can issue the build and, I think, run commands, so if the correct target could be selected, then issuing those commands would, in theory, install and run the app on a connected device.

Regards

Mark Warren

[Krukow]

On Jun 22, 2:14 am, Pete Hodgson <phodg...@thoughtworks.com> wrote:

I've email Andrey Zaytsev from Jetbriains (AppCode).

He indicated that there *is* a way using the private framework:
MobileDevice.framework

See:
http://stackoverflow.com/questions/6172688/how-does-jetbrains-appcode-launch-the-ios-simulator

Let us see if he responds. Otherwise, we can start exploring
MobileDevice.framework ourselves.

/Karl

[Krukow]

On Jun 21, 4:41 pm, Pete Hodgson <phodg...@thoughtworks.com> wrote:

> Hi Karl,
>
> We have most of what you describe set up in our CI, which consists of
> Jenkins plus some rake tasks.

[snip]

Pete, what about supporting multiple languages and region settings?

This is something that is important in our project.

What is needed is a sim_launcher command to make the simulator switch
language, and then run a particular feature.
Also, if we want device support I suppose we have problems again :)

Does anyway have any experience with Frank and multiple languages?

/Karl

[David van der Bokke]

I think there is a way to swap languages by loading/editing a settings plist with desktop based code.  I haven't tried this yet but it is in theory possible.  This won't work for device though.

As for simlauncher.. has anyone captured their app console output using simlauncher?  I ended up modifying frank to use waxsim instead in order to get the console output into a log file. 

[Krukow]

On Jun 22, 2:26 pm, David van der Bokke <dav...@groupon.com> wrote:
> I think there is a way to swap languages by loading/editing a settings plist
> with desktop based code.  I haven't tried this yet but it is in theory
> possible.  This won't work for device though.

OK, I'll have to research this. Unless of course someone already knows
how to :)

> As for simlauncher.. has anyone captured their app console output using
> simlauncher?  I ended up modifying frank to use waxsim instead in order to
> get the console output into a log file.
>

No, but that is interesting in case the test fails. I don't know
anything about waxsim, but sim_launcher looks very powerful and
flexible to me, so I wouldn't want to change to something else.

/Karl

[David van der Bokke]

I did some quick research.. it all has to do with:

~/Library/Application\ Support/iPhone\ Simulator/4.3/Library/Preferences/.GlobalPreferences.plist

You can update this file and restart the emulator to change languages.  I haven't exactly narrowed down the change in the file yet that needs to be made.. but it is read/writable through desktop objective-c.  I also don't know if this is consistent for different OS versions.  I will do more research later on today and try to hack something together as a start.

[David van der Bokke]

Here is my hack job.. this will change the language to French for the 4.3 simulator if you close and reopen the simulator.  The tricky part is going to be mapping languages to keyboards for the lists and seeing if they are different between the simulators.

        NSString *path = @"/Users/davidv/Library/Application Support/iPhone Simulator/4.3/Library/Preferences/.GlobalPreferences.plist";
       
        NSMutableDictionary *plistData = [NSMutableDictionary dictionaryWithContentsOfFile:path];
        [self printDictionary:plistData];
       
        // add french to the keyboards list
        NSMutableArray *keyboards = [[NSMutableArray alloc] init];
        [keyboards addObject:@"fr_FR@hw=French;sw=AZERTY"];
        [keyboards addObject:@"en_US@hw=US;sw=QWERTY"];
        [plistData setObject:keyboards forKey:@"AppleKeyboards"];
       
        // pull french to the top of the languages list
        NSMutableArray *languages = [plistData objectForKey:@"AppleLanguages"];
        [languages removeObject:@"fr"];
        NSMutableArray *newlanguages = [[NSMutableArray alloc] init];
        [newlanguages addObject:@"fr"];
        [newlanguages addObjectsFromArray:languages];
        [plistData setObject:newlanguages forKey:@"AppleLanguages"];
       
        [self printDictionary:plistData];
        [plistData writeToFile:path atomically:YES];
       
        [keyboards release];
        [languages release];
        [newlanguages release];
        [plistData release];
        [path release];

[Krukow]

On Jun 22, 5:14 pm, David van der Bokke <dav...@groupon.com> wrote:
> Here is my hack job.. this will change the language to French for the 4.3
> simulator if you close and reopen the simulator.  The tricky part is going
> to be mapping languages to keyboards for the lists and seeing if they are
> different between the simulators.
>

[snip..]

Excellent, will try it out immediately.

/Karl

[Krukow]

Thanks David. I took your solution and extended iphonesim to support
changing language for a particular sdk.

Tested on my system, please be careful :)

https://github.com/moredip/iphonesim/pull/1

/Karl