Reset app without resetting simulator

[Jeff Hatfield]

Ever since 0.10.0 every single time I start a feature, the entire simulator resets. Content, Settings, and window. It doesn't matter if I have RESET_BETWEEN_SCENARIOS set to 1 or 0. I've been looking around, but each thread I see talking about resetting the simulator or the app is worried about RESET_BETWEEN_SCENARIOS and app data instead of the entire simulator. 

Again, I DO NOT want to reset the simulator or it's settings. Am I missing something? 

[Derek Dongwook Chung]

While leaving RESET_BETWEEN_SCENARIOS set to 0, try deleting the App Directory and Preference File before running tests.

App Directory: $HOME/Library/Developer/CoreSimulator/Devices/$SIMULATOR_UDID/data/Applications/

Preference File: $HOME/Library/Developer/CoreSimulator/Devices/$SIMULATOR_UDID/data/Library/Preferences/$PREFERENCE_FILE

This would clear the App settings while keeping the Simulator settings such as images, permissions, etc.

[Jeff Hatfield]

Setting RESET_BETWEEN_SCENARIOS to 0 does nothing and deleting those directories did nothing. It's still closing the iOS simulator and restarting it each time no matter what

[Joshua Moody]

> It's still closing the iOS simulator and restarting it each time no matter what

This has been the behavior since 0.9.169 when running against Xcode 5.1.1.  The simulator is quit before each Scenario.  We did this starting with Xcode 5.1.1 because the iOS Simulator environment was incredibly unstable.  You can see the Xcode 5.1.1 release notes that Apple acknowledged that this was known problem.  The suggested work around was to quit the simulator.  We did some internal testing a found that quitting simulator actually _improved_ test runtimes.

The Xcode 6* release notes also acknowledge that this is a problem and suggest quitting the simulator.

It is on our todo list to try to avoid quitting the simulator in the CoreSimulator / Xcode 6 environment.

> every single time I start a feature, the entire simulator resets. Content, Settings, and window

If this is actually happening, then it is a bug.  How are you determining that the content and settings are being reset?  What is the symptom?

If you are not seeing your NSUserDefaults persist between runs, see this case: https://github.com/calabash/calabash-ios/issues/536

[Jeff Hatfield]

This has been the behavior since 0.9.169 when running against Xcode 5.1.1.

Ahh. That would be why I'm seeing it so suddenly. I upgraded form 0.9.167 to 0.10.0  

It is on our todo list to try to avoid quitting the simulator in the CoreSimulator / Xcode 6 environment.

That would be much appreciated! 

If this is actually happening, then it is a bug.  How are you determining that the content and settings are being reset?  What is the symptom?

Well I use the Faker gem and in it are a lot of words that are not recognized by the iOS simulator and if the spellcheck is not turned off, it will auto-correct certain words. Each time the simulator resets, it will auto-capitalize and auto-correct everything that is typed. I can go in to the settings, turn off all of the settings under 'Keyboard', run a calabash test, have it fail (due to a word being auto-corrected), and come back to the 'Keyboard' settings and see that the settings have all been turned back on. 

[Joshua Moody]

>> It is on our todo list to try to avoid quitting the simulator in the CoreSimulator / Xcode 6 environment.
>>
> That would be much appreciated!

I am not optimistic that the CoreSimulator environment will be stable enough. :(

>> If this is actually happening, then it is a bug. How are you determining that the content and settings are being reset? What is the symptom?

> Each time the simulator resets, it will auto-capitalize and auto-correct everything that is typed. I can go in to the settings, turn off all of the settings under 'Keyboard', run a calabash test, have it fail (due to a word being auto-corrected), and come back to the 'Keyboard' settings and see that the settings have all been turned back on.

Can you confirm that you are not calling reset_simulator_content_and_settings from somewhere?

If not, please file an issue here: https://github.com/calabash/calabash-ios/issues
and we will investigate.

I believe there was some traffic about this issue on the Apple developer forums. It might be a known bug; there seem to be many issues with defaults/settings in the simulators.

[Jeff Hatfield]

I am not optimistic that the CoreSimulator environment will be stable enough. :( 

That's unfortunate, but understandable. What exactly do you mean by stable enough if you don't mind me asking?  

 

Can you confirm that you are not calling reset_simulator_content_and_settings from somewhere? 

I can 

If not, please file an issue here: https://github.com/calabash/calabash-ios/issues and we will investigate. 

I'll do that first thing tomorrow morning 

[Jeff Hatfield]

Good news (except for me)! The Content and Settings of the iOS Simulator ARE NOT being reset with each test. I've been going back and forth between iOS 7.1 and 8.0 which does reset the Content and Settings. Staying with the same OS seems to retain the Content and Settings

[Joshua Moody]

between iOS 7.1 and 8.0 which does reset the Content and Settings

Each simulator has its own content and settings.  If, from the iOS Simulator UI, reset the content and settings, it would only reset the content and setting for that simulator.

$ calabash-ios sim reset 

Will reset the content and setting of _all_ simulators.

If you use RESET_BETWEEN_SCENARIOS, the reset_app_sandbox will try (very hard) to figure out what iOS version you are targeting and:

* reset only that simulator's content and settings  # Xcode >= 6

* reset only the app's content and settings # Xcode < 6

### Links

- http://calabashapi.xamarin.com/ios/file.ENVIRONMENT_VARIABLES.html#RESET_BETWEEN_SCENARIOS

- http://calabashapi.xamarin.com/ios/Calabash/Cucumber/Launcher.html#reset_app_sandbox-instance_method

- https://github.com/calabash/calabash-ios/pull/526

[Tester]

Hi Joshua,

I'm seeing a lot of instability on my Calabash tests because sometimes the simulator takes longer to launch and then it times out and the test fails because it didn't execute.

Do you know if:

1) I can increase the timeouts used by Calabash to wait for simulator/install/connect to the app?

2) I can stop the simulator from restarting at every scenario (and then I will check if my tests become more stable or not)?

Thanks in advance.

[Teresa Peters]

I was wondering the same thing for #2. It's been awhile since we've used simulator.

[Joshua Moody]

Starting in Calabash 0.17.0, RESET_BETWEEN_SCENARIOS will no longer reset the simulator content and settings.

Run-loop 2.0.0, available only to Calabash >= 0.17.0, provides options you can use to control various timeouts.

  # These options control various aspects of an app's life cycle on the iOS
  # Simulator.
  #
  # You can override these values if they do not work in your environment.
  #
  # For cucumber users, the best place to override would be in your
  # features/support/env.rb.
  #
  # For example:
  #
  # RunLoop::CoreSimulator::DEFAULT_OPTIONS[:install_app_timeout] = 60
  DEFAULT_OPTIONS = {
    # In most cases 30 seconds is a reasonable amount of time to wait for an
    # install.  When testing larger apps, on slow machines, or in CI, this
    # value may need to be higher.  120 is the default for CI.
    :install_app_timeout => RunLoop::Environment.ci? ? 120 : 30,
    :uninstall_app_timeout => RunLoop::Environment.ci? ? 120 : 30,
    :launch_app_timeout => RunLoop::Environment.ci? ? 120 : 30,
    :wait_for_state_timeout => RunLoop::Environment.ci? ? 120 : 30
  }

> stop the simulator restarting at every Scenario

You can do this: don't call relaunch in your hooks.  It will be your responsibility to ensure that each Scenario starts in a good known state.

Periodically, we test whether we can avoid relaunching the simulator.  The honest truth is that the iOS Simulators are just not stable enough to remain open for any more than a few tests.  You actually spend more time detecting a clean launch and then retrying to launch the simulator.

The best option we've seen is to use a backdoor to reset your app to a good known state between Scenarios.