Resetting state between acceptance tests

[Liron Yahdav]

I'm about half way through GOOS book and I'm thoroughly enjoying it. I've been trying to apply the process of writing a failing acceptance test first to my job. I'm an iOS app developer.

My question is, does it make sense to reset an app's state entirely between each acceptance test that is run or is it better to have each test continue where the previous test left off?

Benefits I see to resetting state between tests:
- You don't have to worry about what state the app is in after each test. You can write each test as if it's a fresh install of your application. Otherwise at the beginning of each test you might have to consider what state the previous test left the app in, or write extra logic that puts the app in a known state after each test.
- The tests tend to be more deterministic when resetting. It's less things that can go wrong, less reasons the tests might be flaky, which can be a big issue with a larger acceptance test suite.

Benefits I see to not resetting:
- You don't have to write the code to reset state, which might be tricky
- You don't take a speed hit of resetting on each test when running your whole test suite
- You might catch bugs due to the interaction between features in your application. For example if you use feature A and then feature B in your app, the app crashes. If you reset state between tests you'll never catch such a bug.

Would love to get some thoughts from the group on this. Thanks!

[Steve Freeman]

Acceptance tests tend to be less “pure” than lower-level tests because of the extra complexity of the component under test. The first question to figure out is just how slow these tests actually are. Perhaps a few small changes to the design would make the system much more testable. Those extra hooks generally turn out to be useful for other purposes.

The next usual response it think about the “pyramid of testing” - do most of the testing at lower levels, and write fewer higher level tests based on a degree of confidence about the next level down. The worst maintenance situation is usually the inverse pyramid where everything is tested from the outside.

Whatever you end up doing, you need to think about making sure that you know exactly what’s happened to the system and you can detect when and why it’s failed.

Finally, the most important thing to remember is to get on with it /and/ to be ready to restructure and even rewrite your tests as your understanding grows. This never gets old.

S