Order dependent test failures

[Jeff McCune]

Hey everyone,

I'm working on fixing up a module and I've written a test that triggers the bug against Puppet 2.7.x.  I'm running into a problem where I get 0 failures when I run `rake test` or `rspec --format d spec/` but I get the failure I'm expecting if I run the spec file directly with rspec spec/unit/application/describe_spec.rb

Here's the gist of what's going on:

https://gist.github.com/2410088

The code in question is:

[1] https://github.com/jeffmccune/puppetlabs-mount-providers/tree/ticket/master/13070_fix_mounttab

And Puppet 2.7.x

What is a common case for this type of order dependent false positive, and what are some strategies I can employ to be a bit more defensive about the context my test is running in?

Thanks,

-Jeff

[Chris Price]

Nick Lewis has a script that you can grab from here:

https://gist.github.com/1217637

If you know the order that your tests ran it, this will help you narrow it down to the particular two that are interfering with one another (accomplished via a sort of binary search, running half of the number of tests from the last run over and over again until it has identified the guilty parties).  In my experience it's usually a matter of stubs/expectations stomping on one another, or global/singleton state variables not being reset properly.

I have a pull request here:

https://github.com/puppetlabs/puppet/pull/676

That shows how you can potentially log the order that the test files were run in, which is a prereq of using Nick's script to debug.  I need to test it a bit further but what I'm hoping is that after I have confidence in it we can use it on the Jenkins jobs so that we are always capturing their order.  Then, we can enable rspec's random order feature so that all of our jenkins runs will be randomized, and if we ever trigger an order-dependent failure we'll know the order so that we can use Nick's script.

Once I've narrowed it down to two specific test files, if nothing obvious jumps out at me I'll just comment out all of the code in the first test and selectively uncomment it until I determine which lines are causing the problems.

hth

chris

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To post to this group, send email to puppe...@googlegroups.com.
To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.

[Randall Hansen]

On Apr 17, 2012, at 5:27 PM, Jeff McCune <je...@puppetlabs.com> wrote:

> Here's the gist of what's going on:

1,000 Internets for the correct use of "gist." :)

r

[Chris Price]

Doh.  I didn't click through to your gist before I replied.  It looks like you've already identified the problematic test, so most of what I posted will be of no use to you.  :(

--

[Daniel Pittman]

Recent rspec supports an `--order random`, as well as `--order
random:${seed}` which runs with a fixed seed. The random order prints
the seed at the end; you can use that to shuffle the tests and help
discover these.

Sadly, Puppet doesn't fully pass with that option, and I have not yet
finished purging them, so you will get some genuine but unconnected
positives. You can have a cookie for fixing them along the way,
though. ;)

--
Daniel Pittman
⎋ Puppet Labs Developer – http://puppetlabs.com
♲ Made with 100 percent post-consumer electrons

[Jeff McCune]

On Tue, Apr 17, 2012 at 6:11 PM, Chris Price <ch...@puppetlabs.com> wrote:

Doh.  I didn't click through to your gist before I replied.  It looks like you've already identified the problematic test, so most of what I posted will be of no use to you.  :(

Well, I know my test works in isolation, but I don't know what other stuff is happening that causes it to break in the larger context, so...  maybe it's still useful?

Specifically, I'm looking for some way to figure out relatively easily what's different between the two scenarios of running rspec spec/ and running the spec file in isolation.

I thought there might be a magic bullet, but I'm guessing there isn't.  In which case I'll just dive into the debugger and start looking around at the state of the system, trying to find the difference between the two.

Whenever I do this, I feel like I'm in a cavern with a book of matches.  I was hoping someone had brought along a spotlight or a magelight spell or something...

-Jeff

[Chris Price]

Yeah, unfortunately, commenting out large swaths and then adding back in a little bit at a time is about the best weapon I've found to combat those situations so far.  Would love to hear if anyone has found anything better.

--

[Daniel Pittman]

I just bisect with `rspec spec/[a-m]* spec/my/failing_spec.rb`, and
then narrow it down that way. The script is a faster way to do that.

I never found debugger inspection very useful compared to bisection
because the source is often very, very opaque and distant - so unless
I knew what I was looking for, I never found it.

--

[Jeff McCune]

On Tue, Apr 17, 2012 at 6:47 PM, Daniel Pittman <dan...@puppetlabs.com> wrote:

I just bisect with `rspec spec/[a-m]* spec/my/failing_spec.rb`, and
then narrow it down that way.  The script is a faster way to do that.

I never found debugger inspection very useful compared to bisection
because the source is often very, very opaque and distant - so unless
I knew what I was looking for, I never found it.

Cool, this has been helpful.  I learned that it's not "just me" or that I'm doing it wrong, which was what I initially suspected.

For this particular issue, the TypeDoc initialize method calls  Puppet::Type.loadall so I'm going to try and figure out another approach to trigger the bug that behaves consistently after other tests might have initialized the Types and Providers.

-Jeff