I'm working on fixing up a module and I've written a test that triggers the bug against Puppet 2.7.x. I'm running into a problem where I get 0 failures when I run `rake test` or `rspec --format d spec/` but I get the failure I'm expecting if I run the spec file directly with rspec spec/unit/application/describe_spec.rb
What is a common case for this type of order dependent false positive, and what are some strategies I can employ to be a bit more defensive about the context my test is running in?
If you know the order that your tests ran it, this will help you narrow it down to the particular two that are interfering with one another (accomplished via a sort of binary search, running half of the number of tests from the last run over and over again until it has identified the guilty parties). In my experience it's usually a matter of stubs/expectations stomping on one another, or global/singleton state variables not being reset properly.
That shows how you can potentially log the order that the test files were run in, which is a prereq of using Nick's script to debug. I need to test it a bit further but what I'm hoping is that after I have confidence in it we can use it on the Jenkins jobs so that we are always capturing their order. Then, we can enable rspec's random order feature so that all of our jenkins runs will be randomized, and if we ever trigger an order-dependent failure we'll know the order so that we can use Nick's script.
Once I've narrowed it down to two specific test files, if nothing obvious jumps out at me I'll just comment out all of the code in the first test and selectively uncomment it until I determine which lines are causing the problems.
On Tue, Apr 17, 2012 at 5:27 PM, Jeff McCune <j...@puppetlabs.com> wrote: > Hey everyone,
> I'm working on fixing up a module and I've written a test that triggers > the bug against Puppet 2.7.x. I'm running into a problem where I get 0 > failures when I run `rake test` or `rspec --format d spec/` but I get the > failure I'm expecting if I run the spec file directly with > rspec spec/unit/application/describe_spec.rb
> What is a common case for this type of order dependent false positive, and > what are some strategies I can employ to be a bit more defensive about the > context my test is running in?
> Thanks, > -Jeff
> -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-dev@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en.
Doh. I didn't click through to your gist before I replied. It looks like you've already identified the problematic test, so most of what I posted will be of no use to you. :(
On Tue, Apr 17, 2012 at 5:27 PM, Jeff McCune <j...@puppetlabs.com> wrote: > Hey everyone,
> I'm working on fixing up a module and I've written a test that triggers > the bug against Puppet 2.7.x. I'm running into a problem where I get 0 > failures when I run `rake test` or `rspec --format d spec/` but I get the > failure I'm expecting if I run the spec file directly with > rspec spec/unit/application/describe_spec.rb
> What is a common case for this type of order dependent false positive, and > what are some strategies I can employ to be a bit more defensive about the > context my test is running in?
> Thanks, > -Jeff
> -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-dev@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en.
On Tue, Apr 17, 2012 at 17:27, Jeff McCune <j...@puppetlabs.com> wrote: > I'm working on fixing up a module and I've written a test that triggers the > bug against Puppet 2.7.x. I'm running into a problem where I get 0 failures > when I run `rake test` or `rspec --format d spec/` but I get the failure I'm > expecting if I run the spec file directly with > rspec spec/unit/application/describe_spec.rb
> What is a common case for this type of order dependent false positive, and > what are some strategies I can employ to be a bit more defensive about the > context my test is running in?
Recent rspec supports an `--order random`, as well as `--order random:${seed}` which runs with a fixed seed. The random order prints the seed at the end; you can use that to shuffle the tests and help discover these.
Sadly, Puppet doesn't fully pass with that option, and I have not yet finished purging them, so you will get some genuine but unconnected positives. You can have a cookie for fixing them along the way, though. ;)
-- Daniel Pittman ⎋ Puppet Labs Developer – http://puppetlabs.com ♲ Made with 100 percent post-consumer electrons
On Tue, Apr 17, 2012 at 6:11 PM, Chris Price <ch...@puppetlabs.com> wrote: > Doh. I didn't click through to your gist before I replied. It looks like > you've already identified the problematic test, so most of what I posted > will be of no use to you. :(
Well, I know my test works in isolation, but I don't know what other stuff is happening that causes it to break in the larger context, so... maybe it's still useful?
Specifically, I'm looking for some way to figure out relatively easily what's different between the two scenarios of running rspec spec/ and running the spec file in isolation.
I thought there might be a magic bullet, but I'm guessing there isn't. In which case I'll just dive into the debugger and start looking around at the state of the system, trying to find the difference between the two.
Whenever I do this, I feel like I'm in a cavern with a book of matches. I was hoping someone had brought along a spotlight or a magelight spell or something...
Yeah, unfortunately, commenting out large swaths and then adding back in a little bit at a time is about the best weapon I've found to combat those situations so far. Would love to hear if anyone has found anything better.
On Tue, Apr 17, 2012 at 6:41 PM, Jeff McCune <j...@puppetlabs.com> wrote: > On Tue, Apr 17, 2012 at 6:11 PM, Chris Price <ch...@puppetlabs.com> wrote:
>> Doh. I didn't click through to your gist before I replied. It looks >> like you've already identified the problematic test, so most of what I >> posted will be of no use to you. :(
> Well, I know my test works in isolation, but I don't know what other stuff > is happening that causes it to break in the larger context, so... maybe > it's still useful?
> Specifically, I'm looking for some way to figure out relatively easily > what's different between the two scenarios of running rspec spec/ and > running the spec file in isolation.
> I thought there might be a magic bullet, but I'm guessing there isn't. In > which case I'll just dive into the debugger and start looking around at the > state of the system, trying to find the difference between the two.
> Whenever I do this, I feel like I'm in a cavern with a book of matches. I > was hoping someone had brought along a spotlight or a magelight spell or > something...
> -Jeff
> -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-dev@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en.
I just bisect with `rspec spec/[a-m]* spec/my/failing_spec.rb`, and then narrow it down that way. The script is a faster way to do that.
I never found debugger inspection very useful compared to bisection because the source is often very, very opaque and distant - so unless I knew what I was looking for, I never found it.
On Tue, Apr 17, 2012 at 18:44, Chris Price <ch...@puppetlabs.com> wrote: > Yeah, unfortunately, commenting out large swaths and then adding back in a > little bit at a time is about the best weapon I've found to combat those > situations so far. Would love to hear if anyone has found anything better.
> On Tue, Apr 17, 2012 at 6:41 PM, Jeff McCune <j...@puppetlabs.com> wrote:
>> On Tue, Apr 17, 2012 at 6:11 PM, Chris Price <ch...@puppetlabs.com> wrote:
>>> Doh. I didn't click through to your gist before I replied. It looks >>> like you've already identified the problematic test, so most of what I >>> posted will be of no use to you. :(
>> Well, I know my test works in isolation, but I don't know what other stuff >> is happening that causes it to break in the larger context, so... maybe >> it's still useful?
>> Specifically, I'm looking for some way to figure out relatively easily >> what's different between the two scenarios of running rspec spec/ and >> running the spec file in isolation.
>> I thought there might be a magic bullet, but I'm guessing there isn't. In >> which case I'll just dive into the debugger and start looking around at the >> state of the system, trying to find the difference between the two.
>> Whenever I do this, I feel like I'm in a cavern with a book of matches. I >> was hoping someone had brought along a spotlight or a magelight spell or >> something...
>> -Jeff
>> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To post to this group, send email to puppet-dev@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-dev+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-dev?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-dev@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en.
-- Daniel Pittman ⎋ Puppet Labs Developer – http://puppetlabs.com ♲ Made with 100 percent post-consumer electrons
On Tue, Apr 17, 2012 at 6:47 PM, Daniel Pittman <dan...@puppetlabs.com>wrote:
> I just bisect with `rspec spec/[a-m]* spec/my/failing_spec.rb`, and > then narrow it down that way. The script is a faster way to do that.
> I never found debugger inspection very useful compared to bisection > because the source is often very, very opaque and distant - so unless > I knew what I was looking for, I never found it.
Cool, this has been helpful. I learned that it's not "just me" or that I'm doing it wrong, which was what I initially suspected.
For this particular issue, the TypeDoc initialize method calls Puppet::Type.loadall so I'm going to try and figure out another approach to trigger the bug that behaves consistently after other tests might have initialized the Types and Providers.
