intermittent failures with database cleaner

[Jason FB]

My test suite is plagued with intermittent test failures that I have been trying to fix for several months now with no success. As far as I can tell, it’s all race condition stuff that has to do with Database cleaner. I use exclusively Factories (FactoryGirl) 

Basically I have set up data using Factories in before(:each) calls at the top of all my tests.

before(:each) do

    FactoryGirl.create(:gallery_bucket)

    FactoryGirl.create(:promo_bucket)

end

When the spec runs, a line of code that is relatively global to my layout calls the following:

 def load_gallery

    @galleries = Spree::Bucket.find_by_slug('gallery').bucket_contents

  end

When this line of code runs,  Spree::Bucket.find_by_slug('gallery’) returns nil.

In some cases I have been able to debug both the before(:each) and have see the records returned correctly by Spree::Bucket.all (returns 2 records), but then when I continue and fall into a debugger inside of the load_gallery method, Spree::Bucket.all returns nil.

Like I said, it is intermittent, so Spree::Bucket.all returns 2 objects and everything passes. Other times, the objects just appear not to be available to the ruby instance. 

When I run the whole suite (about 80 tests) I get about  a 1 in 4 chance of at least 1 failure. When I run tests individually, they usually (although not always) pass.

Although I have tried different transaction/truncation strategies, the one I currently have can be seen here:

https://gist.github.com/jasonfb/953fca4c4788a70bb661

Interesting to note, when I switch to an all-transaction strategy, most of my test suite fails, particularly the Webkit (:js => true) tests. Using a transaction strategy for the non-Webkit tests and a truncation strategy for the Webkit tests seems to produce the best (least like to fail) results, but even that still has these intermittent failures. 

Any tips for where I should look next would be greatly appreciated.

-Jason

[Aaron Kromer]

Interesting to note, when I switch to an all-transaction strategy, most of my test suite fails, particularly the Webkit (:js => true) tests.

This is not all that surprising if you are using Capybara. Per the README:

If you are using a SQL database, it is common to run every test in a transaction, which is rolled back at the end of the test, rspec-rails does this by default out of the box for example. Since transactions are usually not shared across threads, this will cause data you have put into the database in your test code to be invisible to Capybara.

It’s not uncommon to have a different strategy for :feature specs. You can use an around(:each, type: :feature) hook to adjust the strategy accordingly. In the gist you linked they do something similar for JS specs: https://gist.github.com/jasonfb/953fca4c4788a70bb661#file-gistfile1-txt-L78

config.before(:each, :js => true) do
  DatabaseCleaner.strategy = :truncation
end

That also depends on the adapter you use.

it’s all race condition stuff that has to do with Database cleaner

I’d say that’s probably unlikely. before hooks run in the order they are defined. Which is based on the order the files are loaded. If you keep your before hooks for Database Cleaner in the spec/rails_helper.rb (as in the gist). This is going to be one of the first files loaded. If your near global stated is loaded before that, then yes you could have a problem, but would likely see it much more frequently.

If you have a spec file with this:

before(:each) do
  FactoryGirl.create(:gallery_bucket)
  FactoryGirl.create(:promo_bucket)
end

it "..." do
  load_gallery

  # ...
end

I would never expect that to fail due to the find returning nil. If it does, that would suggest Spree may be caching results from somewhere else. I’d be sure to clear any Spree caches before my specs run each time.

If you have say:

before(:each) do
  FactoryGirl.create(:gallery_bucket)
  FactoryGirl.create(:promo_bucket)
end

before(:each) do
  load_gallery
end

it "..." do
  # ...

end

I would expect the same behavior, it should not fail.

However, if you’re using nested context / describe blocks, you can quickly get yourself into problems:

describe "something..." do
  before(:each) do

    load_gallery
  end

  describe "nested something..." do

    before(:each) do
      FactoryGirl.create(:gallery_bucket)
      FactoryGirl.create(:promo_bucket)
    end

    it "..." do

      # ...

    end

  end
end

I would expect that to fail. Since load_gallery is called first. While it’s easy to see in such a short and simple example, with a large spec file it may be much more difficult to debug with load_gallery being set far away from your deeply nested context.

You can use the following to try to track down where things are defined and loaded, be sure to load this as the first thing RSpec sees:

module DebugBefore
  def before(*args, &block)
    code_source = caller[0]
    super(*args) do |ex|
      puts "before hook: #{code_source}"
      block.call(ex)
    end
  end

  def around(*args, &block)
    code_source = caller[0]
    super(*args) do |ex|
      puts "before hook: #{code_source}"
      block.call(ex)
    end
  end
end

RSpec.configuration.singleton_class.include DebugBefore
RSpec.configuration.extend DebugBefore

This will show you were the different hooks are defined and the order they run in when the spec runs.

--
You received this message because you are subscribed to the Google Groups "rspec" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rspec+un...@googlegroups.com.
To post to this group, send email to rs...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rspec/9f3382d8-2e7b-45e9-ae46-8920f0c2c1e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.