[Cruisecontrolrb-users] CC running tests twice with Passenger 3

[Daniel Berger]

Hi,

Ruby 1.8.7
Cruise Control 1.4.0
Passenger 3.0.2
git

Since upgrading to Passenger 3 we've noticed that our Cruise projects
try to run the test suite twice which is, in itself, causing some of our
tests to fail (on the second run).

Note that our tests pass (and only run once) on our local boxes, so it's
not some hidden fork call. Also, they pass if run manually on the server
hosting CC (and only run once). It's only the automated testing that
exhibits this behavior.

Any ideas?

Regards,

Dan

_______________________________________________
Cruisecontrolrb-users mailing list
Cruisecont...@rubyforge.org
http://rubyforge.org/mailman/listinfo/cruisecontrolrb-users

[Alexey Verkhovsky]

Are you running two builder processes for the same project?

--
Alexey Verkhovsky
http://alex-verkhovsky.blogspot.com/
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]

[Daniel Berger]

Hi,

Can you help me determine if that's the case? Where is that configured?

Dan

[Alexey Verkhovsky]

It's not configured anywhere, moreover is not supposed to happen. But then, running the same build twice is not supposed to happen, either.
ps -ef | grep build should tell you all you need.

[Daniel Berger]

Not that I can see. We do have multiple watchers setup that are pointing
at the same project but different branches.

Hm, is it possible we have projects setup pointing at the same branch by
mistake? Is there a way to tell? Would that cause it?

Regards,

Dan

[Daniel Berger]

BTW, I tried running this command (that showed up in ps -ef) from the
command line manually in the work directory:

ruby -e "require 'rubygems' rescue nil; require 'rake'; load
'/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake';
ARGV << '--nosearch' << 'cc:build'; Rake.application.run; ARGV.clear"

That worked fine.

Regards,

Dan

[Daniel Berger]

BTW, here's what I see after install a local instance of cruise and
building via the web interface:

dberger:/home/dberger >ps -ef | grep build | grep -v grep | grep village

dberger 25625 19459 0 14:45 ? 00:00:00
ruby /home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/cruise build
globe_village
dberger 28170 25625 0 15:07 ? 00:00:00 sh -c
echo /home/dberger/.cruise/projects/globe_village/work dberger$ ruby\ -e

\ \"require\ \'rubygems\'\ rescue\ nil\;\ require\ \'rake\'\;\ load\
\'/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake
\'\;\ ARGV\ \<\<\ \'--nosearch\'\ \<\<\ \'cc:build\'\;\

Rake.application.run\;\ ARGV.clear\"
>> /home/dberger/.cruise/projects/globe_village/build-0dca885.2/build.log && ruby -e "require 'rubygems' rescue nil; require 'rake'; load '/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run; ARGV.clear" >> /home/dberger/.cruise/projects/globe_village/build-0dca885.2/build.log 2>&1

On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote:

[Daniel Berger]

Hi again,

Invoking the build process from a local instance I see this:

dberger 9330 25625 0 16:10 ? 00:00:00 sh -c
echo /home/dberger/.cruise/projects/globe_village/work dberger$ ruby\ -e

\ \"require\ \'rubygems\'\ rescue\ nil\;\ require\ \'rake\'\;\ load\
\'/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake
\'\;\ ARGV\ \<\<\ \'--nosearch\'\ \<\<\ \'cc:build\'\;\

Rake.application.run\;\ ARGV.clear\"
>> /home/dberger/.cruise/projects/globe_village/build-28091f9.8/build.log && ruby -e "require 'rubygems' rescue nil; require 'rake'; load '/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run; ARGV.clear" >> /home/dberger/.cruise/projects/globe_village/build-28091f9.8/build.log 2>&1

dberger 9331 9330 16 16:10 ? 00:00:09 ruby -e require

'rubygems' rescue nil; require 'rake'; load
'/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake';
ARGV << '--nosearch' << 'cc:build'; Rake.application.run; ARGV.clear

dberger 22962 1 0 14:23 ? 00:00:01

ruby /home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/cruise build
globe_village

So, it looks like 2 processes there, 9330 and 9331. The latter has the
former as its parent pid.

I also tried switching to Passenger 2.x, but the result was the same.

In addition, I tried altering the :cruise task to just do "puts hello"
but it still seems to try to run the entire test suite. I slapped a
debug print statement in the cruise control source to make sure it was
picking up that there's a custom cruise task. It is.

Any ideas?

Regards,

Dan

[Alexey Verkhovsky]

On Mon, Jan 24, 2011 at 4:17 PM, Daniel Berger <dbe...@globe.gov> wrote:

So, it looks like 2 processes there, 9330 and 9331. The latter has the
former as its parent pid.

Looks exactly how it should. One is shell, the other one is ruby that it invoked.

> On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote:
> > Not that I can see. We do have multiple watchers setup that are pointing
> > at the same project but different branches.

Accessing the same database and/or some other resource(s), I presume? I'd bet that's the problem. To fix it, you want to configure CC.rb to serialize builds (i.e., make it so that only one build is running at any given time). There is a commented out setting for this in the site_config.rb, you just need to find and uncomment it.

--Alex

[Daniel Berger]

I tried that but it didn't solve my problem unfortunately. This is tied
to Passenger somehow I think, because when I run the build using WEBrick
it only runs the tests once, and they pass.

I'll play with our :cruise task some more and see if I can narrow it
down.

[Daniel Berger]

On Tue, 2011-01-25 at 09:54 -0700, Daniel Berger wrote:
> On Mon, 2011-01-24 at 18:26 -0700, Alexey Verkhovsky wrote:
> > On Mon, Jan 24, 2011 at 4:17 PM, Daniel Berger <dbe...@globe.gov>
> > wrote:
> > So, it looks like 2 processes there, 9330 and 9331. The latter
> > has the
> > former as its parent pid.
> >
> > Looks exactly how it should. One is shell, the other one is ruby that
> > it invoked.
> >
> >
> > > On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote:
> > > > Not that I can see. We do have multiple watchers setup
> > that are pointing
> > > > at the same project but different branches.
> >
> >
> > Accessing the same database and/or some other resource(s), I presume?
> > I'd bet that's the problem. To fix it, you want to configure CC.rb to
> > serialize builds (i.e., make it so that only one build is running at
> > any given time). There is a commented out setting for this in the
> > site_config.rb, you just need to find and uncomment it.
>
> I tried that but it didn't solve my problem unfortunately. This is tied
> to Passenger somehow I think, because when I run the build using WEBrick

Scratch that. It happens with WEBrick, too, but -only- if I daemonize
it. Very peculiar.

Investigating...

[Alexey Verkhovsky]

When you catch it in the act of running the same build twice, I suggest that you look into the process tree and find out who started who. If it's not two concurrent builds running of different branches clashing with one another, then one of the build processes is probably an orphan.

When you run this stuff under Passenger, it starts and stops frontend workers at will. These, in turn, start and stop builder processes, one per project. There is a protection mechanism in the form of a builder lock file, that is intended to prevent CC.rb from running multiple builders ob the same project directory; HOWEVER, if a builder process starts a build, which then doesn't die when the builder dies (say, a rescue Exception clause eating the stop signals, which are Exceptions in Ruby), you can end up with concurrent builds per project, I guess.

Personally, I simply don't run CC.rb frontend under Passenger because of these oddball child process management scenarios. What I do instead is follow the instructions in daemon/cruise and let init run a single frontend process with a single set of builders.

[Daniel Berger]

On Tue, 2011-01-25 at 13:21 -0700, Alexey Verkhovsky wrote:
> When you catch it in the act of running the same build twice, I
> suggest that you look into the process tree and find out who started
> who. If it's not two concurrent builds running of different branches
> clashing with one another, then one of the build processes is probably
> an orphan.
>
> When you run this stuff under Passenger, it starts and stops frontend
> workers at will. These, in turn, start and stop builder processes, one
> per project. There is a protection mechanism in the form of a builder
> lock file, that is intended to prevent CC.rb from running multiple
> builders ob the same project directory; HOWEVER, if a builder process
> starts a build, which then doesn't die when the builder dies (say, a
> rescue Exception clause eating the stop signals, which are Exceptions
> in Ruby), you can end up with concurrent builds per project, I guess.
>
> Personally, I simply don't run CC.rb frontend under Passenger because
> of these oddball child process management scenarios. What I do instead
> is follow the instructions in daemon/cruise and let init run a single
> frontend process with a single set of builders.

I don't think this is related, but we've narrowed it down to this bit of
code, added to a test helper library, that was added to dynamically add
routes for testing:

begin
_routes = Our::Application.routes
_routes.disable_clear_and_finalize = true
_routes.clear!

Our::Application.routes_reloader.paths.each{ |path| load(path) }

_routes.draw do
match "request404" => "application#request404"
match "request500" => "application#request500"
match "generic_request" => "application#generic_request"
match "login_required_request" =>
"application#login_required_request"
match "authorize_school_request" =>
"application#authorize_school_request"
match "authorize_school_without_redirect_request" =>
"application#authorize_school_without_redirect_request"
match "redirect_back_request" => "application#redirect_back_request"
match "render_data_request" => "application#render_data_request"
match "cancant_request" => "application#cancant_request"

match "redirect" => "legacy#redirect"
end
ActiveSupport.on_load(:action_controller) do
Our::Application.routes.finalize!
end
ensure
_routes.disable_clear_and_finalize = false
end

If we comment this out, and the tests that use the helper, the problem
goes away.

BTW, I think the code came from http://openhood.com/rails/rails%
203/2010/07/20/add-routes-at-runtime-rails-3/ originally.

Still tinkering with it...

[Chad Woolley]

On Tue, Jan 25, 2011 at 1:21 PM, Alexey Verkhovsky
<alexey.v...@gmail.com> wrote:
> Personally, I simply don't run CC.rb frontend under Passenger because of
> these oddball child process management scenarios. What I do instead is
> follow the instructions in daemon/cruise and let init run a single frontend
> process with a single set of builders.

Also, if you use the init script './daemon/cruise [start|stop]', it is
smart enough to kill all the child processes of your builders. Very
handy.

[Daniel Berger]

<snip>

I decided to remove this code and simply put the routes in the
config/routes.rb file wrapped in an if block. That wasn't the issue,
though.

Based on my experiments, any attempt to reopen the ApplicationController
class causes the double run behavior. There were some instance variables
we wanted to get at directly, so we reopened the class and added some
accessors. If I comment that code and the related tests, CC is suddenly
happy again, even though they work fine from the command line.

Why? I haven't a clue. Very bizarre.

For now we're just going to comment them out and move on.