Ruby 1.8.7 Cruise Control 1.4.0 Passenger 3.0.2 git
Since upgrading to Passenger 3 we've noticed that our Cruise projects try to run the test suite twice which is, in itself, causing some of our tests to fail (on the second run).
Note that our tests pass (and only run once) on our local boxes, so it's not some hidden fork call. Also, they pass if run manually on the server hosting CC (and only run once). It's only the automated testing that exhibits this behavior.
> Since upgrading to Passenger 3 we've noticed that our Cruise projects > try to run the test suite twice which is, in itself, causing some of our > tests to fail (on the second run).
> Note that our tests pass (and only run once) on our local boxes, so it's > not some hidden fork call. Also, they pass if run manually on the server > hosting CC (and only run once). It's only the automated testing that > exhibits this behavior.
> Since upgrading to Passenger 3 we've noticed that our Cruise > projects > try to run the test suite twice which is, in itself, causing > some of our > tests to fail (on the second run).
> Note that our tests pass (and only run once) on our local > boxes, so it's > not some hidden fork call. Also, they pass if run manually on > the server > hosting CC (and only run once). It's only the automated > testing that > exhibits this behavior.
It's not configured anywhere, moreover is not supposed to happen. But then, running the same build twice is not supposed to happen, either. ps -ef | grep build should tell you all you need.
> > Since upgrading to Passenger 3 we've noticed that our Cruise > > projects > > try to run the test suite twice which is, in itself, causing > > some of our > > tests to fail (on the second run).
> > Note that our tests pass (and only run once) on our local > > boxes, so it's > > not some hidden fork call. Also, they pass if run manually on > > the server > > hosting CC (and only run once). It's only the automated > > testing that > > exhibits this behavior.
On Mon, 2011-01-24 at 10:41 -0700, Alexey Verkhovsky wrote: > It's not configured anywhere, moreover is not supposed to happen. But > then, running the same build twice is not supposed to happen, either. > ps -ef | grep build should tell you all you need.
> On Mon, Jan 24, 2011 at 10:26 AM, Daniel Berger <dber...@globe.gov> > wrote: > Hi,
> Can you help me determine if that's the case? Where is that > configured?
> Dan
> On Mon, 2011-01-24 at 10:14 -0700, Alexey Verkhovsky wrote: > > Are you running two builder processes for the same project?
> > On Mon, Jan 24, 2011 at 9:28 AM, Daniel Berger > <dber...@globe.gov> > > wrote: > > Hi,
> > Since upgrading to Passenger 3 we've noticed that > our Cruise > > projects > > try to run the test suite twice which is, in itself, > causing > > some of our > > tests to fail (on the second run).
> > Note that our tests pass (and only run once) on our > local > > boxes, so it's > > not some hidden fork call. Also, they pass if run > manually on > > the server > > hosting CC (and only run once). It's only the > automated > > testing that > > exhibits this behavior.
On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > Not that I can see. We do have multiple watchers setup that are pointing > at the same project but different branches.
> Hm, is it possible we have projects setup pointing at the same branch by > mistake? Is there a way to tell? Would that cause it?
> Regards,
> Dan
> On Mon, 2011-01-24 at 10:41 -0700, Alexey Verkhovsky wrote: > > It's not configured anywhere, moreover is not supposed to happen. But > > then, running the same build twice is not supposed to happen, either. > > ps -ef | grep build should tell you all you need.
> > On Mon, Jan 24, 2011 at 10:26 AM, Daniel Berger <dber...@globe.gov> > > wrote: > > Hi,
> > Can you help me determine if that's the case? Where is that > > configured?
> > Dan
> > On Mon, 2011-01-24 at 10:14 -0700, Alexey Verkhovsky wrote: > > > Are you running two builder processes for the same project?
> > > On Mon, Jan 24, 2011 at 9:28 AM, Daniel Berger > > <dber...@globe.gov> > > > wrote: > > > Hi,
> > > Since upgrading to Passenger 3 we've noticed that > > our Cruise > > > projects > > > try to run the test suite twice which is, in itself, > > causing > > > some of our > > > tests to fail (on the second run).
> > > Note that our tests pass (and only run once) on our > > local > > > boxes, so it's > > > not some hidden fork call. Also, they pass if run > > manually on > > > the server > > > hosting CC (and only run once). It's only the > > automated > > > testing that > > > exhibits this behavior.
>> /home/dberger/.cruise/projects/globe_village/build-0dca885.2/build.log && ruby -e "require 'rubygems' rescue nil; require 'rake'; load '/home/dberger/Downloads/Ruby/cruisecontrol-1.4.0/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run; ARGV.clear" >> /home/dberger/.cruise/projects/globe_village/build-0dca885.2/build.log 2>&1 On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > Not that I can see. We do have multiple watchers setup that are pointing > at the same project but different branches.
> Hm, is it possible we have projects setup pointing at the same branch by > mistake? Is there a way to tell? Would that cause it?
> Regards,
> Dan
> On Mon, 2011-01-24 at 10:41 -0700, Alexey Verkhovsky wrote: > > It's not configured anywhere, moreover is not supposed to happen. But > > then, running the same build twice is not supposed to happen, either. > > ps -ef | grep build should tell you all you need.
> > On Mon, Jan 24, 2011 at 10:26 AM, Daniel Berger <dber...@globe.gov> > > wrote: > > Hi,
> > Can you help me determine if that's the case? Where is that > > configured?
> > Dan
> > On Mon, 2011-01-24 at 10:14 -0700, Alexey Verkhovsky wrote: > > > Are you running two builder processes for the same project?
> > > On Mon, Jan 24, 2011 at 9:28 AM, Daniel Berger > > <dber...@globe.gov> > > > wrote: > > > Hi,
> > > Since upgrading to Passenger 3 we've noticed that > > our Cruise > > > projects > > > try to run the test suite twice which is, in itself, > > causing > > > some of our > > > tests to fail (on the second run).
> > > Note that our tests pass (and only run once) on our > > local > > > boxes, so it's > > > not some hidden fork call. Also, they pass if run > > manually on > > > the server > > > hosting CC (and only run once). It's only the > > automated > > > testing that > > > exhibits this behavior.
So, it looks like 2 processes there, 9330 and 9331. The latter has the former as its parent pid.
I also tried switching to Passenger 2.x, but the result was the same.
In addition, I tried altering the :cruise task to just do "puts hello" but it still seems to try to run the entire test suite. I slapped a debug print statement in the cruise control source to make sure it was picking up that there's a custom cruise task. It is.
On Mon, 2011-01-24 at 15:21 -0700, Daniel Berger wrote: > BTW, I tried running this command (that showed up in ps -ef) from the > command line manually in the work directory:
> On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > > Not that I can see. We do have multiple watchers setup that are pointing > > at the same project but different branches.
> > Hm, is it possible we have projects setup pointing at the same branch by > > mistake? Is there a way to tell? Would that cause it?
> > Regards,
> > Dan
> > On Mon, 2011-01-24 at 10:41 -0700, Alexey Verkhovsky wrote: > > > It's not configured anywhere, moreover is not supposed to happen. But > > > then, running the same build twice is not supposed to happen, either. > > > ps -ef | grep build should tell you all you need.
> > > On Mon, Jan 24, 2011 at 10:26 AM, Daniel Berger <dber...@globe.gov> > > > wrote: > > > Hi,
> > > Can you help me determine if that's the case? Where is that > > > configured?
> > > Dan
> > > On Mon, 2011-01-24 at 10:14 -0700, Alexey Verkhovsky wrote: > > > > Are you running two builder processes for the same project?
> > > > On Mon, Jan 24, 2011 at 9:28 AM, Daniel Berger > > > <dber...@globe.gov> > > > > wrote: > > > > Hi,
> > > > Since upgrading to Passenger 3 we've noticed that > > > our Cruise > > > > projects > > > > try to run the test suite twice which is, in itself, > > > causing > > > > some of our > > > > tests to fail (on the second run).
> > > > Note that our tests pass (and only run once) on our > > > local > > > > boxes, so it's > > > > not some hidden fork call. Also, they pass if run > > > manually on > > > > the server > > > > hosting CC (and only run once). It's only the > > > automated > > > > testing that > > > > exhibits this behavior.
On Mon, Jan 24, 2011 at 4:17 PM, Daniel Berger <dber...@globe.gov> wrote: > So, it looks like 2 processes there, 9330 and 9331. The latter has the > former as its parent pid.
Looks exactly how it should. One is shell, the other one is ruby that it invoked.
> On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > > > Not that I can see. We do have multiple watchers setup that are > pointing > > > at the same project but different branches.
Accessing the same database and/or some other resource(s), I presume? I'd bet that's the problem. To fix it, you want to configure CC.rb to serialize builds (i.e., make it so that only one build is running at any given time). There is a commented out setting for this in the site_config.rb, you just need to find and uncomment it.
On Mon, 2011-01-24 at 18:26 -0700, Alexey Verkhovsky wrote: > On Mon, Jan 24, 2011 at 4:17 PM, Daniel Berger <dber...@globe.gov> > wrote: > So, it looks like 2 processes there, 9330 and 9331. The latter > has the > former as its parent pid.
> Looks exactly how it should. One is shell, the other one is ruby that > it invoked.
> > On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > > > Not that I can see. We do have multiple watchers setup > that are pointing > > > at the same project but different branches.
> Accessing the same database and/or some other resource(s), I presume? > I'd bet that's the problem. To fix it, you want to configure CC.rb to > serialize builds (i.e., make it so that only one build is running at > any given time). There is a commented out setting for this in the > site_config.rb, you just need to find and uncomment it.
I tried that but it didn't solve my problem unfortunately. This is tied to Passenger somehow I think, because when I run the build using WEBrick it only runs the tests once, and they pass.
I'll play with our :cruise task some more and see if I can narrow it down.
On Tue, 2011-01-25 at 09:54 -0700, Daniel Berger wrote: > On Mon, 2011-01-24 at 18:26 -0700, Alexey Verkhovsky wrote: > > On Mon, Jan 24, 2011 at 4:17 PM, Daniel Berger <dber...@globe.gov> > > wrote: > > So, it looks like 2 processes there, 9330 and 9331. The latter > > has the > > former as its parent pid.
> > Looks exactly how it should. One is shell, the other one is ruby that > > it invoked.
> > > On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > > > > Not that I can see. We do have multiple watchers setup > > that are pointing > > > > at the same project but different branches.
> > Accessing the same database and/or some other resource(s), I presume? > > I'd bet that's the problem. To fix it, you want to configure CC.rb to > > serialize builds (i.e., make it so that only one build is running at > > any given time). There is a commented out setting for this in the > > site_config.rb, you just need to find and uncomment it.
> I tried that but it didn't solve my problem unfortunately. This is tied > to Passenger somehow I think, because when I run the build using WEBrick
Scratch that. It happens with WEBrick, too, but -only- if I daemonize it. Very peculiar.
When you catch it in the act of running the same build twice, I suggest that you look into the process tree and find out who started who. If it's not two concurrent builds running of different branches clashing with one another, then one of the build processes is probably an orphan.
When you run this stuff under Passenger, it starts and stops frontend workers at will. These, in turn, start and stop builder processes, one per project. There is a protection mechanism in the form of a builder lock file, that is intended to prevent CC.rb from running multiple builders ob the same project directory; HOWEVER, if a builder process starts a build, which then doesn't die when the builder dies (say, a rescue Exception clause eating the stop signals, which are Exceptions in Ruby), you can end up with concurrent builds per project, I guess.
Personally, I simply don't run CC.rb frontend under Passenger because of these oddball child process management scenarios. What I do instead is follow the instructions in daemon/cruise and let init run a single frontend process with a single set of builders.
On Tue, Jan 25, 2011 at 9:54 AM, Daniel Berger <dber...@globe.gov> wrote: > On Mon, 2011-01-24 at 18:26 -0700, Alexey Verkhovsky wrote: > > On Mon, Jan 24, 2011 at 4:17 PM, Daniel Berger <dber...@globe.gov> > > wrote: > > So, it looks like 2 processes there, 9330 and 9331. The latter > > has the > > former as its parent pid.
> > Looks exactly how it should. One is shell, the other one is ruby that > > it invoked.
> > > On Mon, 2011-01-24 at 15:03 -0700, Daniel Berger wrote: > > > > Not that I can see. We do have multiple watchers setup > > that are pointing > > > > at the same project but different branches.
> > Accessing the same database and/or some other resource(s), I presume? > > I'd bet that's the problem. To fix it, you want to configure CC.rb to > > serialize builds (i.e., make it so that only one build is running at > > any given time). There is a commented out setting for this in the > > site_config.rb, you just need to find and uncomment it.
> I tried that but it didn't solve my problem unfortunately. This is tied > to Passenger somehow I think, because when I run the build using WEBrick > it only runs the tests once, and they pass.
> I'll play with our :cruise task some more and see if I can narrow it > down.
On Tue, 2011-01-25 at 13:21 -0700, Alexey Verkhovsky wrote: > When you catch it in the act of running the same build twice, I > suggest that you look into the process tree and find out who started > who. If it's not two concurrent builds running of different branches > clashing with one another, then one of the build processes is probably > an orphan.
> When you run this stuff under Passenger, it starts and stops frontend > workers at will. These, in turn, start and stop builder processes, one > per project. There is a protection mechanism in the form of a builder > lock file, that is intended to prevent CC.rb from running multiple > builders ob the same project directory; HOWEVER, if a builder process > starts a build, which then doesn't die when the builder dies (say, a > rescue Exception clause eating the stop signals, which are Exceptions > in Ruby), you can end up with concurrent builds per project, I guess.
> Personally, I simply don't run CC.rb frontend under Passenger because > of these oddball child process management scenarios. What I do instead > is follow the instructions in daemon/cruise and let init run a single > frontend process with a single set of builders.
I don't think this is related, but we've narrowed it down to this bit of code, added to a test helper library, that was added to dynamically add routes for testing:
begin _routes = Our::Application.routes _routes.disable_clear_and_finalize = true _routes.clear!
_routes.draw do match "request404" => "application#request404" match "request500" => "application#request500" match "generic_request" => "application#generic_request" match "login_required_request" => "application#login_required_request" match "authorize_school_request" => "application#authorize_school_request" match "authorize_school_without_redirect_request" => "application#authorize_school_without_redirect_request" match "redirect_back_request" => "application#redirect_back_request" match "render_data_request" => "application#render_data_request" match "cancant_request" => "application#cancant_request"
match "redirect" => "legacy#redirect" end ActiveSupport.on_load(:action_controller) do Our::Application.routes.finalize! end ensure _routes.disable_clear_and_finalize = false end
If we comment this out, and the tests that use the helper, the problem goes away.
On Tue, Jan 25, 2011 at 1:21 PM, Alexey Verkhovsky
<alexey.verkhov...@gmail.com> wrote: > Personally, I simply don't run CC.rb frontend under Passenger because of > these oddball child process management scenarios. What I do instead is > follow the instructions in daemon/cruise and let init run a single frontend > process with a single set of builders.
Also, if you use the init script './daemon/cruise [start|stop]', it is smart enough to kill all the child processes of your builders. Very handy. _______________________________________________ Cruisecontrolrb-users mailing list Cruisecontrolrb-us...@rubyforge.org http://rubyforge.org/mailman/listinfo/cruisecontrolrb-users
> I don't think this is related, but we've narrowed it down to this bit of > code, added to a test helper library, that was added to dynamically add > routes for testing:
> _routes.draw do > match "request404" => "application#request404" > match "request500" => "application#request500" > match "generic_request" => "application#generic_request" > match "login_required_request" => > "application#login_required_request" > match "authorize_school_request" => > "application#authorize_school_request" > match "authorize_school_without_redirect_request" => > "application#authorize_school_without_redirect_request" > match "redirect_back_request" => "application#redirect_back_request" > match "render_data_request" => "application#render_data_request" > match "cancant_request" => "application#cancant_request"
> match "redirect" => "legacy#redirect" > end > ActiveSupport.on_load(:action_controller) do > Our::Application.routes.finalize! > end > ensure > _routes.disable_clear_and_finalize = false > end
> If we comment this out, and the tests that use the helper, the problem > goes away.
I decided to remove this code and simply put the routes in the config/routes.rb file wrapped in an if block. That wasn't the issue, though.
Based on my experiments, any attempt to reopen the ApplicationController class causes the double run behavior. There were some instance variables we wanted to get at directly, so we reopened the class and added some accessors. If I comment that code and the related tests, CC is suddenly happy again, even though they work fine from the command line.
Why? I haven't a clue. Very bizarre.
For now we're just going to comment them out and move on.
