Puppetmasterd looses contact with the mysql database (using storeconfigs)

[Juri Rischel Jensen]

Hi All

I've just bootstrapped my new, shining puppet setup (replacing an 2,5 year old, messy one), with (almost) all the bells and whistles: A Passenger-based setup with storeconfigs enabled.

For a start I've only got three node definitions; two specific ones and node default. The default one doesn't do anything yet, and only one of the specific nodes has actual classes enabled.

Now, when I startup mysql and puppetmasterd, the first one or two runs of puppetd on the client is fine. Puppetmasterd stores the configs, but suddenly (and this is with only one host contacting the puppetmaster) the log says:

Could not store configs: Mysql::Error: MySQL server has gone away: SELECT * FROM `hosts`   WHERE (`hosts`.`name` = 'web01.mydomain.com')  LIMIT 1

At first I thought it was MySQL drowning, but I've found that a restart of puppetmasterd (with /etc/init.d/apache2 restart, as it run through Passenger) fixes the problem - but only temporarily. Next time a host contacts the puppetmaster, I get the error stated above. :-( And it fails until I restart puppetmaster again.

I have the following optimizations in my passenger configuration (cut from Nigel Kerstens mails about his Passengerbased setup):

PassengerLogLevel 1

PassengerMaxRequests 3000

PassengerMaxPoolSize 15

PassengerStatThrottleRate 600

I've also tried to optimize the MySQL configuration, but as a restart of Apache/Passenger solves the problem (for one clientrun), I don't think that the problem lies with MySQL.

Do any of you have a clue of what can be wrong here?

--
Med venlig hilsen/Best regards
Juri Rischel Jensen

Fab:IT ApS
Vesterbrogade 50
DK-1620 København
Tlf: +45 70 202 407
www.fab-it.dk

[Brice Figureau]

Hi,

First make sure you installed the mysql Ruby Gem (or the correct package
on your system).

On 5/06/09 11:58, Juri Rischel Jensen wrote:
> Now, when I startup mysql and puppetmasterd, the first one or two runs
> of puppetd on the client is fine. Puppetmasterd stores the configs, but
> suddenly (and this is with only one host contacting the puppetmaster)
> the log says:
>
> Could not store configs: Mysql::Error: MySQL server has gone away:
> SELECT * FROM `hosts` WHERE (`hosts`.`name` = 'web01.mydomain.com')
> LIMIT 1

Server has gone away usually means MySQL has crashed. Check your
mysql.err file or syslog on this system to see what happened.
Which version of MySQL is it?

> At first I thought it was MySQL drowning, but I've found that a restart
> of puppetmasterd (with /etc/init.d/apache2 restart, as it run through
> Passenger) fixes the problem - but only temporarily. Next time a host
> contacts the puppetmaster, I get the error stated above. :-( And it
> fails until I restart puppetmaster again.

Usually MySQL restarts itself after a crash (it runs through a wrapper
called mysqld_safe usually).

But maybe, the mysql crash leaves the puppetmaster in an unstable state
which prevents it to connect back to MySQL.

> I have the following optimizations in my passenger configuration (cut
> from Nigel Kerstens mails about his Passengerbased setup):
>
> PassengerLogLevel 1
> PassengerMaxRequests 3000
> PassengerMaxPoolSize 15
> PassengerStatThrottleRate 600
>
> I've also tried to optimize the MySQL configuration, but as a restart of
> Apache/Passenger solves the problem (for one clientrun), I don't think
> that the problem lies with MySQL.
>
> Do any of you have a clue of what can be wrong here?

I'd suggest you run your master outside of passenger to see if you can
replicate the issue.
Run the master on the console with --no-daemonize --debug --trace.
Then connect your clients.
If it crashes, you can rule out passenger as the culprit.
If it works, then it is a bad interaction between passenger, rails
and/or MySQL.

Good luck,
--
Brice Figureau
My Blog: http://www.masterzen.fr/

[Juri Rischel Jensen]

Hi Brice

Thanks for your answer - it helped me a bit further in the
troubleshooting. Running puppetmaster outside of passenger gives no
errors at all. All my hosts check in and puppetmaster stores their
config. So it seems that there's a bug when using Passenger together
with storeconfigs. Maybe it's the version of rails (2.0.5) that's the
culprit? Is it compatible with passenger?

My list of gems:

actionmailer (2.0.5)
actionpack (2.0.5)
activerecord (2.0.5)
activeresource (2.0.5)
activesupport (2.0.5)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
fastthread (1.0.4, 1.0.1)
gem_plugin (0.2.3)
hoe (1.11.0)
mongrel (1.1.5)
mysql (2.7)
passenger (2.2.2, 2.1.2)
rack (0.4.0)
rails (2.0.5)
rake (0.8.7, 0.8.4)
rubyforge (1.0.3)
rubygems-update (1.3.3)

I'm using v. 0.4.0 of rack as the documentation describes (http://reductivelabs.com/trac/puppet/wiki/UsingPassenger
). When calling puppetmaster directly on the console, I bypass Rack
too, don't I...?

--

[Juri Rischel Jensen]

Hi everyone

Am I the only one that has stumbled upon this bug....?

I've tried to raise the loglevel to info in config.ru, but it gives me
no further info, unfortunately. :-(

On Jun 6, 2009, at 23:05 , Juri Rischel Jensen wrote:

> Thanks for your answer - it helped me a bit further in the
> troubleshooting. Running puppetmaster outside of passenger gives no
> errors at all. All my hosts check in and puppetmaster stores their
> config. So it seems that there's a bug when using Passenger together
> with storeconfigs. Maybe it's the version of rails (2.0.5) that's the
> culprit? Is it compatible with passenger?

--

[Juri Rischel Jensen]

Hi everyone

I've now tried to change database to postgresql, and unfortunately: I
get the same result:

Jun 9 12:03:13 puppet puppetmasterd[31869]: Could not store configs:
PGError: server closed the connection unexpectedly#012#011This
probably means the server terminated abnormally#012#011before or while
processing the request.#012: BEGIN

So something in the combination of Passenger, activerecord,
puppetmaster, rack and maybe more, does not work. Damn! I really don't
know how I should proceed with this, as running puppetmasterd outside
passenger doesn't produce the error, and raising the errorlevel in
config.ru doesn't provide me with more info than the line above.

How can I track down such an error...?

On Jun 6, 2009, at 23:05 , Juri Rischel Jensen wrote:

> Thanks for your answer - it helped me a bit further in the
> troubleshooting. Running puppetmaster outside of passenger gives no
> errors at all. All my hosts check in and puppetmaster stores their
> config. So it seems that there's a bug when using Passenger together
> with storeconfigs. Maybe it's the version of rails (2.0.5) that's the
> culprit? Is it compatible with passenger?

--