With Mongrel, external node script inherits listen socket, leading to problems

6 views
Skip to first unread message

Jos Backus

unread,
Jul 21, 2008, 8:08:04 PM7/21/08
to puppe...@googlegroups.com
Hi all,

I'm using Puppet with Mongrel and the Apache proxy_balancer_module. The
puppetmasterd process listens on port 28140.

It seems that when using Mongrel, the external node script is run with the
puppetmasterd listen socket being passed in as Mongrel doesn't set
close-on-exec on the listen socket, unlike the Webrick wrapper in Puppet. This
occasionally creates problems as it appears that Apache will sometimes connect
to the socket in the external node script instead of the puppetmasterd. I
accidentally found this because my external node script got stuck and I saw
Apache connect to it.

Observe the following lsof output of the external node script, taken at the
end of its run:

Without the close-on-exec in Mongrel:

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
lw-puppet 32042 puppet cwd DIR 8,6 4096 2731042 /var/service/puppetmaster-mongrel-1
lw-puppet 32042 puppet rtd DIR 8,5 4096 2 /
lw-puppet 32042 puppet txt REG 8,5 3508 830133 /usr/bin/ruby
lw-puppet 32042 puppet mem REG 8,5 911912 823589 /usr/lib/libruby.so.1.8.5
lw-puppet 32042 puppet mem REG 8,5 206336 2541714 /lib/libm-2.5.so
lw-puppet 32042 puppet mem REG 8,5 1598720 2541699 /lib/libc-2.5.so
lw-puppet 32042 puppet mem REG 8,5 25992 2541702 /lib/libcrypt-2.5.so
lw-puppet 32042 puppet mem REG 8,5 43036 934412 /usr/lib/ruby/1.8/i386-linux/socket.so
lw-puppet 32042 puppet mem REG 8,5 113480 934415 /usr/lib/ruby/1.8/i386-linux/syck.so
lw-puppet 32042 puppet mem REG 8,5 119212 2541726 /lib/libpthread-2.5.so
lw-puppet 32042 puppet mem REG 8,5 120368 2539715 /lib/ld-2.5.so
lw-puppet 32042 puppet mem REG 8,5 14644 2541712 /lib/libdl-2.5.so
lw-puppet 32042 puppet mem REG 8,5 17376 934413 /usr/lib/ruby/1.8/i386-linux/stringio.so
lw-puppet 32042 puppet mem REG 8,6 217016 2158472 /var/db/nscd/hosts
lw-puppet 32042 puppet 0r CHR 1,3 1178 /dev/null
lw-puppet 32042 puppet 1u REG 8,5 0 22036510 /tmp/puppet.29537.0
lw-puppet 32042 puppet 2u REG 8,5 0 22036510 /tmp/puppet.29537.0
lw-puppet 32042 puppet 3w REG 8,6 23690777 1994973 /var/log/puppet/puppetmaster.log
lw-puppet 32042 puppet 4u IPv4 1871182332 TCP localhost.localdomain:51554->localhost.localdomain:8141 (CLOSE_WAIT)
lw-puppet 32042 puppet 5u IPv4 1871182334 TCP localhost.localdomain:51555->localhost.localdomain:8141 (CLOSE_WAIT)
lw-puppet 32042 puppet 6w REG 8,5 0 22036514 /tmp/lwn.out
lw-puppet 32042 puppet 7u IPv4 1871174582 TCP localhost.localdomain:28140 (LISTEN)
lw-puppet 32042 puppet 8r FIFO 0,6 1871182337 pipe
lw-puppet 32042 puppet 9u IPv4 1871182327 TCP localhost.localdomain:28140->localhost.localdomain:42951 (ESTABLISHED)
lw-puppet 32042 puppet 10u REG 8,5 0 22036510 /tmp/puppet.29537.0

With the close-on-exec in Mongrel (see attached patch):

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
lw-puppet 19703 puppet cwd DIR 8,6 4096 2731042 /var/service/puppetmaster-mongrel-1
lw-puppet 19703 puppet rtd DIR 8,5 4096 2 /
lw-puppet 19703 puppet txt REG 8,5 3508 830133 /usr/bin/ruby
lw-puppet 19703 puppet mem REG 8,5 14644 2541712 /lib/libdl-2.5.so
lw-puppet 19703 puppet mem REG 8,5 113480 934415 /usr/lib/ruby/1.8/i386-linux/syck.so
lw-puppet 19703 puppet mem REG 8,5 120368 2539715 /lib/ld-2.5.so
lw-puppet 19703 puppet mem REG 8,5 1598720 2541699 /lib/libc-2.5.so
lw-puppet 19703 puppet mem REG 8,5 43036 934412 /usr/lib/ruby/1.8/i386-linux/socket.so
lw-puppet 19703 puppet mem REG 8,5 25992 2541702 /lib/libcrypt-2.5.so
lw-puppet 19703 puppet mem REG 8,5 911912 823589 /usr/lib/libruby.so.1.8.5
lw-puppet 19703 puppet mem REG 8,5 17376 934413 /usr/lib/ruby/1.8/i386-linux/stringio.so
lw-puppet 19703 puppet mem REG 8,5 206336 2541714 /lib/libm-2.5.so
lw-puppet 19703 puppet mem REG 8,5 119212 2541726 /lib/libpthread-2.5.so
lw-puppet 19703 puppet mem REG 8,6 217016 2158472 /var/db/nscd/hosts
lw-puppet 19703 puppet 0r CHR 1,3 1178 /dev/null
lw-puppet 19703 puppet 1u REG 8,5 0 22036510 /tmp/puppet.14976.0
lw-puppet 19703 puppet 2u REG 8,5 0 22036510 /tmp/puppet.14976.0
lw-puppet 19703 puppet 3w REG 8,6 23693843 1994973 /var/log/puppet/puppetmaster.log
lw-puppet 19703 puppet 4u IPv4 1871248672 TCP localhost.localdomain:55953->localhost.localdomain:8141 (CLOSE_WAIT)
lw-puppet 19703 puppet 5u IPv4 1871248674 TCP localhost.localdomain:55954->localhost.localdomain:8141 (CLOSE_WAIT)
lw-puppet 19703 puppet 6w REG 8,5 7700 22036514 /tmp/lwn.out
lw-puppet 19703 puppet 7r FIFO 0,6 1871248676 pipe
lw-puppet 19703 puppet 9u IPv4 1871248662 TCP localhost.localdomain:28140->localhost.localdomain:42850 (ESTABLISHED)
lw-puppet 19703 puppet 10u REG 8,5 0 22036510 /tmp/puppet.14976.0

Note the absence of the `localhost.localdomain:28140 (LISTEN)' socket in the
second case.

I don't see a way to add this change to the Puppet code easily because unlike
in the Webrick situation, the Mongrel wrapper doesn't seem to expose the
listen socket.

--
Jos Backus
jos at catnook.com

mongrel-cloexec.diff

Luke Kanies

unread,
Jul 21, 2008, 10:46:39 PM7/21/08
to puppe...@googlegroups.com
On Jul 21, 2008, at 7:08 PM, Jos Backus wrote:

> I'm using Puppet with Mongrel and the Apache proxy_balancer_module.
> The
> puppetmasterd process listens on port 28140.
>
> It seems that when using Mongrel, the external node script is run
> with the
> puppetmasterd listen socket being passed in as Mongrel doesn't set
> close-on-exec on the listen socket, unlike the Webrick wrapper in
> Puppet. This
> occasionally creates problems as it appears that Apache will
> sometimes connect
> to the socket in the external node script instead of the
> puppetmasterd. I
> accidentally found this because my external node script got stuck
> and I saw
> Apache connect to it.
>
> Observe the following lsof output of the external node script, taken
> at the
> end of its run:


Hmm, seems like a clear bug, but it's a bug with Mongrel, right?

We obviously can use the patch in the meantime, but have you filed it
against Mongrel?

--
Whenever I hear anyone arguing for slavery, I feel a strong impulse to
see it tried on him personally. -- Abraham Lincoln
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

Jos Backus

unread,
Jul 22, 2008, 12:49:33 AM7/22/08
to puppe...@googlegroups.com
On Mon, Jul 21, 2008 at 09:46:39PM -0500, Luke Kanies wrote:
> Hmm, seems like a clear bug, but it's a bug with Mongrel, right?

Yes. In fact, unless there's a way to access the listen socket from within
Puppet it can only be fixed in Mongrel.

> We obviously can use the patch in the meantime, but have you filed it
> against Mongrel?

I will for sure, tomorrow. Thanks for the feedback, Luke.

Luke Kanies

unread,
Jul 22, 2008, 1:13:23 AM7/22/08
to puppe...@googlegroups.com

Great, thanks, and thanks for tracking this down.

--
Some people are afraid of heights. I'm afraid of widths.
-- Stephen Wright

Reply all
Reply to author
Forward
0 new messages