puppet apply stops with message "Killed"

[Ádám Sándor]

Hi,

I'm having a problem with my puppet script hanging for a while then dying with the message "Killed" printed to the console. There is no further explanation of what happened but I'm suspecting some event propagation might get stuck and puppet may have some mechanism to kill the process if it hangs for too long. That's just a guess though, so please let me know if anyone has some insight on this problem. Thanks!

I'm running on Debian 7, Puppet 3.5.1.

Tail of the output of "puppet apply --verbose --debug --modulepath=/root/puppet/modules /root/puppet/ks-prod.pp" is:

...

Notice: /Stage[main]/Ks::Database/Exec[create-blog-db]/returns: executed successfully

Debug: /Stage[main]/Ks::Database/Exec[create-blog-db]: The container Class[Ks::Database] will propagate my refresh event

Debug: Class[Ks::Database]: The container Stage[main] will propagate my refresh event

Debug: Class[Ks::Database]: The container Class[Ks::Server] will propagate my refresh event

Debug: /Stage[main]/Apache::Mod::Autoindex/Apache::Mod[autoindex]/File[autoindex.load symlink]/mode: Not managing symlink mode

Debug: /Stage[main]/Apache::Mod::Autoindex/Apache::Mod[autoindex]/File[autoindex.conf symlink]/mode: Not managing symlink mode

Debug: Exec[concat_/etc/apache2/ports.conf](provider=posix): Executing check '/var/lib/puppet/concat/bin/concatfragments.sh -o /var/lib/puppet/concat/_etc_apache2_ports.conf/fragments.concat.out -d /var/lib/puppet/concat/_etc_apache2_ports.conf -t    '

Debug: Executing '/var/lib/puppet/concat/bin/concatfragments.sh -o /var/lib/puppet/concat/_etc_apache2_ports.conf/fragments.concat.out -d /var/lib/puppet/concat/_etc_apache2_ports.conf -t    '

Debug: /Stage[main]/Apache::Mod::Prefork/Apache::Mpm[prefork]/File[/etc/apache2/mods-enabled/prefork.conf]/mode: Not managing symlink mode

Debug: /Stage[main]/Apache::Mod::Php/Apache::Mod[php5]/File[php5.conf symlink]/mode: Not managing symlink mode

Debug: Exec[own apache log directory](provider=posix): Executing 'chown -R -f www-data:www-data /var/log/apache2'

Debug: Executing 'chown -R -f www-data:www-data /var/log/apache2'

Notice: /Stage[main]/Ks::Web/Exec[own apache log directory]/returns: executed successfully

Debug: /Stage[main]/Ks::Web/Exec[own apache log directory]: The container Class[Ks::Web] will propagate my refresh event

Debug: Class[Ks::Web]: The container Stage[main] will propagate my refresh event

Debug: Class[Ks::Web]: The container Class[Ks::Server] will propagate my refresh event

Debug: Class[Ks::Server]: The container Stage[main] will propagate my refresh event

Killed

[Felix Frank]

Hi,

can you find out wether that is reproducible with Puppet 3.4? If not,
you may just want to hold out for 3.6, wait if it works better.

For debugging this, I would try two things.

First, keep an eye on 'ps auxwf' output while puppet is seemingly
frozen, see wether and what is being forked.

If that doesn't help, you can try with `strace -f`, although there's a
fair chance that it won't yield very conclusive traces.

HTH,
Felix

[Ádám Sándor]

Thanks for the reply. It seems that the Killed message is coming from the vserver manager (or whatever it's called, I don't know much about the technology), because the process is using too much memory. Which is a problem because it means that puppet is using more memory then the application actually needs. I'll look into it more closely and try the stuff you posted see what that shows.

[Ádám Sándor]

Viewing the process tree doesn't show anything new, it doesn't look like puppet is forking any new processes. It's just using more and more memory and 94% of CPU. This happens even if I run the provisioning after everything is set up so it doesn't even have to do much.

Something I left out from my original post is that I'm running in a vserver environment. When I tried to replicate that environment locally in my vagrant box by disabling swap space and setting memory to 512MB puppet didn't have a problem, so this could be vserver related. Does anyone have a similar experience?

[Juan Sierra Pons]

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/e4ebdcdc-4bc2-46d0-8e79-24370336ba17%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Hi,

I had a similar problem with another software and the reponsible of
killing the process was the "Out-of-Memory Process Killer"

Probably your server has not enough memory and no swap and when the
puppet process runs the Out-of-Memory Process Killer kills it.

Solutions /workarounds:

Add some swap to the server
Add some memory

Hope it helps

Best regards

--------------------------------------------------------------------------------------
Juan Sierra Pons ju...@elsotanillo.net
Linux User Registered: #257202
Web: http://www.elsotanillo.net Git: http://www.github.com/juasiepo
GPG key = 0xA110F4FE
Key Fingerprint = DF53 7415 0936 244E 9B00 6E66 E934 3406 A110 F4FE
--------------------------------------------------------------------------------------

[Ádám Sándor]

I managed to solve the problem :) First we added more memory to the machine. With 1GB of memory the script ran further but still failed. In this case it became apparent that it's doing something with file permissions. It was setting group 70 (don't even know what kind of group that is) to some scripts it was supposed to copy for execution. It was doing this very slowly - tens of seconds for each file - and eating more and more memory in the process, until it reached 1GB and the vserver killed it.

The fix was very simple, but I'm still intrigued as to what went wrong, especially since it was such a hard-to-debug issue. I changed this resource:

file { 'migration-scripts':

    path => $scriptPath,

    ensure => directory,

    source => 'puppet:///modules/ks/migration',

    recurse => true,

    owner => 'root'

  }

to this:

file { 'migration-scripts':

    path => $scriptPath,

    ensure => directory,

    source => 'puppet:///modules/ks/migration',

    recurse => true

  } 

If someone can give an explanation I would be very happy to read it. Also because I only got 1GB of memory on the server temporarily and I'm suspecting another similar problem in my manifest since with 512GB memory the execution didn't get to the part where I now spotted the group permission setting.

[Felix Frank]

Not an explanation off the top of my hat, but seeing as none of this is
supposed to happen, let's start at the base. What version of Linux and
which vserver patch are you running?

[jcbollinger]

On Tuesday, April 29, 2014 2:53:18 PM UTC-5, Ádám Sándor wrote:

I managed to solve the problem :) First we added more memory to the machine. With 1GB of memory the script ran further but still failed. In this case it became apparent that it's doing something with file permissions. It was setting group 70 (don't even know what kind of group that is) to some scripts it was supposed to copy for execution. It was doing this very slowly - tens of seconds for each file - and eating more and more memory in the process, until it reached 1GB and the vserver killed it.

When you say you fixed the problem, do you mean that the new failure occurs while applying a different resource?

 

The fix was very simple, but I'm still intrigued as to what went wrong, especially since it was such a hard-to-debug issue. I changed this resource:

file { 'migration-scripts':

    path => $scriptPath,

    ensure => directory,

    source => 'puppet:///modules/ks/migration',

    recurse => true,

    owner => 'root'

  }

to this:

file { 'migration-scripts':

    path => $scriptPath,

    ensure => directory,

    source => 'puppet:///modules/ks/migration',

    recurse => true

  } 

If someone can give an explanation I would be very happy to read it. Also because I only got 1GB of memory on the server temporarily and I'm suspecting another similar problem in my manifest since with 512GB memory the execution didn't get to the part where I now spotted the group permission setting.

As Felix said, this is not generally expected or documented behavior.  The best we can do is guess, and we're working with incomplete information when it comes to that.  I observe, however, that recursive File resources can be a source of trouble.  They do work, but they require memory on the agent proportional to the total number of files, plus a bit.  Also, if your recursive File is following links ('links' parameter) then it is possible to give it a logically infinite directory tree to attempt to sync.  You don't specify 'links' in your example, and I don't recall offhand what the default behavior is in that regard, but it's something to watch out for.

In addition, File resources in general are expensive when the managed content is large and you use the default checksum algorithm.  The choice of algorithm is a trade-off between expense of the test and likelihood of a wrong result, and Puppet defaults to the least likelihood of a wrong result.

If you are managing large collections of files or file collections having large aggregate size, then you would be better off avoiding recursive File resources.  The usual recommendation is to bundle up the files into a package (e.g. an RPM or DEB), put that in a local repository, and manage it via a Package resource.

As for the group assigned to newly-created files, it is probably the (numeric) group the source files have on the master.  If you don't specify a group for a source'd file, and that file does not initially exist on the target node, then the group assigned to it at creation time is the one the source file has.  Similarly for the file owner.  If that's indeed what you are seeing then that aspect of the observed behavior is as expected.  (In contrast, if you don't specify the group (or owner), and the file does already exist, then Puppet does not modify its group (or owner).)

I cannot explain why declining to manage the file owner makes a significant difference in memory consumption, but it's reasonable to suppose that the large memory consumption is a factor in the poor performance.  The 'physical' memory assigned to your virtual machine is not necessarily matched with real physical memory on the underlying vserver host.  Therefore, processes in the VM that consume a lot of memory may cause a lot of page faults on the host, with the attending performance impact.  The negative performance effects of such page faulting might even be magnified in the VM.


John

[Len Rugen]

Did puppet hit a ulimit?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/569c0062-4372-4428-a865-51e0d587b127%40googlegroups.com.