Could not retrieve file metadata ... end of file reached

[James Green]

We occasionally have an agent fail because of this. I'm told by others running the agents more frequently that it appears to be at random and not on anything particularly large.

Looks like server and agents are 3.7.4, both Ubuntu. Can't see much in the way of server logs.

Any ideas where to go to debug this? A few minutes on, and the agent in this case has now succeeded...

Thanks,

James

[Christopher Wood]

It seems like other people have had this issue, and maybe it's when the puppetmaster process restarts in the middle of somebody's agent run?

https://projects.puppetlabs.com/issues/3019
https://projects.puppetlabs.com/issues/4808

In your place I would probably do the following:

a) turn up the log level a bit on the master and agent
b) packet capture on agent->master connections

(You have the keys/certs for both ends so you can decrypt the traffic: http://wiki.wireshark.org/SSL)

Then when it happens again I would correlate the time in the logs with the packet capture to find the message that discussed the "end of file". Then I would see if the agent actually received an EOF character or similar oddity from the master. If the logs are detailed enough I might be able to track that down to something happening on the master.

I tried grepping through my local puppet installation for those error messages but I'm not skilled enough to tell how the things I found are relevant.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bax-gWpyZLWHpB19_EWtD3OnG8U%3DHEvkJFtyK1HGzkAHHA%40mail.gmail.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bax-gWpyZLWHpB19_EWtD3OnG8U%3DHEvkJFtyK1HGzkAHHA%40mail.gmail.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[Henrik Lindberg]

On 2015-05-03 12:02, James Green wrote:
> We occasionally have an agent fail because of this. I'm told by others
> running the agents more frequently that it appears to be at random and
> not on anything particularly large.
>

If you are using webrick then it is most likely a concurrency problem
(more than one agent calling in at the same time). Webrick is not
recommended for production use because of this.

- henrik
--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[James Green]

Hi,

Sorry for the delayed response.

In our case we are using Passenger.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/md9sfc%24r1e%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.

[James Green]

Error: /Stage[main]/Our_unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/our_unattended_upgrades/etc/apt/apt.conf.d/20auto-upgrades: end of file reached

Wrapped exception:

end of file reached

Versions on the server:

ii  facter                           2.4.1-1puppetlabs1               all          Ruby module for collecting simple facts about a host operating system

ii  hiera                            1.3.4-1puppetlabs1               all          A simple pluggable Hierarchical Database.

ii  puppet                           3.7.4-1puppetlabs1               all          Centralized configuration management - agent startup and compatibility scripts

ii  puppet-common                    3.7.4-1puppetlabs1               all          Centralized configuration management

ii  puppetdb                         2.2.2-1puppetlabs1               all          PuppetDB Centralized Storage.

ii  puppetdb-terminus                2.2.2-1puppetlabs1               all          Connect Puppet to PuppetDB by setting up a terminus for PuppetDB.

ii  puppetlabs-release               1.0-11                           all          "Package to install Puppet Labs gpg key and apt repo"

ii  puppetmaster-common              3.7.4-1puppetlabs1               all          Puppet master common scripts

ii  puppetmaster-passenger           3.7.4-1puppetlabs1               all          Centralised configuration management - master setup to run under mod passenger

Are we looking at a known bug or are we really going to need to debug this?

James

[James Green]

I was running puppet agent -t --noop repeatedly with this error.

Running again, this time omitting --noop, it succeeded. Not entirely clear what this tells me...

[Josh Cooper]

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2Bay6EoB1zCZ587s%2B-gWno%3DGdHBR%3DKkhmD1aVHHXn66TsVw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

The fact that you are on puppet 3.7 and the error occasionally occurs, but not always, makes me think it's https://tickets.puppetlabs.com/browse/PUP-3238. Starting in puppet 3.7, we've added support for persistent HTTP connections, see https://docs.puppetlabs.com/puppet/latest/reference/subsystem_agent_master_comm.html#persistent-connections--keepalive.

This eliminates the TCP/SSL handshake for every file metadata and content request, and greatly reduces the load on the master. But there is a possibility that a loadbalancer or puppetmaster may close an idle connection. Unfortunately, ruby does not surface socket errors when writing a request, only when reading. So often this issue manifests itself with "end of file" reached.

To confirm if this is the case, make sure the puppet agent's http_keep_alive setting is less than the keep alive on your master. See https://docs.puppetlabs.com/references/3.7.latest/configuration.html#httpkeepalivetimeout. You can also disable keepalive on the client by setting the timeout to 0.

Most likely your catalog is performing a long'ish operation, e.g. installing a package, the master closes the idle connection, the agent then tries to retrieve a file whose source is a puppet URI, which triggers a metadata request, and the agent receives the EOF exception.

Josh

--

Josh Cooper

Developer, Puppet Labs

PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.

Register now to take advantage of the Early Adopter discount —save $349!

[James Green]

I am not convinced that this is to do with an agent being busy then attempting the next connection.

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Error: Could not retrieve catalog from remote server: end of file reached

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

The above all happened within a few seconds. I might argue within 4-5 seconds.

Unless of course I'd misunderstood your hypothesis?

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2Bu97ukpopZZOcHp1bB5V_Ae34o_%3Dn5vcTakBx3rFccWO5trbg%40mail.gmail.com.

[Josh Cooper]

On Tue, Mar 17, 2015 at 4:52 AM, James Green <james.m...@gmail.com> wrote:

I am not convinced that this is to do with an agent being busy then attempting the next connection.

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Error: Could not retrieve catalog from remote server: end of file reached

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

The above all happened within a few seconds. I might argue within 4-5 seconds.

Unless of course I'd misunderstood your hypothesis?

Puppet's persistent http connection feature will attempt to reuse the connection for the duration of the run. There are several "opportunities" for the connection to remain idle while the puppet agent is busy doing other work, e.g. downloading a package from a repo, executing a long running process, evaluating facts (like your example above), etc. If the connection is idle longer than the master (or load-balancer's) keep alive or idle timeouts, then the server side will close the connection. For example, apache 2.4 defaults HTTP KeepAliveTimeout is 5 seconds[1], and I've seen customer environments where haproxy idle timeouts were set to 60 seconds.

One way to check if the server side is closing the connection, is to disable keep alive on the agent as I mentioned earlier. If the problem seems to go away, then I'd try re-enabling keep alive on the agent, and running tcpdump during an agent run. You'll likely see the master/load-balancer sending a TCP RST some exact number of seconds after the agent last wrote to the socket.

To resolve the issue, make sure your server side keep alive timeouts are greater than the agent's, either by increasing the server's timeout, or decreasing the agent's.

[1] http://httpd.apache.org/docs/2.4/mod/core.html#keepalivetimeout

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMH6%2BawYM087vXbgMN%3DL1xAZ_6kgDrwwzQJXtO%3Dd9%3DjJGMJQKw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--