Jira (PUP-8115) Puppet randomly corrupts files (it puts json/pson instead of file content)

[Ivan Bayan (JIRA)]

Ivan Bayan created an issue

Puppet / PUP-8115 Puppet randomly corrupts files (it puts json/pson instead of file content) Issue Type: Bug Assignee: Unassigned Created: 2017/11/02 2:54 AM Environment: client: puppet 3.8.7 CentOS release 6.6 (Final) ruby 1.8.7 server: puppetserver version: 2.8.0 Ubuntu 16.04.3 LTS foreman 1.15.6-1 openjdk-8-jdk-headless 8u131-b11-2ubuntu1.16.04.3 openjdk-8-jre-headless 8u131-b11-2ubuntu1.16.04.3 ruby 2.3.1 Priority: Normal Reporter: Ivan Bayan Few weeks ago we have started to notice corrupted files which managed by puppet. Every time content of file changed to something like json. It happens rarely and unpredictable. For example, file resource definition: file { '/etc/puppet/etckeeper-commit-pre': mode => '755', owner => root, group => root, source => 'puppet:///modules/amoeba/etckeeper-commit-pre', } It was regular shell script, but content changed to: {{{"path":"/etc/puppetlabs/code/environments/fcc_production/modules/amoeba/files/etckeeper-commit-pre","relative_path":null,"links":"manage","owner":0,"group":752,"mode":420,"checksum":{"type":"md5","value":" {md5}20814627a1afc8495d264bd1403b498c"},"type":"file","destination":null}}} Last modification date: {{~# ls -l /etc/puppet/etckeeper-commit-pre -rwxr-xr-x 1 root root 279 Oct 30 09:35 /etc/puppet/etckeeper-commit-pre}} For that puppet run, syslog contains next: {{Oct 30 09:30:50 lou-webproxy puppet-agent[16977]: Local environment: "production" doesn't match server specified node environment "fcc_production", switching agent to "fcc_production". Oct 30 09:33:30 lou-webproxy puppet-agent[16977]: Could not retrieve catalog from remote server: execution expired Oct 30 09:33:30 lou-webproxy puppet-agent[16977]: Using cached catalog Oct 30 09:33:35 lou-webproxy puppet-agent[16977]: (/Stage[main]/Security_updates/File[/etc/yum/yum-cron.conf]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/security_updates/yum-cron.conf: Could not intern from text/pson: Paths must be fully qualified Oct 30 09:34:48 lou-webproxy puppet-agent[16977]: (/Stage[main]/Bash::Logout/File[/etc/bash.bash_logout_fcc]/content) content changed '{md5} c3b5264e7fe471a18f30f8468af159ea' to ' {md5}93b20bfff221414cdbaf45bedc8db6f2' Oct 30 09:34:56 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/epel/RPM-GPG-KEY-EPEL-6: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream) Oct 30 09:35:20 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-pre]/content) content changed '{md5} 20814627a1afc8495d264bd1403b498c' to ' {md5}d865e6b948a74cb03bc3401c0b01b785' Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6]) Skipping because of failed dependencies Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-debuginfo]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-debuginfo]) Skipping because of failed dependencies Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-source]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-source]) Skipping because of failed dependencies Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel]) Skipping because of failed dependencies Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-post]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/amoeba/etckeeper-commit-post: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream) Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper]) Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper]) Skipping because of failed dependencies Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source]) Skipping because of failed dependencies Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: blockdevice_sr0_model Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sr0_model]/message) defined 'message' as 'blockdevice_sr0_model' Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo]) Skipping because of failed dependencies Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Puppettidy/Exec[empty dirs under /var/lib/puppet/clientbucket]/returns) executed successfully Oct 30 09:35:59 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:59 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing]) Skipping because of failed dependencies Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Service[puppet]) Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Service[puppet]) Skipping because of failed dependencies Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: blockdevice_sda_model Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sda_model]/message) defined 'message' as 'blockdevice_sda_model' Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: Finished catalog run in 149.49 seconds Oct 30 09:36:06 lou-webproxy puppet-agent[16977]: Could not send report: undefined method `from_data_hash' for Array:Class}} puppetserver.log {{Oct 30 09:30:50 lou-webproxy puppet-agent[16977]: Local environment: "production" doesn't match server specified node environment "fcc_production", switching agent to "fcc_production". Oct 30 09:33:30 lou-webproxy puppet-agent[16977]: Could not retrieve catalog from remote server: execution expired Oct 30 09:33:30 lou-webproxy puppet-agent[16977]: Using cached catalog Oct 30 09:33:35 lou-webproxy puppet-agent[16977]: (/Stage[main]/Security_updates/File[/etc/yum/yum-cron.conf]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/security_updates/yum-cron.conf: Could not intern from text/pson: Paths must be fully qualified Oct 30 09:34:48 lou-webproxy puppet-agent[16977]: (/Stage[main]/Bash::Logout/File[/etc/bash.bash_logout_fcc]/content) content changed '{md5} c3b5264e7fe471a18f30f8468af159ea' to ' {md5}93b20bfff221414cdbaf45bedc8db6f2' Oct 30 09:34:56 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/epel/RPM-GPG-KEY-EPEL-6: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream) Oct 30 09:35:20 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-pre]/content) content changed '{md5} 20814627a1afc8495d264bd1403b498c' to ' {md5} d865e6b948a74cb03bc3401c0b01b785' Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6]) Skipping because of failed dependencies Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-debuginfo]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-debuginfo]) Skipping because of failed dependencies Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-source]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-source]) Skipping because of failed dependencies Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:21 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel]) Skipping because of failed dependencies Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-post]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/amoeba/etckeeper-commit-post: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream) Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper]) Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper]) Skipping because of failed dependencies Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source]) Skipping because of failed dependencies Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: blockdevice_sr0_model Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sr0_model]/message) defined 'message' as 'blockdevice_sr0_model' Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo]) Skipping because of failed dependencies Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Puppettidy/Exec[empty dirs under /var/lib/puppet/clientbucket]/returns) executed successfully Oct 30 09:35:59 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:59 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing]) Skipping because of failed dependencies Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Service[puppet]) Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Service[puppet]) Skipping because of failed dependencies Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: blockdevice_sda_model Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sda_model]/message) defined 'message' as 'blockdevice_sda_model' Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: Finished catalog run in 149.49 seconds Oct 30 09:36:06 lou-webproxy puppet-agent[16977]: Could not send report: undefined method `from_data_hash' for Array:Class}} About same time when it happened first time, we tried to improve puppet server performance, so we tried to tune 'max-active-instances', but returned it back. Why it happens? Add Comment

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)

[Eric Sorenson (JIRA)]

Eric Sorenson updated an issue

Puppet / PUP-8115 Puppet randomly corrupts files (it puts json/pson instead of file content)

Change By: Eric Sorenson

Few weeks ago we have started to notice corrupted files which managed by puppet. Every time content of file changed to something like json. It happens rarely and unpredictable. For example, file resource definition:

{code}

file { '/etc/puppet/etckeeper-commit-pre':     mode   => '755',     owner  => root,     group  => root,     source => 'puppet:///modules/amoeba/etckeeper-commit-pre',   }

{code}

It was regular shell script, but content changed to:

{ code} { { "path":"/etc/puppetlabs/code/environments/fcc_production/modules/amoeba/files/etckeeper-commit-pre","relative_path":null,"links":"manage","owner":0,"group":752,"mode":420,"checksum":{"type":"md5","value":"{md5}20814627a1afc8495d264bd1403b498c"},"type":"file","destination":null} }} Last modification date:

{{ ~# ls -l /etc/puppet/etckeeper-commit-pre -rwxr-xr-x 1 root root 279 Oct 30 09:35 /etc/puppet/etckeeper-commit-pre

{code } }

For that puppet run, syslog contains next:

{ { code} Oct 30 09:30:50 lou-webproxy puppet-agent[16977]: Local environment: "production" doesn't match server specified node environment "fcc_production", switching agent to "fcc_production".

{code } } puppetserver.log { { code}

{code } }

About same time when it happened first time, we tried to improve puppet server performance, so we tried to tune 'max-active-instances', but returned it back. Why it happens?

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson assigned an issue to Ivan Bayan

Puppet / PUP-8115 Puppet randomly corrupts files (it puts json/pson instead of file content)

Change By: Eric Sorenson Assignee: Ivan Bayan

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson commented on PUP-8115

Re: Puppet randomly corrupts files (it puts json/pson instead of file content) Hi Ivan - this is going to be really hard to dig into without a stand-alone reproduction case. The content of that file is the puppet server's HTTP response for file metadata - so something has gone awry where it put that response into the content instead of using it to determine whether to retrieve the actual file content. The errors from the agent are interesting:

Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-post]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/amoeba/etckeeper-commit-post: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream)

I think the second log is not actually your puppetserver, maybe a mis-paste? It's the same content as the agent one. This suggests the agent and the server are not talking the same language - what version of puppet is on your master?

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-8115

Re: Puppet randomly corrupts files (it puts json/pson instead of file content)

It'd also be useful to run the agent with - http_debug and -debug to see what REST requests it's making.

Add Comment

[Ivan Bayan (JIRA)]

Ivan Bayan updated an issue

Puppet / PUP-8115

Puppet randomly corrupts files (it puts json/pson instead of file content)

Change By: Ivan Bayan

Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-post]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/amoeba/etckeeper-commit-post: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream)

Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper]) Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper]) Skipping because of failed dependencies Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:51 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source]) Skipping because of failed dependencies Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: blockdevice_sr0_model Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sr0_model]/message) defined 'message' as 'blockdevice_sr0_model' Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo]) Skipping because of failed dependencies Oct 30 09:35:58 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Puppettidy/Exec[empty dirs under /var/lib/puppet/clientbucket]/returns) executed successfully Oct 30 09:35:59 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing]) Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 30 09:35:59 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing]) Skipping because of failed dependencies Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Service[puppet]) Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Amoeba::Etckeeper/Service[puppet]) Skipping because of failed dependencies Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: blockdevice_sda_model Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: (/Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sda_model]/message) defined 'message' as 'blockdevice_sda_model' Oct 30 09:36:00 lou-webproxy puppet-agent[16977]: Finished catalog run in 149.49 seconds Oct 30 09:36:06 lou-webproxy puppet-agent[16977]: Could not send report: undefined method `from_data_hash' for Array:Class {code} puppetserver.log {code}

Oct 2017-10-  30 09: 35:02,983 INFO  [qtp1978657657-1639189] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10- 30  09 : 50 lou 35:03,311 INFO  [qtp1978657657 - webproxy puppet 1639168] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017 - agent 10-30 09:35:03,418 INFO [ 16977 qtp1978657657-1639554 ]  [puppetserver] Puppet Not Found :  Could not find file_metadata plugins 2017-10-30 09:35:03,433 INFO  [qtp1978657657-1647678] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:03,441 INFO  [qtp1978657657-1631368] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:03,540 INFO  [qtp1978657657-1639593] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:03,746 INFO  [qtp1978657657-1642277] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:03,752 INFO  [qtp1978657657-1644916] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:03,958 INFO  [qtp1978657657-1647662] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:04,056 INFO  [qtp1978657657-1644918] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:35:04,060 INFO  [qtp1978657657-1634004] [puppetserver] Puppet Not Found: Could not find file_metadata plugins 2017-10-30 09:36:10,278 WARN  [qtp1978657657-1644994] [puppetserver] //lou-webproxy.int/Puppet  Local environment: "production" doesn't match server specified node environment "fcc_production", switching agent to "fcc_production". Oct 2017-10-  30 09: 33 36 : 30 lou 10,278 ERROR [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] :  //lou-webproxy.int/Puppet  Could not retrieve catalog from remote server: execution expired Oct 2017-10-  30 09: 33 36 : 30 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] :  //lou-webproxy.int/Puppet  Using cached catalog Oct 2017-10-  30 09: 33 36 : 35 lou 10,278 ERROR [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Security_updates/File[/etc/yum/yum-cron.conf] )  Could not evaluate: Could not retrieve file metadata for puppet:///modules/security_updates/yum-cron.conf: Could not intern from text/pson: Paths must be fully qualified Oct 2017-10-  30 09: 34 36 : 48 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Bash::Logout/File[/etc/bash.bash_logout_fcc]/content )  content changed '{md5}c3b5264e7fe471a18f30f8468af159ea' to '{md5}93b20bfff221414cdbaf45bedc8db6f2' Oct 2017-10-  30 09: 34 36 : 56 lou 10,278 ERROR [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] )  Could not evaluate: Could not retrieve file metadata for puppet:///modules/epel/RPM-GPG-KEY-EPEL-6: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream) Oct 2017-10-  30 09: 35 36 : 20 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-pre]/content )  content changed '{md5}20814627a1afc8495d264bd1403b498c' to '{md5}d865e6b948a74cb03bc3401c0b01b785' Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Epel::Rpm_gpg_key[EPEL-6]/Exec[import-EPEL-6] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-debuginfo] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-debuginfo] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-source] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-source] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 21 lou 10,278 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 51 lou 10,279 ERROR [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/File[/etc/puppet/etckeeper-commit-post] )  Could not evaluate: Could not retrieve file metadata for puppet:///modules/amoeba/etckeeper-commit-post: Could not intern from application/octet-stream: No format match the given format name or mime-type (application/octet-stream) Oct 2017-10-  30 09: 35 36 : 51 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper] )  Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 2017-10-  30 09: 35 36 : 51 lou 10,279 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Augeas[puppet.conf.etckeeper] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 51 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing-source] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 51 lou 10,279 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing- source so urce ] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 58 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] :  //lou-webproxy.int/Puppet  blockdevice_sr0_model Oct 2017-10-  30 09: 35 36 : 58 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sr0_model]/message )  defined 'message' as 'blockdevice_sr0_model' Oct 2017-10-  30 09: 35 36 : 58 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 58 lou 10,279 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-debuginfo] )  Skipping because of failed dependencies Oct 2017-10-  30 09: 35 36 : 58 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Puppettidy/Exec[empty dirs under /var/lib/puppet/clientbucket]/returns )  executed successfully Oct 2017-10-  30 09: 35 36 : 59 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing] )  Dependency File[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] has failures: true Oct 2017-10-  30 09: 35 36 : 59 lou 10,279 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Epel/Yumrepo[epel-testing] )  Skipping because of failed dependencies Oct 2017-10-  30 09:36: 00 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Service[puppet] )  Dependency File[/etc/puppet/etckeeper-commit-post] has failures: true Oct 2017-10-  30 09:36: 00 lou 10,279 WARN  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Amoeba::Etckeeper/Service[puppet] )  Skipping because of failed dependencies Oct 2017-10-  30 09:36: 00 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] :  //lou-webproxy.int/Puppet  blockdevice_sda_model Oct 2017-10-  30 09:36: 00 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] : ( / /lou-webproxy.int// Stage[main]/Lsi_raid::Zabbix/Notify[blockdevice_sda_model]/message )  defined 'message' as 'blockdevice_sda_model' Oct 2017-10-  30 09:36: 00 lou 10,279 INFO  [qtp1978657657 - webproxy puppet-agent 1644994] [ 16977 puppetserver ] :  //lou-webproxy.int/Puppet  Finished catalog run in 149.49 seconds

Oct 30 09:36:06 lou-webproxy puppet-agent[16977]: Could not send report: undefined method `from_data_hash' for Array:Class {code} About same time when it happened first time, we tried to improve puppet server performance, so we tried to tune 'max-active-instances', but returned it back. Why it happens?

Add Comment

[Ivan Bayan (JIRA)]

Ivan Bayan commented on PUP-8115

Re: Puppet randomly corrupts files (it puts json/pson instead of file content) Sorry for that mistake, i fixed log from puppetserver.log I put info in 'environment' description, here is it:

client: puppet 3.8.7 CentOS release 6.6 (Final) ruby 1.8.7

server: puppetserver version: 2.8.0 Ubuntu 16.04.3 LTS foreman 1.15.6-1 openjdk-8-jdk-headless 8u131-b11-2ubuntu1.16.04.3 openjdk-8-jre-headless 8u131-b11-2ubuntu1.16.04.3 ruby 2.3.1

Also i found one similar bug: https://projects.puppetlabs.com/issues/1010 It's very old, but it's very close to problem which occurred in our environment.

Add Comment

[Ivan Bayan (JIRA)]

Ivan Bayan commented on PUP-8115

Re: Puppet randomly corrupts files (it puts json/pson instead of file content)

I could run puppet with option -debug , but as i said before it happens rarely, here is very low chance that puppet will corrupt anything when i will run it manually.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-8115

Puppet randomly corrupts files (it puts json/pson instead of file content)

Change By: Josh Cooper Team: Coremunity

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-8115

Re: Puppet randomly corrupts files (it puts json/pson instead of file content) This is probably an issue with persistent http connections introduced in puppet 3.7. In the comments above, puppet tries to retrieve file metatdata, but gets an http response containing binary content (application/octet-stream). Presumably the binary content is from a previous content request. If the connection raises an exception (such as a timeout, connection reset, etc), then ruby should raise an exception and puppet will not reuse the connection. However, it is possible ruby is swallowing an exception. I've seen 2 cases so far: ruby silently truncates the response (see PUP-9911), and doesn't raise exceptions when writing to the socket. Puppet also doesn't read the response body when handling redirects. I think one option is to ensure response body's are drained before we add the connection back to the pool.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-8115

Re: Puppet randomly corrupts files (it puts json/pson instead of file content)

Puppet's networking code has been completely rewritten from scratch and we ensure that persistent connections are drained before adding them back to the connection pool. It's possible there's a ruby bug, but since this ticket hasn't been updated in a while, I'm going to close it. Please reopen if you see this issue again.

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)