Puppet dumped multiple 1G files into folder after reboot

[Stephen S.]

We have a puppet server running 3.8.7 on RHEL 6.10. It experienced an OOM event this morning then the server recovered without intervention. While looking into this I noticed the server became very slow to respond. I saw that it suddenly had over 3200 processes running and was about to crash. I was able to reboot the box and it came back up fine. Everything now seems normal except for the fact that there are about 5 files now in /var/lib/pgsql/data/base// that are eating up 1G each. These are all timestamped with todays date and were not there until after the reboot. Does anyone know if these files are safe to delete? I am not familar with this particular folder or how it interacts with puppetmaster. We do not have puppetdb installed on this system so I am not sure why it would write to this folder.

Files created today.

-rw------- 1 postgres postgres 1.0G Feb 16 10:02 /var/lib/pgsql/data/base/16385/17240.2
-rw------- 1 postgres postgres 1.0G Feb 16 10:05 /var/lib/pgsql/data/base/16385/17240.4
-rw------- 1 postgres postgres 1.0G Feb 16 10:06 /var/lib/pgsql/data/base/16385/17240.3
rw-------. 1 postgres postgres 1.0G Feb 16 10:07 /var/lib/pgsql/data/base/16385/17240
-rw------- 1 postgres postgres 1.0G Feb 16 10:04 /var/lib/pgsql/data/base/16385/17240.1

[David Schmitt]

Hi Stephen,

while I can't say anything about what might have caused your issues in the first place, I can say with high confidence that the files you're seeing there *are* part of a postgresql database, and deleting them straight up will destroy something. To figure out what that "something" is, you'll need to figure out what databases are running on your system, and who is using it.

Good Luck,

David

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/476c2298-d121-4d6f-80c2-f17e3508a956%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers, David

https://twitter.com/dev_el_ops

[Stephen S.]

Yes, I was able to get into the psql db and poke around. I see legit data there so I'm not comfortable removing any files. But, I am still concerned with what causes this to happen. At 5G a pop this could quickly fill up a partition.

[David Schmitt]

Stephen,

apologies for the late response - I've been out sick last week.

I've poked around a bit in old bug reports and the only thing I could find is https://tickets.puppetlabs.com/browse/PDB-1812 , which says that PuppetDB 3.0.2 fixed an issue with autovaccuming that was not properly garbage collecting space in postgresql.

If you're on a version even older than that, and can't upgrade, it might make sense to look into manually configuring some regular database maintenance. That could possibly limit the growth of the files on disk to what the database actually needs for it's day-to-day work.

To get off puppet 3 please see the Upgrade Guide at https://docs.puppet.com/upgrade/upgrade_steps.html .

Good luck, David

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f68cb873-dae7-4731-b287-332d814902d7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.