puppetdb '[replace catalog]' postgresql errors with index 'catalog_resources_pkey'

[Mathew Crane]

I am moving our environment from puppet 2.7.1 with puppet dashboard to puppet 3.5.1 with puppetdb 2.0. During testing on a separate host with the identical setup as our puppet CA, everything seems to be functioning properly: puppet agents are able to be added on the master and receive everything they should, and I'm able to query the pgsl db and API and view information and facts. However, I've noticed two concerning things in the logs below. 

I am by no means a pgsl expert, and after enabling debug logging I'm unable to tell why it's firing these alerts.  Do I need to tune the db somehow, or is there something off in my configs? 

1.) After inserting node information into catalog_resources, an error occurs in /var/log/puppetdb/puppetdb.log:

2014-05-30 08:20:15,328 ERROR [c.p.p.command] [...] [replace catalog] Retrying after attempt 6, due to: org.postgresql.util.PSQLException: ERROR: index row size 4296 exceeds maximum 2712 for index "catalog_resources_pkey"

  Hint: Values larger than 1/3 of a buffer page cannot be indexed.

Consider a function index of an MD5 hash of the value, or use full text indexing.

2.) Additionally, I am seeing a lot of deprecation warnings in /var/log/puppetdb/puppetdb.log, like below. I would think that puppet 3.5.1 with puppetdb 2.0 wouldn't fire these? Are they safe to ignore?

'2014-05-30 08:39:36,021 WARN  [c.p.p.command] command 'replace catalog' version 3 is deprecated, use the latest version'

Here is my config on the test CA:

/etc/puppet/puppet.conf:

[main]

    logdir = /var/log/puppet

    rundir = /var/run/puppet

    ssldir = $vardir/ssl

[agent]

    server = puppetca.fqdn

    environment = live

    classfile = $vardir/classes.txt

    localconfig = $vardir/localconfig

    pluginsync = true

[master]

    environment = live

    certname = puppetca.fqdn

    dns_alt_names = puppetca

    ca_name = 'TEST CA'

    storeconfigs = true

    storeconfigs_backend = puppetdb

    reports = store,puppetdb

    ssl_client_header = SSL_CLIENT_S_DN

    ssl_client_verify_header = SSL_CLIENT_VERIFY

    pluginsync=true

[Ken Barber]

> I am moving our environment from puppet 2.7.1 with puppet dashboard to
> puppet 3.5.1 with puppetdb 2.0. During testing on a separate host with the
> identical setup as our puppet CA, everything seems to be functioning
> properly: puppet agents are able to be added on the master and receive
> everything they should, and I'm able to query the pgsl db and API and view
> information and facts. However, I've noticed two concerning things in the
> logs below.
>
> I am by no means a pgsl expert, and after enabling debug logging I'm unable
> to tell why it's firing these alerts. Do I need to tune the db somehow, or
> is there something off in my configs?
>
> 1.) After inserting node information into catalog_resources, an error occurs
> in /var/log/puppetdb/puppetdb.log:
>
> 2014-05-30 08:20:15,328 ERROR [c.p.p.command] [...] [replace catalog]
> Retrying after attempt 6, due to: org.postgresql.util.PSQLException: ERROR:
> index row size 4296 exceeds maximum 2712 for index "catalog_resources_pkey"
> Hint: Values larger than 1/3 of a buffer page cannot be indexed.
> Consider a function index of an MD5 hash of the value, or use full text
> indexing.

Now this is a new one for me ... the catalog_resources_pkey is
actually a combination of, the catalog_id (bigint), the type and title
(both text fields).

Sounds like either the type or title has exceeded the index maximum
size, which is very uncommon but I can see how this might happen.

It would be very hard to create a large resource type, but not
impossible. Whats more likely is that the title of one of your
resources is excessively large. This is possible most likely with an
exec {} or something similar that uses a very large command as the
title of the exec, instead of putting it in the command parameter like
so: http://docs.puppetlabs.com/references/latest/type.html#exec-attribute-command.
Either that, or something very similar.

Without seeing your catalog however I can only guess. However you
might be able to track this down yourself. If you go onto the node
that is seemingly throwing the error, you can analyze the catalog file
and track down any elongated titles or types that might be triggering
this. Here is an example of how I would go about it:
https://gist.github.com/kbarber/b60798cc0ba4f262ae1e. To get a
compiled catalog you may need to temporarily disable puppetdb for this
to be created, either way - if you can get a hold of the catalog I
think we can find our answers.

BTW this might be considered a bug, and might call for better
normalisation in our schema to avoid this. Having said that, a type
and title combination this excessive is very abnormal, so it might be
something we want to stop from happening in Puppet anyway :-). So
weither this is a bug for puppet or puppetdb we'll have to determine.

> 2.) Additionally, I am seeing a lot of deprecation warnings in
> /var/log/puppetdb/puppetdb.log, like below. I would think that puppet 3.5.1
> with puppetdb 2.0 wouldn't fire these? Are they safe to ignore?
>
> '2014-05-30 08:39:36,021 WARN [c.p.p.command] command 'replace catalog'
> version 3 is deprecated, use the latest version'

This looks like you haven't upgraded puppetdb-terminus to the latest
version (2.0.0) on your puppet master. We've bumped the command
version to 4 now, this looks like its just the old code. Also people
often forget to restart the Apache process when upgrading the terminus
code, don't forget to do that.

ken.

[Mathew Crane]

 

It would be very hard to create a large resource type, but not
impossible. Whats more likely is that the title of one of your
resources is excessively large. This is possible most likely with an
exec {} or something similar that uses a very large command as the
title of the exec, instead of putting it in the command parameter like
so: http://docs.puppetlabs.com/references/latest/type.html#exec-attribute-command.
Either that, or something very similar.

Thank you for the quick, thorough, and incredibly helpful reply. You pointed me in the right direction. Turns out we had an older module that was doing just what you suspected.  There were two titles of over 370 characters apiece.  Here is the original module: https://github.com/puppet-modules/puppet-common/blob/master/manifests/defines/line.pp

Command used to troubleshoot:

cat /var/lib/puppet/client_data/catalog/puppetca.fqdn.json | python -mjson.tool | grep title

I resolved this issue by changing all execs to named execs, differentiated by the '${file}' variable passed into the script to avoid duplicate declarations. For example, exec { name: "d_line_ensure_present_${file}": command => ... } 

Checking the logs, I no longer see the errors. 

BTW this might be considered a bug, and might call for better
normalisation in our schema to avoid this. Having said that, a type
and title combination this excessive is very abnormal, so it might be
something we want to stop from happening in Puppet anyway :-). So
weither this is a bug for puppet or puppetdb we'll have to determine.

Let me know if you need any more information from my end on this.

 

This looks like you haven't upgraded puppetdb-terminus to the latest
version (2.0.0) on your puppet master. We've bumped the command
version to 4 now, this looks like its just the old code. Also people
often forget to restart the Apache process when upgrading the terminus
code, don't forget to do that.

puppetdb-terminus was updated along with puppetdb, but httpd had not been restarted.  

Thanks again, Ken!

[Ken Barber]

No problem :-).

Can you raise a bug on the original exec {} issue for me?
https://tickets.puppetlabs.com/browse/PDB

ken.

[Mathew Crane]

No problem :-).

Can you raise a bug on the original exec {} issue for me?
https://tickets.puppetlabs.com/browse/PDB

ken.

https://tickets.puppetlabs.com/browse/PDB-742

 

[Ken Barber]

Thanks mate :-).

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/cffe9c19-f419-47d2-8793-65ba4506db89%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.