PuppetDB errors

[Paul Seymour]

Hello,

Getting a couple of errors thrown with PuppetDB

2014-11-19 08:03:49,120 ERROR [c.p.p.command] [22329b48-7c20-4ec4-beb2-15cd02e11bff] [replace facts] Retrying after attempt 8, due to: org.postgresql.util.PSQLException: ERROR: update or delete on table "fact_paths" violates foreign key constraint "fact_values_path_id_fk" on table "fact_values"

  Detail: Key (id)=(512) is still referenced from table "fact_values".

org.postgresql.util.PSQLException: ERROR: update or delete on table "fact_paths" violates foreign key constraint "fact_values_path_id_fk" on table "fact_values"

  Detail: Key (id)=(512) is still referenced from table "fact_values".

Which is similar to PDB-1031 albeit with a different fact (ours is a custom one not _timestamp).

puppetdb=# select * from fact_paths where id = 512;

 id  | value_type_id | depth |                  name                  |                  path

-----+---------------+-------+----------------------------------------+----------------------------------------

 512 |             0 |     0 | XXXXXX_ethernet_eth0_intralink_unit_id | XXXXXX_ethernet_eth0_intralink_unit_id

(1 row)

puppetdb=# select * from fact_values where path_id = 512;

  id   | path_id | value_type_id |                value_hash                | value_integer | value_float | value_string | value_boolean | value_json

-------+---------+---------------+------------------------------------------+---------------+-------------+--------------+---------------+------------

 50319 |     512 |             0 | 80e54cb487aed67b98d7dc92feaac988bf95bcc7 |               |             | 107827       |               |

 22128 |     512 |             0 | 47e3876c8d334accbeb827775e728643581bbb83 |               |             | 107292       |               |

Is it just noise or something to be concerned about ? This is with v2.2.1 and PostGres 9.3.5

Thanks

Paul

[Wyatt Alt]

Hey Paul,

That's some kind of DB corruption. Shouldn't be happening. Do you have
any sense of when this started or whether it was tied to a recent
upgrade? Is there a stacktrace in your logs that you could gist? Does it
happen on every agent run or only occasionally? Is it always the same
path_id like in the other ticket?

Also would you mind reporting the output of this?

select * from facts where fact_value_id in (50319,22128);

Wyatt

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/08bf4cb9-6c80-489d-a0ef-26627f395f5e%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/08bf4cb9-6c80-489d-a0ef-26627f395f5e%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Paul Seymour]

On Wednesday, 19 November 2014 17:56:23 UTC, Wyatt Alt wrote:

Hey Paul,

That's some kind of DB corruption. Shouldn't be happening. Do you have
any sense of when this started or whether it was tied to a recent
upgrade? Is there a stacktrace in your logs that you could gist? Does it
happen on every agent run or only occasionally? Is it always the same
path_id like in the other ticket?

Also would you mind reporting the output of this?

select * from facts where fact_value_id in (50319,22128);

Thanks Wyatt,

I cannot be sure if this is tied to an upgrade specifically but I just happened to be keeping an eye

on the logs a little more lately (working on an ELK thing to monitoring them).

A stack trace is available at https://gist.github.com/PsychoSid/b1a479d7d6fec7477263

The output from the select is:-

$ psql

psql (9.3.5)

Type "help" for help.

postgres=# \c puppetdb

You are now connected to database "puppetdb" as user "postgres".

puppetdb=# select * from facts where fact_value_id in (50319,22128);

 factset_id | fact_value_id

------------+---------------

         22 |         50319

(1 row)

Hope this helps.

Cheers

Paul 

[Wyatt Alt]

Paul,

Thanks for that. It looks like 22128 has not been cleaned up properly for some reason, and it's likely being skipped over in the gc process because the facts cleanup identifies values to delete from that column in the facts table.

Are there failed commands in your /var/lib/puppetdb/mq/discarded directory you can tie this to? I'd like to take a look at that if you can send me a tarball.

This issue would likely disappear if you dropped and recreated the puppetdb database and allowed the nodes to repopulate it, but that could take some maneuvering depending on your situation/data needs.

Wyatt

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7baf1653-d0b4-4fcf-8bcb-039b43af99a5%40googlegroups.com.

[Wyatt Alt]

Manually deleting that row (id=22128) in fact_values might also do the trick if it's only one affected node/value, but wouldn't tell much about how you got to this state so could be leaving something broken.  It would be interesting to get the discarded tarball in any case.

[Paul Seymour]

Thanks Wyatt,

I have sent the tarballs from a couple of instances where we are seeing this to your inbox.

Cheers

Paul