sporadic error searching puppetdb

[dma...@gmail.com]

Hi,

I'm using puppetdb 5.2.18 as a backend for puppetserver 5.3.12 and I get sporadic errors trying to run a search:

Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Failed to execute '/pdb/query/v4' on at least 1 of the following 'server_urls'

If I leave it for 30 seconds and run it again, it works fine.

Originally I did find that puppetdb was being OOM killed so I've adjusted those settings and that is no longer happening, however I can't find any evidence of what's going on in the logs now.

The only thing I can think of is that puppetdb (well, jruby) is shutting down threads while the search is being run, and then of course, when I run it again, new threads have been started up and work fine.

I tried setting the debug level per https://puppet.com/docs/puppetdb/latest/trouble_session_logging.html but I can't figure out what the logs are trying to tell me.

Does anyone have any suggestions?

Thanks,

Chris.