Jira (PDB-4264) GC error (stderr) is swallowed in PE install

15 views
Skip to first unread message

Robert Roland (JIRA)

unread,
Feb 11, 2019, 6:29:03 PM2/11/19
to puppe...@googlegroups.com
Robert Roland created an issue
 
PuppetDB / Bug PDB-4264
GC error (stderr) is swallowed in PE install
Issue Type: Bug Bug
Affects Versions: PDB 5.2.7
Assignee: Unassigned
Components: PuppetDB
Created: 2019/02/11 3:28 PM
Priority: Major Major
Reporter: Robert Roland

On a system where puppetdb does not have a large enough heap setting, puppetdb can fail, and it will fail silently.

systemd log:

Feb 11 22:23:05 ip-10-227-1-110.amz-dev.puppet.net systemd[1]: Starting pe-puppetdb Service...
 
Feb 11 22:23:26 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: WARNING: pos-int? already refers to: #'clojure.core/pos-int? in namespace: taoensso.encore, being replaced by: #'taoensso.encore/pos-int?
 
Feb 11 22:23:26 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: WARNING: bytes? already refers to: #'clojure.core/bytes? in namespace: taoensso.encore, being replaced by: #'taoensso.encore/bytes?
 
Feb 11 22:23:36 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: The [database] classname config option has been retired and will be ignored.
 
Feb 11 22:23:36 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: The [database] log-slow-statements config option has been retired and will be ignored.
 
Feb 11 22:23:36 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: The [database] subprotocol config option has been retired and will be ignored.
 
Feb 11 22:23:40 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: The [database] classname config option has been retired and will be ignored.
 
Feb 11 22:23:40 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: The [database] log-slow-statements config option has been retired and will be ignored.
 
Feb 11 22:23:40 ip-10-227-1-110.amz-dev.puppet.net puppetdb[2358]: The [database] subprotocol config option has been retired and will be ignored.
 
Feb 11 22:23:42 ip-10-227-1-110.amz-dev.puppet.net systemd[1]: Started pe-puppetdb Service.
 
Feb 11 22:23:54 ip-10-227-1-110.amz-dev.puppet.net systemd[1]: pe-puppetdb.service: main process exited, code=killed, status=9/KILL
 
Feb 11 22:23:54 ip-10-227-1-110.amz-dev.puppet.net systemd[1]: Unit pe-puppetdb.service entered failed state.
 
Feb 11 22:23:54 ip-10-227-1-110.amz-dev.puppet.net systemd[1]: pe-puppetdb.service failed.

In the puppetdb log:

2019-02-11T23:06:47.981Z INFO  [p.p.c.services] Finished sweep of stale nodes (threshold: 7 days)
 
2019-02-11T23:06:47.983Z INFO  [p.p.c.services] Starting purge deactivated and expired nodes (threshold: 14 days)
 
2019-02-11T23:06:48.100Z INFO  [p.p.c.services] Finished purge deactivated and expired nodes (threshold: 14 days)
 
2019-02-11T23:06:48.110Z INFO  [p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
 
2019-02-11T23:06:48.189Z INFO  [p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
 
2019-02-11T23:06:48.217Z INFO  [p.p.c.services] Starting gc packages
 
2019-02-11T23:06:48.224Z INFO  [p.p.c.services] Finished gc packages
 
2019-02-11T23:06:48.225Z INFO  [p.p.c.services] Starting database garbage collection
 
2019-02-11T23:06:49.852Z INFO  [p.p.c.services] Finished database garbage collection

This error was only visible by running puppetdb in the foreground and watching the output:

2019-02-11T23:17:14.299Z ERROR [p.p.threadpool] Error processing command on thread cmd-proc-thread-3
 
java.lang.OutOfMemoryError: GC overhead limit exceeded
 
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
 
        at java.lang.String.<init>(String.java:207)
 
        at com.fasterxml.jackson.core.sym.CharsToNameCanonicalizer.findSymbol(CharsToNameCanonicalizer.java:428)
 
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseName(ReaderBasedJsonParser.java:1677)
 
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:682)
 
        at cheshire.parse$parse_STAR_.invokeStatic(parse.clj:63)
 
        at cheshire.parse$parse_STAR_.invoke(parse.clj:61)
 
        at cheshire.parse$parse_STAR_.invokeStatic(parse.clj:64)
 
        at cheshire.parse$parse_STAR_.invoke(parse.clj:61)
 
        at cheshire.parse$parse_STAR_.invokeStatic(parse.clj:63)
 
        at cheshire.parse$parse_STAR_.invoke(parse.clj:61)
 
        at cheshire.parse$parse.invokeStatic(parse.clj:98)
 
        at cheshire.parse$parse.invoke(parse.clj:86)
 
        at cheshire.core$parse_stream.invokeStatic(core.clj:250)
 
        at cheshire.core$parse_stream.invoke(core.clj:232)
 
        at cheshire.core$parse_stream.invokeStatic(core.clj:247)
 
        at cheshire.core$parse_stream.invoke(core.clj:232)
 
        at puppetlabs.puppetdb.queue$stream__GT_json.invokeStatic(queue.clj:49)
 
        at puppetlabs.puppetdb.queue$stream__GT_json.invoke(queue.clj:44)
 
        at puppetlabs.puppetdb.queue$cmdref__GT_cmd.invokeStatic(queue.clj:274)
 
        at puppetlabs.puppetdb.queue$cmdref__GT_cmd.invoke(queue.clj:252)
 
        at puppetlabs.puppetdb.command$process_cmdref.invokeStatic(command.clj:499)
 
        at puppetlabs.puppetdb.command$process_cmdref.invoke(command.clj:494)
 
        at puppetlabs.puppetdb.command$message_handler$fn__41447.invoke(command.clj:582)
 
        at puppetlabs.puppetdb.threadpool$dochan$fn__41183$fn__41184.invoke(threadpool.clj:117)
 
        at puppetlabs.puppetdb.threadpool$call_on_threadpool$fn__41178.invoke(threadpool.clj:95)
 
        at clojure.lang.AFn.run(AFn.java:22)
 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 
        at java.lang.Thread.run(Thread.java:748)
 
2019-02-11T23:17:15.925Z INFO  [p.p.command] [24458-1549862453165] [1624 ms] 'replace catalog' command processed for ip-10-227-3-65.amz-dev.puppet.net

PE uses:

 -XX:OnOutOfMemoryError=kill -9 %p

so, to systemd, it looks like the process just died.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Brett Gray (JIRA)

unread,
Sep 18, 2019, 1:10:03 AM9/18/19
to puppe...@googlegroups.com
Brett Gray commented on Bug PDB-4264
 
Re: GC error (stderr) is swallowed in PE install

I have experienced a similar issue and the investigation was painful part. I had to run in the foreground to find out what the real issue was.

2019-09-18T05:00:07.438Z ERROR [p.p.threadpool] Error processing command on thread cmd-proc-thread-1
 
java.lang.OutOfMemoryError: Java heap space

Zachary Kent (Jira)

unread,
Feb 17, 2021, 12:58:03 PM2/17/21
to puppe...@googlegroups.com
Zachary Kent updated an issue
 
Change By: Zachary Kent
Labels: needs-investigation
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

David McTavish (Jira)

unread,
Jan 12, 2022, 3:32:02 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Labels: has_workaround needs-investigation
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

David McTavish (Jira)

unread,
Jan 12, 2022, 3:33:02 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Priority: Major Normal

David McTavish (Jira)

unread,
Jan 12, 2022, 7:20:02 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Priority: Normal Low
Reply all
Reply to author
Forward
0 new messages