Jira (PDB-5633) Investigate why (some?) command discards aren't including the stack

[Rob Browning (Jira)]

Rob Browning created an issue

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack Issue Type: Task Assignee: Unassigned Created: 2023/04/27 11:19 AM Priority: Normal Reporter: Rob Browning When pdb discards a command, it includes the stacktrace via .printStackTrace (see dlo.clj), but during an escalation, the json.gz_err.txt file just had something like: Attempt 3 @ ... java.lang.NullPointerException Attempt 2 @ ... java.lang.NullPointerException Attempt 1 @ ... java.lang.NullPointerException Is that expected/intended? Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

[Charmaine Pritchett (Jira)]

Charmaine Pritchett updated an issue

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Charmaine Pritchett Labels: jira_escalated

Add Comment

[Charmaine Pritchett (Jira)]

Charmaine Pritchett updated an issue

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Charmaine Pritchett Zendesk Ticket Count: 1 Zendesk Ticket IDs: 51495

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack Hi Rob - let m know if there is anything you need me to grab from our customer that may be of use here? Gareth...

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Rob - just as an aside  - Blackrock seem to be suffering from files building up in the discards dir constantly  - so knowing what the issue is would be really useful. g....

Add Comment

[Andy Troup (Jira)]

Andy Troup updated an issue

PuppetDB / PDB-5633

Investigate why (some?) command discards aren't including the stack

Change By: Andy Troup Labels: BlackRock jira_escalated

Add Comment

[Saurabh Dikshit (Jira)]

Saurabh Dikshit updated an issue

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Saurabh Dikshit Zendesk Ticket Count: 1 2 Zendesk Ticket IDs: 51495 ,52057

Add Comment

This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack Hi Rob Browning  - let us know if there's anything else we need to provide here for further investigation  - We had another case of this with SCB in https://tickets.puppetlabs.com/browse/PE-36108

Add Comment

[Cas Donoghue (Jira)]

Cas Donoghue updated an issue

PuppetDB / PDB-5633

Investigate why (some?) command discards aren't including the stack

Change By: Cas Donoghue Sprint: Skeletor 06/07/2023

Add Comment

[Rob Browning (Jira)]

Rob Browning assigned an issue to Rob Browning

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Rob Browning Assignee: Rob Browning

Add Comment

[Rob Browning (Jira)]

Rob Browning updated an issue

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Rob Browning Sprint: Skeletor 06/07/2023

Add Comment

[Rob Browning (Jira)]

Rob Browning assigned an issue to Unassigned

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Rob Browning Assignee: Rob Browning

Add Comment

[Rob Browning (Jira)]

Rob Browning commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack So this ticket was originally intended to (at least) figure out how to fix the (presumed) pdb bug that was preventing the stack traces from printing, but after I investigated for a bit, it didn't look like there was a pdb bug. My search for other culprits didn't hit the mark, but then we think Austin probably found the cause: https://bugs.openjdk.org/browse/JDK-8273392 Assuming that's right, and perhaps even as a test to see if it's right, we thought one possible approach would be to turn that option off at one of the sites and see if the stack traces appear. That would also (more importantly in the short term) hopefully allow us to see what's causing the command failures. We think we should be able to disable the optimization, which should restore the stack traces by adding this JVM argument to the pdb invocation: -XX:-OmitStackTraceInFastThrow

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Rob Browning  - it might be tricky to get Blackrock to make this change  - I can go back and see if they can but they'll not be able to change this directly in production so it could be some time before the change would hot "production" where the issue lies for them. SCB may be in a similar position also  - they are just large orgs and making changes is difficult

Add Comment

[Andy Troup (Jira)]

Andy Troup commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Gareth McGrillan theres no harm in asking. I'm also speaking with them today

Add Comment

[Andy Troup (Jira)]

Andy Troup updated an issue

PuppetDB / PDB-5633

Investigate why (some?) command discards aren't including the stack

Change By: Andy Troup Labels: BlackRock jira_escalated

Add Comment

[Andy Troup (Jira)]

Andy Troup updated an issue

PuppetDB / PDB-5633 Investigate why (some?) command discards aren't including the stack

Change By: Andy Troup Labels: TAM0001 jira_escalated

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack Rob Browning  - forgive my ignorance  - do we add this in the `/etc/sysconfig/pe-puppetdb` invocation or is this added as part of their heira? g...

Add Comment

[Joshua Partlow (Jira)]

Joshua Partlow commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Hi Gareth; yes, that's the file that has pe-puppetdb's JAVA_ARGS. One thing to note is that that env var is managed by puppet; so if you need any reliability or permanence around the change, I believe you will want to adjust puppet_enterprise::profile::puppetdb::java_args where they probably already have memory settings set. I think you'd be adding a "XX:-OmitStackTraceInFastThrow" => "" to the map. But I would double check with someone else in support that I'm not leading you astray on the best way to provide these kind of one off java args overrides for customers.

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Joshua Partlow  - do you guys have a way to provoke the system to create a discard file ?  We can make the change to config bugt we can't see reproduce the actual issue G...

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Joshua Partlow  - is this a configuration option that's available  to set in the console  - I've tried *_ {"XX":"-OmitStackTraceInFastThrow"} * under Class: puppet_enterprise::profile::puppetdb_ but it keeps dropping the : Would you have the correct syntax to configure in the console ? I would like to do it there as this is something that i would like the customers to add and then remove until we fix it in a release g...

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

I think I've got it sorted :   {"XX:-OmitStackTraceInFastThrow":""} is the syntax... Do you know of any way to provoke the discards to see if it actually works  - I'd like to test it before giving it out to BlackRock or SCB (Saurabh Dikshit) Thanks, G...

Add Comment

[Joshua Partlow (Jira)]

Joshua Partlow commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Sorry, Gareth, there must be something going on with Jira. I didn't get notified about any of these comments until today for some reason. Yes, your syntax is correct now, but I'm not sure about the best way to force a discard. Rob Browning?

Add Comment

[Gareth McGrillan (Jira)]

Gareth McGrillan commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

No worries - any pointers to poke would be gratefully received!

Add Comment

[Rob Browning (Jira)]

Rob Browning commented on PDB-5633

Re: Investigate why (some?) command discards aren't including the stack

Hmm, I don't know if there's any reliable way to provoke the behavior since it's a JVM optimization that may happen "under some circumstances": https://docs.oracle.com/javase/8/docs/api/java/lang/Throwable.html#getStackTrace-- As a practical matter, with our JVM, it sounds like it depends on various things, including whether or not the jit has decided the code is "hot" enough.  I suppose you could try submitting a bunch of broken commands, and maybe it'd eventually start happening on the discards.

Add Comment