Jira (PDB-5107) SPIKE - Investigate PDB work required for PE-31775

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-5107 SPIKE - Investigate PDB work required for PE-31775 Change By: Austin Blatt Summary: SPIKE - Investigate PDB work required for PE-31775 Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-5107 SPIKE - Investigate PDB work required for PE-31775

Change By: Austin Blatt Sprint: HA 2020-05-05 , HA 2020-05-19

Add Comment

[Austin Blatt (Jira)]

Austin Blatt assigned an issue to Austin Blatt

PuppetDB / PDB-5107 SPIKE - Investigate PDB work required for PE-31775

Change By: Austin Blatt Assignee: Austin Blatt

Add Comment

[Austin Blatt (Jira)]

Austin Blatt commented on PDB-5107

Re: SPIKE - Investigate PDB work required for PE-31775 ruby->puppetserver conversion A java.lang.String instance containing the HTTP request body submitted by the puppet agent and recieved by Java. The content of this string is UTF-16 encoded, which means it uses twice the memory a UTF-8 encoded string would need to store the same ASCII data. Retains 39,835,016 bytes. ruby->puppetserver conversion A org.jruby.RubyString instance containing a copy of the HTTP request body after conversion from Java to a Puppet::Network::HTTP::Request. Retains 21,909,328 bytes. This is in Puppet An org.jruby.RubyHash instance representing the report data after the Puppet::Network::HTTP::Request body is parsed to create a Puppet::Transaction::Report instance. Retains 26,189,328 bytes. This is being held onto by both the original report object submitted for processing, and PuppetDB's duplicate, so we can't remove this entirely via the terminus, and is likely necessary for the report processor to function properly. An org.jruby.RubyArray instance holding the log entries of the report. Created when the Puppet::Transaction::Report instance is duplicated before processing by PuppetDB. Retains 8,705,088 bytes. This one we can likely remove but so far I haven't been able. An org.jruby.RubyHash instance representing a copy of the report data, transformed by the PuppetDB report processor. Retains 27,387,848 bytes. This one is necessary at command submission An org.jruby.RubyString instance created by serializing the above hash to JSON for submission to PuppetDB. Retains 13,334,192 bytes. This one can be easily removed. An org.jruby.RubyString instance created by duplicating the above string and adding some metadata. Used soley for computing a PuppetDB command checksum. Retains 13,334,272 bytes. ruby->puppetserver conversion A com.puppetlabs.http.client.RequestOptions instance used to make the actual POST request to PuppetDB that contains a copy of the above strings as the request body. The request body in this object is a java.lang.String which also pays the UTF-16 tax. Retains 26,668,368 bytes.

Add Comment

[Austin Blatt (Jira)]

Austin Blatt commented on PDB-5107

Re: SPIKE - Investigate PDB work required for PE-31775

Did some rough estimates and this reduced overall puppetserver memory usage by 12.2% (in this test case). And of the large objects in memory associated with the processing of a report, that memory usage was reduced by 23.0%. <!-- target: terminus-savings --> ```python total = 39835016 + 21909328 + 26189328 + 8705088 + 27387848 + 13334192 + 13334272 + 26668368 saved_mem = 27387848 + 13334272 print("{0:.1%}".format(saved_mem / total)) ```   <!-- name: terminus-savings --> ``` 23.0% ```   <!-- target: puppetserver-savings --> ```python total_mb_before = 311 total_mb_after = 273 print("{0:.1%}".format((total_mb_before - total_mb_after) / total_mb_before)) ```   <!-- name: puppetserver-savings --> ``` 12.2% ```

Add Comment