Jira (PDB-4502) PuppetDB command size metrics are missing

11 views
Skip to first unread message

Charlie Sharpsteen (JIRA)

unread,
Sep 6, 2019, 9:11:03 PM9/6/19
to puppe...@googlegroups.com
Charlie Sharpsteen updated an issue
 
PuppetDB / Bug PDB-4502
PuppetDB command size metrics are missing
Change By: Charlie Sharpsteen
CS Priority: Needs Priority
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Charlie Sharpsteen (JIRA)

unread,
Sep 6, 2019, 9:11:03 PM9/6/19
to puppe...@googlegroups.com
Charlie Sharpsteen created an issue
Issue Type: Bug Bug
Affects Versions: PDB 6.5.0, PDB 5.2.9
Assignee: Unassigned
Created: 2019/09/06 6:10 PM
Priority: Normal Normal
Reporter: Charlie Sharpsteen

PuppetDB has a global.size message queue metric that provides the distribution of command sizes that have been submitted:

https://puppet.com/docs/puppetdb/6.5/api/metrics/v1/mbeans.html#message-queue-metrics

This information is important for determining PuppetDB memory use and detecting changes that have caused an abnormal amount of command data to be submitted. In current configurations, this metric is not receiving any data.

Reproduction Case

  • Install Puppet, Puppet Server, and PuppetDB 5 on a CentOS 7 system:

yum install -y http://yum.puppetlabs.com/puppet5-release-el-7.noarch.rpm
yum install -y puppetserver
 
source /etc/profile.d/puppet-agent.sh
 
puppet config set server $(puppet config print certname)
systemctl start puppetserver
puppet module install puppetlabs-puppetdb
 
puppet apply <<EOF
include puppetdb
class { 'puppetdb::master::config':
  manage_config =>  true,
  enable_reports => true,
}
EOF

  • Run the puppet agent to submit data to PuppetDB: puppet agent -t
  • Check the global.size metric:

Outcome

The metric returns no data:

curl -sS 'http://localhost:8080/metrics/v2/read/puppetlabs.puppetdb.mq:name=global.size' |python -m json.tool
{
    "request": {
        "mbean": "puppetlabs.puppetdb.mq:name=global.size",
        "type": "read"
    },
    "status": 200,
    "timestamp": 1567818278,
    "value": {
        "50thPercentile": 0.0,
        "75thPercentile": 0.0,
        "95thPercentile": 0.0,
        "98thPercentile": 0.0,
        "999thPercentile": 0.0,
        "99thPercentile": 0.0,
        "Count": 0,
        "Max": 0,
        "Mean": 0.0,
        "Min": 0,
        "SnapshotSize": 0,
        "StdDev": 0.0
    }
}

Despite the PuppetDB access log showing that command data was submitted:

# fgrep /pdb/cmd/ /var/log/puppetlabs/puppetdb/puppetdb-access.log
10.16.126.201 - - [07/Sep/2019:01:00:11 +0000] "POST
/pdb/cmd/v1?checksum=03c4cb56d666fdac92d41124d98c4b11c0563d83&version=5&certname=bwoy7ty3hsskdj0.delivery.puppetlabs.net&command=replace_facts&producer-timestamp=2019-09-07T01:00:11.496Z
HTTP/1.1" 200 53 "-" "Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_222)" 53
10.16.126.201 - - [07/Sep/2019:01:00:12 +0000] "POST
/pdb/cmd/v1?checksum=2f074c729b6180641c927fcf5293bf25d69cdce5&version=9&certname=bwoy7ty3hsskdj0.delivery.puppetlabs.net&command=replace_catalog&producer-timestamp=2019-09-07T01:00:12.042Z
HTTP/1.1" 200 53 "-" "Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_222)" 64

Expected Outcome

The Count component of the global.size metric is equal to the number of
requests to the /pdb/cmd API recorded in the access log.

Charlie Sharpsteen (JIRA)

unread,
Sep 6, 2019, 9:59:03 PM9/6/19
to puppe...@googlegroups.com
Charlie Sharpsteen commented on Bug PDB-4502
 
Re: PuppetDB command size metrics are missing

Looks like the metric is based on the Content-Length header:

https://github.com/puppetlabs/puppetdb/blob/6.5.0/src/puppetlabs/puppetdb/middleware.clj#L268-L276

We should change this to measure bytes written to stockpile.

Nick Walker (JIRA)

unread,
Sep 9, 2019, 11:20:04 AM9/9/19
to puppe...@googlegroups.com
Nick Walker assigned an issue to Rob Browning
 
Change By: Nick Walker
Assignee: Rob Browning

Jarret Lavallee (JIRA)

unread,
Sep 12, 2019, 7:30:03 PM9/12/19
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
Change By: Jarret Lavallee
CS Priority: Needs Priority Reviewed

Zachary Kent (JIRA)

unread,
Sep 16, 2019, 6:26:04 PM9/16/19
to puppe...@googlegroups.com
Zachary Kent assigned an issue to Zachary Kent
Change By: Zachary Kent
Assignee: Rob Browning Zachary Kent

Zachary Kent (JIRA)

unread,
Oct 2, 2019, 12:29:05 PM10/2/19
to puppe...@googlegroups.com
Zachary Kent updated an issue
Change By: Zachary Kent
Release Notes Summary: Fixed a bug that prevented command size metrics from being recorded and the max-command-size config option from working properly. This was done by adding a custom 'X-Uncompressed-Length' header to compressed requests sent from the terminus. The value of this header is the size in bytes of the command before it is compressed. This value is used to update the size metric and to decide when to discard commands based on
'max-command-size'.
Release Notes: Bug Fix

Zachary Kent (JIRA)

unread,
Oct 2, 2019, 12:31:03 PM10/2/19
to puppe...@googlegroups.com
Zachary Kent updated an issue
Change By: Zachary Kent
Fix Version/s: PDB 6.8.0
Fix Version/s: PDB 6.3.5
Fix Version/s: PDB 5.2.10

Austin Blatt (JIRA)

unread,
Oct 9, 2019, 2:26:04 PM10/9/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Fix Version/s: PDB 6.8.0
Fix Version/s: PDB 6.7.1
Reply all
Reply to author
Forward
0 new messages