Jira (PDB-4268) Add some kind of metric tracking the number of concurrent enqueues

19 views
Skip to first unread message

Rob Browning (JIRA)

unread,
Feb 15, 2019, 12:26:05 PM2/15/19
to puppe...@googlegroups.com
Rob Browning created an issue
 
PuppetDB / New Feature PDB-4268
Add some kind of metric tracking the number of concurrent enqueues
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2019/02/15 9:25 AM
Priority: Normal Normal
Reporter: Rob Browning
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Nick Walker (JIRA)

unread,
Feb 28, 2019, 1:27:04 PM2/28/19
to puppe...@googlegroups.com
Nick Walker updated an issue
Change By: Nick Walker
This metric is about getting a count of the number of `/cmd` API requests that are waiting to write to disk because all the slots provided by the `concurrent-writes` setting are full.

A continuously positive number for this metric would indicate a need to increase `concurrent-writes`, get a faster disk, or separate your queue onto a different disk than other I/O heavy processes. 

Zachary Kent (Jira)

unread,
Feb 7, 2021, 4:22:05 PM2/7/21
to puppe...@googlegroups.com
Zachary Kent commented on New Feature PDB-4268
 
Re: Add some kind of metric tracking the number of concurrent enqueues

For this ticket we'll need to add a new metric to the mq-metrics-registry found in command.cl_j. The metric will likely need to be updated in the do-enqueue-command func where it acquires a token from the _write-semaphore. Each command should inc this metric before attempting to acquire the token and dec the metric once the token is acquired. 

 

Acceptance criteria: 

    * An added metric to the mq-metrics-registry which tracks the number of commands waiting to acquire a token from the write semaphore at any given time. 

    * A test which shows that this metric increases and decreases as expected. 

 

Testing notes: 

It may be useful to override the concurrent-writes setting during the test and set it to one. Then using with-redefs or some other method to pause command processing in the do-enqueue-command func after an incoming command has acquired the one token from the write-semaphore we should be able to submit more commands which will block on the (.acquire write-semaphore) call. Once the system is in this state we can check that the metric was updated properly. We should also check that once all the commands are allowed to be processed the metric drops to a zero value.

 

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Zachary Kent (Jira)

unread,
Feb 7, 2021, 4:23:03 PM2/7/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Feb 8, 2021, 10:39:02 AM2/8/21
to puppe...@googlegroups.com

Zachary Kent (Jira)

unread,
Feb 8, 2021, 5:55:04 PM2/8/21
to puppe...@googlegroups.com
Zachary Kent commented on New Feature PDB-4268
 
Re: Add some kind of metric tracking the number of concurrent enqueues

See related slack conversation here for more detail on what the approach described above will be measuring.

Zachary Kent (Jira)

unread,
Feb 8, 2021, 7:17:04 PM2/8/21
to puppe...@googlegroups.com
Zachary Kent updated an issue
 
Change By: Zachary Kent
Acceptance Criteria:
    * An added metric to the mq-metrics-registry which tracks the number of commands waiting to acquire a token from the write semaphore at any given time.

    * A test which shows that this metric increases and decreases as expected.

Bogdan Irimie (Jira)

unread,
Feb 10, 2021, 9:43:02 AM2/10/21
to puppe...@googlegroups.com

Oana Tanasoiu (Jira)

unread,
Feb 10, 2021, 9:43:03 AM2/10/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Feb 10, 2021, 9:43:03 AM2/10/21
to puppe...@googlegroups.com

Austin Blatt (Jira)

unread,
Feb 22, 2021, 12:13:03 PM2/22/21
to puppe...@googlegroups.com

Austin Blatt (Jira)

unread,
Feb 22, 2021, 12:13:03 PM2/22/21
to puppe...@googlegroups.com

Claudia Petty (Jira)

unread,
Jun 21, 2023, 10:57:09 AM6/21/23
to puppe...@googlegroups.com
Claudia Petty updated an issue
Change By: Claudia Petty
Labels: new-feature tsr-pdb-backlog
This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages