Jira (PDB-4599) Remove the "HTTP request received while in maintenance mode" message

Charlie Sharpsteen (Jira)

unread,

Mar 19, 2020, 7:52:04 PM3/19/20

to puppe...@googlegroups.com

Charlie Sharpsteen commented on

PDB-4599

Re: Remove the "HTTP request received while in maintenance mode" message

Another report of this from Community Slack today. User was trying to determine why PuppetDB had not come up in 15 minutes after an upgrade and the only apparant messages in puppetdb.log were "HTTP request received while in maintenance mode"

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

Charlie Sharpsteen (Jira)

unread,

Mar 20, 2020, 2:11:03 PM3/20/20

to puppe...@googlegroups.com

Charlie Sharpsteen commented on

PDB-4599

Re: Remove the "HTTP request received while in maintenance mode" message

Looks like we do have a maintenance_mode? flag in the /status output that indicates PuppetDB is in maintenance mode:

# curl -s http://localhost:8080/status/v1/services/puppetdb-status |python -m json.tool

    "active_alerts": [],

    "detail_level": "info",

    "service_name": "puppetdb-status",

    "service_status_version": 1,

    "service_version": "6.9.1",

    "state": "starting",

    "status": {

        "maintenance_mode?": true,

        "queue_depth": 0,

        "read_db_up?": false,

        "write_db_up?": false

Add Comment

Charlie Sharpsteen (Jira)

unread,

Apr 27, 2020, 2:36:06 PM4/27/20

to puppe...@googlegroups.com

Charlie Sharpsteen updated an issue

PuppetDB /

PDB-4599

Remove the "HTTP request received while in maintenance mode" message

Change By:	Charlie Sharpsteen

PuppetDB currently logs the following message at INFO level every time it receives an API request while in maintenance mode:

{quote}

HTTP request received while in maintenance mode

{quote}

https://github.com/puppetlabs/puppetdb/blob/6.7.3/src/puppetlabs/puppetdb/pdb_routing.clj#L30

PuppetDB can end up in maintenance mode for tens of minutes or hours if a large migration or replication sync is occurring during starting startup . For deployments with a large volume of traffic, the above message repeats extremely often which bloats the logs and drowns out any indication of migration progress or stalls that are keeping the service in maintenance.

Additionally, we log a HTTP 503 status to {{puppetdb-access.log}}, so the message is mostly redundant. We should remove this message provided:

- We have clear logging for when the service enters maintenance mode that indicates PDB API requests will be refused.

- A request to the {{/status/v1/services}} API shows the system is in maintenance mode.