Jira (PUP-9213) Puppet agent should indicate which HTTP timeout has expired

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen created an issue

Puppet / PUP-9213 Puppet agent should indicate which HTTP timeout has expired Issue Type: Improvement Affects Versions: PUP 6.0.2, PUP 5.5.6 Assignee: Unassigned Created: 2018/10/08 9:32 AM Priority: Normal Reporter: Charlie Sharpsteen The Puppet agent has two configurable timeouts for HTTP connections: http_connect_timeout: The amount of time allowed for the connection to start. http_read_timeout: The maximum amount of time allowed between reading blocks of data in the server's response. The first timeout is usually tripped if a network issue is causing handshake packets to be dropped. The second timeout is usually tripped by an overloaded Puppet Server. However, we use the Ruby Timeout exception's generic "execution expired" message which does not indicate which timeout was hit. Having this information available in the error message would enable users to choose appropriate debugging methods. Reproduction Case Install the puppet-agent package on CentOS 7: rpm -Uvh http://yum.puppetlabs.com/puppet6/puppet-release-el-7.noarch.rpm yum install -y puppet-agent Add a firewall rule to drop inbound packets for port 8140: iptables -A INPUT -p tcp --dport 8140 -j DROP Run the puppet agent against localhost with a reduced connection timeout: /opt/puppetlabs/bin/puppet agent -t --http_connect_timeout 5s --server localhost Outcome Puppet agent's first HTTP connection fails with a generic "execution expired" message: # /opt/puppetlabs/bin/puppet agent -t --http_connect_timeout 5s --server localhost Info: Creating a new SSL key for o8nyrzjogl8gnjf.delivery.puppetlabs.net Error: Could not request certificate: execution expired Exiting; failed to retrieve certificate and waitforcert is disabled [root@o8nyrzjogl8gnjf ~]# rpm -Uvh http://yum.puppetlabs.com/puppet6/pup Expected Outcome The error message should indicate that a connection timeout was triggered instead of a read timeout. Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen updated an issue

Puppet / PUP-9213 Puppet agent should indicate which HTTP timeout has expired

Change By: Charlie Sharpsteen Team: Coremunity

Add Comment

[Gene Liverman (JIRA)]

Gene Liverman updated an issue

Puppet / PUP-9213 Puppet agent should indicate which HTTP timeout has expired

Change By: Gene Liverman Labels: customer0

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-9213 Puppet agent should indicate which HTTP timeout has expired

Change By: Josh Cooper Sprint: Coremunity Grooming

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-9213

Re: Puppet agent should indicate which HTTP timeout has expired Grooming Notes: 1. Target master 2. Update lib/puppet/http/client.rb to rescue OpenTimeout separate from ReadTimeout and update the corresponding message. 3. Update lib/puppet/network/http/connection.rb the same way.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-9213

Puppet agent should indicate which HTTP timeout has expired

Change By: Josh Cooper Sprint: Coremunity Grooming Hopper

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone updated an issue

Puppet / PUP-9213 Puppet agent should indicate which HTTP timeout has expired

Change By: Melissa Stone Sprint: Coremunity Hopper Platform Core KANBAN

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone assigned an issue to Melissa Stone

Puppet / PUP-9213 Puppet agent should indicate which HTTP timeout has expired

Change By: Melissa Stone Assignee: Melissa Stone

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone commented on PUP-9213

Re: Puppet agent should indicate which HTTP timeout has expired This may already be done. I just tested the latest puppet-agent release, and it's reporting a timeout error. [root@slim-bestseller ~]# rpm -Uvh http://yum.puppetlabs.com/puppet6/puppet-release-el-7.noarch.rpm Retrieving http://yum.puppetlabs.com/puppet6/puppet-release-el-7.noarch.rpm warning: /var/tmp/rpm-tmp.QejvPQ: Header V4 RSA/SHA256 Signature, key ID ef8d349f: NOKEY Preparing... ################################# [100%] Updating / installing... 1:puppet-release-1.0.0-7.el7 ################################# [100%] [root@slim-bestseller ~]# yum install -y puppet-agent Loaded plugins: fastestmirror localmirror-extras | 2.9 kB 00:00:00 localmirror-os | 3.6 kB 00:00:00 localmirror-updates | 2.9 kB 00:00:00 puppet | 2.5 kB 00:00:00 (1/5): localmirror-extras/primary_db | 159 kB 00:00:00 (2/5): puppet/x86_64/primary_db | 240 kB 00:00:00 (3/5): localmirror-os/group_gz | 165 kB 00:00:00 (4/5): localmirror-os/primary_db | 6.0 MB 00:00:00 (5/5): localmirror-updates/primary_db | 5.9 MB 00:00:00 Determining fastest mirrors Resolving Dependencies --> Running transaction check ---> Package puppet-agent.x86_64 0:6.12.0-1.el7 will be installed --> Finished Dependency Resolution   Dependencies Resolved   =========================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================== Installing: puppet-agent x86_64 6.12.0-1.el7 puppet 23 M   Transaction Summary =========================================================================================================================================== Install 1 Package   Total download size: 23 M Installed size: 23 M Downloading packages: warning: /var/cache/yum/x86_64/7/puppet/packages/puppet-agent-6.12.0-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID ef8d349f: NOKEY Public key for puppet-agent-6.12.0-1.el7.x86_64.rpm is not installed puppet-agent-6.12.0-1.el7.x86_64.rpm | 23 MB 00:00:01 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppet-release Importing GPG key 0xEF8D349F: Userid : "Puppet, Inc. Release Key (Puppet, Inc. Release Key) <rel...@puppet.com>" Fingerprint: 6f6b 1550 9cf8 e59e 6e46 9f32 7f43 8280 ef8d 349f Package : puppet-release-1.0.0-7.el7.noarch (installed) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-puppet-release Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : puppet-agent-6.12.0-1.el7.x86_64 1/1 Verifying : puppet-agent-6.12.0-1.el7.x86_64 1/1   Installed: puppet-agent.x86_64 0:6.12.0-1.el7   Complete! [root@slim-bestseller ~]# iptables -A INPUT -p tcp --dport 8140 -j DROP [root@slim-bestseller ~]# /opt/puppetlabs/bin/puppet agent -t --http_connect_timeout 5s --server localhost Error: Request to https://localhost:8140/puppet-ca/v1 timed out connect operation after 5.001 seconds Wrapped exception: execution expired Error: No more routes to ca Error: Could not run: No more routes to ca

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone commented on PUP-9213

Re: Puppet agent should indicate which HTTP timeout has expired

We also already have this code in https://github.com/puppetlabs/puppet/blob/9bdbdc1f605adad3f1363edacd1dfa91551cc6a9/lib/puppet/http/client.rb#L31-L34 in the client connect method.

Add Comment

[Gene Liverman (JIRA)]

Gene Liverman commented on PUP-9213

Re: Puppet agent should indicate which HTTP timeout has expired

Is there a separate process that’s getting rid of the lib/puppet one Melissa Stone ?

Add Comment

[Melissa Stone (JIRA)]

Melissa Stone commented on PUP-9213

Re: Puppet agent should indicate which HTTP timeout has expired

We're re-doing the http request implementation in Puppet. You can check on the progress we're making in https://tickets.puppetlabs.com/browse/PUP-8550 We're switching everything over piecemeal, so it'll be a gradual change. Once everything is cut over, the plan is to delete all of the old implementations.

Add Comment

[Gene Liverman (JIRA)]

Gene Liverman commented on PUP-9213

Re: Puppet agent should indicate which HTTP timeout has expired

Awesome thanks.

Add Comment