Jira (PUP-10772) fresh puppet agent installation fails when msgpack is enabled on puppetserver

0 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Nov 2, 2020, 2:51:03 PM11/2/20
to puppe...@googlegroups.com
Josh Cooper moved an issue
 
Puppet / Bug PUP-10772
fresh puppet agent installation fails when msgpack is enabled on puppetserver
Change By: Josh Cooper
Key: PA PUP - 3441 10772
Project: Puppet Agent
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Nov 2, 2020, 3:29:03 PM11/2/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Nov 2, 2020, 3:29:04 PM11/2/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Nov 3, 2020, 2:43:03 PM11/3/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10772
 
Re: fresh puppet agent installation fails when msgpack is enabled on puppetserver

Also wanted to mention there is an issue with JRuby sorting arrays PUP-8615. While that ticket is about sorting formats used in pops serialization, it may explain some non-deterministic behavior with how puppetserver is sorting network formats based on their weight?

Josh Cooper (Jira)

unread,
Nov 3, 2020, 2:47:03 PM11/3/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Bug Fix
Release Notes Summary: The agent claimed to accept catalogs serialized using the "rich_data_msgpack" format, but if the "msgpack" gem wasn't installed, then the agent would fail to deserialize the catalog and fail the run. Now the agent only claims to support that format when the "msgpack" gem is installed.

Josh Cooper (Jira)

unread,
Nov 3, 2020, 3:03:03 PM11/3/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Nov 3, 2020, 3:03:03 PM11/3/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Nov 3, 2020, 4:28:02 PM11/3/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10772
 
Re: fresh puppet agent installation fails when msgpack is enabled on puppetserver

Merged to master in https://github.com/puppetlabs/puppet/commit/09405d485a3411597773f2e8a2873fc09f77caba. Note this fixes the agent side of the problem. It would still be good to understand why puppetserver used `rich_data_msgpack` when `Puppet[:preferred_serialization_format]` defaults to json, so `rich_data_json` should have taken precedence.

Tim Meusel (Jira)

unread,
Nov 4, 2020, 10:57:04 AM11/4/20
to puppe...@googlegroups.com
Tim Meusel commented on Bug PUP-10772

So some testing. First let me explain my current setup:

  • agents send all requests to an haproxy
  • haproxy forwards requests with /puppet-ca to one specific puppetserver
  • all other requests go to another group of puppetservers

 

the puppetserver for CA requests has the msgpack gem installed and was restarted after it was installed. I purged it on all other puppetservers and restarted them. Log from an existing/working puppet agent, where I also purged the msgpack gem:

 

Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: catalog supports formats: rich_data_json json rich_data_msgpack pson yaml dot
Debug: Closing connection for https://*:8140
Debug: Creating new connection for https://*:8140
Debug: Starting connection for https://*:8140
Debug: Using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384
Debug: HTTP POST https://*:8140/puppet/v3/catalog/*?environment=production returned 200 OK
Debug: Caching connection for https://*:8140

besides that, the run works fine.

Now I took the same agent, without msgpack installed, and it was talking directly to the CA server with msgpack installed:

Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: catalog supports formats: rich_data_json json rich_data_msgpack pson yaml dot
Debug: Closing connection for https://*:8140
Debug: Creating new connection for https://*:8140
Debug: Starting connection for https://*:8140
Debug: HTTP POST https://*:8140/puppet/v3/catalog/*?environment=production returned 200 OK

agent run works fine as well.

Now I took the same agent, without msgpack installed, and it was talking directly to the CA server with msgpack purged (and puppetserver restarted):

Debug: catalog supports formats: rich_data_json json rich_data_msgpack pson yaml dot
Debug: Closing connection for https://*:8140
Debug: Creating new connection for https://*:8140
Debug: Starting connection for https://*:8140
Debug: Using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256
Debug: HTTP POST https://*:8140/puppet/v3/catalog/*?environment=production returned 200 OK
Debug: Caching connection for https://*:8140

rich_data_msgpack should not be listed here. I repeated the debug run and added --http_debug

<- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 38071\r\n\r\n"
... long text is long...
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.18.0\r\n"
-> "Date: Wed, 04 Nov 2020 15:05:37 GMT\r\n"
-> "Content-Type: application/vnd.puppet.rich+json; charset=utf-8\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Connection: keep-alive\r\n"
-> "X-Puppet-Version: 6.19.1\r\n"
-> "Vary: Accept-Encoding, User-Agent\r\n"
-> "Content-Encoding: gzip\r\n"

  • it looks like puppetserver listens on localhost and nginx terminates the TLS. This was setup so long ago that I forgot it
  • msgpack is still listed as supported, but msgpack gem isn't present on the puppetserver and also not on the agent
  • the run seems to work.

I purged puppet with yum, deleted /opt/puppetlabs/puppet and reinstalled the agent. msgpack isn't installed. It tried to talk to a puppserver where msgpack is installed:

ebug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: catalog supports formats: rich_data_json json rich_data_msgpack pson yaml dot
Debug: Closing connection for https://*:8140
Debug: Creating new connection for https://*:8140
Debug: Starting connection for https://*:8140
Debug: Using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256
Debug: HTTP POST https://*:8140/puppet/v3/catalog/*?environment=production returned 200 OK
Debug: Caching connection for https://*:8140

this passes as well? I will try to get a completley new system in the next days and that with that. Somehow it should be possible to reproduce the initial error I had while opening the ticket.

Josh Cooper (Jira)

unread,
Nov 4, 2020, 6:32:03 PM11/4/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-10772

Passed CI in f664d6a216.

It is possible there is some non-deterministic behavior when puppetserver/JRuby sorts the array of formats based on their weights (as we've seen something similar in PUP-8615). I'm going to mark this agent-side issue as resolved. Tim Meusel please do let us know if you can repro the server side issue, and create a PUP ticket linked to this one. Thanks for your help debugging this!

Josh Cooper (Jira)

unread,
Nov 4, 2020, 6:33:03 PM11/4/20
to puppe...@googlegroups.com

Claire Cadman (Jira)

unread,
Nov 9, 2020, 6:48:03 AM11/9/20
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages