Jira (PUP-9566) Allow to send extra headers when requesting a catalog compilation

31 views
Skip to first unread message

David Moreno García (JIRA)

unread,
Mar 18, 2019, 11:46:03 AM3/18/19
to puppe...@googlegroups.com
David Moreno García created an issue
 
Puppet / New Feature PUP-9566
Allow to send extra headers when requesting a catalog compilation
Issue Type: New Feature New Feature
Assignee: David Moreno García
Created: 2019/03/18 8:45 AM
Priority: Normal Normal
Reporter: David Moreno García

Pointing a set of Puppet Agents to different Puppet Servers is only possible by changing the "server" or "port" values in the puppet.conf. In this new approach, we are trying to deploy our servers in a Kubernetes cluster and point all the agents to the same server and port (the cluster's Ingress). For this to work, we need a mechanism to indicate where the traffic should be redirected. The easiest way to do this is by setting a header in the Puppet Agent's request.

This change will allow to set extra headers to be send every time Puppet Agent is executed on the node. Our set up is just a use case for this new feature but the community could also benefit form it.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

David Moreno García (JIRA)

unread,
Mar 18, 2019, 12:26:03 PM3/18/19
to puppe...@googlegroups.com

Jorie Tappa (JIRA)

unread,
Mar 18, 2019, 12:40:03 PM3/18/19
to puppe...@googlegroups.com

Jacob Helwig (JIRA)

unread,
Mar 20, 2019, 5:23:02 PM3/20/19
to puppe...@googlegroups.com
Jacob Helwig commented on New Feature PUP-9566
 
Re: Allow to send extra headers when requesting a catalog compilation

David Moreno García I'm trying to fully understand the use-case here, as the PR would also add the header(s) to HTTP(S) requests to non-Puppet servers, which seems like it might be troublesome.

Is there a reason that splitting out what used to be separate Puppet servers into their own services in Kubernetes wouldn't work? This would give them each their own unique ingress port.

You mentioned sending all the agents to the same "server" (Kubernetes service) and trying to redirect to individual instances in the pod, but I'm a bit confused why the requests would need to be redirected to specific instances within the pod, when all instances within a pod are supposed to be identical. I'm not sure what I'm missing about the setup you're describing.

David Moreno García (JIRA)

unread,
Mar 20, 2019, 5:32:03 PM3/20/19
to puppe...@googlegroups.com

David Moreno García (JIRA)

unread,
Mar 20, 2019, 5:32:03 PM3/20/19
to puppe...@googlegroups.com

David Moreno García (JIRA)

unread,
Mar 20, 2019, 5:33:02 PM3/20/19
to puppe...@googlegroups.com

David Moreno García (JIRA)

unread,
Mar 20, 2019, 5:33:03 PM3/20/19
to puppe...@googlegroups.com
David Moreno García commented on New Feature PUP-9566
 
Re: Allow to send extra headers when requesting a catalog compilation

The goal here is to have a single k8s cluster with a namespace for each of our top level hostgroups. Those namespaces will have several Puppet Servers to attend requests from nodes of that top level hostgroup. To loadbalance the traffic to those pods we have a k8s service and in front of all these namespaces with have Ingres.

What we are trying to do is to have a single entry point to the cluster (single server and single port) for all our agents independently of their top level hostgroup. The header will be use in the ingress controller to decide to which service/namespace to send the request.

I attached a diagram to show it a bit more clearly. Let me know if that helps.

Charlie Sharpsteen (JIRA)

unread,
Mar 20, 2019, 6:08:02 PM3/20/19
to puppe...@googlegroups.com

Is the Ingress service able to make decisions based on attributes in the client's TLS certificate? If so, then Puppet's trusted facts might be a way to accomplish this.

David Moreno García (JIRA)

unread,
Mar 20, 2019, 6:17:02 PM3/20/19
to puppe...@googlegroups.com

Charlie Sharpsteen Correct me if I'm mistaken but if using trusted facts, changing a node to a different hostgroup would require to regenerate the certificate, right? We have an external CA. Headers would be a much easier approach.

Charlie Sharpsteen (JIRA)

unread,
Mar 20, 2019, 6:52:02 PM3/20/19
to puppe...@googlegroups.com

That is correct, changing trusted data would require the certificate to be re-issued. If that is a thing that happens frequently, then headers would be a better approach.

Nacho Barrientos (JIRA)

unread,
Mar 21, 2019, 3:04:05 AM3/21/19
to puppe...@googlegroups.com

I agree with David that in our case using headers is much more lightweight in terms of operational cost and maintenance. As he mentioned, we're planning on using an agent-generated header to give a "hint" to the load balancing layer and then verify on the backend that the node is actually authorised to make such request doing an RPC to a trusted source of information.

Apart from the issue when changing the hostgroup, persuading the managers of the external CA to sign CSRs containing the extra header (that must be validated) might be a PITA for us.

 

Rob Braden (JIRA)

unread,
May 1, 2019, 5:05:02 PM5/1/19
to puppe...@googlegroups.com
Rob Braden commented on New Feature PUP-9566

David Moreno García We're going to take a step back and work with our product management team to review the wider use cases of running puppet infrastructure on kubernetes and possible improvements to make that a better experience.

In that context, this is not going to be a short term priority but any input on use cases and problems is very welcome.

Nacho Barrientos (JIRA)

unread,
May 2, 2019, 3:08:02 AM5/2/19
to puppe...@googlegroups.com

Hi Rob,

Thanks for your input. Does that mean that you're not going to consider the proposed patch? If so, please let us know as soon as possible as in that case unfortunately we'll have to fork.

 

Thanks.

Jorie Tappa (JIRA)

unread,
May 21, 2019, 12:07:04 PM5/21/19
to puppe...@googlegroups.com
Jorie Tappa commented on New Feature PUP-9566

Hi Nacho Barrientos, this does mean for now we won't be accepting the patch. I'll close the PR and this ticket, and in the case that this gets prioritized and scoped out at some point, we can always reopen the ticket.

Rob Braden (JIRA)

unread,
Jun 5, 2019, 7:37:03 PM6/5/19
to puppe...@googlegroups.com
Rob Braden commented on New Feature PUP-9566

Nacho Barrientos David Moreno García - I think we may want to get some additional context regarding your use case. The primary concern right now is that this adds headers to all requests made by Puppet, not just requests to Puppet servers. If we do adopt this patch we'll want to be careful to document the behavior, as there is some potential for information leakage if users don't understand the behavior. It seems like it would be possible to configure puppet to only add the custom header when connecting to a defined set of infrastructure, but the way the http client code is currently implemented, it would be messy and complex.

We do have some improvements planned that will help, but that doesn't address your immediate needs.

The option exists now to change the User Agent string, does that help with your use case? If not, can you provide any other specifics that may help us understand what you're trying to accomplish?

Nacho Barrientos (JIRA)

unread,
Jun 6, 2019, 5:36:03 AM6/6/19
to puppe...@googlegroups.com

Hi Rob,

Thanks for coming back to us.

In our case actually, as explained by David, we only need to add custom headers to the requests sent by the agents when they chat to the Puppet servers. It'd be perhaps interesting to configure the HTTP client to restrict what type of requests will be modified (by path, for instance) but indeed it's not worth to spend time on this due to how the client is implemented. This is totally an agent-oriented change proposal. In our case our agents only talk to the Puppet servers so to make use of this new feature the only thing that we had to do configuration-wise was to add the new setting to the [agent] section of the puppet.conf available on the agents and that was it. Puppet servers were left untouched as we don't need them to send any extra headers to any endpoint.

I personally don't see a risk if the default behavior is to not to send anything extra so nothing changes, that indeed combined with a good description in the documentation. Obviously administrators could send anything they wanted but that's their choice. On top of that, there's nothing harmful that you could naively do (like when using a switch), users must explicitly configure what data to send in those arbitrary headers.

Overloading the User-Agent could be a solution but I think it's far from being elegant. Actually we're already using that header for something else and adding non-standard data there could be painful as it'd require doing some post-processing on our monitoring pipelines to remove undesired stuff.

We cannot provide much more information than what has been described in previous posts, not because we can't share more details but because there isn't much more really. We want to send metadata from the agent side that will be consumed by the ingress layer to make routing decisions. As Puppet talks HTTP we think that HTTP headers is a good fit for this. Using extended attributes in the client's certificate is unfortunately not an option for us at the moment. This new feature was born in the context of deploying our masters on Kubernetes clusters, however it could be useful in many more situations outside of the containers world.

Josh Cooper (JIRA)

unread,
Jun 11, 2019, 3:11:03 PM6/11/19
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9566

One of our main concerns is people assuming puppet uses HTTPS to connect to puppetservers, and thinking it's ok to add sensitive headers, such as session ids. However, if puppet sends headers everywhere, then it could leak sensitive info when retrieving file content from insecure HTTP file servers. I think restricting the extra headers to puppet-y requests is possible, it's just unfortunately not as clean as it could be due to the various ways the agent makes http requests.

Ethan Brown (JIRA)

unread,
Jun 17, 2019, 3:31:04 PM6/17/19
to puppe...@googlegroups.com
Ethan Brown commented on New Feature PUP-9566

Thanks for the comments Josh Cooper

What other HTTP requests does Puppet make outside of requests to Puppetserver and HTTP backed file resources? Would hostname matching be a good mechanism for restricting headers for specific use cases?

I'm thinking about something like csr_attributes - i.e. a separate YAML map of host -> array of additional headers. That's probably more complexity than we would want to add for this and it may become difficult to keep the server hostname in puppet.conf synchronized with a new sidecar file (rather than just using the servers hostname automatically to restrict) - but trying to come up with something that isn't so hardcoded around the identity of puppetserver.

Josh Cooper (JIRA)

unread,
Aug 13, 2019, 8:08:05 PM8/13/19
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Unassigned
 
Change By: Josh Cooper
Assignee: David Moreno García

Josh Cooper (JIRA)

unread,
Sep 24, 2019, 7:54:03 PM9/24/19
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9566
 
Re: Allow to send extra headers when requesting a catalog compilation

We are working on improvements to the agent's HTTP code, and it will be fairly straightforward to add additional HTTP headers when the agent is talking to the puppetserver service.

Nacho Barrientos (JIRA)

unread,
Sep 25, 2019, 3:00:06 AM9/25/19
to puppe...@googlegroups.com

Hi,

For info we've had our proposed patch in production for a while now. We'll see if we can de-fork in the future, depending on how you implement the feature. Thanks!

Rob Braden (JIRA)

unread,
Feb 3, 2020, 1:02:04 PM2/3/20
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Feb 10, 2020, 12:22:04 PM2/10/20
to puppe...@googlegroups.com

Melissa Stone (JIRA)

unread,
Feb 14, 2020, 12:07:06 PM2/14/20
to puppe...@googlegroups.com

Melissa Stone (JIRA)

unread,
Feb 14, 2020, 12:07:06 PM2/14/20
to puppe...@googlegroups.com
Melissa Stone updated an issue
Change By: Melissa Stone
Sprint: Coremunity Hopper Platform Core KANBAN

Melissa Stone (JIRA)

unread,
Feb 25, 2020, 7:07:05 PM2/25/20
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Feb 28, 2020, 10:18:03 PM2/28/20
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9566

Passed CI in 4092d14208bbcb1bba7969266813f0ac6a46185a

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Feb 29, 2020, 12:10:57 AM2/29/20
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9566

This change has been merged, but won't take effect until PUP-10260 is fully implemented to use the new http client. For example:

$ bx puppet agent -t --http_debug --http_extra_headers foo:bar
opening connection to undying-naming.delivery.puppetlabs.net:8140...
opened
starting SSL for undying-naming.delivery.puppetlabs.net:8140...
SSL established
<- "GET /puppet/v3/node/whatcom.vpn.puppet.net?environment=production&configured_environment=production&transaction_uuid=de3d8b65-1b23-48ef-a973-d9497c530c26 HTTP/1.1\r\nX-Puppet-Version: 6.14.0\r\nUser-Agent: Puppet/6.14.0 Ruby/2.3.8-p459 (x86_64-darwin18)\r\nAccept: application/json, application/x-msgpack, text/pson\r\nFoo: bar\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: undying-naming.delivery.puppetlabs.net:8140\r\n\r\n"

Josh Cooper (Jira)

unread,
Mar 2, 2020, 5:32:03 PM3/2/20
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9566

Passed CI in 031a0fecefc4e3d8962a7799d6b7e836b9e917e2

Melissa Stone (Jira)

unread,
Mar 2, 2020, 5:59:03 PM3/2/20
to puppe...@googlegroups.com
Melissa Stone updated an issue
Change By: Melissa Stone
Release Notes: Enhancement
Release Notes Summary: Users can now define custom headers to send with http requests to puppet infrastructure. Use the `:http_extra_headers` setting to define these. They should be a comma separated string of `key:value` pairs.
Reply all
Reply to author
Forward
0 new messages