Jira (PUP-9316) implement no_proxy in puppet.conf

2 views
Skip to first unread message

Thomas Kishel (JIRA)

unread,
Nov 15, 2018, 5:22:04 PM11/15/18
to puppe...@googlegroups.com
Thomas Kishel created an issue
 
Puppet / Improvement PUP-9316
implement no_proxy in puppet.conf
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 2018/11/15 2:21 PM
Priority: Normal Normal
Reporter: Thomas Kishel

Puppet implements the following settings in puppet.conf:

 http_proxy_host
 
 http_proxy_password
 
 http_proxy_port
 
 http_proxy_user

Puppet also reads those same settings from ENV.
But it only reads the associated no_proxy setting from ENV.

This requires configuring puppet proxy settings in two disparate locations.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Eric Thompson (JIRA)

unread,
Nov 15, 2018, 6:50:02 PM11/15/18
to puppe...@googlegroups.com
Eric Thompson updated an issue
Change By: Eric Thompson
Team: Coremunity

Josh Cooper (JIRA)

unread,
Aug 7, 2019, 7:46:03 PM8/7/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.8.0
Fix Version/s: PUP 6.4.4
Fix Version/s: PUP 6.0.11
Fix Version/s: PUP 5.5.17

Josh Cooper (JIRA)

unread,
Aug 7, 2019, 7:46:04 PM8/7/19
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (JIRA)

unread,
Aug 7, 2019, 7:48:02 PM8/7/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes Summary: Allows `no_proxy` to be specified as a puppet setting which is consistent with other http_proxy_* puppet settings. The NO_PROXY environment variable takes precedence over the no_proxy puppet setting.
Release Notes: Enhancement

Josh Cooper (JIRA)

unread,
Aug 7, 2019, 7:48:03 PM8/7/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Platform Core KANBAN

Josh Cooper (JIRA)

unread,
Aug 12, 2019, 3:58:04 PM8/12/19
to puppe...@googlegroups.com

Alexandru Popa (JIRA)

unread,
Aug 14, 2019, 9:22:02 AM8/14/19
to puppe...@googlegroups.com

Josh Cooper can you please add release notes if needed. Also I think this ticket can be closed.

Thanks,

Alex

Josh Cooper (JIRA)

unread,
Aug 14, 2019, 11:14:03 AM8/14/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-9316

Passed CI in c956ad95fcd

Jean Bond (JIRA)

unread,
Aug 19, 2019, 7:19:02 PM8/19/19
to puppe...@googlegroups.com
Jean Bond updated an issue
Change By: Jean Bond
Labels: resolved-issue-added

Josh Cooper (JIRA)

unread,
Sep 16, 2019, 2:59:04 PM9/16/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.0.11

Elof Ofel (JIRA)

unread,
Oct 24, 2019, 8:20:04 AM10/24/19
to puppe...@googlegroups.com
Elof Ofel commented on Improvement PUP-9316

When I upgraded from 5.5.16 to 5.5.17, puppet stopped working on all my Debian 9 machines. (haven't tried on any others yet)

I think it has to do with this change.

# env | grep -i proxy
 
no_proxy=localhost,127.0.0.1 
HTTP_PROXY=http://10.x.x.x:3128/
 
FTP_PROXY=http://10.x.x.x:3128/
 
https_proxy=http://10.x.x.x:3128/
 
http_proxy=http://10.x.x.x:3128/
 
HTTPS_PROXY=http://10.x.x.x:3128/
 
ftp_proxy=http://10.x.x.x:3128/
 
 
 
# puppet agent -t --noop
 
Warning: Unable to fetch my node definition, but the agent run will continue:
 
Warning: 403 "Forbidden"
 
Info: Retrieving pluginfacts
 
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': 403 "Forbidden"
 
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: 403 "Forbidden"
 
Info: Retrieving plugin
 
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': 403 "Forbidden"
 
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: 403 "Forbidden"
 
Info: Loading facts
 
Error: Could not retrieve catalog from remote server: 403 "Forbidden"
 
Warning: Not using cache on failed catalog
 
Error: Could not retrieve catalog; skipping run
 
Error: Could not send report: 403 "Forbidden"

 

It seems like the new code in 5.5.17 is now using the system environment variables which it didn't before:

My puppet.conf contain no proxy configuration at all.
 
My system environment have the above proxy variables.
 
I run:
 
# unset HTTP_PROXY
 
# unset HTTPS_PROXY
 
# unset https_proxy
 
# unset http_proxy
 
# export no_proxy=localhost,127.0.0.1
 
# puppet agent -t --noop
 
 
 
Now it works!

 

Problem!

 

I don't want to use the proxy for the puppet traffic, but I do want to have the system environment proxy variables.

If I set the proxy-variables to the empty string in puppet.conf I assume this has no effect since ENV has preceedence, so that is not a solution either.

 

Am I supposed to add my puppetserver to the no_proxy ENV (no_proxy=localhost,127.0.0.1,<puppetserver>)?

(that will unfortunetly break some modules that fetch files from a webserver running on port 80 on the puppetserver - they are fetched via the proxy, so I can't add it to no_proxy.)

Thomas Kishel (JIRA)

unread,
Oct 24, 2019, 4:27:03 PM10/24/19
to puppe...@googlegroups.com
Thomas Kishel commented on Improvement PUP-9316

It seems like the new code in 5.5.17 is now using the system environment variables which it didn't before.

The http_proxy.rb code in 5.5.16 deferred to the system environment proxy variables before this change:

https://puppet.com/docs/puppet/latest/configuration.html#httpproxyhost

The HTTP proxy host to use for outgoing connections.
Note: You may need to use a FQDN for the server hostname when using a proxy.
Environment variable http_proxy or HTTP_PROXY will override this value

https://github.com/puppetlabs/puppet/blob/5.5.15/lib/puppet/util/http_proxy.rb#L82

The same is true after this change:

https://github.com/puppetlabs/puppet/blob/5.5.17/lib/puppet/util/http_proxy.rb#L82

 

With this change, Puppet.settings[:no_proxy] was instantiated (but still defers to the system environment no_proxy variable):

https://puppet.com/docs/puppet/latest/configuration.html#noproxy

List of domain names that should not go through http_proxy_host.
Environment variable no_proxy or NO_PROXY will override this value.

https://github.com/puppetlabs/puppet/blob/5.5.17/lib/puppet/util/http_proxy.rb#L35
https://github.com/puppetlabs/puppet/blob/5.5.17/lib/puppet/util/http_proxy.rb#L134

 

Before this change, only the system environment no_proxy variable was used:

https://github.com/puppetlabs/puppet/blob/5.5.15/lib/puppet/util/http_proxy.rb#L36

 

Given your configuration where system environment proxy variables are set, I would expect ...

In 5.5.16:

  • Line 007 to call self.no_proxy?()
    Line 033 in self.no_proxy?() to set its no_proxy_env variable to "localhost,127.0.0.1" from ENV
    self.no_proxy? to return false unless the dest parameter is in the no_proxy variable
    Line 010 to setup the proxy if self.no_proxy?() returned false

In 5.5.17:

  • Line 007 to call self.no_proxy?()
    Line 036 in self.no_proxy?() to call self.no_proxy() and set its no_proxy variable to "localhost,127.0.0.1" from ENV
    self.no_proxy? to return false unless the dest parameter is in the no_proxy_env variable
    Line 010 to setup the proxy if self.no_proxy?() returned false

Could you run "puppet agent -t --noop --debug --http_debug" and reply with just the output related this error?

Josh Cooper (JIRA)

unread,
Oct 24, 2019, 6:27:05 PM10/24/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-9316

Elof Ofel In 5.5.16 and prior, puppet claimed to observe the no_proxy environment variable, but it didn't actually do that, because the connection factory called Puppet::Util::HttpProxy.no_proxy?(Site) passing a site instance instead of a String or URI. The reason this wasn't noticed before is because the no_proxy? silently rescues URI::InvalidURIError and returns false.

This was fixed in puppet 5.5.17 in commit cb4a9b7da148f97ed9ce02f859ad6a90115c0079.

As a result, puppet now correctly observes the no_proxy environment variable. Since your no_proxy environment variable doesn't include your puppetserver, puppet is trying to use the proxy. You'll want to add your puppetserver FQDN to the no_proxy environment variable.

However, there are 2 other issues to be aware of:

PUP-9990 - Puppet could end up using a proxy even though it matched the no_proxy exception list. This was due to Puppet incorrectly creating the Net::HTTP object when running with ruby 2 and up. It was fixed in 54a3e6bfe1f907bbe5505b39f4d9cdfec2be428f, but needs to be backported to 5.5.x. I've filed that as PUP-10112.

PUP-10106 - Puppet and ruby interpret the no_proxy environment variable differently when specifying a domain. Puppet accepts *.example.com and .example.com, whereas ruby only accepts example.com.

Both of these issues can be avoided by specifying the FQDN of the puppetserver in the no_proxy environment variable.

Elof Ofel (JIRA)

unread,
Oct 25, 2019, 5:44:03 AM10/25/19
to puppe...@googlegroups.com
Elof Ofel commented on Improvement PUP-9316 

Actually, the problem here was that puppet 5.5.16 and prior versions did not work in the manner the manual stated.
 
The manual for http_proxy_host says "Environment variable http_proxy or HTTP_PROXY will override this value" but apparently it never did. But now in 5.5.17 this is fixed.
 
 
 
So this discussion doesn't really have to do explicitly with the 'no_proxy' variable, but rather with the rest of the environment variable handling that was changed in the commit.
 
 
 
In my case this fix became a major problem even though it was just a minor version upgrade.
 
 
 
 
 
Some logs just because you asked for them:
 
 
 
I have the new version 5.5.17:
 
#  dpkg -l | grep puppet
 
ii  puppet-agent                          5.5.17-1stretch                   amd64
 
ii  puppet5-release                       5.0.0-7stretch                    all
 
 
 
I don't even have any 'no_proxy' ENV (but I do have a http_proxy):
 
# env | grep -i proxy
 
In puppet.conf I specify no proxy.
 
# grep -i proxy /etc/puppetlabs/puppet/puppet.conf
 
<no match>
 
 
 
According to the manual, puppet should use the proxy, but in 5.5.16 and earlier it didn't.
 
But now in 5.5.17 it does.
 
Debug output:
 
# puppet agent -t --noop --debug --http_debug
 
...
 
...
 
opening connection to 10.30.8.5:3128...
 
opened
 
<- "CONNECT mypuppetserver.foo.bar.se:8140 HTTP/1.1\r\nHost: mypuppetserver.foo.bar.se:8140\r\n\r\n"
 
-> "HTTP/1.1 403 Forbidden\r\n"
 
-> "Server: squid\r\n"
 
-> "Mime-Version: 1.0\r\n"
 
-> "Date: Fri, 25 Oct 2019 08:54:52 GMT\r\n"
 
-> "Content-Type: text/html;charset=utf-8\r\n"
 
-> "Content-Length: 3479\r\n"
 
-> "X-Squid-Error: ERR_ACCESS_DENIED 0\r\n"
 
-> "Vary: Accept-Language\r\n"
 
-> "Content-Language: en\r\n"
 
-> "X-Cache: MISS from proxy4\r\n"
 
-> "X-Cache-Lookup: NONE from myproxy4:3128\r\n"
 
-> "Connection: keep-alive\r\n"
 
-> "\r\n"
 
Conn close because of connect error 403 "Forbidden"
 
Warning: Unable to fetch my node definition, but the agent run will continue:
 
Warning: 403 "Forbidden"
 
...
 
...
 
 
 
So the traffic is now sent to my proxy (which it has never done before).
 
The proxy says ERR_ACCESS_DENIED, which is correct since the puppet traffic has never passed through the proxy before.
 
 
 
Just to verify, I unset the proxy and try again:
 
# unset HTTP_PROXY
 
# unset HTTPS_PROXY
 
# unset https_proxy
 
# unset http_proxy
 
# puppet agent -t --noop --debug --http_debug
 
 
 
Now 5.5.17 works fine using a direct connection (GET, not CONNECT to a proxy).
 
 
 
 
 
Just for comparison, I downgrade puppet back to 5.5.16:
 
#  dpkg -l | grep puppet
 
ii  puppet-agent                          5.5.16-1stretch                   amd64
 
ii  puppet5-release                       5.0.0-7stretch                    all
 
 
 
I restore the ENV back to normal:
 
# env | grep -i proxy
 
I run puppet again:
 
# puppet agent -t --noop --debug --http_debug
 
It works. It uses direct GET, not CONNECT to a proxy.
 
 
 
So puppet 5.5.16 (and prior versions) has always been accessing the puppetserver directly,
 
disregarding the ENV proxy settings even though the manual said they would be used.
 
 
 
All in all, the new behavior is the correct one, 'cause now puppet honor the ENV as it always should have.
 
In my case this introduced a problem. To solve this, I will either allow puppet via the proxy or add my puppetserver IP and fqdn to the 'no_proxy' ENV variable.

 

Austin Boyd (Jira)

unread,
Mar 11, 2020, 4:43:03 AM3/11/20
to puppe...@googlegroups.com
Austin Boyd updated an issue
Change By: Austin Boyd
Zendesk Ticket Count: 1
Zendesk Ticket IDs: 38358
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Austin Boyd (Jira)

unread,
Mar 11, 2020, 4:43:04 AM3/11/20
to puppe...@googlegroups.com
Austin Boyd updated an issue
Change By: Austin Boyd
Labels: jira_escalated resolved-issue-added
Reply all
Reply to author
Forward
0 new messages