Jira (PUP-9990) no_proxy config setting is ignored by puppet agent

0 views
Skip to first unread message

Ben Roberts (JIRA)

unread,
Aug 29, 2019, 9:41:04 AM8/29/19
to puppe...@googlegroups.com
Ben Roberts created an issue
 
Puppet / Bug PUP-9990
no_proxy config setting is ignored by puppet agent
Issue Type: Bug Bug
Affects Versions: PUP 6.8.0
Assignee: Unassigned
Created: 2019/08/29 6:40 AM
Labels: proxy
Priority: Normal Normal
Reporter: Ben Roberts

Puppet Version: 6.8.0
Puppet Server Version: 6.5.0
OS Name/Version: CentOS 6/7

I'm seeing puppet-agent fail to respect the no_proxy setting when defined in the config file. Having checked the diff that merged this support into 6.8.0, I've added some debug logging to identify the code path being taken, and it correctly determines a proxy should be bypassed, but then tries to use the proxy to connect to the puppetserver anyway, which is blocked by local policy

Desired Behavior:

Puppet agent should respect the no_proxy setting in puppet.conf

Actual Behavior:

puppet agent --test --server_list master1 --noop
# Lots of debug output omitted
Debug: LOCAL Using settings no_proxy 10.0.0.0/8,example.com,*.example.com,local,*.local,127.0.0.1,localhost,master1,master2
Debug: LOCAL not using proxy because destination (?-mix:^master1$) matched a no_proxy entry for master1
Debug: LOCAL not using proxy because no_proxy? returned True
Debug: Starting connection for https://master1:8140
Debug: Unable to connect to server from server_list setting: 403 "Forbidden"
Error: Could not run Puppet configuration client: Could not select a functional puppet master from server_list: 'master1'
# 403 Forbidden is coming from local webproxy

(hostnames and actual value of no_proxy obfuscated. Lines containing LOCAL are local modifications to the puppet code to expose the behaviour).

I only see this happen when the no_proxy setting is being read from puppet.conf. If no_proxy is set as env var, it properly bypasses the proxy for the unqualified hostname which matches the no_proxy value.

It feels like puppet agent is not using the proxy settings from puppet.conf at all. It does all the work to decide whether it should and that answer isn't respected by the thing that actually makes the requests to the puppetserver

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
Sep 3, 2019, 1:10:03 PM9/3/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 3, 2019, 1:32:03 PM9/3/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9990
 
Re: no_proxy config setting is ignored by puppet agent

I'm not able to reproduce. If I set the proxy host to a bogus value, then the connection will only succeed if the proxy matches the `no_proxy` list:

$ bundle exec puppet config print no_proxy --section agent
10.0.0.0/8,example.com,*.example.com,local,*.local,127.0.0.1,localhost,master1,master2,g327wjgy9sjg0ut.delivery.puppetlabs.net
$ bundle exec puppet agent -t --server_list g327wjgy9sjg0ut.delivery.puppetlabs.net --http_proxy_host doesntexist.example.com
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for XXX
Info: Applying configuration version '1567531578'
Notice: Applied catalog in 0.01 seconds

But if I remove my server from the no_proxy setting, then the agent tries to use the proxy and fails to connect:

$ bundle exec puppet config set no_proxy '10.0.0.0/8,example.com,*.example.com,local,*.local,127.0.0.1,localhost,master1,master2' --section agent
$ bundle exec puppet config print no_proxy --section agent
10.0.0.0/8,example.com,*.example.com,local,*.local,127.0.0.1,localhost,master1,master2
$ bundle exec puppet agent -t --server_list g327wjgy9sjg0ut.delivery.puppetlabs.net --http_proxy_host doesntexist.example.com
Error: Could not run Puppet configuration client: Could not select a functional puppet master from server_list: 'g327wjgy9sjg0ut.delivery.puppetlabs.net'

I've tried setting no_proxy in the main and agent sections with the same results. What is in your puppet.conf?

Ben Roberts (JIRA)

unread,
Sep 4, 2019, 5:30:03 AM9/4/19
to puppe...@googlegroups.com
Ben Roberts commented on Bug PUP-9990

I have re-tested this. The circumstance under which it is using the proxy unexpectedly is when:

  • puppet.conf contains http_proxy_host, http_proxy_port and no_proxy variables
  • env var http_proxy is set
  • env var no_proxy is not set

The configuration reference docs state that env var http_proxy or HTTP_PROXY overrides the http_proxy_host and http_proxy_port config setting, but makes no note that these env vars also affect the no_proxy setting, which is counter-intuitive. no_proxy is not currently mentioned at all in https://puppet.com/docs/puppet/latest/configuration.html, the only place it's documented is https://puppet.com/docs/puppet/latest/release_notes_puppet.html#puppet-release-notes-x.8.0 which also does not indicate any relation to the http_proxy env var overriding it.

This is perhaps a somewhat contrived example, we would normally have both http_proxy and no_proxy env vars set together, or not at all. The reason I found myself in the state of having one and not the other was while investigating the behaviour where the puppet config and env var for no_proxy had different values, so I unset the environment variable to try and debug, only to find the config file value was then not being used at all.

Can you re-create this, and if so do you agree it's confusion and should be fixed (either via a behaviour change, or a documentation change)?

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 10:04:04 AM9/4/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9990

The no_proxy setting should be added to the configuration reference. /cc Jean Bond

But I can't reproduce the issue as described:

$ cat ~/.puppetlabs/etc/puppet/puppet.conf
[main]
server = puppet.delivery.puppetlabs.net
http_proxy_host = doesntexist.example.com
http_proxy_port=8888
no_proxy=*.delivery.puppetlabs.net
$ env http_proxy=doesntexist.example.com bundle exec puppet agent -t --server_list puppet.delivery.puppetlabs.net
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for XXX
Info: Applying configuration version '1567605099'
Notice: Applied catalog in 0.01 seconds

If I comment out the no_proxy in puppet.conf, then it tries to use the proxy, and fails as expected. Note the -verbose and -http_debug shows it trying to connect to the proxy:

$ env http_proxy=doesntexist.example.com bundle exec puppet agent -t --server_list puppet.delivery.puppetlabs.net --http_debug --verbose
opening connection to doesntexist.example.com:8888...
Error: Could not run Puppet configuration client: Could not select a functional puppet master from server_list: 'puppet.delivery.puppetlabs.net'

Ben Roberts (JIRA)

unread,
Sep 4, 2019, 10:36:03 AM9/4/19
to puppe...@googlegroups.com
Ben Roberts commented on Bug PUP-9990

When set only in puppet.conf, and nothing in the environment, it connect fine.

[root ~]# printenv | grep -i proxy
[root ~]# grep proxy /etc/puppetlabs/puppet/puppet.conf
http_proxy_host = webproxy.example.com
http_proxy_port = 3128
no_proxy = 10.0.0.0/8,local,*.local,127.0.0.1,localhost,*.example.com,example.com,master1,master2
[root ~]# puppet agent --test --server_list master1 --noop --verbose --http_debug     opening connection to master1:8140...
opened
starting SSL for master1:8140...
SSL established
<- "GET /status/v1/simple/master HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Puppet/6.8.0 Ruby/2.5.3-p105 (x86_64-linux)\r\nHost: master1:8140\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Date: Wed, 04 Sep 2019 14:11:21 GMT\r\n"
-> "Content-Type: text/plain;charset=utf-8\r\n"
-> "Vary: Accept-Encoding, User-Agent\r\n"
-> "Content-Length: 7\r\n"
-> "\r\n"
reading 7 bytes...

When http_proxy env var is set, but no_proxy env var is not set, I expect it to still work, because no_proxy is in the config file, and has not been overridden, but it does not. I don't think your test case above covered this combination:

[root ~]# env http_proxy=http://webproxy.example.com:3128 puppet agent --test --server_list master1 --noop --verbose --http_debug
opening connection to webproxy.example.com:3128...
opened
<- "CONNECT master1:8140 HTTP/1.1\r\nHost: master1:8140\r\n\r\n"
-> "HTTP/1.0 403 Forbidden\r\n"
-> "Server: squid\r\n"
-> "Mime-Version: 1.0\r\n"
-> "Date: Wed, 04 Sep 2019 14:09:26 GMT\r\n"
-> "Content-Type: text/html\r\n"
-> "Content-Length: 3094\r\n"
-> "X-Squid-Error: ERR_ACCESS_DENIED 0\r\n"
-> "Vary: Accept-Language\r\n"
-> "Content-Language: en\r\n"
-> "X-Cache: MISS from squid1\r\n"
-> "X-Cache-Lookup: NONE from squid1:3128\r\n"
-> "Via: 1.0 squid1 (squid)\r\n"
-> "Connection: keep-alive\r\n"
-> "\r\n"
Conn close because of connect error 403 "Forbidden"
Error: Could not run Puppet configuration client: Could not select a functional puppet master from server_list: 'master1'

If I set both the http_proxy and no_proxy env vars, it works again:

[root ~]# env http_proxy=http://webproxy.example.com:3128 no_proxy=master1 puppet agent --test --server_list master1 --noop
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin

I didn't expect the http_proxy env var to have any bearing on whether the no_proxy config setting was used.

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 11:26:04 AM9/4/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9990

Thanks Ben Roberts. I can reproduce using puppet-agent 6.8.0 on debian stretch. I'll continue looking.

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 11:30:03 AM9/4/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9990

root@ps5omle63p3320c:~# puppet --version
6.8.0
root@ps5omle63p3320c:~# /opt/puppetlabs/puppet/bin/ruby --version
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
root@ps5omle63p3320c:~# printenv | grep -i proxy
root@ps5omle63p3320c:~# grep proxy /etc/puppetlabs/puppet/puppet.conf
http_proxy_host = webproxy.example.com
http_proxy_port = 3128
no_proxy = *.delivery.puppetlabs.net
root@ps5omle63p3320c:~#  env http_proxy=http://webproxy.example.com:3128 puppet agent --test --server_list k83emhbfl9imev6.delivery.puppetlabs.net
Error: Could not run Puppet configuration client: Could not select a functional puppet master from server_list: 'k83emhbfl9imev6.delivery.puppetlabs.net'
root@ps5omle63p3320c:~# puppet agent --test --server_list k83emhbfl9imev6.delivery.puppetlabs.net
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for ps5omle63p3320c.delivery.puppetlabs.net
Info: Applying configuration version '1567610920'
Notice: Applied catalog in 0.02 seconds

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 12:05:04 PM9/4/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9990

In ruby < 2.5, the Net::HTTP.start method did not automagically read proxy environment variables, but Net::HTTP.new did. This was changed in https://github.com/ruby/ruby/commit/67723c1e4673253b2f4a2c7204ccab9d0daaaeb9#diff-8c2ab8e0fb4f052e1d95ab6334e192c1 so that both start and new observe the env variables by default.

When combined with https://github.com/puppetlabs/puppet/commit/64ec44b6331c978ca3bde47f5ae4bc672949ada8#diff-be7012428c14ff64a2e7e58a45ebecd3 for PUP-4470, ruby will try to use proxy settings from the environment even when puppet doesn't want to use a proxy.

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 12:49:02 PM9/4/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.9.0

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 12:49:03 PM9/4/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 1:00:15 PM9/4/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9990
 
Re: no_proxy config setting is ignored by puppet agent

I wasn't able to reproduce earlier because puppet.delivery.puppetlabs.net resolves to 127.0.0.1 (I'm running puppetserver as a docker container), and Ruby has logic to ignore the proxy env variables when connecting to the loopback: https://github.com/ruby/ruby/blob/f6da4a544760e00d932fea9a586bd869e82ad339/lib/uri/generic.rb#L1533

Josh Cooper (JIRA)

unread,
Sep 4, 2019, 2:04:03 PM9/4/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Platform Core KANBAN
Reply all
Reply to author
Forward
0 new messages