Jira (PUP-10292) external trusted data is executed for file_content requests

[Dylan Ratcliffe (JIRA)]

Dylan Ratcliffe created an issue

Puppet / PUP-10292 external trusted data is executed for file_content requests Issue Type: Bug Affects Versions: PUP 6.12.0 Assignee: Unassigned Created: 2020/02/12 5:52 AM Priority: Major Reporter: Dylan Ratcliffe Puppet Version: 6.12.0 Puppet Server Version: 6.8.0 OS Name/Version: RHEL When using the trusted_external_command setting, the command is executed for every file_content request, not just catalogs. This info can't possibly be required for a file_content request and this will cause a huge performance impact to the puppetserver as this could easily be executed 1000 times for a given node if it's new and needs to do a pluginsync, or if there are a lot of files managed. The error you see below is just from my trusted_external_command and is irrelevant other than show that it does execute it each time   172.17.0.2 - - - 12/Feb/2020:11:52:37 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/chomp.rb?environment=production& HTTP/1.1" 200 1295 172.17.0.2 172.17.0.2 8140 96 2020-02-12 11:52:37,545 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned172.17.0.2 - - - 12/Feb/2020:11:52:37 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/chop.rb?environment=production& HTTP/1.1" 200 1343 172.17.0.2 172.17.0.2 8140 107 2020-02-12 11:52:37,632 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned172.17.0.2 - - - 12/Feb/2020:11:52:37 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/clamp.rb?environment=production& HTTP/1.1" 200 1330 172.17.0.2 172.17.0.2 8140 84 2020-02-12 11:52:37,730 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned172.17.0.2 - - - 12/Feb/2020:11:52:37 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/concat.rb?environment=production& HTTP/1.1" 200 1331 172.17.0.2 172.17.0.2 8140 85 2020-02-12 11:52:37,811 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned172.17.0.2 - - - 12/Feb/2020:11:52:37 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/convert_base.rb?environment=production& HTTP/1.1" 200 1935 172.17.0.2 172.17.0.2 8140 87 2020-02-12 11:52:37,916 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned172.17.0.2 - - - 12/Feb/2020:11:52:37 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/count.rb?environment=production& HTTP/1.1" 200 1373 172.17.0.2 172.17.0.2 8140 86 2020-02-12 11:52:38,013 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned172.17.0.2 - - - 12/Feb/2020:11:52:38 +0000 "GET /puppet/v3/file_content/plugins/puppet/parser/functions/create_ini_settings.rb?environment=production& HTTP/1.1" 200 1711 172.17.0.2 172.17.0.2 8140 94 2020-02-12 11:52:38,107 WARN [c.p.p.ShellUtils] Executed an external process which logged to STDERR: loading of puppet_x/external_data/multiplexer failed, no external data will be returned   Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-10292

Re: external trusted data is executed for file_content requests That's unexpected. If it's happening with file content, then I'd expect it to happen with file metadata(s) too.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-10292

external trusted data is executed for file_content requests

Change By: Josh Cooper Team: Coremunity

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10292

Re: external trusted data is executed for file_content requests It happens for all REST requests, not just file content. So a typical steady-state agent makes a node request, 3 file_metadatas requests (plugins, pluginfacts, locales), catalog and report. For some reason, puppet server executes the command twice for each file_metadatas request, so in all there are 1+3(2)+1+1=9 executions per agent run. Also if the agent makes a file_content request, like the pluginsync case, then there is N more executions. Now that I think about it more, I think this is working as designed, but I understand it's an issue. Puppetserver does not preserve state across requests, so it doesn't know what the external data is for a node across a run. It seems possible to keep an in-memory cache of data based on the client cert (after it's been authenticated). But that is going to need puppet and puppetserver coordination I think. /cc Maggie Dreyer

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Charlie Sharpsteen (Jira)]

Charlie Sharpsteen commented on PUP-10292

Re: external trusted data is executed for file_content requests

As an aside, the execution of the external command desperately needs to be wrapped in a profiler. So we know when it starts adding seconds of latency to each request

Add Comment

[Mike Fröhner]

Mike Fröhner commented on PUP-10292

Re: external trusted data is executed for file_content requests

We noticed this the last days too. The pluginsync is also affected by that.

Add Comment

[Mike Fröhner]

Mike Fröhner commented on PUP-10292

Re: external trusted data is executed for file_content requests

Ah sorry - you already mentioned the pluginsync.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-10292

external trusted data is executed for file_content requests

Change By: Josh Cooper Sprint: Coremunity Hopper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper assigned an issue to Reid Vandewiele

Puppet / PUP-10292 external trusted data is executed for file_content requests

Change By: Josh Cooper Assignee: Reid Vandewiele

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele assigned an issue to Unassigned

Puppet / PUP-10292 external trusted data is executed for file_content requests

Change By: Reid Vandewiele Assignee: Reid Vandewiele

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-10292 external trusted data is executed for file_content requests

Change By: Josh Cooper Fix Version/s: PUP 6.16.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10292

Re: external trusted data is executed for file_content requests Thanks Reid Vandewiele, we'll review the PR and get this in 6.16.0

Add Comment

[Melissa Stone (Jira)]

Melissa Stone updated an issue

Puppet / PUP-10292

external trusted data is executed for file_content requests

Change By: Melissa Stone Sprint: Coremunity Hopper Platform Core KANBAN

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10292

Re: external trusted data is executed for file_content requests Merged to master in https://github.com/puppetlabs/puppet/commit/af9cb4749a7ac8b1f7d629d29bd9906b46aff591

Add Comment

[Melissa Stone (Jira)]

Melissa Stone commented on PUP-10292

Re: external trusted data is executed for file_content requests

This has passed ci as a part of puppet-agent 6.15.0.195.g95e09983

Add Comment

[Melissa Stone (Jira)]

Melissa Stone updated an issue

Puppet / PUP-10292

external trusted data is executed for file_content requests

Change By: Melissa Stone Release Notes: Enhancement Release Notes Summary: The `trusted_external_command` is only run once when external data is requested, rather than every time a request is made.

Add Comment

[Claire Cadman (Jira)]

Claire Cadman updated an issue

Puppet / PUP-10292 external trusted data is executed for file_content requests

Change By: Claire Cadman Labels: doc_reviewed

Add Comment