Jira (PUP-10014) Unvendor deep

Issue Type:	Task
Assignee:	Unassigned
Created:	2019/09/09 11:50 AM
Priority:	Normal
Reporter:	Josh Cooper

Puppet vendors the deep_merge gem version 1.0.1. Puppet also implements deep_merge logic (https://github.com/puppetlabs/puppet/blob/ca796215b95825f22b0ea30f56a27af181eda53a/lib/puppet/parser/scope.rb#L821-L838). And there are other places where we should deeply merging objects.

This ticket is to update calls sites to use the deep_merge gem, and unvendor the gem. It will need to be added as a runtime dependency to .gemspec and ext/project_data.yaml. Also add to puppet-agent as a shared gem (so it's useable by puppet-agent and puppetserver).

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

Josh Cooper (JIRA)

unread,

Sep 9, 2019, 4:21:02 PM9/9/19

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Puppet vendors the deep_merge gem version 1.0.1. Puppet also implements deep_merge logic (https://github.com/puppetlabs/puppet/blob/ca796215b95825f22b0ea30f56a27af181eda53a/lib/puppet/parser/scope.rb#L821-L838). And there are other places where we should deeply merging objects.

This ticket is to update calls sites to use the deep_merge gem, and unvendor the gem. It will need to be added as a runtime dependency to .gemspec and ext/project_data.yaml. Also add to puppet- agent runtime as a shared gem (so it's useable by puppet-agent and puppetserver). < /td>

Josh Cooper (JIRA)

unread,

Sep 13, 2019, 5:29:03 PM9/13/19

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Puppet vendors the deep_merge gem version 1.0.1. Puppet also implements deep_merge logic (https://github.com/puppetlabs/puppet/blob/ ca796215b95825f22b0ea30f56a27af181eda53a 34b208f04f7415f9a99bf6eafd6d101b303b920d /lib/puppet/ parser vendor / scope load_deep_merge .rb# L821-L838 L1 ). And there are other places where we should deeply merging objects.

This ticket is to update calls sites to use the deep_merge gem, and unvendor the gem. It will need to be added as a runtime dependency to .gemspec and ext/project_data.yaml. Also add to puppet-runtime as a shared gem (so it's useable by puppet-agent and puppetserver).

Jorie Tappa (JIRA)

unread,

Sep 23, 2019, 1:21:03 PM9/23/19

to puppe...@googlegroups.com

Jorie Tappa updated an issue

Puppet /

Change By:	Jorie Tappa
Team:	Coremunity

Jorie Tappa (JIRA)

unread,

Sep 23, 2019, 1:22:03 PM9/23/19

to puppe...@googlegroups.com

Jorie Tappa updated an issue

Puppet /

Change By:	Jorie Tappa
Sprint:	Coremunity Grooming

Rob Braden (JIRA)

unread,

Oct 14, 2019, 4:07:03 PM10/14/19

to puppe...@googlegroups.com

Rob Braden updated an issue

Puppet /

Change By:	Rob Braden
Sprint:	Coremunity Grooming Hopper

Jorie Tappa (JIRA)

unread,

Oct 14, 2019, 4:23:04 PM10/14/19

to puppe...@googlegroups.com

Jorie Tappa assigned an issue to Jorie Tappa

Puppet /

Change By:	Jorie Tappa
Assignee:	Jorie Tappa

Jorie Tappa (JIRA)

unread,

Oct 15, 2019, 2:57:03 PM10/15/19

to puppe...@googlegroups.com

Jorie Tappa updated an issue

Puppet /

Change By:	Jorie Tappa

Puppet vendors the deep_merge gem version 1.0.1. Puppet also implements deep_merge logic ( [ https://github.com/puppetlabs/puppet/blob/34b208f04f7415f9a99bf6eafd6d101b303b920d/lib/puppet/vendor/load_deep_merge.rb#L1 ] ). And there are other places where we should deeply merging objects.

This ticket is to update calls sites to use the deep_merge gem, and unvendor the gem. It will need to be added as a runtime dependency to .gemspec and ext/project_data.yaml. Also add to puppet-runtime as a shared gem (so it's useable by puppet-agent and puppetserver).

Need to work backwards -
1. add deep_merge gem to artifactory with kermintor in release-new-new
2. add deep_merge gem component to puppet-runtime - at this point you need to know if it's client-side, server-side, or both

if you're using something that might be used on the server, see ([https://github.com/puppetlabs/puppet-runtime/blob/6d8fb954c44f3d6c79c3756b2372b0dcc2136aca/configs/components/rubygem-semantic_puppet.rb#L17]) - this prefers the {{settings[:puppet_gem_vendor_dir]}} if set, otherwise falls back to {{settings[:gem_home]}}

if only client side, refer to something such as https://github.com/puppetlabs/puppet-runtime/blob/master/configs/components/rubygem-httpclient.rb

3. add component to the different projects in puppet-runtime repo (semantic-puppet is a good example)

4. dockerfile in puppet-agent installs the gem when building the puppet-agent docker image (https://github.com/puppetlabs/puppet-agent/blob/master/docker/puppet-agent-alpine/windows-build/Dockerfile.build#L41)
5. add your gem to test to make sure it gets installed correctly: [https://github.com/puppetlabs/puppet-agent/blob/master/acceptance/tests/validate_vendored_ruby.rb#L154]
6. setting in puppet-runtime for destination directories that controls which directory it gets installed into. ([https://github.com/puppetlabs/puppet-runtime/blob/d5d4ee5835251beef3decfbf4ace790d3da139f0/configs/projects/agent-runtime-6.4.x.rb#L18]) where we want gems to go if shared with server: /opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems
_*Caveat: can't contain native code*_

7. if the compiler or hiera code depends on it, it needs to be added for jruby: [https://github.com/puppetlabs/puppetserver/blob/e87fbaf09112aa7ffa9f31c192d879d12c9562ff/resources/ext/build-scripts/jruby-gem-list.txt]
discover there's already a gem there and cry
Once it has been added everywhere it needs to be, actually remove vendored code from Puppet

Jorie Tappa (JIRA)

unread,

Oct 15, 2019, 3:04:03 PM10/15/19

to puppe...@googlegroups.com

Jorie Tappa updated an issue

Puppet /

Change By:	Jorie Tappa
Sprint:	Coremunity Hopper Platform Core KANBAN

Josh Cooper (JIRA)

unread,

Oct 15, 2019, 5:14:03 PM10/15/19

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Puppet vendors the deep_merge gem version 1.0.1. Puppet also implements deep_merge logic ([https://github.com/puppetlabs/puppet/blob/34b208f04f7415f9a99bf6eafd6d101b303b920d/lib/puppet/vendor/load_deep_merge.rb#L1]). And there are other places where we should deeply merging objects.

This ticket is to update calls sites to use the deep_merge gem, and unvendor the gem. It will need to be added as a runtime dependency to .gemspec and ext/project_data.yaml. Also add To avoid breaking CI, we need to work backwards from the consumers of the puppet -runtime as a shared gem (so it's useable by to puppet -agent and puppetserver). itself:

Need to work backwards -

1. add deep_merge gem to artifactory with kermintor in release-new-new

2. add deep_merge gem component to puppet-runtime - at this point you need to know if it's client-side only , or both client and server -side , and whether the gem is pure ruby or both contains native extensions:

if you're using something adding a pure-ruby gem that might must be used on the server, available everywhere see ([https://github.com/puppetlabs/puppet-runtime/blob/6d8fb954c44f3d6c79c3756b2372b0dcc2136aca/configs/components/rubygem-semantic_puppet.rb#L17]) - this . This prefers the {{settings[:puppet_gem_vendor_dir]}} if set (for example {{/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems}}) , otherwise falls back to {{settings[:gem_home]}} (for example {{/opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/}}).

if only If you're adding client - side gem (eg for a provider) or a gem with native extensions , refer to something such as see https://github.com/puppetlabs/puppet-runtime/blob/master/configs/components/rubygem- httpclient CFPropertyList .rb

3. add component to the different projects in puppet-runtime repo (semantic , see https://github.com/puppetlabs/puppet - runtime/blob/master/configs/projects/_shared-agent-components.rb, https://github.com/puppetlabs/ puppet is a good example) -runtime/blob/master/configs/projects/bolt-runtime.rb, https://github.com/puppetlabs/puppet-runtime/blob/master/configs/projects/pdk-runtime.rb, etc

4. dockerfile in puppet-agent installs the gem when building the puppet-agent docker image (https://github.com/puppetlabs/puppet-agent/blob/master/docker/puppet-agent-alpine/windows-build/Dockerfile.build#L41)

5. add your gem to test to make sure it gets installed correctly: [https://github.com/puppetlabs/puppet-agent/blob/master/acceptance/tests/validate_vendored_ruby.rb#L154]

6.

setting in puppet-runtime for destination directories that controls which directory it gets installed into. ([https://github.com/puppetlabs/puppet-runtime/blob/d5d4ee5835251beef3decfbf4ace790d3da139f0/configs/projects/agent-runtime-6.4.x.rb#L18]) where we want gems to go if shared with server: /opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems
_*Caveat: can If the gem isn 't contain native code*_

7. if the compiler or hiera code depends on it client-only , add it needs to be added for : jruby: [https://github.com/puppetlabs/puppetserver/blob/e87fbaf09112aa7ffa9f31c192d879d12c9562ff/resources/ext/build-scripts/jruby-gem-list.txt]

discover there's already a gem there and cry

7. Once it has been added everywhere it needs to be, actually remove vendored code from Puppet

Josh Cooper (JIRA)

unread,

Oct 15, 2019, 5:15:02 PM10/15/19

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

Change By:	Josh Cooper

Puppet vendors the deep_merge gem version 1.0.1. Puppet also implements deep_merge logic ([https://github.com/puppetlabs/puppet/blob/34b208f04f7415f9a99bf6eafd6d101b303b920d/lib/puppet/vendor/load_deep_merge.rb#L1]). And there are other places where we should deeply merging objects.

This ticket is to update calls sites to use the deep_merge gem, and unvendor the gem. It will need to be added as a runtime dependency to .gemspec and ext/project_data.yaml. To avoid breaking CI, we need to work backwards from the consumers of the puppet gem to puppet itself:

1. add deep_merge gem to artifactory with kermintor in release-new-new

2. add deep_merge gem component to puppet-runtime - at this point you need to know if it's client-side only, or both client and server, and whether the gem is pure ruby or contains native extensions:

if you're adding a pure-ruby gem that must be available everywhere see ([https://github.com/puppetlabs/puppet-runtime/blob/6d8fb954c44f3d6c79c3756b2372b0dcc2136aca/configs/components/rubygem-semantic_puppet.rb#L17]). This prefers {{settings[:puppet_gem_vendor_dir]}} if set ( , for example {{/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems}} ) , otherwise falls back to {{settings[:gem_home]}} ( , for example {{/opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/}} ) .

If you're adding client-side gem (eg for a provider) or a gem with native extensions, see https://github.com/puppetlabs/puppet-runtime/blob/master/configs/components/rubygem-CFPropertyList.rb

3. add component to the different projects in puppet-runtime repo, see https://github.com/puppetlabs/puppet-runtime/blob/master/configs/projects/_shared-agent-components.rb, https://github.com/puppetlabs/puppet-runtime/blob/master/configs/projects/bolt-runtime.rb, https://github.com/puppetlabs/puppet-runtime/blob/master/configs/projects/pdk-runtime.rb, etc

4. dockerfile in puppet-agent installs the gem when building the puppet-agent docker image (https://github.com/puppetlabs/puppet-agent/blob/master/docker/puppet-agent-alpine/windows-build/Dockerfile.build#L41)

5. add your gem to test to make sure it gets installed correctly: [https://github.com/puppetlabs/puppet-agent/blob/master/acceptance/tests/validate_vendored_ruby.rb#L154]

6. If the gem isn't client-only, add it to: jruby: [https://github.com/puppetlabs/puppetserver/blob/e87fbaf09112aa7ffa9f31c192d879d12c9562ff/resources/ext/build-scripts/jruby-gem-list.txt]

7. Once it has been added everywhere it needs to be, actually remove vendored code from Puppet

Josh Cooper (JIRA)

unread,

Oct 15, 2019, 7:25:03 PM10/15/19

to puppe...@googlegroups.com

Josh Cooper commented on

Re: Unvendor deep_merge

For this particular ticket, it turns out we're already including deep_merge in all the places. So it may be as simple as removing the vendored code from puppet, the deep_merge load hook, and updating puppet's gem dependencies.

Kris Bosland (JIRA)

unread,

Oct 29, 2019, 4:56:03 PM10/29/19

to puppe...@googlegroups.com

Kris Bosland commented on

Re: Unvendor deep_merge

Merged to master at a0e7c84.

Josh Cooper (JIRA)

unread,

Oct 29, 2019, 10:35:03 PM10/29/19

to puppe...@googlegroups.com

Josh Cooper commented on

Re: Unvendor deep_merge

Passed CI in a0e7c84236

Josh Cooper (JIRA)

unread,

Oct 29, 2019, 10:36:02 PM10/29/19

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /