Bug#1050340: puppetserver: incompatibility with system hiera-eyaml

[Cyril Brulebois]

Package: puppetserver
Version: 7.9.5-2
Severity: important

Hi,

I totally lost hiera eyaml support while upgrading from bullseye to
bookworm. Neither the old hiera configuration file or the new one
worked. Given the upstream upgrade path, I totally understand that
there's little puppet packagers can do to ease the pain…

https://www.puppet.com/docs/puppet/7/hiera_migrate#hiera_migrate

Just in case it helps others, here's what I ended up using, which
lets me use nodes/*.eyaml files:

,---[ /etc/puppet/hiera.yaml ]---
| ---
| # Hiera 5 Global configuration file
|
| version: 5
|
| defaults:
| data_hash: yaml_data
| datadir: code/hiera
|
| hierarchy:
| - name: "Per-node data"
| paths:
| - "nodes/%{trusted.certname}.yaml"
| - "common.yaml"
|
| - name: "Per-node data (encrypted)"
| path: "nodes/%{trusted.certname}.eyaml"
| lookup_key: eyaml_lookup_key
| options:
| pkcs7_private_key: /var/lib/puppet/keys/private_key.pkcs7.pem
| pkcs7_public_key: /var/lib/puppet/keys/public_key.pkcs7.pem
`---

But now I'm facing a bigger issue, which is that any use of hiera
triggers this error, and dozens of log lines:

Lookup using eyaml lookup_key function is only supported when the hiera_eyaml library is present

I'm attaching a log excerpt with a trace.

Since I didn't want to keep a broken puppet {master,server} for too
long, I bit the bullet and tried installing the gem, which worked around
the immediate problem:

puppetserver gem install hiera-eyaml

But it'd be great if that problem could be debugged and a proper
solution found, only using Debian packages… This is a rather small
setup, changes don't happen very often, there's nothing really
mission-critical, so I can happily assist debugging/running tests
if instructed.

Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[Antoine Beaupré]

On 2023-08-23 13:55:21, Cyril Brulebois wrote:
> Package: puppetserver
> Version: 7.9.5-2
> Severity: important
>
> Hi,
>
> I totally lost hiera eyaml support while upgrading from bullseye to
> bookworm. Neither the old hiera configuration file or the new one
> worked. Given the upstream upgrade path, I totally understand that
> there's little puppet packagers can do to ease the pain…
>
> https://www.puppet.com/docs/puppet/7/hiera_migrate#hiera_migrate
>
> Just in case it helps others, here's what I ended up using, which
> lets me use nodes/*.eyaml files:
>

[...]

> But now I'm facing a bigger issue, which is that any use of hiera
> triggers this error, and dozens of log lines:
>
> Lookup using eyaml lookup_key function is only supported when the hiera_eyaml library is present
>
> I'm attaching a log excerpt with a trace.
>
>
> Since I didn't want to keep a broken puppet {master,server} for too
> long, I bit the bullet and tried installing the gem, which worked around
> the immediate problem:
>
> puppetserver gem install hiera-eyaml
>
> But it'd be great if that problem could be debugged and a proper
> solution found, only using Debian packages… This is a rather small
> setup, changes don't happen very often, there's nothing really
> mission-critical, so I can happily assist debugging/running tests
> if instructed.

That's a great bug report, thanks!

Do you think this is a version problem, in other words would it be
sufficient to upgrade the hiera-eyaml package to 3.4.0?

--
feature, n: a documented bug | bug, n: an undocumented feature
- Mario S F Ferreira <li...@FreeBSD.org>

[Cyril Brulebois]

Antoine Beaupré <ana...@debian.org> (2023-08-23):

> Do you think this is a version problem, in other words would it be
> sufficient to upgrade the hiera-eyaml package to 3.4.0?

That thought crossed my mind, but seeing the same version everywhere in
rmadison's output, I considered I would stop there. Gut feelings:
- maybe there's a version mismatch;
- maybe there's some search path problem (e.g. only/mostly looking into
some vendorized libraries, but then it really should have vendorized
that particular one).

I'm happy to test a newer version once you have a tentative package. I'm
not sure I can reasonably commit to trying and preparing the update
myself.

[Antoine Beaupré]

On 2023-08-23 17:55:19, Cyril Brulebois wrote:
> Antoine Beaupré <ana...@debian.org> (2023-08-23):
>> Do you think this is a version problem, in other words would it be
>> sufficient to upgrade the hiera-eyaml package to 3.4.0?
>
> That thought crossed my mind, but seeing the same version everywhere in
> rmadison's output, I considered I would stop there. Gut feelings:
> - maybe there's a version mismatch;
> - maybe there's some search path problem (e.g. only/mostly looking into
> some vendorized libraries, but then it really should have vendorized
> that particular one).

Oh i meant the upstream version, the package in Debian is obviously out
of date here. :)

> I'm happy to test a newer version once you have a tentative package. I'm
> not sure I can reasonably commit to trying and preparing the update
> myself.

Same here, I was just curious if you thought the manual gem update fixed
this because it's a new upstream or just because "some gem thing".

a.

--
We must learn to live together as brothers or perish together as fools.
- Martin Luther King, Jr.

[Cyril Brulebois]

Antoine Beaupré <ana...@debian.org> (2023-08-23):

> Oh i meant the upstream version, the package in Debian is obviously out
> of date here. :)

Sure, I meant I would have tried (package relationship permitting) a
newer package if there was one available.

> Same here, I was just curious if you thought the manual gem update fixed
> this because it's a new upstream or just because "some gem thing".

“Yes.” ;p

[Georg Faerber]

Hi all,

I ran into this issue as well and debugged a bit:

On 23-08-23 12:16:10, Antoine Beaupré wrote:
> Same here, I was just curious if you thought the manual gem update
> fixed this because it's a new upstream or just because "some gem
> thing".

I believe the issue is not about the new upstream, as according to the
changelog there were no relevant changes in between 3.3.0 and 3.4.0, but rather
it's about the directories, which are searched and code is loaded from.

Due to this, initially, I took the "easy route" and did this:

+++ a/etc/puppet/puppetserver/conf.d/puppetserver.conf
+++ b/etc/puppet/puppetserver/conf.d/puppetserver.conf
@@ -11,7 +11,7 @@ jruby-puppet: {
# This setting defines the complete "GEM_PATH" for jruby. If set, it should include
# the gem-home directory as well as any other directories that gems can be loaded
# from (including the vendored gems directory for gems that ship with puppetserver)
- gem-path: [${jruby-puppet.gem-home}, "/usr/lib/puppetserver/vendored-jruby-gems"]
+ gem-path: [${jruby-puppet.gem-home}, "/usr/lib/puppetserver/vendored-jruby-gems", "/usr/share/rubygems-integration/all"]

# PLEASE NOTE: Use caution when modifying the below settings. Modifying
# these settings will change the value of the corresponding Puppet settings

However, this leads to the following:

Exception in thread "main" java.lang.IllegalStateException: Unable to borrow JRubyInstance from pool
[...]
Caused by: org.jruby.embed.EvalFailedException: (LoadError) Java extensions are required for JRuby.
no such file to load -- concurrent/concurrent_ruby.jar

concurrent_ruby.jar is part of the puppetserver package and installed to
/usr/lib/puppetserver/vendored-jruby-gems/gems/concurrent-ruby-1.1.5/lib/concurrent/concurrent_ruby.jar.

The ordering of the directory paths in puppetserver.conf doesn't make a
difference, the error stays the same.

So, symlinks to the rescue:

ln -s /usr/share/rubygems-integration/all/gems/hiera-eyaml-3.3.0/ /usr/lib/puppetserver/vendored-jruby-gems/gems/hiera-eyaml-3.3.0
ln -s /usr/share/rubygems-integration/all/gems/optimist-3.0.0/ /usr/lib/puppetserver/vendored-jruby-gems/gems/optimist-3.0.0
ln -s /usr/lib/ruby/vendor_ruby/highline /usr/lib/puppetserver/vendored-jruby-gems/gems/highline
ln -s /usr/share/rubygems-integration/all/specifications/hiera-eyaml-3.3.0.gemspec /usr/lib/puppetserver/vendored-jruby-gems/specifications/hiera-eyaml-3.3.0.gemspec
ln -s /usr/share/rubygems-integration/all/specifications/optimist-3.0.0.gemspec /usr/lib/puppetserver/vendored-jruby-gems/specifications/optimist-3.0.0.gemspec
ln -s /usr/share/rubygems-integration/all/specifications/highline-2.0.3.gemspec /usr/lib/puppetserver/vendored-jruby-gems/specifications/highline-2.0.3.gemspec

This made the error go away.

Maybe that's helpful,
all the best,
Georg

[Dominique Martinet]

Georg Faerber wrote on Tue, Sep 19, 2023 at 10:19:57AM +0000:
> So, symlinks to the rescue:
>

> [...]

This is horrible.
So I obviously did the same. Thanks a lot!

I'd be interested in a better solution that'll survive upgrades
though -- I agree wth Georg's analysis that this is "just" a problem of
loading paths, so either the puppetserver package could add hiera-eyaml
in its vendored-jruby-gems (which probably isn't too hard, but debian
frowns on vendored stuff in general so I don't see why it'd be better
for ruby deps..), or we'll need to look into why system gems cannot be
used with jruby...

--
Dominique Martinet | Asmadeus