Hiera doesn't work with JSON

[Stack Kororā]

Greetings,

I am having an issue with Hiera again. Bangin my head on it for a few hours and not getting anywhere. Since this group was able to help last time, I have my fingers crossed you can help me again. :-D

I am trying to use JSON with Hiera this time.

First, fully updated dev box.

$ cat /etc/redhat-release

Scientific Linux release 6.7 (Carbon)

$ sudo yum update

Loaded plugins: refresh-packagekit, security

Setting up Update Process

No Packages marked for Update

$ rpm -qa | egrep '(puppet|fact|hiera|ruby)'

facter-2.4.4-1.el6.x86_64

puppet-3.8.2-1.el6.noarch

hiera-1.3.4-1.el6.noarch

puppet-server-3.8.2-1.el6.noarch

ruby-libs-1.8.7.374-4.el6_6.x86_64

ruby-rdoc-1.8.7.374-4.el6_6.x86_64

ruby-shadow-2.2.0-2.el6.x86_64

ruby-1.8.7.374-4.el6_6.x86_64

rubygems-1.3.7-5.el6.noarch

ruby-augeas-0.4.1-3.el6.x86_64

rubygem-deep_merge-1.0.0-2.el6.noarch

libselinux-ruby-2.0.94-5.8.el6.x86_64

rubygem-json-1.5.5-3.el6.x86_64

ruby-irb-1.8.7.374-4.el6_6.x86_64

Let's use the puppet documentation to get started, shall we?

https://docs.puppetlabs.com/hiera/1/data_sources.html

$ cat /etc/puppet/hiera.yaml

---

:backends:

   - json

:hierarchy:

   - common

:merge_behavior: "deeper"

:json:

   :datadir: /etc/puppet/hieradata

BTW: the doc's actual JSON example is broken as it is missing the [ ] around it. I included it. below and if you don't you fail the parser and Hiera just ignores everything instead of giving an error.

$ cat /etc/puppet/hieradata/common.json

[

{

   "apache-packages" : [

   "apache2",

   "apache2-common",

   "apache2-utils"

   ],

    "hosts_entry" :  "sandbox.%{fqdn}",

    "sshd_settings" : {

                       "root_allowed" : "no",

                       "password_allowed" : "no"

                     }

}

]

Verify valid JSON

$ cat /etc/puppet/hieradata/common.json | jq '.[0]'

{

  "sshd_settings": {

    "password_allowed": "no",

    "root_allowed": "no"

  },

  "hosts_entry": "sandbox.%{fqdn}",

  "apache-packages": [

    "apache2",

    "apache2-common",

    "apache2-utils"

  ]

}

Awesome!  Lets validate in Hiera:

$ hiera --version

1.3.4

$ hiera -c /etc/puppet/hiera.yaml hosts_entry

/usr/lib/ruby/site_ruby/1.8/hiera/filecache.rb:56:in `read_file': Data retrieved from /etc/puppet/hieradata/common.json is String not Hash (TypeError)

from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:24:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:76:in `datasources'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `map'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `datasources'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:17:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:206:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `each'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera.rb:60:in `lookup'

from /usr/bin/hiera:225

And here is where things went wrong. The worst part is that it is such a generic error I can't find anything of real value in my online searching. Even worse, the only directly relevant hit I found basically said to stop using JSON and use YAML instead. Not really a solution is it? 

But whatever. It is probably good to verify that YAML is at least working with Heira in the dev environment. Using the puppet documentation again for guidance. 

$ mv /etc/puppet/hieradata/common.json /etc/puppet/hieradata/common.yaml
$ vim /etc/puppet/hieradata/common.yaml

$ cat /etc/puppet/hieradata/common.yaml 

---

# array

apache-packages:

    - apache2

    - apache2-common

    - apache2-utils

# string

apache-service: apache2

# interpolated facter variable

hosts_entry: "sandbox.%{fqdn}"

# hash

sshd_settings:

    root_allowed: "no"

    password_allowed: "yes"

# alternate hash notation

sshd_settings: {root_allowed: "no", password_allowed: "yes"}

# to return "true" or "false"

sshd_settings: {root_allowed: no, password_allowed: yes}

$ vim /etc/puppet/hiera.yaml 

$ cat /etc/puppet/hiera.yaml

---

:backends:

    - yaml

:hierarchy:

    - common

:merge_behavior: "deeper"

:yaml:

    :datadir: /etc/puppet/hieradata

$ hiera -c /etc/puppet/hiera.yaml hosts_entry

sandbox.

Awesome! So YAML works.

Back to the JSON version....

$ hiera -c /etc/puppet/hiera.yaml hosts_entry

/usr/lib/ruby/site_ruby/1.8/hiera/filecache.rb:56:in `read_file': Data retrieved from /etc/puppet/hieradata/common.json is String not Hash (TypeError)

from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:24:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:76:in `datasources'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `map'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:74:in `datasources'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend/json_backend.rb:17:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:206:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `each'

from /usr/lib/ruby/site_ruby/1.8/hiera/backend.rb:203:in `lookup'

from /usr/lib/ruby/site_ruby/1.8/hiera.rb:60:in `lookup'

from /usr/bin/hiera:225

Dah!

No matter how simple I make it or what tweaks I do to the common.json file, I always get that exact error. If I run puppet agent, I get a very similar error too! 

"So just use YAML like the other post said" 

1) That doesn't fix JSON and it avoids the problem ("Hey doc, my legs are broken!" "Have you tried walking on your hands instead? Why not just do that?" :-P)

2) The fancy new tool that my security team is using to manage and monitor system wide variables reads/writes JSON but doesn't do YAML. Meaning we either have to keep doing the manual checks we are doing, or stick a parser in between. Why? Puppet and Heira are supposed to work with JSON, it read/writes JSON, this should theoretically just work.

Does anyone know why hiera isn't working with JSON? I really feel like this is probably a simple solution that I am just not seeing. I am hoping someone else might be able to recognize the answer.

Thanks!

~Stack~

[Matthaus Owens]

Stack,

Adding [ ] around your json is what is causing the problem you're
seeing. It turns the data you are writing into an Array instead of a
Hash. Hiera is expecting it to be a hash, as the error message
indicates. The error message here may not be very helpful though,
because the data isn't a string, but is an Array. What parse error do
you get without the [ ]?

I was able to get a correct return from hiera by removing the [ ].

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/2703e86d-f1e8-40a3-9d0c-24fd85bbbfe4%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Matthaus Owens
Puppet Labs

PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.
Register now to take advantage of the Early Bird discount —save $249!

[~Stack~]

Greetings Matthaus!

On 09/17/2015 05:24 PM, Matthaus Owens wrote:
> Stack,
[snip]

>> BTW: the doc's actual JSON example is broken as it is missing the [ ] around
>> it. I included it. below and if you don't you fail the parser and Hiera just
>> ignores everything instead of giving an error.
>>
>
> Adding [ ] around your json is what is causing the problem you're
> seeing. It turns the data you are writing into an Array instead of a
> Hash. Hiera is expecting it to be a hash, as the error message
> indicates. The error message here may not be very helpful though,
> because the data isn't a string, but is an Array. What parse error do
> you get without the [ ]?
>
> I was able to get a correct return from hiera by removing the [ ].

On one hand, I feel really dumb. That should have been obvious. I was
comparing against the output I got from that other program and I see how
and why it is wrapping it with [] now. Plus the example I was told to
use to verify the JSON output was "jq[0]". 15 seconds on the jq tutorial
website shows why that is broken. Some how that fact just didn't click
in my head. Duh.

On the other hand, scrolling back through my terminal history I see the
pre-[] testing where hiera simply just didn't print anything or returned
nil. I am failing to see the difference between then and now...My guess
is that I had another hiera error, and in my struggles for fixing this
issue, I fixed that one...but I'm not 100% on that...

On the bright side, it is all working now!! :-)

I must be dense or have done something to make Hiera not like me. It
seems we brawl for a long time, then suddenly everything starts
working... :-D

Thank you so much for pointing out my silly goof up. I really REALLY
appreciate the second pair of eyes pointing out my mistake.

~Stack~