Custom Facts using awk

[warron.french]

Hello, I have finally learned how to write a Custom Fact; and duplicated the syntax several times over inside the same .rb file.

I am using syntax that looks like the following:

Facter.add('qty_monitors_total') do

      setcode  do

             Facter::Util::Resolution.exec('/bin/grep " connected" /var/log/Xorg.0.log | cut -d\) -f2,3,4 | grep GPU |sort -u | wc -l')

      end

end

I don't know of any other way to do this yet; but that's not my concern yet.

What I would like to know is how can I use an awk command within the Facter::Util::Resolution.exec('.........') line.

I have a need to essentially reproduce the line above but drop   wc -l and add awk '{ print $2"_"$3"_on_"$1$4 }'

I need the awk command to pretty much look like this; the problem is awk uses its own single quotes (') and it will break the ruby code.

I am not a ruby developer; so if someone could either tell me:

1. It's just not possible; or
2. do it this way

That would be greatly appreciated.  Thank you,

--------------------------
Warron French

[Peter Bukowinski]

Hi Warron,

I'd consider using an external, executable fact to avoid ruby altogether.

https://docs.puppet.com/facter/3.6/custom_facts.html#executable-facts-----unix

Basically, you can write a bash script (or use any language you want),
drop it into '<MODULEPATH>/<MODULE>/facts.d/' on your puppet server,
and it will be synced to all your nodes (assuming you use pluginsync).

The only requirement for executable fact scripts is that they must
return key/value pairs in the format 'key=value'. Multiple keys/values
get their own line. In your case, you could do something like this:

-----
#!/bin/bash

key="qty_monitors_total"
value=$(your parsing command for /var/log/Xorg.0.log here)

echo "${key}=${value}"
-----

Save the file as an executable script in the above mentioned path and
it should be available on the next puppet run.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAJdJdQmZXQAd%2Bo%2Bnp-NHqxGHnXubf%2Bac-dP5FPoy4QYMEVuBuA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Rob Nelson]

That's probably one of the best ways to do this. But...

You CAN use double quotes around a string. You will need to escape characters that will otherwise be interpolated like double quotes and dollar signs. I'm going off memory but I think `"awk '{print \$1_\$2}'"` should interpolate to `awk '{print $1_$2}'` properly. This is often tedious and may require some trial and error to ensure both the double quotes and the system call that uses it don't interpolate too much but it can work.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson

[warron.french]

Oh wow!  That cool!  Thanks for the different method Peter!

--------------------------
Warron French

> email to puppet-users+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAJdJdQmZXQAd%2Bo%2Bnp-NHqxGHnXubf%2Bac-dP5FPoy4QYMEVuBuA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.

[warron.french]

Thanks Rob.  I will try both approaches; for me there is more appeal in simply using a shell script.

Peter, can I generate multiple key=value pairs inside the same shell script?  I don't explicitly have to do it the way you presented with 

key="key_name"

value="evaluated_expression"

echo "${key}=${value}"         Do I?

Can I simple skip the key="key_name" part and just do the expression_evaluation assigned to a variable and then echo them together?

Perhaps I can write shell functions and execute them all?

--------------------------
Warron French

> email to puppet-users+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAJdJdQmZXQAd%2Bo%2Bnp-NHqxGHnXubf%2Bac-dP5FPoy4QYMEVuBuA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT8Sduj%2BL%3DR3hZfSqkqeUgp9eBsQLjuBWEMrW6L1xxpE0Q%40mail.gmail.com.

[Michael Watters]

You can do everything you need entirely in ruby.  For example, here's a custom fact that returns the number of screens reported by the X server.

Facter.add(:screen_count) do
    confine :kernel => 'Linux'
    setcode do
        @screens = Facter::Core::Execution.exec("/usr/bin/xrandr -display :0").split("\n")
        @screens.count { |x| x =~ /connected/ }
    end
end

Note that this only counts screens currently in use by X which may not match the graphics card's total capabilities.  A second option would be to look at /sys/class/drm directly for any devices detected by the kernel.  For example:

Facter.add(:screen_count2) do
    confine :kernel => 'Linux'
    setcode "ls -d /sys/class/drm/*-DP-* | /usr/bin/wc -w"
end

 

[Branan Riley]

On Wed, Mar 22, 2017 at 4:07 PM Peter Bukowinski <pmb...@gmail.com> wrote:

Hi Warron,

I'd consider using an external, executable fact to avoid ruby altogether.

  https://docs.puppet.com/facter/3.6/custom_facts.html#executable-facts-----unix

Basically, you can write a bash script (or use any language you want),
drop it into '<MODULEPATH>/<MODULE>/facts.d/' on your puppet server,
and it will be synced to all your nodes (assuming you use pluginsync).

The only requirement for executable fact scripts is that they must
return key/value pairs in the format 'key=value'. Multiple keys/values
get their own line. In your case, you could do something like this:

We actually now support JSON and YAML output from executable facts as well. This made it into the release notes, but it looks like the main docs didn't get updated for the feature. I'll work with our documentation team to get that updated!

For a bash script doing awk transformations, though, the key=value syntax is likely easier.

Branan Riley

Software Engineer, Puppet Inc. 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.

[Joshua Schaeffer]

Yes, you can put multiple key=value pairs in a single file. Puppet v4 supports three known types:

1. YAML (must end in .yaml)
2. JSON (must end in .json)
3. Text (must end in .txt)

When working with .txt files you can only define string values. Arrays and hashes are not supported. Also. If your file is a program or script it must be executable (execute bit set). The script itself must output key=value. So what is required is the:

echo "my_fact=my_value"

This would create the fact call "my_fact" and it would set the value of it to "my_value". I don't mess around with external facts to much so I'm not 100% positive if you can perform the evaluation in the fact declaration, but my assumption is you can. What is required is that the key=value statement is echo'd to the stdout and that what is echo'd to stdout is the literal fact name equal to the literal value. As long as bash performs its operations properly then the literal values would be output, not the expression. Puppet then will pickup the stdout stream and set the fact accordingly. Again, this all an educated guess. Probably should have tested it myself, but I'll let that to you. :)

Hope that helps,

Joshua Schaeffer

P.S.

Executable facts on Windows should be known extension types as well and can include:

• .com or .exe
• .psl (PowerShell)
• .cmd or .bat (batch script)

--------------------------
Warron French

> email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAJdJdQmZXQAd%2Bo%2Bnp-NHqxGHnXubf%2Bac-dP5FPoy4QYMEVuBuA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[Joshua Schaeffer]

Sorry, I may have been ambiguous with the file extensions.

A text file should end in .txt. This means you define external facts like this:

# This is a plain text file that defines two external facts
my_fact1=my_value1
my_fact2=my_value2

An executable program or script should end in whatever extension it is written in (a.k.a. .sh, .py, .java, etc). When defining external facts with executable scripts or programs what is necessary is that the execute bit is set and that the puppet agent can execute them.

[warron.french]

Peter, perhaps I misunderstood you; but, I thought I was supposed to be able to use bash or sh scripting to generate facters of my own without the use of Ruby.

The link you provided refers to a python script example.  It also adds a shebang line at the top of the script; do I need the shebang line, or will Puppet simply execute the shell script with:

sh scriptname.sh

Thanks for the feedback,

--------------------------
Warron French

On Wed, Mar 22, 2017 at 7:07 PM, Peter Bukowinski <pmb...@gmail.com> wrote:

> email to puppet-users+unsubscribe@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAJdJdQmZXQAd%2Bo%2Bnp-NHqxGHnXubf%2Bac-dP5FPoy4QYMEVuBuA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.

[Peter Bukowinski]

Hi Warron,

Puppet executes the script directly, so you need the shebang line and you must ensure the file is executable.

-- Peter

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJdJdQnAbguKzz0S2O_NJfp2nzjeev77Ld7PHBEAOBH8_CZPsw%40mail.gmail.com.

[warron.french]

OK, done, and done.  But it still isn't showing up.

Is this potentially because I am using PE-3.8 as a component of Red Hat Satellite?

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJdJdQnAbguKzz0S2O_NJfp2nzjeev77Ld7PHBEAOBH8_CZPsw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/F53C3C7F-916F-4DB1-896F-8FF0F7F54BD6%40gmail.com.

[Joshua Schaeffer]

External facts are a Puppet v4 feature only. You have to use Ruby to create custom facts in Puppet v3.

--------------------------
Warron French

--------------------------
Warron French

> email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAJdJdQmZXQAd%2Bo%2Bnp-NHqxGHnXubf%2Bac-dP5FPoy4QYMEVuBuA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJA1CN9aFH4Eza-FoxzrfXDWiGCUHXE%2BGFt2Nu%3DjK2eDzV4upg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJdJdQnAbguKzz0S2O_NJfp2nzjeev77Ld7PHBEAOBH8_CZPsw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[Gabriel Schuyler]

Never fear, external facts work just fine in Puppet 3.

[Joshua Schaeffer]

Excellent. Good to know. I was told otherwise.

Thanks,

Joshua Schaeffer

[warron.french]

Joshua, thanks for this feedback.  I don't really know ruby; can you offer some ideas of where I can find other Puppet Facts written in Ruby that don't look like my originally posted example?

Thank you sir.

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/73bd7507-03a3-4421-bf51-af2e9f37d62c%40googlegroups.com.

[Luke Bigum]

Puppet modules on Github are a good source. I've found a simple one:

https://github.com/puppetlabs/puppetlabs-apache/blob/master/lib/facter/apache_version.rb

And one of my own that's a little more complicated:

https://github.com/LMAX-Exchange/puppet-networking-example/blob/master/lib/facter/interface_ringbuffer.rb

-Luke

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[warron.french]

Hey, thanks for the examples Luke!  I am looking at them now.

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f28aa960-21c9-4d8f-a086-2e72d0893060%40googlegroups.com.

[warron.french]

Hi Luke, I have some questions for you.

First, the link= https://github.com/puppetlabs/puppetlabs-apache/blob/master/lib/facter/apache_version.rb didn't have any reference to awk at all, was this the file you intended to suggest?

Secondly, the link= https://github.com/LMAX-Exchange/puppet-networking-example/blob/master/lib/facter/interface_ringbuffer.rb did have a reference to awk; thank you.

However, the syntax:
      ethtool_g = %x{/sbin/ethtool -g #{int} 2>/dev/null | grep -P '^(RX|TX):' | awk '{print $2}'}

Looks like something other than just plain shell scripting, so can you break this down for me just a little bit?

I recognize what looks like a variable, called ethtool_g, and then it continues with assignement based on %x{.......}  where the "...." is your shell scripting.

What is the %x a reference for/to?  Can I simply replace your variable with one of my own, and your shell scripting between the curly braces with my own shell scripting?

Is that legal, and is this in the language of ruby (so I have a reference point of where to go to look up more examples?

Sorry for the load of questions.  Thank you for the information.

--------------------------
Warron French

[Luke Bigum]

On Thursday, 30 March 2017 16:11:35 UTC+1, Warron French wrote:

Hi Luke, I have some questions for you.

First, the link= https://github.com/puppetlabs/puppetlabs-apache/blob/master/lib/facter/apache_version.rb didn't have any reference to awk at all, was this the file you intended to suggest?

Oh, I wasn't giving you Awk examples specifically, I was giving you one with a small amount of Ruby, and one with a bit more Ruby and some string manipulation in it. The use of Awk and piped shell commands in my Fact there is 100% pure laziness, it would be more "robust" to do all of the string manipulation in Ruby.

 

Secondly, the link= https://github.com/LMAX-Exchange/puppet-networking-example/blob/master/lib/facter/interface_ringbuffer.rb did have a reference to awk; thank you.

However, the syntax:
      ethtool_g = %x{/sbin/ethtool -g #{int} 2>/dev/null | grep -P '^(RX|TX):' | awk '{print $2}'}

Looks like something other than just plain shell scripting, so can you break this down for me just a little bit?

I recognize what looks like a variable, called ethtool_g, and then it continues with assignement based on %x{.......}  where the "...." is your shell scripting.

What is the %x a reference for/to?  Can I simply replace your variable with one of my own, and your shell scripting between the curly braces with my own shell scripting?

 

Correct, ethtool_g is a Ruby variable.

%x{} is one of the ways of executing something in a Shell and getting it's STDOUT, there are other ways, this post explains it well:  http://stackoverflow.com/questions/2232/calling-shell-commands-from-ruby

the #{int} is embedding a Ruby variable called 'int' defined earlier into the string.

Is that legal, and is this in the language of ruby (so I have a reference point of where to go to look up more examples?

Yes, you can.

What I would recommend is copy one of those Facts to your homedir, then set an environment variable FACTERLIB=/home/$USERNAME, which sets an extra Facter search path to your homedir. If you then run "facter -p" you should see the new Fact in the list. Then you can edit your Fact to your heart's content, and Google every crash or error message you come up with ;-) Once it actually works you can add the Fact to a module and distribute it to servers.

-Luke

 

--------------------------
Warron French

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[Michael Watters]

%x is a ruby method which captures command output.  IMO you can do most of what you need using native ruby methods, there's no need to pipe output to grep/sed/awk since ruby has built in pattern matching functions and if you're using ruby you might as well do it the ruby way.

For example, this fact uses the =~ operator to do pattern matching on an array of strings.

Facter.add(:screen_count) do
    confine :kernel => 'Linux'
    setcode do
        @screens = Facter::Core::Execution.exec("/usr/bin/xrandr -display :0").split("\n")
        @screens.count { |x| x =~ /connected/ }
    end
end

IMO piping grep to awk is never necessary.  awk has a built-in search function, just use it!  The command used in that fact could be rewritten as follows.

 ethtool_g = %x{ethtool -g #{int} 2>/dev/null  | awk '/^(RX|TX)/ {print $2}'}

Still messy and a more native way to do it would be like this.

%x{ethtool -g #{int} 2>/dev/null}.each_line do |line|

    puts line.split("\s")[-1] if line =~ /^(RX|TX):/

end

--------------------------
Warron French

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[warron.french]

Wow is that a lot to take in.  I simply need to learn Ruby to cut down on noise, or so it seems.

Thanks Michael.

--------------------------
Warron French

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/69e989c8-1c87-4601-a992-81988b363021%40googlegroups.com.