Historical reporting, BI from puppetdb?

[Ryan Anderson]

I use puppetdb + puppetboard, which are very useful to see the current state of my environment. Puppetboard also provides a very nice representation of each agent's most recent reports. However, I want to take it to the next level and create custom historical reports for business intelligence (eg How many changes per day over the last month? How many RHEL5 virtual systems as of $DATE?). I've looked online and not found many solutions on this. Might Foreman provide this? Even if it does....I'm not sure I'm ready for that commitment. Since puppetdb uses PostgreSQL, I'd imagine most any BI tool that supports it (eg Pentah) could be configured to get what I want. However, is anyone successfully doing this?

[Ken Barber]

So something to note, while this is an interesting idea, you have to
be careful. We make zero commitments about the stability of the
database schema today ... so expect an upgrade to break things,
especially if we have a migration. In short, accessing the data
directly is not a supported stable API. Not saying it won't work, but
just letting you know the risks involved here.

Not sure how useful it is, we do have a lot of complex query
capability just built into PDB via the REST interface which might be
able to provide what you need:

http://docs.puppetlabs.com/puppetdb/2.0/api/query/v4/query.html

ken.

[Spencer Krum]

You absolutely want to build that against the puppetdb rest interface and not by hitting postgres directly.

I'd be interested in getting at this data as well, if you find a way or build a neat little application to do it.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTmU6MhRDXN2mQ0_4T%3DOTpz-ZboWfUeMZ7%3Dzd-V8nW%3DGvg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Spencer Krum
(619)-980-7820

[Ryan Anderson]

Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more.

[Ken Barber]

> Thanks, good to know. While the REST API would be the method to get at the
> data, my issue is that I'm not capable of writing a web app + data
> repository that can generate web-based reports, etc. I've actually gotten
> into the habit of running one-off queries using the API with curl to get
> YAML-formatted reports, which in part got me wanting more.

Question, this BI tool Pentah, what formats/apis is it able to ingest
beyond SQL? Are there any ETL capabilities built into this tool?

Would something like CSV work?

ken.

[Ken Barber]

Sorry, do you mean Pentaho?

[Ken Barber]

Ryan,

What about something like this?

http://wiki.pentaho.com/display/EAI/Rest+Client

This page seems to mix in general actions with integration steps, but
there are more integration types available here:

http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+Steps

ken.

[Ken Barber]

And others:

http://wiki.pentaho.com/display/EAI/JSON+Input
http://wiki.pentaho.com/display/EAI/HTTP+Client

Perhaps, you could combine the JSON input perhaps on the contents of a
'puppetdb export' tarball if you wanted to analyze the data from a
previous backup :-).

ken.

[Ryan Anderson]

I mentioned Pentaho (albeit spelled incorrectly) as an example, as I saw it mentioned a number of times online due to its open source community variant and full feature set. Thanks for the links, it appears that slurping data into Pentaho via puppetdb's REST API or otherwise is quite possible if enough time is put into it. 

I'm still wondering if anyone is doing the type of BI I'm talking about today against puppetdb? If so, with what tools? I'm willing to try out a couple solutions, but if there are others doing this successfully, I'd rather not reinvent the wheel.

Thanks,

RCA