Lightweight PuppetDB replacement

[Bomber]

Is any plans to implement lightweight puppetdb replacement for storeconfigs? PuppetDB is ugly, i.e. has junkie API and _very_ heavy by itself because of JVM, henve it is almost impossible to use it for testing masterless environments, old active_record implementation was a solution, PuppetDB is not.

Yes, I've read a lot about active_record 'slowness' but never saw any test (seems it was not published) proving this point. May be those test was made using ruby 1.8 or something. Anyway, requesting for lightweight PuppetDB replacement in pure ruby.

[Luke Kanies]

> On Sep 8, 2014, at 6:44 AM, Bomber <andrew.do...@gmail.com> wrote:
>
> Is any plans to implement lightweight puppetdb replacement for storeconfigs? PuppetDB is ugly, i.e. has junkie API and _very_ heavy by itself because of JVM, henve it is almost impossible to use it for testing masterless environments, old active_record implementation was a solution, PuppetDB is not.
>
> Yes, I've read a lot about active_record 'slowness' but never saw any test (seems it was not published) proving this point. May be those test was made using ruby 1.8 or something. Anyway, requesting for lightweight PuppetDB replacement in pure ruby.

No, there are no such plans.

Store configs isn't just slow, it's unusable at even small scale. The
list archives should make it pretty clear that people couldn't use it.
I don't agree that using the JVM makes PuppetDB too heavyweight to use
for testing or in masterless environments. At least, that's not been
the experience of our customers.

PuppetDB works well for most of the use cases we know about, and we're
definitely not going to invest in a second production-worthy service
to compete with it. The API is pretty simple if you want to build one,
though.

[Daniele Sluijters]

Hi,

Maybe it's me but I find the way you started this discussion extremely rude and at some point offensive. You jettison in here with random claims of ugliness and slowness and that acitve_record was a better fit. I don't know what you're trying to achieve but you might want to work on how you ask people for help or input in the future.

As far as the PuppetDB API's ugliness go, I can only disagree, As the author of pypuppetdb, puppetb-stencil and Puppetboard and probably the largest consumer of the PuppetDB API, it's been nice to work with. Yes it has a few things that could use improvement but that's mainly because it was designed as an API for machines to get data, not for humans to read.

So, what exactly is your problem, how does the JVM worsen this and what is you gripe with the PuppetDB API? I'd like to see some metrics that prove your point about PuppetDB and the JVM. I'd also like to see some real examples of your gripes with the PuppetDB API. I'd also like to hear how PuppetDB complicates your testing? In both cases you need a database, PuppetDB even makes this easier because it can run in-memory.

-- 

Daniele Sluijters

[Eric Shamow]

It’s not just you.

Further as someone who remains a JVM skeptic (and caveat: former PL employee), PuppetDB is one of the cleanest/nicest pieces of engineering I’ve seen PL produce.

That’s not to say I don’t have issues with bits of it (what software do I not have issues with?) PuppetDB is some really great craftsmanship.

-Eric

-- 
Eric Shamow
Sent with Airmail

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/a4debfd7-968a-4f3d-ae46-2beed02f0312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Turnbull]

Eric Shamow wrote:
> It’s not just you.
>
> Further as someone who remains a JVM skeptic (and caveat: former PL
> employee), PuppetDB is one of the cleanest/nicest pieces of engineering
> I’ve seen PL produce.
>
> That’s not to say I don’t have issues with bits of it (what software do
> I not have issues with?) PuppetDB is some really great craftsmanship.
>

I'll be blunter than that: anyone who wants stored configs back
fundamentally hates themselves. :)

Regards

James

--
* The Docker Book (http://dockerbook.com)
* The LogStash Book (http://logstashbook.com)
* Pro Puppet (http://tinyurl.com/ppuppet2 )
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

[Pete Brown]

I have to agree with everyone.

I used stored configs for a few years but switched to PuppetDB the day
it got released.
Stored configs was a good concept and allowed me to do some pretty
amazing things at the time and without it a things were so much
harder.
I do admit I was a bit sceptical about PuppetDB at first due to it
needing a JVM but once I got over that and embraced to future it made
my setup even more amazing.
Adding lovely tools like Puppetboard and puppetdb-query, on top makes
it the best tool for the job.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5410383A.7040500%40lovedthanlost.net.

> For more options, visit https://groups.google.com/d/optout.

--
Pete Brown
Director and Primary Systems Engineer
Abstract IT Pty Ltd.

[Michael Stahnke]

Additionally, from a performance perspective, we couldn't even run the test suites we run with PuppetDB against the ActiveRecord based system. It falls over, we run out of memory and the slowness is unreal. If you want to see this in action, Look for "PuppetConf Deepak" on youtube and you'll see talks where he shows this in live demos. 

Beyond that, Ruby (MRI at least) take more memory than many PuppetDB installations. I was never a huge fan of the JVM, but when I saw what PuppetDB could do with only a 128MB heap, I was sold. It's a constant memory reservation from the OS, and just works. MRI would jump from little usage to hundreds of megs (even sometimes gigs) of memory during runs and checkins from clients. Another advantage of the JVM is are the metrics you can get from it to do things like tuning and monitoring all via JMX. 

If the API is the main concern, there are binding for the API in python, ruby, go, even haskell. There are probably lots more I'm just not aware of.  

From an installation point of view (your heavy comment) installing a jar and jre isn't too difficult. (We even provide a module to do this for you).  It's also a little easier to ensure dependencies all work together compared to Ruby with ActiveRecord versions changing/breaking API between 2.x, 3.x and 4.x. Also, after 3.x, active record started requiring many more gems, IIRC.

If there are specific problems, I'm sure somebody would be happy to help you a bit. If you only have complaints, I hope some of this is food for thought. 

[Jason Antman]

If this was just a troll, I'll speak the unfortunate truth - please keep that on the puppet-users list, which is increasingly filling with it.

If this was a legitimate question, I'd guess you have some horrible misconfiguration

- ActiveRecord was barely functional at best, and riddled with problems (performance being one of the biggest ones)

- I use a bunch (all?) of Daniele's code that hits the API, and have written a *lot* using pypuppetdb myself. With a few minor features that I'd like to see in the API (most of which are planned), it's been a joy for me to work with

- I've had no trouble running PuppetDB *and postgres* alongside a master in a relatively small VM for testing purposes

-Jason

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAMto7LL01OivipRv1GhfT932wJO-LtddcFVa96TqXxheDn1KXw%40mail.gmail.com.

[Luke Kanies]

On Sep 15, 2014, at 5:57 PM, Jason Antman <ja...@jasonantman.com> wrote:

If this was just a troll, I'll speak the unfortunate truth - please keep that on the puppet-users list, which is increasingly filling with it.

Hi Jason,

That’s the first I’ve heard anyone say that puppet-users is filling with trolls.  I’d love to hear more if you have that concern, because it doesn’t match the experience of me or my team, but it’s obviously a big concern for us that the list stay effective and useful, which means it should be low on trolls.

I’d appreciate a follow-up to me and/or Dawn (da...@puppetlabs.com).

Thanks,

Luke

[…]

-- 

http://puppetlabs.com/ | http://about.me/lak | @puppetmasterd

[Hunter Haugen]

On Mon, Sep 8, 2014 at 6:28 AM, Bomber <andrew.do...@gmail.com> wrote:

Is any plans to implement lightweight puppetdb replacement for storeconfigs?

https://github.com/nightfly19/minidb is an implementation that I came across. I'm not sure if it was a hobby project or used in production. 

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> - I've had no trouble running PuppetDB *and postgres* alongside a
> master in a relatively small VM for testing purposes

The only problem I see with deprecating active_record storeconfigs is:
How are you going to use exported resources in puppet apply
environments, without having to do all the SSL dance?

https://groups.google.com/forum/#!msg/puppet-users/L4CAHh3eYag/To9nHlAvA34J

While I might not yet have fully figured out that puppetdb can also be
used without SSL nowadays, it looks like no one could answer my question.

Why does one want to use storeconfigs/exported resources in a puppet
apply environment? Actually you don't, but if you share modules, you
might end up with modules that require them.

And currently - up to my knowledge - it is not possible to use them in
apply-mode without getting any kind of warning or doing all the heavy
lifting of installing puppetdb. Which is quite easy, but still it's
not just yum install puppet; git clone mymodules; git apply mymodules.

~pete

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAlQcSUYACgkQbwltcAfKi38h4ACgoSA93CCCIaVE6KClQ7QtzSay
2bQAoIvDTyjy4its2T8bqvHJQAjmMvk7
=ouh9
-----END PGP SIGNATURE-----

[Ken Barber]

> The only problem I see with deprecating active_record storeconfigs is:
> How are you going to use exported resources in puppet apply
> environments, without having to do all the SSL dance?
>
> https://groups.google.com/forum/#!msg/puppet-users/L4CAHh3eYag/To9nHlAvA34J

Well, we could simplify the SSL for puppetdb, An option to let it use
server side validation only basically (so still SSL, but no client
auth). CA validation is also on the table.

Of course, just to mention: reducing authentication requirements is
what it is, its a compromise to security somewhere and one has to make
the correct decisions around this.

Another entirely different approach for production environments is to
remove PKI handling, so its independant of puppet apply, and make it
easier and closer to system bootstrap time for everyone. ie. make ssl
easier, so we don't have to drop security features.

> While I might not yet have fully figured out that puppetdb can also be
> used without SSL nowadays, it looks like no one could answer my question.

It can, but the terminus won't use cleartext. Probably a bad idea to
ever let this happen in production. I can see use cases for it in
test/debugging.

> Why does one want to use storeconfigs/exported resources in a puppet
> apply environment? Actually you don't, but if you share modules, you
> might end up with modules that require them.

Exported resources can still be a thing for masterless environments,
plus searching based solutions like puppetdbquery. I can totally see
this - all the same rules apply to mastered environments in this case,
generally speaking.

Plus some users just want facts or reports, not exported resources or
searching. So that their tools can analyze this out-of-band. Thats
also a viable use-case.

> And currently - up to my knowledge - it is not possible to use them in
> apply-mode without getting any kind of warning or doing all the heavy
> lifting of installing puppetdb. Which is quite easy, but still it's
> not just yum install puppet; git clone mymodules; git apply mymodules.

If we presume PuppetDB would already exist somewhere its about making
the client install easier. I think if we allow server side validation
only, then perhaps the dance would become somewhat smaller.

If we presume no PuppetDB, then its a little more trickier to make
this fast. I'm guessing this is a dev environment requirement in most
cases. Me? I'd probably want tooling around the puppetdb subcommand to
help me fire up PuppetDB easier, like puppetdb dev <insert params
here> where params could be port etc. Or maybe a jar file I can fire
up. Helpers in tooling like beaker come to mind, especially if you
wanted to test a module that requires exported resources for example,
end-to-end.

ken.