Announce: Puppet Server 0.2.0

[Nate Wolfe]

We are thrilled to announce the preview release of Puppet Server, our newest open source project.

Puppet Server is a next-generation alternative to our current Puppet master, which builds on the

successful Clojure technology stack underlying projects like PuppetDB.

Packages are available in the Puppet Labs package repositories, so you can try it out today as a

drop-in replacement for the existing Puppet master.

As the version number 0.2.0 should imply, Puppet Server is not production ready (yet), but please

do try it out in your favorite sandbox. Additionally, the API will not be considered stable until

Puppet Server reaches 1.0.0.

Install Puppet Server from packages:

https://github.com/puppetlabs/puppet-server/blob/b6a76e/documentation/install_from_packages.markdown

Submit issues to:

https://tickets.puppetlabs.com/browse/SERVER

Source:

https://github.com/puppetlabs/puppet-server

[Dominic Cleal]

On 23/09/14 17:11, Nate Wolfe wrote:
> We are thrilled to announce the preview release of Puppet Server, our
> newest open source project.
> Puppet Server is a next-generation alternative to our current Puppet
> master, which builds on the
> successful Clojure technology stack underlying projects like PuppetDB.
>
> Packages are available in the Puppet Labs package repositories, so you
> can try it out today as a
> drop-in replacement for the existing Puppet master.

Very neat, it works well as a drop-in replacement.

The only hitch I've had was with the Foreman report processor, which
makes an HTTPS connection to Apache with mod_ssl. On new OSes with
modern mod_ssl versions (e.g. EL7 or Ubuntu 14.04), the report processor
fails to make an HTTPS connection from the JVM with the error:

2014-09-26 08:56:09,984 ERROR [puppet-server] Report processor failed:
Could not send report to Foreman at
https://foreman.example.com/api/reports: Could not generate DH keypair
["sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1287)", ...]

This is a well-known problem between JVM clients and recent mod_ssl
versions, as the DH prime length supported by the JVM is limited.
Adding the DH parameter limits to the server's certificate worked around
the problem.

http://httpd.apache.org/docs/current/ssl/ssl_faq.html#javadh

Java 8 worked slightly better in that it accepts 2048 bit parameters,
but the default combination is still a problem. I guess it might affect
others using HTTPS from the master.

--
Dominic Cleal
Red Hat Engineering

[Darin Perusich]

Is there any documention on compiling puppet-server from source? I
need to be able to build it in an environment w/o network access, Open
Build Service, and I'm trying figure out all the dependencies I'm
missing.
--
Later,
Darin

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/CAB%2Bty33iME6m4uE5uz%2BdDU7deTFTY%3DVu-Rn8Kph3m2kjm1%3DYHw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Andy Parker]

On Fri, Sep 26, 2014 at 11:35 AM, Darin Perusich <da...@darins.net> wrote:

Is there any documention on compiling puppet-server from source? I
need to be able to build it in an environment w/o network access, Open
Build Service, and I'm trying figure out all the dependencies I'm
missing.

There is a document in the repository on how to run the server from source, which also includes the steps for compiling it[1]. 

It uses Leiningen to manage dependencies and so if you need to get all of the dependencies without network access you'll need to look into how to get leiningen to work from local caches (sorry, I don't know how to do that). You can see all of the dependencies listed out in the project.clj file[2].

 

1: https://github.com/puppetlabs/puppet-server/blob/master/documentation/dev_running_from_source.markdown

2: https://github.com/puppetlabs/puppet-server/blob/master/project.clj

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CADaviKuDwMfSzLpdzsrkpQZbWhRmzDXmWceWp6AZEh-QDGPhcg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 20-24 in San Francisco - www.puppetconf.com 

[Nate Wolfe]

I'm trying figure out all the dependencies I'm missing. 

Leiningen has a couple commands that can help with this.  When you have a network connection, `lein deps` will pull down all dependencies and cache them locally (under ~/.m2).  After that, you're good to go with no network access.  If you want to see a comprehensive list of all dependencies (including transitive dependencies!), you can use `lein deps: tree`.

Dependencies will be pulled down from www.clojars.org (for Clojure libraries), and search.maven.org (for Java libraries).

- Nate 

[Niels Abspoel]

For the Archlinux users, an aur package is available:

https://aur.archlinux.org/packages/puppetserver/

Op dinsdag 23 september 2014 18:12:29 UTC+2 schreef Nate Wolfe:

[Darin Perusich]

On Fri, Sep 26, 2014 at 3:21 PM, Nate Wolfe <nwo...@puppetlabs.com> wrote:
>> I'm trying figure out all the dependencies I'm missing.
>
>
> Leiningen has a couple commands that can help with this. When you have a
> network connection, `lein deps` will pull down all dependencies and cache
> them locally (under ~/.m2). After that, you're good to go with no network
> access. If you want to see a comprehensive list of all dependencies
> (including transitive dependencies!), you can use `lein deps: tree`.
>
> Dependencies will be pulled down from www.clojars.org (for Clojure
> libraries), and search.maven.org (for Java libraries).
>

Yes, I'm using 'lein deps' to download the deps locally, which is how
I also handle it for PuppetDB also.

The issue I'm seeing is that when i run 'lein uberjar' the resulting
jar doesn't contain all the classes need to run, jetty and
trapperkeeper are not included. When I attempt to start puppetserver
it fails and throws the following exception.

Any idea what's going on here?

Exception in thread "main" java.lang.IllegalArgumentException: Unable
to load service:
puppetlabs.trapperkeeper.services.webserver.jetty9-service/jetty9-service
at puppetlabs.trapperkeeper.bootstrap$resolve_service_BANG_$fn__5967.invoke(bootstrap.clj:47)
at puppetlabs.trapperkeeper.bootstrap$resolve_service_BANG_.invoke(bootstrap.clj:44)
at puppetlabs.trapperkeeper.bootstrap$parse_bootstrap_config_BANG_$iter__5990__5994$fn__5995.invoke(bootstrap.clj:157)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.RT.next(RT.java:598)
at clojure.core$next.invoke(core.clj:64)
at clojure.core$every_QMARK_.invoke(core.clj:2423)
at puppetlabs.trapperkeeper.bootstrap$parse_bootstrap_config_BANG_.invoke(bootstrap.clj:151)
at puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke(core.clj:113)
at puppetlabs.trapperkeeper.core$run.invoke(core.clj:144)
at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:159)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at clojure.lang.Var.invoke(Var.java:427)
at clojure.lang.AFn.applyToHelper(AFn.java:172)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.core$apply.invoke(core.clj:617)
at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:7)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at clojure.lang.Var.invoke(Var.java:427)
at clojure.lang.AFn.applyToHelper(AFn.java:172)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.core$apply.invoke(core.clj:617)
at clojure.main$main_opt.invoke(main.clj:335)
at clojure.main$main.doInvoke(main.clj:440)
at clojure.lang.RestFn.invoke(RestFn.java:512)
at clojure.lang.Var.invoke(Var.java:435)
at clojure.lang.AFn.applyToHelper(AFn.java:185)
at clojure.lang.Var.applyTo(Var.java:532)
at clojure.main.main(main.java:37)
Caused by: java.io.FileNotFoundException: Could not locate
puppetlabs/trapperkeeper/services/webserver/jetty9_service__init.class
or puppetlabs/trapperkeeper/services/webserver/jetty9_service.clj on
classpath:
at clojure.lang.RT.load(RT.java:443)
at clojure.lang.RT.load(RT.java:411)
at clojure.core$load$fn__5018.invoke(core.clj:5530)
at clojure.core$load.doInvoke(core.clj:5529)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5336)
at clojure.core$load_lib$fn__4967.invoke(core.clj:5375)
at clojure.core$load_lib.doInvoke(core.clj:5374)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$load_libs.doInvoke(core.clj:5413)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$require.doInvoke(core.clj:5496)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at puppetlabs.trapperkeeper.bootstrap$resolve_service_BANG_$fn__5967.invoke(bootstrap.clj:44)
... 30 more

>
> On Tuesday, September 23, 2014 9:12:22 AM UTC-7, Nate Wolfe wrote:
>>
>> We are thrilled to announce the preview release of Puppet Server, our
>> newest open source project.
>> Puppet Server is a next-generation alternative to our current Puppet
>> master, which builds on the
>> successful Clojure technology stack underlying projects like PuppetDB.
>>
>> Packages are available in the Puppet Labs package repositories, so you can
>> try it out today as a
>> drop-in replacement for the existing Puppet master.
>>
>> As the version number 0.2.0 should imply, Puppet Server is not production
>> ready (yet), but please
>> do try it out in your favorite sandbox. Additionally, the API will not be
>> considered stable until
>> Puppet Server reaches 1.0.0.
>>
>> Install Puppet Server from packages:
>>
>> https://github.com/puppetlabs/puppet-server/blob/b6a76e/documentation/install_from_packages.markdown
>>
>> Submit issues to:
>> https://tickets.puppetlabs.com/browse/SERVER
>>
>> Source:
>> https://github.com/puppetlabs/puppet-server
>>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/71c7c144-9e43-4c26-8b04-0780555e26e1%40googlegroups.com.

[Ken Barber]

> Yes, I'm using 'lein deps' to download the deps locally, which is how
> I also handle it for PuppetDB also.
>
> The issue I'm seeing is that when i run 'lein uberjar' the resulting
> jar doesn't contain all the classes need to run, jetty and
> trapperkeeper are not included. When I attempt to start puppetserver
> it fails and throws the following exception.
>
> Any idea what's going on here?

Can you show your full working, in a gist or something? That is -
every command from start to finish until you fail? Might help track it
down.

ken.

[Darin Perusich]

Here's a quick recreation of the steps. This is on OpenSUSE 13.1 with
OpenJDK 1.7.0_51

https://gist.github.com/deadpoint/900a2d92911705023986
--
Later,
Darin

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAE4bNTnqR43%2BheNDKsODmDN0f__%3D3THrCfc7NHv7jANokOjyFQ%40mail.gmail.com.

[Chris Price]

On Mon, Sep 29, 2014 at 7:17 AM, Darin Perusich <da...@darins.net> wrote:

Here's a quick recreation of the steps. This is on OpenSUSE 13.1 with
OpenJDK 1.7.0_51

https://gist.github.com/deadpoint/900a2d92911705023986
--
Later,
Darin

Hi Darin,

The issue you're seeing is because the "tk-jetty9" dependency is specified in the 'dev' profile in project.clj.  Dependencies that appear there instead of in the "main" deps section are available during testing / development, but they don't get bundled into the uberjar.

The reason we do it this way (for now) is because we don't use the 'puppet-server' git repo as the source for building the final package artifacts; we now have a separate packaging project that can be re-used to build all of our clojure-based packages without having to repeat the packaging code everywhere.  This also allows us to do things like compose multiple projects together (e.g., puppet-server + jetty + PE web apps) for development purposes.

We might be able to open-source that packaging repo at some point if folks need it; I'll talk to our release team.  In the meantime, you can work around the issue by simply moving the 'tk-jetty9' dependency up in the project.clj, out of the 'dev' profile and into the 'main' dependencies.  That'll cause it to get included in the uberjar.

[Josko Plazonic]

Hullo,

On 09/23/2014 12:11 PM, Nate Wolfe wrote:
> As the version number 0.2.0 should imply, Puppet Server is not
> production ready (yet), but please
> do try it out in your favorite sandbox.

I didn't find so far any feature comparison between puppet server and
the regular (purely ruby based) puppet master. I realize that puppet
server work is far from finished and that it probably loads at least
some of the puppet's ruby code (I assume, given dependency on having
regular puppet installed, so maybe that's being relied on heavily? or
not?) - but it would be useful to have a table/chart comparing
features/problems/discrepancies/works/doesn't of new code base vs old one.

Is there some documentation about that somewhere? Having that
information would greatly help in deciding when to jump ship and embark
on the clojure/jruby adventure, production ready or not.

Thanks,

JP

[Trevor Vaughan]

Any docs on the configuration files?

Is there a way to update webserver.conf to run a CA on a different port and, instead of having 'want' in the client_auth section having something like 'need' with the CA having 'none' (or something)?

Goal: Run a CA separately from the main daemon using a *single* startup script. The CA would sign and the Master would trust but the trust would be required and absolute.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5429BA61.90308%40princeton.edu.

For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Andy Parker]

On Mon, Sep 29, 2014 at 1:00 PM, Josko Plazonic <plaz...@princeton.edu> wrote:

Hullo,

On 09/23/2014 12:11 PM, Nate Wolfe wrote:

As the version number 0.2.0 should imply, Puppet Server is not production ready (yet), but please
do try it out in your favorite sandbox.

I didn't find so far any feature comparison between puppet server and the regular (purely ruby based) puppet master.  I realize that puppet server work is far from finished and that it probably loads at least some of the puppet's ruby code (I assume, given dependency on having regular puppet installed, so maybe that's being relied on heavily? or not?) - but it would be useful to have a table/chart comparing features/problems/discrepancies/works/doesn't of new code base vs old one.

Because of the way it has been written (JRuby running puppet), puppet-server should be able to do just about anything that a current puppet master can do. There are a couple things that we know that it can't do:

  * Because of JRuby, most gems with native extensions won't work. This should only affect you if you are using custom functions that use ruby code that ties into native extensions (https://github.com/jruby/jruby/wiki/C-Extension-Alternatives)

  * The puppet-server expects to be the SSL terminator. If you are running something in front of your master that terminates SSL and forwards headers on to the master, then the puppet-server won't work. I believe Chris knows about this and is working on it.

Other than that, puppet-server should be complete. It actually runs and passes all of the same system tests that we run against the webrick and passenger setups. Chris can correct me if there are any other things that he knows about. I think he is also working on a blog post and so maybe he'll address some of these questions in that.

 

Is there some documentation about that somewhere?  Having that information would greatly help in deciding when to jump ship and embark on the clojure/jruby adventure, production ready or not.

Thanks,

JP

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5429BA61.90308%40princeton.edu.

For more options, visit https://groups.google.com/d/optout.

--

[Darin Perusich]

Progress...after making that change to the project.clj and building
it's now failing to start with this error. Hopefully I can dig into
this a bit more tomorrow, but puppetdb has been working fine on this
system.

ArgumentError: Could not find terminus puppetdb for indirection facts
validate_terminus_class at
/usr/lib64/ruby/vendor_ruby/2.0.0/puppet/indirector/indirection.rb:161
terminus_class= at
/usr/lib64/ruby/vendor_ruby/2.0.0/puppet/indirector/indirection.rb:153
configure_routes at
/usr/lib64/ruby/vendor_ruby/2.0.0/puppet/indirector.rb:27
each at org/jruby/RubyHash.java:1341
configure_routes at
/usr/lib64/ruby/vendor_ruby/2.0.0/puppet/indirector.rb:17
configure_indirector_routes at
file:/usr/share/puppetserver/puppet-server-release.jar!/puppet-server-lib/puppet/server/master.rb:238
initialize at
file:/usr/share/puppetserver/puppet-server-release.jar!/puppet-server-lib/puppet/server/master.rb:77
invoke at jruby_puppet_core.clj:140
invoke at jruby_puppet_core.clj:121
invoke at jruby_puppet_core.clj:222
invoke at jruby_puppet_core.clj:210
invoke at jruby_puppet_service.clj:29
invoke at internal.clj:246
invoke at internal.clj:230
shutdown_on_error at internal.clj:285
invoke at internal.clj:254
invoke at internal.clj:254
invoke at core.clj:619
doInvoke at core.clj:2396
invoke at jruby_puppet_service.clj:28
invoke at core.clj:1836

[Chris Price]

On Fri, Sep 26, 2014 at 3:06 AM, Dominic Cleal <dcle...@redhat.com> wrote:

The only hitch I've had was with the Foreman report processor, which
makes an HTTPS connection to Apache with mod_ssl.  On new OSes with
modern mod_ssl versions (e.g. EL7 or Ubuntu 14.04), the report processor
fails to make an HTTPS connection from the JVM with the error:

Thanks for the good info on this!  I've copypasta'd your description into this Jira ticket so we can track it:

https://tickets.puppetlabs.com/browse/SERVER-17

[Chris Price]

On Mon, Sep 29, 2014 at 1:33 PM, Andy Parker <an...@puppetlabs.com> wrote:

On Mon, Sep 29, 2014 at 1:00 PM, Josko Plazonic <plaz...@princeton.edu> wrote:

Hullo,

On 09/23/2014 12:11 PM, Nate Wolfe wrote:

As the version number 0.2.0 should imply, Puppet Server is not production ready (yet), but please
do try it out in your favorite sandbox.

I didn't find so far any feature comparison between puppet server and the regular (purely ruby based) puppet master.  I realize that puppet server work is far from finished and that it probably loads at least some of the puppet's ruby code (I assume, given dependency on having regular puppet installed, so maybe that's being relied on heavily? or not?) - but it would be useful to have a table/chart comparing features/problems/discrepancies/works/doesn't of new code base vs old one.

Because of the way it has been written (JRuby running puppet), puppet-server should be able to do just about anything that a current puppet master can do. There are a couple things that we know that it can't do:

  * Because of JRuby, most gems with native extensions won't work. This should only affect you if you are using custom functions that use ruby code that ties into native extensions (https://github.com/jruby/jruby/wiki/C-Extension-Alternatives)

  * The puppet-server expects to be the SSL terminator. If you are running something in front of your master that terminates SSL and forwards headers on to the master, then the puppet-server won't work. I believe Chris knows about this and is working on it.

Other than that, puppet-server should be complete. It actually runs and passes all of the same system tests that we run against the webrick and passenger setups. Chris can correct me if there are any other things that he knows about. I think he is also working on a blog post and so maybe he'll address some of these questions in that.

Andy hit on the two that I'm currently aware of.  We will be working on some additional docs over the course of this week, so keep an eye on the repo if you're interested.  I'll try to make sure we include an FAQ or "known issues" section in the docs that includes the info above.

[Chris Price]

On Mon, Sep 29, 2014 at 1:25 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Any docs on the configuration files?

Coming this week!  :)  Keep an eye on the repo.

 

Is there a way to update webserver.conf to run a CA on a different port and, instead of having 'want' in the client_auth section having something like 'need' with the CA having 'none' (or something)?

Not yet, but this is one of the major goals of the upcoming work that we have slated for the new CA service.

 

Goal: Run a CA separately from the main daemon using a *single* startup script. The CA would sign and the Master would trust but the trust would be required and absolute.

Yeah, this should be doable in the not-too-distant future; I presume in this scenario you'd direct the agents to the CA port via the agent-side puppet.conf settings?

[Chris Price]

On Mon, Sep 29, 2014 at 1:39 PM, Darin Perusich <da...@darins.net> wrote:

Progress...after making that change to the project.clj and building
it's now failing to start with this error. Hopefully I can dig into
this a bit more tomorrow, but puppetdb has been working fine on this
system.

ArgumentError: Could not find terminus puppetdb for indirection facts
      validate_terminus_class at

This sounds like the puppetdb terminus code isn't on the ruby load path (as perceived by the embedded JRuby instances).

In a Cent/RHEL install, e.g., there will be a config file that gets created that has a setting that looks like this:

https://github.com/puppetlabs/puppet-server/blob/master/dev/sample-configs/puppet-server.sample.conf#L21

It will be populated with the list of ruby paths where the puppet/facter packages will install the ruby code for that platform, and the puppetdb terminus should put its ruby code in the same place.  You may need to add additional paths to that setting for your system.

Sorry we don't have more docs on the settings yet, they're on their way :)