problems with puppetdb

[Peter Brown]

Hi everyone,

I got the new version installed and am now having a strange issue.
my puppetdb server seems to start but then dies after a few seconds
with no logging that i can find.
I tried setting the log level to DEBUG but still nothing.

I tried running by hand and discovered it couldn't read it's config
files so i fixed that.
It also wants write access to /usr/share/puppetdb
This seems like strange place it would need write access to.
Is the really necessary?
It is also ringing security bells in my head so I am a bit hesitant to
give it write access without knowing why.

My puppetdb is running on latest ubuntu with latest openjdk 6

Thanks in advance.

Pete.

[Peter Brown]

I gave puppetdb write permissions to /usr/share/puppetdb an it starts
now but it's not making me happy leaving it with write access to a
system directory.

[Peter Brown]

It also seems to be ignoring my port settings.
It seems to be listening on 1099 and 55752.
are these the new default ports?

[Peter Bukowinski]

How did you install/upgrade it? Mine runs on RHEL6. The config files I set when I installed the initial version did not get modified during the upgrade process. My ports are still as expected.

-- Peter

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

[Peter Brown]

All I did was upgrade it.
The config files are as they were. ( i even tried catting the files
together and it does the same thing )

I just downgraded to the old version and i am getting the same errors
so I have no idea what's happening.

It is reading the config file[s] because it is using the database settings.
setting the log level to DEBUG isn't helping at all as the only extra
thing it logs is that it is connecting to the database which i can see
anyway with netstat.

[Deepak Giridharagopal]

I've created ticket http://projects.puppetlabs.com/issues/15536 to track this; we'll take a look at that this iteration. Would you mind adding yourself as a "watcher" on that ticket?

Thanks for the report!

deepak

--

Deepak Giridharagopal / Puppet Labs / grim_radical

[Chris Price]

Also, would you mind sharing a bit more info about your setup?  I presume:

* You are installing from the puppetlabs apt repos?

* Your upgrade was from puppetdb 0.9.1 to 0.9.2?

[Peter Brown]

On 17 July 2012 03:15, Chris Price <ch...@puppetlabs.com> wrote:
> Also, would you mind sharing a bit more info about your setup? I presume:
>
> * You are installing from the puppetlabs apt repos?

yes

> * Your upgrade was from puppetdb 0.9.1 to 0.9.2?

and yes.

thanks again Deepak.
I shall add myself to the ticket.
It seems it's a problem with jetty because all the other settings seem
to be working fine.
I am guessing Jetty is part of the puppetdb install?
I tried installing jetty as well but still get the same results.

> On Monday, July 16, 2012 9:36:40 AM UTC-7, Deepak Giridharagopal wrote:
>>
>> On Sun, Jul 15, 2012 at 7:42 PM, Peter Brown <rendh...@gmail.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> I got the new version installed and am now having a strange issue.
>>> my puppetdb server seems to start but then dies after a few seconds
>>> with no logging that i can find.
>>> I tried setting the log level to DEBUG but still nothing.
>>>
>>> I tried running by hand and discovered it couldn't read it's config
>>> files so i fixed that.
>>> It also wants write access to /usr/share/puppetdb
>>> This seems like strange place it would need write access to.
>>> Is the really necessary?
>>> It is also ringing security bells in my head so I am a bit hesitant to
>>> give it write access without knowing why.
>>>
>>> My puppetdb is running on latest ubuntu with latest openjdk 6
>>
>>
>> I've created ticket http://projects.puppetlabs.com/issues/15536 to track
>> this; we'll take a look at that this iteration. Would you mind adding
>> yourself as a "watcher" on that ticket?
>>
>> Thanks for the report!
>>
>> deepak
>>
>> --
>> Deepak Giridharagopal / Puppet Labs / grim_radical
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/UF5A0GCai9UJ.

[Deepak Giridharagopal]

On Mon, Jul 16, 2012 at 5:13 PM, Peter Brown <rendh...@gmail.com> wrote:

On 17 July 2012 03:15, Chris Price <ch...@puppetlabs.com> wrote:
> Also, would you mind sharing a bit more info about your setup?  I presume:
>
> * You are installing from the puppetlabs apt repos?

yes

> * Your upgrade was from puppetdb 0.9.1 to 0.9.2?

and yes.

thanks again Deepak.
I shall add myself to the ticket.
It seems it's a problem with jetty because all the other settings seem
to be working fine.
I am guessing Jetty is part of the puppetdb install?
I tried installing jetty as well but still get the same results.

Indeed, that's part of the install. It's embedded in PuppetDB as a library. The ports thing is super-weird...we definitely didn't change them as part of the upgrade. In fact, your configs should have remained unchanged. How are you determining what ports it's listening on? What does /var/log/puppetdb/puppetdb.log say during startup?

Thanks!

[Peter Brown]

i am using netstat -nap and grepping for the PID to find out what
ports it's using.

I am not not getting any logging into /var/log/puppetdb/puppetdb.log
It was working yesterday but nothing there today.
I just tried running it under the official oracle jre 1.6.0.33 and same results.
I have also tried reinstalling from scratch and get the same results.

I am at a loss as to what to try next.

>
> Thanks!
> deepak
>
> --
> Deepak Giridharagopal / Puppet Labs / grim_radical
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

[Peter Brown]

I have gotten around the current problem by installing puppetdb on one
of me CentOS 6 nodes and it appears to be working as expected.
I even got it talking to my postgres database on my master node and
that seems to be working as well.
Now I can apply some new catalogs i need to roll out.

I am happy to help further but it's not as urgent at this stage.

Thanks guys.

Pete.

[Chris Price]

Hello Pete,

Do you have a reliable way to reproduce the port issue that you mentioned?  We'd like to troubleshoot it further but haven't found any way to reproduce it on our end yet.

If you're not getting log output in /var/log/puppetdb, can you please try the following?:

1. Edit the file /etc/puppetdb/conf.d/config.ini, and comment out the logging-config line.

2. Run this command:

java -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

This should allow you to see the output on the console as puppetdb is trying to start up.  If you can copy and paste that output for us, it may be helpful.

Thanks!

Chris

> puppet-users+unsubscribe@googlegroups.com.

[Peter Brown]

On 20 July 2012 11:20, Chris Price <ch...@puppetlabs.com> wrote:
> Hello Pete,
>
> Do you have a reliable way to reproduce the port issue that you mentioned?
> We'd like to troubleshoot it further but haven't found any way to reproduce
> it on our end yet.
>
> If you're not getting log output in /var/log/puppetdb, can you please try
> the following?:
>
> 1. Edit the file /etc/puppetdb/conf.d/config.ini, and comment out the
> logging-config line.
> 2. Run this command:
>
> java -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d

I tried that but only got INFO level logging
I edited the log4j.properties file and turned on debug level console
logging and got this output


>root@korolev:~# java -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.ini
2012-07-20 13:03:40,982 INFO [puppetlabs.jetty] Removing buggy
security provider SunPKCS11-NSS version 1.7
2012-07-20 13:03:46,916 INFO [cli.services] PuppetDB version 0.9.2
2012-07-20 13:03:48,490 INFO [cli.services] Starting broker
2012-07-20 13:03:51,892 INFO [cli.services] Starting 2 command
processor threads
2012-07-20 13:03:51,896 INFO [cli.services] Starting query server
2012-07-20 13:03:51,897 INFO [cli.services] Starting database
compactor (60 minute interval)
2012-07-20 13:03:52,015 INFO [cli.services] Beginning database compaction
org.eclipse.jetty.ssl.password : 2012-07-20 13:03:52,420 INFO
[cli.services] Finished database compaction
^C2012-07-20 13:08:27,976 INFO [cli.services] Shutdown request
received; puppetdb exiting.
root@korolev:~# java -jar /usr/share/puppetdb/puppetdb.jar services -c
/etc/puppetdb/conf.ini
2012-07-20 13:08:44,944 INFO [puppetlabs.jetty] Removing buggy
security provider SunPKCS11-NSS version 1.7
2012-07-20 13:08:50,008 INFO [main] [cli.services] PuppetDB version 0.9.2
2012-07-20 13:08:50,008 DEBUG [main] [bonecp.BoneCPDataSource] JDBC
URL = jdbc:postgresql://127.0.0.1:5432/puppetdb, Username = puppet,
partitions = 5, max (per partition) = 10, min (per partition) = 1,
helper threads = 3, idle max age = 60 min, idle test period = 240 min
2012-07-20 13:08:50,348 INFO [main] [cli.services] Starting broker
2012-07-20 13:08:54,677 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:54,744 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:54,748 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:54,748 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:54,748 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:54,757 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:55,652 INFO [main] [journal.Journal] ignoring zero
length, partially initialised journal data file: db-1.log number = 1 ,
length = 0
2012-07-20 13:08:56,247 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:56,247 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:56,250 DEBUG [main] [index.BTreeIndex] loading
2012-07-20 13:08:56,310 INFO [main] [cli.services] Starting 2 command
processor threads
2012-07-20 13:08:56,315 INFO [main] [cli.services] Starting query server
2012-07-20 13:08:56,316 INFO [main] [cli.services] Starting database
compactor (60 minute interval)
2012-07-20 13:08:56,351 INFO [clojure-agent-send-off-pool-3]
[cli.services] Beginning database compaction
2012-07-20 13:08:56,372 DEBUG [clojure-agent-send-off-pool-2]
[component.Container] Container
org.eclipse.jetty.server.Server@3c716932 +
SelectChannelConnector@localhost:8080 as connector
org.eclipse.jetty.ssl.password : 2012-07-20 13:08:56,415 INFO
[clojure-agent-send-off-pool-3] [cli.services] Finished database
compaction
2012-07-20 13:08:56,480 DEBUG [clojure-agent-send-off-pool-0]
[listener.DefaultMessageListenerContainer] Established shared JMS
Connection
2012-07-20 13:08:56,480 DEBUG [clojure-agent-send-off-pool-1]
[listener.DefaultMessageListenerContainer] Established shared JMS
Connection

from my limited knowledge it seems to me is is trying to open port
8080 but i can't see that port open at all.

netstat -nap |grep java tells me this (yes this is the only java
application running on the machine).

root@korolev:~# netstat -nap|grep java
tcp6 0 0 :::1099 :::*
LISTEN 11083/java
tcp6 0 0 :::34810 :::*
LISTEN 11083/java
tcp6 0 0 127.0.0.1:51702 127.0.0.1:5432
ESTABLISHED 11083/java
tcp6 0 0 127.0.0.1:51700 127.0.0.1:5432
ESTABLISHED 11083/java
tcp6 0 0 127.0.0.1:51701 127.0.0.1:5432
ESTABLISHED 11083/java
tcp6 0 0 127.0.0.1:51703 127.0.0.1:5432
ESTABLISHED 11083/java
unix 2 [ ] STREAM CONNECTED 6211563 11083/java

Hope that helps somewhat.

let me know if you need me to try anything else.

Pete.

>> > puppet-users...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/puppet-users?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/puppet-users/-/BoEHNeXgvsUJ.

>
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to

> puppet-users...@googlegroups.com.

[Chris Price]

OK, yes, the most important bit from that log output is this:

SelectChannelConnector@localhost:8080

If you hit port 8080 with a browser and/or via telnet, it should be open.  Ideally there should also be some output about opening port 8081 for SSL connections, but it's entirely possible that the packaging issues that you brought to our attention are the reason that that did not get configured properly.  We can dig into your jetty.ini file to investigate that if needed.

As for the netstat output: what arguments are you calling it with?  On my system, I get (filtered down to the relevant bits):

cprice@localhost ~/work/puppetdb $ netstat -ntpl

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name     

tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      24802/java      

tcp6       0      0 192.168.101.134:8081    :::*                    LISTEN      24802/java      

So, I can see port 8080 and 8081 open.  It's possible that there may be a few additional IPv4/tcp ports open that are owned by java processes--these could perhaps be attributed to ActiveMQ or our embedded database.  Would love to see how this output matches up with yours.

Thanks

Chris

>> > puppet-users+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/puppet-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/BoEHNeXgvsUJ.
>
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to

> puppet-users+unsubscribe@googlegroups.com.

[Peter Brown]

On 18 December 2012 15:50, Ryan Bond <drb...@gmail.com> wrote:

Was this issue ever resolved? I've been experiencing the same symptoms, ever since I switched over to postgresql.

Interestingly enough, when I run java -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.ini  instead of the init script, puppetdb fires up with no problems.

Any feedback would be appreciated.

I had the issue on you home testing setup and haven't looked at it for a while.

I use puppetdb with a postgresql backed on CentOS at work and it works pretty well.

 

Thanks,

Ryan

>> > puppet-users...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/puppet-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/BoEHNeXgvsUJ.
>

> To post to this group, send email to puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> puppet-users...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/jjxvnTGP7K4J.

To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

[Ryan Bond]

It seems that once I cut over to postgres the puppet user was no longer able to launch puppetdb.

The easy fix was changing the user to "root" in /etc/default/puppetdb...but I doubt this is something I'll want to leave in place.

[Chris Price]

Hi Ryan,

The puppetdb process shouldn't be trying to manipulate the postgres process in any way--it should only be communicating with it via a socket, and the authentication should be based on the postgres database user account rather than the system user account.  Are you seeing any errors in your puppetdb logs when you run it as the 'puppet' user?

Thanks!

Chris

[Ryan Bond]

Hi Chris,

When it runs as the pe-puppetdb user, it dies off with zero logging.  I have Debug enabled as well... 

Thanks

Ryan

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/exxHeQhbIUQJ.

[Michael Henry]

I ran across this same problem after upgrading.  Puppetdb would slowly run, then die without a single log.

I don't remember what version of puppetdb I upgraded from, but I upgraded to 1.0.5.

Check your /etc/puppetdb/conf.d  It's likely empty.

On ubuntu I solved the problem (a little overkill) by:
(note: will delete any non-postgres database you have)

$ apt-get purge puppetdb && rm -rf /etc/puppetdb /var/lib/puppetdb && apt-get install puppetdb

Once it was reinstalled, I configured it and it worked properly.  There's probably less of a sledgehammer approach to do it, but I'm working on a test box.  ;-)

I did also run across the issue with it listening on TCP *:1099 and *:55752 at Pete mentioned.  Created another thread for that problem as it's unrealted to puppetdb not starting.

-- Mike