Puppetdb java Library

[Alex Wacker]

Hello,

Does anyone here happen to know if https://github.com/puppetlabs/puppetdb-javaclient still happens to work with puppetdb 1.5 via puppet enterprise? I have having difficulty making the connection from the java client.

Specifically

Jul 11, 2014 6:32:11 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFO: I/O exception (java.net.SocketException) caught when connecting to {s}->https://127.0.0.1:8081: Software caused connection abort: recv failed
Jul 11, 2014 6:32:11 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFO: Retrying connect to {s}->https://127.0.0.1:8081
Jul 11, 2014 6:32:11 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFO: I/O exception (java.net.SocketException) caught when connecting to {s}->https://127.0.0.1:8081: Software caused connection abort: recv failed
Jul 11, 2014 6:32:11 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFO: Retrying connect to {s}->https://127.0.0.1:8081
Jul 11, 2014 6:32:11 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFO: I/O exception (java.net.SocketException) caught when connecting to {s}->https://127.0.0.1:8081: Software caused connection abort: recv failed
Jul 11, 2014 6:32:11 AM org.apache.http.impl.client.DefaultHttpClient tryConnect

[Thomas Hallgren]

The current version of puppetdb-javaclient is designed to work with puppetdb 1.5. A couple of questions:

1. Have you tried using http instead of https and port 8080? Being able to use plain http was one of the features that were added when we made it compatible with 1.5.
2. If you're using Java 1.7 or later, can you try starting the JVM with -Djsse.enableSNIExtension=false ? That might help you get passed problems with servers that aren't completely honest about their hostnames.
3. Since you're using https (and hence ssl) it will be required that certificates and private key has been configured correctly. Are you using the same credentials setup in some other configuration that works?

[Alex Wacker]

Have had no luck with HTTP same general error:

Jul 15, 2014 1:59:01 PM org.apache.http.impl.client.DefaultHttpClient tryExecute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:8080: Software caused connection abort: recv failed
Jul 15, 2014 1:59:01 PM org.apache.http.impl.client.DefaultHttpClient tryExecute
INFO: Retrying request to {}->http://127.0.0.1:8080
Jul 15, 2014 1:59:01 PM org.apache.http.impl.client.DefaultHttpClient tryExecute
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://127.0.0.1:8080: The target server failed to respond
Jul 15, 2014 1:59:01 PM org.apache.http.impl.client.DefaultHttpClient tryExecute
INFO: Retrying request to {}->http://127.0.0.1:8080
Jul 15, 2014 1:59:01 PM org.apache.http.impl.client.DefaultHttpClient tryExecute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:8080: Connection reset
Jul 15, 2014 1:59:01 PM org.apache.http.impl.client.DefaultHttpClient tryExecute
INFO: Retrying request to {}->http://127.0.0.1:8080
org.apache.http.NoHttpResponseException: 127.0.0.1:8080 failed to respond

Just tried adding the recommended flag with no improvement.

key and cert information is setup like so:

        BasicAPIPreferences bap = new BasicAPIPreferences();
       
        bap.setPrivateKeyPEM(new File("C:\\Users\\test_user\\ssl_stuff\\private_keys\\server.pem"));
        bap.setCertPEM(new File("C:\\Users\\test_user\\ssl_stuff\\certs\\server.pem"));
        bap.setCaCertPEM(new File("C:\\Users\\test_user\\ssl_stuff\\ca\\ca.pem"));
        bap.setServiceHostname("127.0.0.1");
        bap.setAllowAllHosts(true);
        bap.setServicePort(8081);
       
        PuppetDBClient client = PuppetDBClientFactory.newClient(bap);

Should also mention that it is connecting to a puppet master inside of a virtualbox VM for testing using vagrant. All required ports should be forwarded.

[Thomas Hallgren]

Are you able to get something from the server using a windows correspondence to linux commands such as curl or wget? Or
perhaps in a browser? It really seems like a hard connection failure rather than a protocol error or authentication problem.

- thomas

[Alex Wacker]

I am able to get a response out of the standard puppet API (not puppetdb) via curl however puppetdb only gives such:

 

While something such as the puppet master API (while not exactly the puppetdb api) will at least respond to me

[Ken Barber]

> I am able to get a response out of the standard puppet API (not puppetdb)
> via curl however puppetdb only gives such:
>
> While something such as the puppet master API (while not exactly the
> puppetdb api) will at least respond to me

I can't help but notice you are connecting from a windows box to
localhost (127.0.0.1) in your tests. Is PuppetDB running on a Windows
box or something? Because this is not a supported nor tested
configuration sorry, and I wouldn't be surprised if its very broken.
PuppetDB only works on Linux and some Unix OS's generally:
http://docs.puppetlabs.com/puppetdb/2.1/#system-requirements.

If its not running on Windows, you'll need to check your puppetdb.log
(usually in /var/log/puppetdb) when that connection attempt is made.
Connection reset is generally caused by an application fault of some
kind and it should be followed up with an exception in the logs.

ken.

[Alex Wacker]

puppetdb is running on a RHEL box inside of virtual box.

required ports a forwarded between the host and VM

[Alex Wacker]

Also not seeing anything related to the failure to connect in the logs

On Wednesday, July 16, 2014 9:51:48 AM UTC-4, Ken Barber wrote:

[Ken Barber]

So a connection reset is pretty fatal, and it makes me feel like your
PuppetDB instance isn't even working today. We should probably start
by determining your PuppetDB instance is working at all.

Is this a production instance? Is anything talking to it correctly at
all? What happens when you restart it for example, any change? You
should be actively seeing commands being submitted to that PuppetDB in
its puppetdb.log as so:

2014-07-13 19:00:21,775 INFO [c.p.p.command]
[a42497d2-7b96-49ae-bcaf-5c9b2020cac2] [replace facts] puppetdb1.vm
2014-07-13 19:00:21,961 INFO [c.p.p.command]
[adadbf63-c5b5-4e81-bdcf-59d187d53451] [replace catalog] puppetdb1.vm
2014-07-13 19:00:22,207 INFO [c.p.p.command]
[7305367c-1c9d-4565-b45d-9a61b6ef88ee] [store report] puppet v3.6.2 -
puppetdb1.vm

Whats more, curl's _should_ be working. Can you provide the results
(in text please, not a screenshot) of running the following:

root@puppetdb1:~# netstat -anp | grep java
tcp6 0 0 :::8080 :::*
LISTEN 10823/java
tcp6 0 0 :::8081 :::*
LISTEN 10823/java
tcp6 0 0 192.168.182.132:53125 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53126 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53124 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53123 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53118 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53121 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53130 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53120 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53117 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53127 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53129 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53119 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53128 192.168.182.196:5432
ESTABLISHED 10823/java
tcp6 0 0 192.168.182.132:53122 192.168.182.196:5432
ESTABLISHED 10823/java
unix 2 [ ] STREAM CONNECTED 15655 10823/java
unix 2 [ ] STREAM CONNECTED 17083 10823/java
root@puppetdb1:~# netstat -anp | grep 8080
tcp6 0 0 :::8080 :::*
LISTEN 10823/java

Then taking the process id (10823):

root@puppetdb1:~# ps auxw | grep 10823
puppetdb 10823 6.7 10.5 747000 217344 ? Ssl 19:04 0:22
/usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xmx192m
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof
-Djava.security.egd=file:/dev/urandom -cp
/usr/share/puppetdb/puppetdb.jar clojure.main -m
com.puppetlabs.puppetdb.core services -c /etc/puppetdb/conf.d --debug

Also ... can you tell me exactly what version of PuppetDB you are
running? Normally you can query this with something like:

root@puppetdb1:~# curl 'http://localhost:8080/v3/version'
{
"version" : "2.1.0"
}

But since your curl requests are failing, perhaps we can gather it
from the logs:

root@puppetdb1:~# grep 'PuppetDB version' /var/log/puppetdb/puppetdb.log
2014-07-13 19:00:03,333 INFO [c.p.p.c.services] PuppetDB version 2.1.0

Please perform the above, and show full working as I've done, that
helps me understand the full results.

ken.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/08ff0fb4-b1b2-4b52-8b61-e2696cbebac2%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Alex Wacker]

I do have the ability to query it locally inside the VM, appears to only have issues querying externally from the host machine. Though the puppetmaster API appears to accept my queries. Does external access to puppetdb need to be granted somewhere? I saw there is a whitelist didn't have much luck.

[root@master pe-httpd]# curl -X GET http://localhost:8080/v3/version
{
  "version" : "1.5.1-pe"

[Ken Barber]

In your configuration:

/etc/puppetdb/conf.d/jetty.ini

There are two settings that affect this:

host =
ssl-host =

host defaults to 127.0.0.1 if its not set, so you must set it to
something like 0.0.0.0 and restart PuppetDB to get it listening
correctly on all interfaces. Same applies to ssl-host if you need to
have that listening externally.

ken.

> https://groups.google.com/d/msgid/puppet-dev/c4d2f098-03e4-41e7-9f34-1f4e11de68bb%40googlegroups.com.

[Alex Wacker]

I appreciate the help. I was able to get connected

One final question. In the example in the readme below methods such as 'eq', 'or' and 'and' are called. Are these defined anywhere in the library as they are missing for me currently?

List<Resource> classes = client.getResources(
    or(eq(Resource.TITLE, "main"), eq(Resource.TITLE, "Settings")));

[Thomas Hallgren]

On 2014-07-17 19:06, Alex Wacker wrote:
> I appreciate the help. I was able to get connected
>
> One final question. In the example in the readme below methods such as
> 'eq', 'or' and 'and' are called. Are these defined anywhere in the
> library as they are missing for me currently?
>
> List<Resource> classes = client.getResources(
> or(eq(Resource.TITLE, "main"), eq(Resource.TITLE, "Settings")));
>

They are defined in com.puppetlabs.puppetdb.javaclient.query.Query. In
order to write code like in the example, try using:

import static com.puppetlabs.puppetdb.javaclient.query.Query.*;

- thomas

[Alex Wacker]

Appreciate that, I appeared to miss it in the docs.

In regards to the getNodeFactsMethod() , I am having a hard time understanding what Parameters<Node> is supposed to contain.

If I am attempting to get all the facts for a specific node as a List<Fact> could you provide me a quick example on what this node statement is supposed to contain?

I attempted basing it off of what other similar methods in the docs but did not have much luck.

[Alex Wacker]

Sorry I forgot to include the current error when attempting to pull with the method:

com.puppetlabs.puppetdb.javaclient.model.Resource cannot be cast to com.puppetlabs.puppetdb.javaclient.model.Fact

[Thomas Hallgren]

Hi Alex,

First of all, you found a bug. The parameter and return type for the getNodeFacts() method were both wrong. I fixed that and pushed a new version (0.1.6) of the API code to our Nexus instance.

The getNodeFacts method calls:

 /v3/nodes/<node>/facts

The call is documented here:

 http://docs.puppetlabs.com/puppetdb/2.1/api/query/v3/nodes.html#get-v3nodesnodefacts

It's a shortcut to the /v3/facts endpoint with a query ["=", "certname", <"NODE">. In addition to this query, the v3/facts endpoint will also accept a query for  "name" or "value". You should hence be able to write expressions like:

 result = getNodeFacts(eq(Fact.NAME, "some name"), "some certname");

or

 result = getNodeFacts(eq(Fact.VALUE, "some value"), "some certname");

HTH,
- thomas

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/f599b660-d175-40b5-afca-ad1c15bc6675%40googlegroups.com.