Stardog-2.2.2 Cluster Deployment

[samir]

Hi all,

I am trying to deploy a Stardog-2.2.2 cluster ( using starman) on three machines (1 master + 2 followers),
whose IPs are inserted in the .starmanconfig file at the master machine.
I do bootstrap and svm install successfuly, but then the deployment process  never ends!!
Am I missing some additional configuration data or something else?!

Here are the last lines from the pallet.log file:

2014-11-01 16:14:21,819 DEBUG [operate-13] p.script-builder sudo-cmd-for {:username "snjoum", :sudo-password nil, :sudo-user nil, :password nil, :no-sudo nil}
2014-11-01 16:14:21,820 DEBUG [operate-13] p.script-builder prefix /usr/bin/sudo -n
2014-11-01 16:14:21,821 DEBUG [operate-13] p.s.execute ssh-script-on-target command {:env-cmd "/usr/bin/env", :env nil, :env-fwd [:SSH_AUTH_SOCK], :prefix ["/usr/bin/sudo" "-n"], :execv ("/bin/bash" "/tmp/palletUmupZ")}
2014-11-01 16:14:21,935 DEBUG [operate-13] p.execute 192.168.98.130   <== Archive:  /opt/stardog/1/runtime/stardog-csr1.zip
2014-11-01 16:14:21,935 DEBUG [operate-13] p.execute 192.168.98.130   <== replace /opt/stardog/1/runtime/stardog-2.2.2/bin/install-service.bat? [y]es, [n]o, [A]ll, [N]one, [r]ename:

In advance thank you very much for your response,
Best regards,
Samir

[Mike Grove]

See the above line, it looks like the node is waiting for user input on the console for the copy operation.  

Cheers,

Mike

 

In advance thank you very much for your response,
Best regards,
Samir

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[samir]

Hello Mike,

I thought so, but it accepted no input!
Any way, somehow I have got it deployed, without figuring out where was the error! :-(
Now, when I try to start the cluster, I have further errors:-
 - ZK is running only on the master node,
 - failed to start the proxy on all nodes!

START CLUSTER - Starting cluster:  1
Starting components in cluster...
> Starting Zookeeper: 192.168.98.130
> Starting Zookeeper: 192.168.98.131
> Starting Zookeeper: 192.168.98.132
> Zookeeper running: 192.168.98.130
> Starting Stardog: 192.168.98.130:6000
> Starting Stardog: 192.168.98.131:6000
> Starting Stardog: 192.168.98.132:6000
> Stardog running: 192.168.98.130:6000
> Stardog running: 192.168.98.131:6000
> Stardog running: 192.168.98.132:6000
> Starting Proxy: 192.168.98.130:5820
> Starting Proxy: 192.168.98.131:5820
> Starting Proxy: 192.168.98.132:5820
> Failed to start Proxy: 192.168.98.131. See /opt/stardog/1/data/proxy.log at 192.168.98.131 for more details.
> Failed to start Proxy: 192.168.98.132. See /opt/stardog/1/data/proxy.log at 192.168.98.132 for more details.
> Failed to start Proxy: 192.168.98.130. See /opt/stardog/1/data/proxy.log at 192.168.98.130 for more details.
Not all components were started, please see the logs.

In proxy.log, there are errors for all nodes. Here is one of them:

WARNING: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

Nov 03, 2014 2:32:16 PM org.apache.zookeeper.ClientCnxn$SendThread logStartConnect
INFO: Opening socket connection to server srv3.localhost/192.168.98.132:2180. Will not attempt to authenticate using SASL (unknown error)
Nov 03, 2014 2:32:16 PM org.apache.zookeeper.ClientCnxn$SendThread run
WARNING: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

KeeperErrorCode = ConnectionLoss for /participants

Any ideas please?

Thanks & cheers,
Samir

[Fernando Hernandez]

Samir,

On Mon, Nov 3, 2014 at 8:53 AM, samir <samir...@gmail.com> wrote:

Hello Mike,

I thought so, but it accepted no input!
Any way, somehow I have got it deployed, without figuring out where was the error! :-(
Now, when I try to start the cluster, I have further errors:-
 - ZK is running only on the master node,
 - failed to start the proxy on all nodes!

The proxy couldn't start because zookeeper didn't start properly. Are there any logs in the nodes where zookeeper couldn't start? It should be in the same directory as the proxy logs.

Cheers,

Fernando

[samir]

Hello Fernando,

Should any ZK configuration data be added to .starmanconfig?

Here is the zookeeper log file:

INFO: Reading configuration from: /opt/stardog/2/data/zookeeper.properties
Nov 03, 2014 2:24:45 PM org.apache.zookeeper.server.quorum.QuorumPeerConfig parseProperties
INFO: Defaulting to majority quorums
Nov 03, 2014 2:24:45 PM org.apache.zookeeper.server.DatadirCleanupManager <init>
INFO: autopurge.snapRetainCount set to 3
Nov 03, 2014 2:24:45 PM org.apache.zookeeper.server.DatadirCleanupManager <init>
INFO: autopurge.purgeInterval set to 0
Nov 03, 2014 2:24:45 PM org.apache.zookeeper.server.DatadirCleanupManager start
INFO: Purge task is not scheduled.
Exception in thread "Thread-1" java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
        at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:50)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:122)
        at com.complexible.stardog.pack.cli.ZkStart$1.run(ZkStart.java:119)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.jmx.HierarchyDynamicMBean
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 4 more

Thank you very much,

Samir

[Fernando Hernandez]

It looks like not all server files were installed; does that server have less jars than the other ones?

[samir]

Under "/opt/stardog/2/runtime/stardog-2.2.2/server/pack/"  each one has 37 jars!

[Fernando Hernandez]

On Mon, Nov 3, 2014 at 10:36 AM, samir <samir...@gmail.com> wrote:

Under "/opt/stardog/2/runtime/stardog-2.2.2/server/pack/"  each one has 37 jars!

Could you attach the pallet.log file? Stardog uses more jars than the ones in the 'pack' directory.

Also, it would probably be easier to re-deploy the cluster than repairing a deployment that went wrong. We're still working on improving these kinds of deployment errors.  If you deployed to a system that was already running, make sure to wipe the /opt/stardog directory before you try again.

Cheers,

Fernando

[samir]

The pallet.log file is too big to be attached. May I post some part of it?

Could you please post the contents that .starmanconfig file should have? I will then repeat  the whole process.

Thanks & cheers,

Samir

[Fernando Hernandez]

On Mon, Nov 3, 2014 at 4:52 PM, samir <samir...@gmail.com> wrote:

The pallet.log file is too big to be attached. May I post some part of it?

Could you try attaching a zipped version?

 

Could you please post the contents that .starmanconfig file should have? I will then repeat  the whole process.

Other than the IPs described in [1] there are no other required configurations.

[1] http://docs.stardog.com/cluster/#sd-Setting-Up-Stardog-Cluster-with-Starman

Fernando

[samir]

Hi,

Yes, I tried with the zipped file, but it was too big to be sent to the group!! 

[Fernando Hernandez]

Samir,

I took a look at the logs you sent along – thanks for that by the way – and it looks like the user you're executing Starman with doesn't have password-less [1] sudo in the remote machines. Starman requires that the user has password-less SSH access and password-less sudo in the remote machines (including the local machine if it's in the deployment list) in order to do some admin operations – see "Other Servers" in the documentation [2].

I presume that ZK is only starting in one of the machines which is your local machine, is that correct? If that's so, it probably didn't require you to input the password for root access sometimes. However, for the other machines – the remote ones –  it's throwing errors such as:

2014-11-05 11:10:36,618 DEBUG [operate-14] p.execute 192.168.98.131   <== mkdir: cannot create directory ‘/opt/stardog/101/data/zookeeper’: Permission denied

2014-11-05 11:10:36,618 DEBUG [operate-14] p.execute 192.168.98.131   <== chown: cannot access ‘/opt/stardog/101/data/zookeeper’: No such file or directory

2014-11-05 11:10:36,618 DEBUG [operate-14] p.execute 192.168.98.131   <== chmod: cannot access ‘/opt/stardog/101/data/zookeeper’: No such file or directory

2014-11-05 11:10:36,618 DEBUG [operate-14] p.execute 192.168.98.131   <== /tmp/palletRpI0R: line 3: /opt/stardog/101/data/zookeeper/myid: No such file or directory

2014-11-05 11:10:36,618 DEBUG [operate-14] p.execute 192.168.98.131   <== chown: cannot access ‘/opt/stardog/101/data/zookeeper/myid’: No such file or directory

2014-11-05 11:10:36,619 DEBUG [operate-14] p.execute 192.168.98.131   <== chmod: cannot access ‘/opt/stardog/101/data/zookeeper/myid’: No such file or directory

2014-11-05 11:10:36,619 WARN [operate-14] p.s.transport 192.168.98.131 Exit status  : 1

2014-11-05 11:10:36,619 DEBUG [operate-12] p.execute 192.168.98.132   <== mkdir: cannot create directory ‘/opt/stardog/101/data/zookeeper’: Permission denied

2014-11-05 11:10:36,619 DEBUG [operate-12] p.execute 192.168.98.132   <== chown: cannot access ‘/opt/stardog/101/data/zookeeper’: No such file or directory

2014-11-05 11:10:36,619 DEBUG [operate-12] p.execute 192.168.98.132   <== chmod: cannot access ‘/opt/stardog/101/data/zookeeper’: No such file or directory

Please make those changes to the user that executes Starman and let us know if that works.

[1] http://askubuntu.com/questions/192050/how-to-run-sudo-command-with-no-password/192062#192062

[2] http://docs.stardog.com/cluster/#sd-Setting-Up-Stardog-Cluster-with-Starman

Cheers,

Fernando

On Mon, Nov 3, 2014 at 4:52 PM, samir <samir...@gmail.com> wrote:

[samir njoum]

Hello Fernando,

Thanks alot for checking those files.

I have set password-less ssh and sudo on all machines before i ran Starman.

I agree with you that those errors give such an indication, but I am sure that I have done everything as desired.

Please check if there are any other possibilities and I will double check all three machines again.

Thanks you very much,

Samir 

[Fernando Hernandez]

Just to add that the local machine on which ZK started is actually the master machine on which I do everything.

Is that correct that the master machine is one of the three machines of which the cluster consist (one master + two follower)?

That ZK that started is in your local machine doesn't necessarily mean that's the master. Same goes for Stardog.

 

---------- Forwarded message ----------
From: samir njoum <samir...@gmail.com>
Date: Wed, Nov 5, 2014 at 3:52 PM
Subject: Re: [stardog-users] Stardog-2.2.2 Cluster Deployment
To: sta...@clarkparsia.com

Hello Fernando,

Thanks alot for checking those files.

I have set password-less ssh and sudo on all machines before i ran Starman.

I agree with you that those errors give such an indication, but I am sure that I have done everything as desired. 

Please check if there are any other possibilities and I will double check all three machines again.

It's not very clear to me what else would be causing this issue. I'll see if I can reproduce locally.

In the meantime, you could try creating a cluster with VirtualBox or Amazon EC2 in order to get you started.

Cheers,

Fernando

[Fernando Hernandez]

On Wed, Nov 5, 2014 at 5:18 PM, Fernando Hernandez <fern...@clarkparsia.com> wrote:

Just to add that the local machine on which ZK started is actually the master machine on which I do everything.

Is that correct that the master machine is one of the three machines of which the cluster consist (one master + two follower)?

That ZK that started is in your local machine doesn't necessarily mean that's the master. Same goes for Stardog.

 

---------- Forwarded message ----------
From: samir njoum <samir...@gmail.com>
Date: Wed, Nov 5, 2014 at 3:52 PM
Subject: Re: [stardog-users] Stardog-2.2.2 Cluster Deployment
To: sta...@clarkparsia.com

Hello Fernando,

Thanks alot for checking those files.

I have set password-less ssh and sudo on all machines before i ran Starman.

I agree with you that those errors give such an indication, but I am sure that I have done everything as desired. 

Please check if there are any other possibilities and I will double check all three machines again.

Did any of the machines had a user or group named 'stardog' before creating a cluster with Starman?

Starman creates a group named stardog in order to associate all created files to that group and associate your login user to that group as well. The problem is that if there's a group with that name before running Starman (using the Other Servers mechanism) it will not be able to associate your current user to that group, hence not being able to run ZK or others.

[samir]

Hello Fernando,

I have checked the machines. They are configured with passwd-less ssh and sudo.

In order to see if those errors are the only reason, I have changed the permissions of /opt/stardog/* and tried again.

And URRRRAAAAAAAAAAA :-)

ZK started on all the machines, but NOT the proxy as well!

About the existing user or group Stardog: 
Yes, you are right. I have a user and a group named Stardog. I have droped them on all the machines and tried again, but the proxy has not started again.

I am cleaning now everything and will repeat the whole thing to post new and clear log files.

Thanks and kindest regards,

Samir

[samir]

Hi,

After the new deployment ZK has successfully started, but not the proxy:

START CLUSTER - Starting cluster:  101

Starting components in cluster...
> Starting Zookeeper: 192.168.98.130
> Starting Zookeeper: 192.168.98.131
> Starting Zookeeper: 192.168.98.132
> Zookeeper running: 192.168.98.130

> Zookeeper running: 192.168.98.131
> Zookeeper running: 192.168.98.132

> Starting Stardog: 192.168.98.130:6000
> Starting Stardog: 192.168.98.131:6000
> Starting Stardog: 192.168.98.132:6000
> Stardog running: 192.168.98.130:6000
> Stardog running: 192.168.98.131:6000
> Stardog running: 192.168.98.132:6000
> Starting Proxy: 192.168.98.130:5820
> Starting Proxy: 192.168.98.131:5820
> Starting Proxy: 192.168.98.132:5820

> Failed to start Proxy: 192.168.98.132. See /opt/stardog/101/data/proxy.log at 192.168.98.132 for more details.
> Failed to start Proxy: 192.168.98.130. See /opt/stardog/101/data/proxy.log at 192.168.98.130 for more details.
> Failed to start Proxy: 192.168.98.131. See /opt/stardog/101/data/proxy.log at 192.168.98.131 for more details.
>>>> All processes running

The proxy error is about:

WARNING: Connection attempt unsuccessful after 64365 (greater than max timeout of 60000). Resetting connection and trying again with a new connection.
Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ClientCnxn$SendThread logStartConnect
INFO: Opening socket connection to server srv1.localhost/192.168.98.130:2180. Will not attempt to authenticate using SASL (unknown error)
Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ZooKeeper close
INFO: Session: 0x0 closed
Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ZooKeeper <init>
INFO: Initiating client connection, connectString=stardog-2:2180,stardog-1:2180,stardog-3:2180 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@ef28a30
Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ClientCnxn$EventThread run
INFO: EventThread shut down
Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ClientCnxn$SendThread logStartConnect

INFO: Opening socket connection to server srv3.localhost/192.168.98.132:2180. Will not attempt to authenticate using SASL (unknown error)

Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ClientCnxn$SendThread run

WARNING: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ClientCnxn$SendThread logStartConnect
INFO: Opening socket connection to server srv1.localhost/192.168.98.130:2180. Will not attempt to authenticate using SASL (unknown error)
Nov 06, 2014 2:04:25 PM org.apache.zookeeper.ClientCnxn$SendThread run

WARNING: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

KeeperErrorCode = ConnectionLoss for /participants

Please check the new uploaded logs if needed.

Have a nice day,

Samir

The new logs are uploaded.

[Fernando Hernandez]

Hi Samir,

I have checked the machines. They are configured with passwd-less ssh and sudo.

In order to see if those errors are the only reason, I have changed the permissions of /opt/stardog/* and tried again.

And URRRRAAAAAAAAAAA :-)

ZK started on all the machines, but NOT the proxy as well!

About the existing user or group Stardog: 
Yes, you are right. I have a user and a group named Stardog. I have droped them on all the machines and tried again, but the proxy has not started again.

I took a look at the logs and I found that even though you changed the permissions to /opt/stardog/* probably the root user still owns them; e.g.:

2014-11-06 14:00:29,378 DEBUG [operate-12] p.execute 192.168.98.132   <== chown: changing ownership of ‘/opt/stardog/101/data/zookeeper’: Operation not permitted

2014-11-06 14:00:29,379 DEBUG [operate-12] p.execute 192.168.98.132   <== chown: changing ownership of ‘/opt/stardog/101/data/zookeeper/myid’: Operation not permitted

2014-11-06 14:00:29,380 DEBUG [operate-15] p.execute 192.168.98.131   <== chown: changing ownership of ‘/opt/stardog/101/data/zookeeper’: Operation not permitted

2014-11-06 14:00:29,380 DEBUG [operate-15] p.execute 192.168.98.131   <== chown: changing ownership of ‘/opt/stardog/101/data/zookeeper/myid’: Operation not permitted

2014-11-06 14:00:29,381 DEBUG [operate-13] p.execute 192.168.98.130   <== chown: changing ownership of ‘/opt/stardog/101/data/zookeeper’: Operation not permitted

2014-11-06 14:00:29,381 DEBUG [operate-13] p.execute 192.168.98.130   <== chown: changing ownership of ‘/opt/stardog/101/data/zookeeper/myid’: Operation not permitted

This is what I think you should do: if there's already a stardog user in those machines, make sure that there is also a group named stardog and add *your* user (snjoum) to that group. This is because Starman will attempt to chmod and chown all those Stardog files and assign them to the stardog *group*, then, when you start up the cluster, it's your user who is in charge of orchestrating the start-up. You can see the relevant code for that here [1].

We'll see how we can make the test-for-user in Starman smarter than this in order to avoid situations like this.

 

[1] https://github.com/clarkparsia/starman/blob/0.6.6/src/starman/pallet/pallet.clj#L119-L126

Cheers,

Fernando

[samir njoum]

Hello Fernando,

I am still here :-)

I had some problems and have moved to a new environment. It's much better.

I have installed everything and deployed a new cluster.

Again I had alot of problems. The last problem is with the proxy. No one could started.

Here is the error log wich is repeated many times for all nodes:

[

Nov 16, 2014 1:10:02 PM org.apache.zookeeper.ClientCnxn$SendThread logStartConnect
INFO: Opening socket connection to server stardog-3/134.21.73.206:2180. Will not attempt to authenticate using SASL (unknown error)
Nov 16, 2014 1:10:02 PM org.apache.zookeeper.ClientCnxn$SendThread run

WARNING: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

Nov 16, 2014 1:10:02 PM org.apache.zookeeper.ClientCnxn$SendThread logStartConnect
INFO: Opening socket connection to server stardog-2/134.21.73.205:2180. Will not attempt to authenticate using SASL (unknown error)
Nov 16, 2014 1:10:02 PM org.apache.zookeeper.ClientCnxn$SendThread run

WARNING: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

]

Please take a look and come back with a solution :-)

Thanks & best regards,

Samir

[Fernando Hernandez]

It's hard to tell what's wrong without more details about your environment and the issue. Could you also share the logs?

Cheers,

Fernando