Nimbus not starting up

[Sanjai Nair]

Hi,

I have Zookeeper running on 2181 (ports are opened):

hduser@hduser-laptop:~/.storm$ netstat -a | grep -e "2181"

tcp        0      0 *:2181                  *:*                     LISTEN     

But when I tried to kick start nimbus, then I got the below exception in the nimbus log file:

2013-05-17 19:17:12 ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@53e6978d

2013-05-17 19:17:12 ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-05-17 19:17:12 ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session

2013-05-17 19:17:13 ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13eb48fd6420000, negotiated timeout = 20000

2013-05-17 19:17:13 zookeeper [INFO] Zookeeper state update: :connected:none

2013-05-17 19:17:13 ZooKeeper [INFO] Session: 0x13eb48fd6420000 closed

2013-05-17 19:17:13 CuratorFrameworkImpl [INFO] Starting

2013-05-17 19:17:13 ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@11c0b8a0

2013-05-17 19:17:13 ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181

2013-05-17 19:17:13 ClientCnxn [INFO] EventThread shut down

2013-05-17 19:17:13 ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session

2013-05-17 19:17:13 ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13eb48fd6420001, negotiated timeout = 20000

2013-05-17 19:17:13 nimbus [INFO] Starting Nimbus server...

2013-05-17 19:22:04 TNonblockingServer [WARN] Got an IOException in internalRead!

java.io.IOException: Connection reset by peer

at sun.nio.ch.FileDispatcher.read0(Native Method)

at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)

at sun.nio.ch.IOUtil.read(IOUtil.java:175)

at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)

at org.apache.thrift7.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)

at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.internalRead(TNonblockingServer.java:669)

at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:458)

at org.apache.thrift7.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)

at org.apache.thrift7.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)

at org.apache.thrift7.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)

I have ensured that even the port 6627 is left unblocked. I have placed the storm.yaml file inside storm/conf folder as well inside .storm folder in my home root folder with al permissions.

Help appreciated.

Cheers

[Viral Bajaria]

2013-05-17 19:17:13 nimbus [INFO] Starting Nimbus server...

2013-05-17 19:22:04 TNonblockingServer [WARN] Got an IOException in internalRead!

Looks like your error happens after almost 5 mins of running. Also the error seems to be with the thrift server and not with zookeeper since the connection established successfully.

Did you try doing submitting a topology or did nimbus just die on its own ?

Can you copy your config.yaml ?

-Viral

[Sanjai Nair]

Did you mean storm.yaml ?

storm.zookeeper.servers:

     - "localhost"

storm.zookeeper.port: 2181

nimbus.host: "localhost"

storm.local.dir: "/home/hduser/storm_dir"

java.library.path: "/usr/local/lib"

Cheers

[Viral Bajaria]

Yeah meant to ask about storm.yaml

Did you try to submit a topology when this exception occurred ?

-Viral

--

You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sanjai Nair]

Nope. I just tried to start up storm and did not try to submit any topology.

[Viral Bajaria]

Nope. I just tried to start up storm and did not try to submit any topology.

Strange that you have a 5-minute gap between the nimbus starting and the error popping up.

Can you try starting it up again and immediately seeing if port 6627 shows up in your netstat ?

What's your environment i.e. OS, storm version ?

[Sanjai Nair]

Yea, I did notice unusual delay when i issue "storm nimbus" from the command line.

I tried to start nimbus again (no topology submitted), and then I checked running processes:

hduser@hduser-laptop:~/zookeeper/zook/bin$ netstat -a | grep "LISTEN"

tcp        0      0 *:38395                 *:*                    LISTEN     

tcp        0      0 *:6627                  *:*                     LISTEN     

tcp        0      0 *:2181                  *:*                     LISTEN 

Let me know if you have any more questions.

Cheers

[Sanjai Nair]

Guest OS: Ubuntu 10.04 (Virtual box)

TCPv6 is disabled

Storm version: 0.8.1

[Sanjai Nair]

Hi,

These are from zookeeper.log file:

2013-05-17 21:31:30,227 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:37093

2013-05-17 21:31:30,229 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer

2013-05-17 21:31:30,250 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:37093 (no session established for client)

2013-05-17 21:33:55,689 [myid:] - INFO  [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x13eb48fd6420005

2013-05-17 21:33:55,789 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:37077 which had sessionid 0x13eb48fd6420005

2013-05-17 21:34:46,759 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:37095

2013-05-17 21:34:46,774 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:37095; will be dropped if server is in r-o mode

2013-05-17 21:34:46,775 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:37095

2013-05-17 21:34:46,868 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@595] - Established session 0x13eb48fd6420006 with negotiated timeout 20000 for client /127.0.0.1:37095

2013-05-17 21:34:46,906 [myid:] - INFO  [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x13eb48fd6420006

2013-05-17 21:34:46,922 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception

EndOfStreamException: Unable to read additional data from client sessionid 0x13eb48fd6420006, likely client has closed socket

        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)

        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

        at java.lang.Thread.run(Thread.java:662)

2013-05-17 21:34:46,925 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:37095 which had sessionid 0x13eb48fd6420006

2013-05-17 21:34:46,944 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:37096

2013-05-17 21:34:46,972 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:37096; will be dropped if server is in r-o mode

2013-05-17 21:34:46,974 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:37096

2013-05-17 21:34:46,983 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@595] - Established session 0x13eb48fd6420007 with negotiated timeout 20000 for client /127.0.0.1:37096

Hope this helps too.

Cheers

[Sanjai Nair]

Solved.

Add "nimbus" to /etc/hosts file as alias to 127.0.0.1 and localhost.

[Sanjai Nair]

FYI

On Friday, May 17, 2013 3:55:05 PM UTC-3, Sanjai Nair wrote: