Zookeeper Fails To Create Server in Local Cluster

[SH]

I've been try to test a Storm topology locally and run into a problem. When I started testing yesterday, everything worked wonderfully and data flowed through the system without any problems. When I came back to day and started things up, I noticed zookeeper was failing to either build the server or accept connections to it--keeping anything from processing--and I'm at a loss as to why.  

I'm running it on Windows 7 with a 3 node topology that reads off a queue, chunks some data, and writes it to a database. When I ran it yesterday, it worked flawlessly. Any suggestions on how to fix this would be appreciated! 

Log dump:

 

0    [main] INFO  backtype.storm.zookeeper  - Starting inprocess zookeeper at port 2181 and dir C:\Temp\/353584c1-1de3-4249-9ce0-a4b9a80eb905
109  [main] INFO  backtype.storm.daemon.nimbus  - Starting Nimbus with conf {"topology.fall.back.on.java.serialization" true, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "nimbus.monitor.freq.secs" 10, "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "storm.local.dir" "C:\\Temp\\/042bb485-26a5-43a7-94d7-383aaca594d7", "supervisor.worker.start.timeout.secs" 240, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "supervisor.enable" true, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.worker.childopts" nil, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "task.heartbeat.frequency.secs" 3, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx1024m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "nimbus.task.timeout.secs" 30, "drpc.invocations.port" 3773, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "topology.ackers" 1, "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
149  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
160  [main-SendThread(localhost:2181)] WARN  org.apache.zookeeper.ClientCnxn  - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Address family not supported by protocol family: connect
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1050)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1077)
791  [main-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none
814  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
817  [main-SendThread(localhost:2181)] WARN  org.apache.zookeeper.ClientCnxn  - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.SocketException: Address family not supported by protocol family: connect
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1050)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1077)
954  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
959  [main-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none
962  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
963  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
971  [main-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none
972  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
990  [main] INFO  backtype.storm.daemon.supervisor  - Starting Supervisor with conf {"topology.fall.back.on.java.serialization" true, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "nimbus.monitor.freq.secs" 10, "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "storm.local.dir" "C:\\Temp\\/59d8ffff-8489-486c-889e-6e5a7e328d37", "supervisor.worker.start.timeout.secs" 240, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "supervisor.enable" true, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.worker.childopts" nil, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "task.heartbeat.frequency.secs" 3, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "supervisor.slots.ports" (1 2 3), "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx1024m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "nimbus.task.timeout.secs" 30, "drpc.invocations.port" 3773, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "topology.ackers" 1, "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}
993  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
996  [main-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none
998  [main] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting

[Nathan Marz]

I'm not sure what the issue is, and it's strange that it was working before. First off, what version of Storm are you testing with? Another thing you can check is to whether there's another instance of ZK on your computer (or anything else on port 2181).

--
Twitter: @nathanmarz
http://nathanmarz.com

[SH]

I'm running version 0.7.1. and there's nothing else running on 2181. I've even looked for other versions of ZooKeeper and this is the only one I've got installed on this box. 

[SH]

For closure, I fixed the problem. It turned out my app server had some bad data on it and once I cleared that out, everything started working again. The zookeeper thing was a red herring. Thanks for the help!

[Ran S.]

Hi SH.

I am now getting the same error when trying to run the storm-starter project.

Any insight that can help me?

Thank you,

Ran

This message may contain confidential and/or privileged information. 

If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. 

If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

[sg]

I am also getting the error with the Storm-Starter using 0.7.1, anybody ever find an answer.

[sg]

I found my answer. 

My host file did not have an entry for localhost. Since the topology in the storm starter fires off with this conf...

96   [main] INFO  backtype.storm.daemon.nimbus  - Starting Nimbus with conf {"topology.fall.back.on.java.serialization" true, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "nimbus.monitor.freq.secs" 10, "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "storm.local.dir" "C:\\Users\\gormanst\\AppData\\Local\\Temp\\/f4beb7d6-9950-42d7-954a-89a08ef72ab7", "supervisor.worker.start.timeout.secs" 240, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "supervisor.enable" true, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.worker.childopts" nil, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "task.heartbeat.frequency.secs" 3, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx1024m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "nimbus.task.timeout.secs" 30, "drpc.invocations.port" 3773, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "topology.ackers" 1, "storm.cluster.mode" "local", "topology.optimize" true, "topology.max.task.parallelism" nil}

you can see it points to localhost as the zookeeper. That didn't resolve on my box to anything.