Getting error "Waiting for zookeeper to be ready as status check returned" while doing seldon-up com

[rishi00...@gmail.com]

Hi,

      I am getting "Waiting for zookeeper to be ready as status check returned" message continuously while running seldon-up command. All pods are in running state as shown in attached image. I have tried "Calico" and "Flannel" kubernetes pods networks. But with both I am getting same error. How can I solve this issue?

Thanks,

Rishikesh

 

[rishi00...@gmail.com]

Hi,

   In Zookeeper logs I am getting following error

2018-03-13 08:45:22,994 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@155] - Failed to resolve address: zookeeper-3

java.net.UnknownHostException: zookeeper-3: Name or service not known

        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:922)

        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1316)

        at java.net.InetAddress.getAllByName0(InetAddress.java:1269)

        at java.net.InetAddress.getAllByName(InetAddress.java:1185)

        at java.net.InetAddress.getAllByName(InetAddress.java:1119)

        at java.net.InetAddress.getByName(InetAddress.java:1069)

        at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:148)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:407)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:426)

        at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)

        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:822)

2018-03-13 08:45:22,995 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000

But I have changed zokeeper port to 2184 in "zookeeper.json", "zookeeper.json,in", "seldon-up" files.  still it is trying to bind to 2181 port. How can I change the port?

[rishikesh bansod]

Hi Clive,

       I had another Kafka zookeeper process running on same servers, hence I changed zookeeper ports to 2184(from 2181). Is this causing issue?

Thanks & Regards,

Rishikesh 

On Tue, Mar 13, 2018 at 5:38 PM, Clive Cox <c...@seldon.io> wrote:

Hi,

 Why have you changed the zookeeper port? This should not need changing.

 

--
You received this message because you are subscribed to the Google Groups "Seldon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seldon-users+unsubscribe@googlegroups.com.
To post to this group, send email to seldon...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/seldon-users/efc35643-fe31-45c0-9e38-ea12f75f6d7d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[rishikesh bansod]

Hi Clive,

         I have again changed ports to 2181(default) but still getting the same error. What might be the issue?

Thanks & Regards,

Rishikesh

On Tue, Mar 13, 2018 at 6:26 PM, Clive Cox <c...@seldon.io> wrote:

I don't think you should need to change the port as everything will be relative to the kubernetes service which is "zookeeper". So not sure why there would be a port clash.

[rishikesh bansod]

Hi, 

     DNS pod is running  in the kubernetes cluster. But why zookeeper-3 service is not recognized? 

Thanks & Regards,

Rishikesh

On Tue, Mar 13, 2018 at 6:50 PM, Clive Cox <c...@seldon.io> wrote:

Are you sure DNS is working in your cluster? Maybe its not finding the zookeeper-3 service DNS name.

java.net.UnknownHostException: zookeeper-3: Name or service not known

[rishikesh bansod]

Hi Clive,

        When I run command "kubectl exec -i zookeeper2-96999fff8-hhq9v -- bash -c "echo srvr | nc localhost  2181" on pod I get message "This ZooKeeper instance is not currently serving requests" for every zookeeper pod. What might be issue. 

Thanks & Regards,

Rishikesh