runtime.properties set up (ZK errors)

[Jay Wang]

Hey guys,

Still trying to get druid to run. 

I have MySQL and ZooKeeper. ZooKeeper seems to be running properly. 

When I run: ./zkServer.sh start, I get "Starting zookeeper ... STARTED"

My runtime.properties is the base one from the wiki, with some minor modifications. 

I switched out: 

# **CHANGED** druid.database.connectURI=

# **CHANGED** druid.database.user=user

# **CHANGED** druid.database.password=pass

with the appropriate connectURI, user, password. 

Right now, druid.zk.service.host=127.0.0.1:2181

and "druid.host=localhost, druid.port=8080"

I didn't touch any of the zookeeper paths in runtime.properties. 

So here's the error, I try to fire up Master with: java -Duser.timezone=UTC -Dfile.encoding=UTF-8 -cp compute-config/:druid-services/target/druid-services-0.2.0-SNAPSHOT-selfcontained.jar com.metamx.druid.http.MasterMain

I can't seem to connect to Zookeeper. Here's the message I'm getting: 

"2013-02-22 06:59:46,682 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181

2013-02-22 06:59:46,684 WARN [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)"

Does anyone know how to proceed? I am running this while zookeeper server has been started (I'm not sure that really matters though...) 

My guess is that my some of my configs are off? I read about using DruidSetup, but didn't get anywhere with that either. 

Another issue could be that my ZooKeeper client isn't working properly. 

When I try to connect to Zookeeper using the Java method: bin/zkCli.sh -server 127.0.0.1:2181

I get errors: 

/Users/zookeeper-3.4.5/bin/zkEnv.sh: line 79: [: too many arguments

Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

at org.apache.zookeeper.ZooKeeperMain.<clinit>(ZooKeeperMain.java:50)

Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory

at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

... 1 more

Sorry for the long post. Any insight would be much appreciated. 

Thanks!

Jay

[Eric Tschetter]

Jay,

Hrm, that is a stumper.  You should be able to connect with the zookeeper tools first.  Hrm, can you try telnetting to the port?  you should be able to do

telnet 127.0.0.1 2181

And then if it connects, type

ruok

and it *should* give you some stuff back.

--Eric

Jay

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/druid-development/-/Qt5QLm18xFIJ.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Marshall Pierce]

I'd try re-downloading the ZooKeeper tarball to make sure you have a
clean install. (Might wish to copy your zookeeper config file before
blowing away your old copy.)

Also, if you're on a mac running Java 7, make sure you're using a recent
version of java 7. Early ones had issues with ZooKeeper. (Or try Java 6.)

-Marshall

[Jay Wang]

Hey Eric,

After starting zookeeper server (whose config file has the port set to 2181), when I try telnet 127.0.0.1 2181, I get: 

Trying 127.0.0.1...

telnet: connect to address 127.0.0.1: Connection refused

telnet: Unable to connect to remote host

Hmm, seems odd. I think I'll try reinstalling ZooKeeper. I don't know if this is relevant, but I can run the druid-test-harness completely fine, which I believe uses Zookeeper... 

Thanks for your help and time,

Jay

[Jay Wang]

UPDATE: reinstalled it and it works now... guess I must have tinkered with the client script somewhere

[Jay Wang]

Yup, reinstalling does the trick! Must have tinkered with a script.... 

[Jay Wang]

Hey Eric, 

Sorry for the barrage of questions! So managed to finally get ZooKeeper working. Now, when I try to start ComputeMain node with this cmd: "java -Duser.timezone=UTC -Dfile.encoding=UTF-8 -cp compute-config/:druid-services/target/druid-services-0.2.0-SNAPSHOT-selfcontained.jar com.metamx.druid.http.ComputeMain", Zookeeper seems to be working nicely. 

It seems that I don't have enough memory on my machine to run this node though... This is the error I am getting: 

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:108)

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:54)

at com.metamx.druid.http.ComputeMain.main(ComputeMain.java:43)

Caused by: com.metamx.common.ISE: Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize or druid.computation.buffer.size: maxDirectMemory[129,957,888], memoryNeeded[4,000,000,000], druid.computation.buffer.size[1,000,000,000], numThreads[3]

On Friday, February 22, 2013 11:16:20 AM UTC-5, Eric Tschetter wrote:

[Hagen Rother]

Yes, my nodes were reporting the same, just add the props to the java call and it runs fine.

~Hagen

http://five.sentenc.es

To view this discussion on the web visit https://groups.google.com/d/msg/druid-development/-/BpiLoOtkL0sJ.

[Eric Tschetter]

As Hagen says, you just need to update your runtime properties

 

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:108)

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:54)

at com.metamx.druid.http.ComputeMain.main(ComputeMain.java:43)

Caused by: com.metamx.common.ISE: Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize or druid.computation.buffer.size: maxDirectMemory[129,957,888], memoryNeeded[4,000,000,000], druid.computation.buffer.size[1,000,000,000], numThreads[3]

Based on the configuration, it needs 4GB of memory for "intermediate computation buffers."  The 4GB number comes from the druid.computation.buffer.size * (number of processing threads + 1).  It looks like your process is starting up with just 128MB of Java heap.  You can reduce the number of threads (druid.processing.threads) and reduce the druid.computation.buffer.size to something more like 30MB.  You can increase your java heap size (-Xmx4g would give you a 4gb heap), or you can just set the "-XX:MaxDirectMemorySize" property that the exception refers to 

(-XX:MaxDirectMemorySize=4g should work).

--Eric

 

To view this discussion on the web visit https://groups.google.com/d/msg/druid-development/-/BpiLoOtkL0sJ.