install from source

[Pat Ferrel]

Following the install from source but not sure how to setup zookeeper. The only zookeeper specific comment in the docs is how to set the zookeeper dataDir in the HBase conf.

Is it supposed to be controlled from pio-start-all or does pio expect it to be running before launch?

[Donald Szeto]

Hi Pat,

By default, HBase will start its own embedded ZooKeeper unless you tell it not to explicitly, thus the need of setting it up through hbase-site.xml.

Regards,

Donald

[Pat Ferrel]

For some reason Zookeeper standalone as managed by Hbase doesn’t run. I’m on a Mac and have installed Kafka and Zookeeper using brew but also uninstalled them before trying the PIO source install. There is no Zookeeper data left on my system and Zookeeper does not lunch automatically so I’m not sure why the error says “ZooKeeper exists". Maybe some trace left in the filesystem but nothing I can see in the environment. 

This error keeps Hbase from starting. I’ll proceed with standalone Hbase troubleshooting but if anyone has seen something like this let me know.

[ERROR] [RecoverableZooKeeper] ZooKeeper exists failed after 1 attempts
[ERROR] [ZooKeeperWatcher] hconnection-0x3bb25aa6, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
[WARN] [ZooKeeperRegistry] Can't retrieve clusterId from Zookeeper
[ERROR] [StorageClient] Cannot connect to ZooKeeper (ZooKeeper ensemble: localhost). Please make sure that the configuration is pointing at the correct ZooKeeper ensemble. By default, HBase manages its own ZooKeeper, so if you have not configured HBase to use an external ZooKeeper, that means your HBase is not started or configured properly.
[ERROR] [Storage$] Error initializing storage client for source HBASE
[ERROR] [Storage$] Can't connect to ZooKeeper
io.prediction.data.storage.StorageClientException: Data source HBASE was not properly initialized.
at io.prediction.data.storage.Storage$$anonfun$9.apply(Storage.scala:182)
at io.prediction.data.storage.Storage$$anonfun$9.apply(Storage.scala:182)
at scala.Option.getOrElse(Option.scala:120)
at io.prediction.data.storage.Storage$.getDataObject(Storage.scala:181)
at io.prediction.data.storage.Storage$.getDataObject(Storage.scala:166)
at io.prediction.data.storage.Storage$.getLEvents(Storage.scala:274)
at io.prediction.data.storage.Storage$.verifyAllDataObjects(Storage.scala:240)
at io.prediction.tools.console.Console$.status(Console.scala:968)
at io.prediction.tools.console.Console$$anonfun$main$1.apply(Console.scala:611)
at io.prediction.tools.console.Console$$anonfun$main$1.apply(Console.scala:573)
at scala.Option.map(Option.scala:145)
at io.prediction.tools.console.Console$.main(Console.scala:573)
at io.prediction.tools.console.Console.main(Console.scala)

[Donald Szeto]

Try cleaning up HBase's ZooKeeper dataDir, and make sure HBASE_MANAGES_ZK is set in $HBASE_HOME/conf/hbase-env.sh, then start HBase again. See if that helps. You may also try using $HBASE_HOME/bin/start-HBASE.sh directly.

Feel free to attach HBase logs as well to aid debugging. They are in $HBASE_HOME/logs.

Regards,

Donald

--

Donald Szeto
PredictionIO

[Pat Ferrel]

It starts and runs “help" but not “status", get the same exception.

1) According to  the Hbase quickstart the dataDir is supposed to not exist so Hbase won’t do a migration and so I’ve tried with an empty one and a non-existent dataDir, same error. Also the default is to use /tmp/… which is fine by me so I tried that too but no help.

2) HBASE_MANAGES_ZK was unset but is defaulted to true, setting it explicitly gives the same error.

3) Some suggestions tell you to remove any ipv6 references to localhost in /etc/hosts so I did that to no avail.

4) I have Hadoop 2.6 installed by not running so it should be using the libs in the Hbase tar.

5) zookeeper is never launched by hbase from the output of jps -l.

This seems to happen a lot from Stackoverflow but no hint has worked yet. Mostly voodoo solutions anyway :-)

[Donald Szeto]

Any raw HBase master logs? We can see if there's any voodoo recipe we can apply. :-)

--

Donald Szeto
PredictionIO

[Pat Ferrel]

Thanks but nothing very interesting there. I get this when running pio-start-all and, of course, a lot of errors with pio status but that’s well after the original error. I get the same error doing things in the hbase shell so not pio specific. I’ve gone through the hbase standalone quickstart too. Just thought maybe someone has seen this. Maybe I’ll try the hbase list.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/pat/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/pat/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[WARN] [NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[INFO] [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[INFO] [log] jetty-6.1.26
[INFO] [log] Started SelectChann...@0.0.0.0:61342
[INFO] [log] jetty-6.1.26
[INFO] [log] Started SelectChann...@0.0.0.0:60010
[FATAL] [HMaster] Unhandled exception. Starting shutdown.
[INFO] [log] Stopped SelectChann...@0.0.0.0:60010

[Pat Ferrel]

OK, I cleaned the install and tried the exact tar specified in the PIO instructions and it works now. The previous binary I got from the hbase site but was named with the same version. Anyway, advice is stick to the PIO instructions.

wget https://archive.apache.org/dist/hbase/hbase-0.98.11/hbase-0.98.11-hadoop2-bin.tar.gz

[Pat Ferrel]

However it does not run from pio-start-all. I think this is because the build-from-source version of pio doesn’t point to the correct hbase locations:

The pio-env.sh contains the following lines, which seem to be incorrect:

HBASE_CONF_DIR=$PIO_HOME/conf

The hbase-site.xml in that location looks incomplete. Change the above to point to your hbase source installation and it runs. 

I’ll create a PR for docs