Re: Installing PIO and UR standalone

[Pat Ferrel]

Sorry to double check these things but this is so like what you see when you don’t add "export HBASE_MANAGES_ZK=true" to hbase/conf/hbase-env.sh.

hbase-site.xml is safest configured with:

<configuration>

    <property>

        <name>hbase.rootdir</name>

        <value>hdfs://some-master:9000/hbase</value>

    </property>

    <property>

        <name>hbase.zookeeper.property.dataDir</name>

        <value>hdfs://some-master:9000/zookeeper</value>

    </property>

    <property>

        <name>hbase.zookeeper.quorum</name>

        <value>localhost</value>

    </property>

    <property>

        <name>hbase.zookeeper.property.clientPort</name>

        <value>2181</value>

    </property>

</configuration>

I think the quorum host is the most important.

On May 13, 2016, at 11:32 AM, Hesham Massoud <heshamham...@gmail.com> wrote:

As I said, this was the first time I used it. Then I realised Its not the right version, after I got the bug. So now I removed it and installed the one here 

https://github.com/actionml/PredictionIO.git

now `pio version` gives me

0.9.7-aml

Its running yes, the console of pio. However, as I said I cannot create a new app, start the Hbase eventserver and also `pio status` command fails all giving the same hbase initialising error.






On Friday, May 13, 2016 at 7:42:21 PM UTC+2, Pat Ferrel wrote:

If you used $ bash -c "$(curl -s https://install.prediction.io/install.sh)”

you did not get the right version. The install instructions I sent uses the new install.sh from a different location.

What does `pio version` say?

Is it running now?

On May 13, 2016, at 9:29 AM, Hesham Massoud <heshamham...@gmail.com> wrote:

Hi Pat,

Thanks for your reply. 

Yes, that's what I read on the forums here on your replies and that's why I am using now the ActionML version. 

I already followed this README. But the thing is, I copied the whole vendors directory that I was using with my old (working) (which didn't have this export HBASE_MANAGES_ZK=true) quick installation version of PIO and it was working.




On Friday, May 13, 2016 at 6:15:03 PM UTC+2, Pat Ferrel wrote:

Sorry but this version of PIO is not supported by the latest UR release. This version (the Salesforce sponsored version) recently released changes that force the use of SSL/TLS/HTTPS with no documentation that I know of. Also you will need new features in the ActionML version of PIO.

The problem below may be that you haven’t set the following in `conf/hbase-env.sh`

     export HBASE_MANAGES_ZK=true

But I strongly recommend that you start here: https://github.com/actionml/cluster-setup/blob/master/install.md

On May 13, 2016, at 8:05 AM, Hesham Massoud <heshamham...@gmail.com> wrote:

Hello guys, 

I've wasted now a day trying to install and run the UR as standalone on my local machine for a project I am working on. So I would be really glad If someone can help me out with this.

First time I installed it, I used the quick install method: using 

$ bash -c "$(curl -s https://install.prediction.io/install.sh)"

After a long of time, I got it to work to the point of training the data. But when I tried deploying the engine. I got this error:

Cannot support TLS_RSA_WITH_AES_256_CBC_SHA with currently installed providers

and after some searching here, I found this: https://groups.google.com/d/msg/predictionio-user/VHr1tYAaUFM/KIUAyCAoBAAJ
which says that I have to do an installation of PIO from the ActionML Fork and follow the manual there. So I had to remove my first installation (the quick one) and followed the manual step by step, eventhough it wasn't that clear at some points because it is targeted more towards distributes/clusters rather than my need (stand alone). But after I followed it as much as I could, I could start PIO with

pio-start-all

 However, when I run any of these commands 

pio app new *myAppName*

or

pio eventserver

or

pio status

I get this error:

[ERROR] [StorageClient] HBase master is not running (ZooKeeper ensemble: localhost). Please make sure that HBase is running properly, and that the configuration is pointing at the correct ZooKeeper ensemble.

[ERROR] [Storage$] Error initializing storage client for source HBASE

Exception in thread "main" io.prediction.data.storage.StorageClientException: Data source HBASE was not properly initialized.

at io.prediction.data.storage.Storage$$anonfun$10.apply(Storage.scala:279)

at io.prediction.data.storage.Storage$$anonfun$10.apply(Storage.scala:279)

at scala.Option.getOrElse(Option.scala:120)

at io.prediction.data.storage.Storage$.getDataObject(Storage.scala:278)

at io.prediction.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:263)

at io.prediction.data.storage.Storage$.getLEvents(Storage.scala:385)

at io.prediction.tools.console.App$.create(App.scala:35)

at io.prediction.tools.console.Console$$anonfun$main$1.apply(Console.scala:739)

at io.prediction.tools.console.Console$$anonfun$main$1.apply(Console.scala:693)

at scala.Option.map(Option.scala:145)

at io.prediction.tools.console.Console$.main(Console.scala:693)

at io.prediction.tools.console.Console.main(Console.scala)

Here is my hbase-site.xml configuration:

 <configuration>
    <property>
        <name>hbase.rootdir</name>
        <value>file:///Users/heshammassoud/pio/PredictionIO-0.9.7-aml/vendors/hbase-1.1.4/data</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/Users/heshammassoud/pio/PredictionIO-0.9.7-aml/vendors/zookeeper</value>
    </property>
</configuration>

and here is my pio-env.sh

#!/usr/bin/env bash

# Copy this file as pio-env.sh and edit it for your site's configuration.

# PredictionIO Main Configuration

#

# This section controls core behavior of PredictionIO. It is very likely that

# you need to change these to fit your site.

PIO_HOME=/Users/heshammassoud/pio/PredictionIO-0.9.7-aml

# SPARK_HOME: Apache Spark is a hard dependency and must be configured.

SPARK_HOME=$PIO_HOME/vendors/spark-1.6.0

#POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-9.4-1204.jdbc41.jar

#MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.37.jar

# ES_CONF_DIR: You must configure this if you have advanced configuration for

#              your Elasticsearch setup.

# ES_CONF_DIR=/opt/elasticsearch

# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO

#                  with Hadoop 2.

# HADOOP_CONF_DIR=/opt/hadoop

# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO

#                 with HBase on a remote cluster.

HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.1.4/conf

# Filesystem paths where PredictionIO uses as block storage.

PIO_FS_BASEDIR=$HOME/.pio_store

PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines

PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp

# PredictionIO Storage Configuration

#

# This section controls programs that make use of PredictionIO's built-in

# storage facilities. Default values are shown below.

#

# For more information on storage configuration please refer to

# https://docs.prediction.io/system/anotherdatastore/

# Storage Repositories

# Default is to use PostgreSQL

PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta

PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH

PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event

PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE

PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model

PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS

# Storage Data Sources

# PostgreSQL Default Settings

# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL

# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and

# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly

#PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc

#PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio

#PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio

#PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio

# MySQL Example

# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc

# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio

# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio

# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio

# Elasticsearch Example

PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch

# PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=<elasticsearch_cluster_name>

# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost

# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300

PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.7.5

# Local File System Example

PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs

PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models

# HBase Example

PIO_STORAGE_SOURCES_HBASE_TYPE=hbase

PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.1.4

I tried a million times to delete the data directories of hbase and zookeeper. Then run

pio-stop-all

which always failed for different reason or timed out. Then I had to run

jps -l

to find what is still not shut down and kill manually.

Then run again 

pio-start-all

and follow the same process but always end up with this hbase initialisation/zookeeper error and cannot build the recommender to use it.

I would really really appreciate it if someone could help me solve this. And I would offer to provide a step-by-step documentation for how to setup the whole thing if I get it to work. Because I wasted a lot of time in it. So I guess it would be really helpful for fresh starters like me not wasting there time.


Thanks a lot in advance!

[Hesham Massoud]

Hey Pat,

No worries at all. In fact, thanks for the support. Actually, suddenly after a fresh install. With all my configurations set as I sent in my first message, I managed to build, train and deploy the app. However, when running pio status it always fails, maybe because of the export HBASE_MANAGES_ZK=true config. I will try to set it again and see if status doesn't throw an exception.

Regarding, hbase-site.xml, I will add the zookeeper.quorum and zookeeper.property.clientPort properties as well as you said. But regarding these two properties:

<property>
        <name>hbase.rootdir</name>
        <value>hdfs://some-master:9000/hbase</value>
    </property>


    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>hdfs://some-master:9000/zookeeper</value>
    </property>

Is it fine to have them as I have them right now like below or is it wrong?

<property>
        <name>hbase.rootdir</name>
        <value>file:///Users/heshammassoud/pio/PredictionIO-0.9.7-aml/vendors/hbase-1.1.4/data</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/Users/heshammassoud/pio/PredictionIO-0.9.7-aml/vendors/zookeeper</value>
    </property>

Thanks in advance,
Hesham.

[Hesham Massoud]

Hey Pat,

Ok event after adding "export HBASE_MANAGES_ZK=true" to hbase/conf/hbase-env.sh.  and after adding these two properties to hbase-sites.xml:

 

<property>
        <name>hbase.zookeeper.quorum</name>
        <value>localhost</value>
    </property>


    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>

'pio-status' throws the same zookeeper error. But at least its working to build, train, deploy the app and query it. But Its the same as without changing the above configurations. A bit confused now about their significance.

[Hesham Massoud]

Although, it used to work two days ago. Now when I run 'pio eventserver'. It throws back the same error. Any ideas?

[Hesham Massoud]

When I restarted my mac. It works again. I really don't get how this thing works though.
Here is my hbase-site.xml for reference:

<configuration>
    <property>
        <name>hbase.rootdir</name>

        <value>file:///Users/heshammassoud/pio/vendors/hbase-1.1.2/data-new</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/Users/heshammassoud/pio/vendors/zookeeper</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>127.0.0.1</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
</configuration>

[Pat Ferrel]

You are running on a Macbook, or something that hibernates? Zookeeper dies during hibernation so you have to restart Hbase when it wakes up. I have an MBP and run pio-stop-all/pio-start-all after a hibernate.

-- 
You received this message because you are subscribed to the Google Groups "actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-use...@googlegroups.com.
To post to this group, send email to action...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/d4ace6cc-6660-472a-b15d-e55a9dc0607a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hesham Massoud]

Ok, thanks Pat. Yeah, apparently this is the problem. My mac slept and it happened again.

Cheers! 

[Pat Ferrel]

FYI you shouldn’t have this problem on servers.

To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/c4c50bfb-8653-48bc-8c79-6c0b0bb4bfb5%40googlegroups.com.