PIO error

bala vivek

unread,

Jan 22, 2018, 1:39:31 AM1/22/18

to actionml-user

Hi,

I'm getting the following error in pio.

pio status gives me the below result,

[INFO] [Console$] Inspecting PredictionIO...

[INFO] [Console$] PredictionIO 0.10.0-incubating is installed at /opt/tools/PredictionIO-0.10.0-incubating

[INFO] [Console$] Inspecting Apache Spark...

[INFO] [Console$] Apache Spark is installed at /opt/tools/PredictionIO-0.10.0-incubating/vendors/spark-1.6.3-bin-hadoop2.6

[INFO] [Console$] Apache Spark 1.6.3 detected (meets minimum requirement of 1.3.0)

[INFO] [Console$] Inspecting storage backend connections...

[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...

[INFO] [Storage$] Verifying Model Data Backend (Source: LOCALFS)...

[INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)...

[INFO] [Storage$] Test writing to Event Store (App Id 0)...

[ERROR] [Console$] Unable to connect to all storage backends successfully. The following shows the error message from the storage backend.

[ERROR] [Console$] Failed after attempts=1, exceptions:

Mon Jan 22 01:00:02 EST 2018, org.apache.hadoop.hbase.client.RpcRetryingCaller@5c5d6175, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2293)

at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2298)

at org.apache.hadoop.hbase.master.HMaster.listNamespaceDescriptors(HMaster.java:2536)

at org.apache.hadoop.hbase.master.MasterRpcServices.listNamespaceDescriptors(MasterRpcServices.java:1100)

at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55734)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

at java.lang.Thread.run(Thread.java:748)

(org.apache.hadoop.hbase.client.RetriesExhaustedException)

[ERROR] [Console$] Dumping configuration of initialized storage backend sources. Please make sure they are correct.

[ERROR] [Console$] Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: TYPE -> elasticsearch, HOME -> /opt/tools/PredictionIO-0.10.0-incubating/vendors/elasticsearch-1.7.3

[ERROR] [Console$] Source Name: LOCALFS; Type: localfs; Configuration: PATH -> /root/.pio_store/models, TYPE -> localfs

[ERROR] [Console$] Source Name: HBASE; Type: hbase; Configuration: TYPE -> hbase, HOME -> /opt/tools/PredictionIO-0.10.0-incubating/vendors/hbase-1.2.4

This setup is running in our production and this is not a new setup. Often I get this error and if do a pio-stop-all and pio-start-all, pio will work fine.

But why often the pio status is showing error. There was no new configuration changes made in the pio-envi.sh file

Pat Ferrel

unread,

Jan 22, 2018, 4:17:26 PM1/22/18

to bala vivek, actionml-user

If you are using a laptop for a dev machine, when it sleeps it can interfere with Zookeeper, which is started and used by HBase. So pio-stop-all then pio-start-all restarts HBase and therefor Zookeeper gracefully to solve this.

Does the stop/start always solve this?

--
You received this message because you are subscribed to the Google Groups "actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-use...@googlegroups.com.
To post to this group, send email to action...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/4d2f0758-3718-4dc1-a844-ce48747d86dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

bala vivek

unread,

Jan 23, 2018, 2:32:17 AM1/23/18

to actionml-user

Hi Pat,

The PIO has installed on the Ubuntu server, the Dev server and production servers are hosted in other countries and we are connecting through VPN from my laptop.

And yes if I do a pio-start-all and pio-stop-all resolves the issue always, but this issue is re-occurring often and sometimes the PIO service is not coming up even after multiple Pio restart.

Not sure with the core reason why the service is often getting down.

Regards,
Bala

Pat Ferrel

unread,

Jan 23, 2018, 2:56:00 PM1/23/18

to bala vivek, actionml-user, us...@predictionio.apache.org

Unfortunately I can’t possibly guess without more information.

What do the logs say when pio cannot be started? Are all these pio instances separate, not in a cluster? In other words does each pio server have all necessary services running on them? I assume none is sleeping like a laptop does?

I you are worrying, when properly configured PIO is quite stable on servers that do not sleep. I have never seen a bug that would cause this and have installed it hundreds of time so lets look through logs and check your pio-env.sh on a particular machine that is having this problem.

To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/4a3ee346-745b-4ca3-a6bd-e6953695f0c9%40googlegroups.com.

bala vivek

unread,

Jan 23, 2018, 11:12:25 PM1/23/18

to actionml-user

Hi Pat,

Yes, I understood PIO was fine working and stable until one week back, But after the jdk update from the version jdk 1.8.0_121 to jdk 1.8.0_151 these errors are happening. And I confirm the jdk upgrade was successful.

We dont have any node like setup, the PredictionIO has its own vendors folder which contains all the hbase, elasticsearch and spark folder.

The pio log says either it cannot connect to the zoopker or it will say "unable to connect backend storage" hope you should be familiar with these error formats.

After the restart Im repeatedly getting any of this error. Some time the pio status says "your system is all ready to go" but if try to do a pio restart the servcie will show some error.

Is there any time interval I need to follow for the pio restart ?

I'm sharing the pio-envi.sh file here.

# SPARK_HOME: Apache Spark is a hard dependency and must be configured.

#SPARK_HOME=$PIO_HOME/vendors/spark-2.1.0-bin-hadoop2.7

SPARK_HOME=$PIO_HOME/vendors/spark-1.6.3-bin-hadoop2.6

#POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-9.4-1204.jdbc41.jar

#MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.37.jar

# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO

# with HBase on a remote cluster.

#HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.2.5/conf

HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.2.4/conf

# Filesystem paths where PredictionIO uses as block storage.

PIO_FS_BASEDIR=$HOME/.pio_store

PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines

PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp

# PredictionIO Storage Configuration

#

# This section controls programs that make use of PredictionIO's built-in

# storage facilities. Default values are shown below.

#

# For more information on storage configuration please refer to

# https://docs.prediction.io/system/anotherdatastore/

# Storage Repositories

# Default is to use PostgreSQL

PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta

PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH

PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event

PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE

PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model

PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS

# Storage Data Sources

# PostgreSQL Default Settings

# Please change "pio" to your database name in # PIO_STORAGE_SOURCES_PGSQL_URL

# Please change # PIO_STORAGE_SOURCES_PGSQL_USERNAME and

# # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly

# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc

# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio

# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio

# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio

# MySQL Example

# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc

# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio

# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio

# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio

# Elasticsearch Example

PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch

# PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=<elasticsearch_cluster_name>

# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost

# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300

PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.7.3

# Local File System Example

PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs

PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models

# HBase Example

PIO_STORAGE_SOURCES_HBASE_TYPE=hbase

#PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.2.5

PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.2.4

bala vivek

unread,

Jan 25, 2018, 3:03:52 AM1/25/18

to actionml-user

hi Pat,

My error msg,

[INFO] [Console$] Inspecting PredictionIO...

[INFO] [Console$] PredictionIO 0.10.0-incubating is installed at /opt/tools/PredictionIO-0.10.0-incubating

[INFO] [Console$] Inspecting Apache Spark...

[INFO] [Console$] Apache Spark is installed at /opt/tools/PredictionIO-0.10.0-incubating/vendors/spark-1.6.3-bin-hadoop2.6

[INFO] [Console$] Apache Spark 1.6.3 detected (meets minimum requirement of 1.3.0)

[INFO] [Console$] Inspecting storage backend connections...

[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...

[INFO] [Storage$] Verifying Model Data Backend (Source: LOCALFS)...

[INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)...

[INFO] [Storage$] Test writing to Event Store (App Id 0)...

[ERROR] [Console$] Unable to connect to all storage backends successfully. The following shows the error message from the storage backend.

[ERROR] [Console$] Failed after attempts=1, exceptions:

Thu Jan 25 03:02:45 EST 2018, org.apache.hadoop.hbase.client.RpcRetryingCaller@5c5d6175, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing