Error when running the UR integration test

[Vaulstein]

Hi,

I was running the command:

./examples/integration-test

All the events were imported and the model was being built and deployed, when I got the below error:

Building and delpoying model

[INFO] [Console$] Using existing engine manifest JSON at /root/ur/manifest.json

[INFO] [Console$] Using command '/root/pio-aml/sbt/sbt' at the current working directory to build.

[INFO] [Console$] If the path above is incorrect, this process will fail.

[INFO] [Console$] Uber JAR disabled. Making sure lib/pio-assembly-0.9.7-aml.jar is absent.

[INFO] [Console$] Going to run: /root/pio-aml/sbt/sbt  package assemblyPackageDependency

[INFO] [Console$] Build finished successfully.

[INFO] [Console$] Looking for an engine...

[INFO] [Console$] Found template-scala-parallel-universal-recommendation_2.10-0.5.0.jar

[INFO] [Console$] Found template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar

[INFO] [RegisterEngine$] Registering engine sOfhWeec3vFg0zFJMxROUe3OV9wlp0o7 08f9b7206d6c4c46c26e6aa45fd73ed15e7de602

[INFO] [Console$] Your engine is ready for training.

[INFO] [Console$] Using existing engine manifest JSON at /root/ur/manifest.json

[INFO] [Runner$] Submission command: /root/pio-aml/vendors/spark-1.5.1-bin-hadoop2.6/bin/spark-submit --driver-memory 4g --executor-memory 4g --class io.prediction.workflow.CreateWorkflow --jars file:/root/ur/target/scala-2.10/template-scala-parallel-universal-recommendation_2.10-0.5.0.jar,file:/root/ur/target/scala-2.10/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar --files file:/root/pio-aml/conf/log4j.properties,file:/usr/local/hadoop/etc/hadoop/core-site.xml --driver-class-path /root/pio-aml/conf:/root/pio-aml/lib/postgresql-9.4-1204.jdbc41.jar:/root/pio-aml/lib/mysql-connector-java-5.1.37.jar:/usr/local/hadoop/etc/hadoop file:/root/pio-aml/assembly/pio-assembly-0.9.7-aml.jar --engine-id sOfhWeec3vFg0zFJMxROUe3OV9wlp0o7 --engine-version 08f9b7206d6c4c46c26e6aa45fd73ed15e7de602 --engine-variant file:/root/ur/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/root/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_STORAGE_SOURCES_HBASE_HOME=/root/pio-aml/vendors/hbase-1.0.0,PIO_HOME=/root/pio-aml,PIO_FS_ENGINESDIR=/root/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/root/pio-aml/vendors/elasticsearch-1.4.4,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/root/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/root/pio-aml/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/predictionio/controller/EngineFactory

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at scala.reflect.runtime.ReflectionUtils$.staticSingletonInstance(ReflectionUtils.scala:60)

at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaModuleMirror.instance(JavaMirrors.scala:473)

at io.prediction.workflow.WorkflowUtils$.getEngine(WorkflowUtils.scala:69)

at io.prediction.workflow.CreateWorkflow$.liftedTree1$1(CreateWorkflow.scala:193)

at io.prediction.workflow.CreateWorkflow$.main(CreateWorkflow.scala:192)

at io.prediction.workflow.CreateWorkflow.main(CreateWorkflow.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: org.apache.predictionio.controller.EngineFactory

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 28 more

After which I tried deploying it separately using pio deploy but it said `No valid Engine instance found for the engine` and asked to run `pio train` again.
But I received the same error on running pio train.
The event data have been added but the training seems to have failed.

Am I missing any file?

[Pat Ferrel]

You are using an old pio with the new Apache compatible UR v0.5.0. This does not work. You have to upgrade to Apache PredictionIO v0.10.0.

Instructions here: http://actionml.com/docs/install

--
You received this message because you are subscribed to the Google Groups "actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-use...@googlegroups.com.
To post to this group, send email to action...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/f427ef96-b8b6-456c-9809-6fe35d61a140%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vaulstein]

Hi Pat,

I upgraded to Apache predictionIO v0.10.0

After that on running the integration test, the training had started this time but it failed I think on the last step as you have said on the below link that
Elasticsearch performs the last step in the algorithm, finding the K Nearest Neighbors (KNN).

https://groups.google.com/forum/#!topic/actionml-user/ZaKuF2H5yDA

As in the link. I checked and I was set pio_meta= PSQL
I changed this to pio_meta = ELASTICSEARCH

But then received the below error:

[aml@stylab elasticsearch-1.4.4]$ pio status

[INFO] [Console$] Inspecting PredictionIO...

[INFO] [Console$] PredictionIO 0.10.0-incubating is installed at /home/aml/apache-predictionio-0.10.0-incubating/PredictionIO-0.10.0-incubating

[INFO] [Console$] Inspecting Apache Spark...

[INFO] [Console$] Apache Spark is installed at /home/aml/apache-predictionio-0.10.0-incubating/PredictionIO-0.10.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6

[INFO] [Console$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)

[INFO] [Console$] Inspecting storage backend connections...

[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...

[ERROR] [Console$] Unable to connect to all storage backends successfully. The following shows the error message from the storage backend.

[ERROR] [Console$] None of the configured nodes are available: [] (org.elasticsearch.client.transport.NoNodeAvailableException)

[ERROR] [Console$] Dumping configuration of initialized storage backend sources. Please make sure they are correct.

[ERROR] [Console$] Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOSTS -> localhost, TYPE -> elasticsearch, HOME -> /home/aml/apache-predictionio-0.10.0-incubating/PredictionIO-0.10.0-incubating/vendors/elasticsearch-1.4.4, PORTS -> 9300

[aml@stylab elasticsearch-1.4.4]$ pio eventserver &

[2] 26527

[aml@stylab elasticsearch-1.4.4]$ [INFO] [Console$] Creating Event Server at 0.0.0.0:7070

Exception in thread "main" org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []

  at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)

at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)

at org.elasticsearch.client.transport.support.InternalTransportIndicesAdminClient.execute(InternalTransportIndicesAdminClient.java:86)

 at org.elasticsearch.client.support.AbstractIndicesAdminClient.exists(AbstractIndicesAdminClient.java:170)

     at org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequestBuilder.doExecute(IndicesExistsRequestBuilder.java:53)

    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)

 at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)

 at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:73)

     at org.apache.predictionio.data.storage.elasticsearch.ESAccessKeys.<init>(ESAccessKeys.scala:43)

       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

     at org.apache.predictionio.data.storage.Storage$.getDataObject(Storage.scala:306)

      at org.apache.predictionio.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:266)

      at org.apache.predictionio.data.storage.Storage$.getMetaDataAccessKeys(Storage.scala:376)

      at org.apache.predictionio.data.api.EventServer$.createEventServer(EventServer.scala:618)

      at org.apache.predictionio.tools.console.Console$.eventserver(Console.scala:893)

       at org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:728)

      at org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:696)

      at scala.Option.map(Option.scala:145)

  at org.apache.predictionio.tools.console.Console$.main(Console.scala:696)

      at org.apache.predictionio.tools.console.Console.main(Console.scala)

[Vaulstein]

Hi Pat,

This issue was probably because the elastic service was failing.
Got that sorted.
However, I have been getting Access key errors on running Integration test.

Another user is getting the same sort of errors. Since Access key isn't being passed manually, this shouldn't be happening, right?

./examples/integration-test

Integration test for The Universal Recommender.

If some step fails check that your engine.json file has been restored

or look for it in 'user-engine.json'

Checking for needed files

Checking status, should exit if pio is not running.

[INFO] [Console$] Inspecting PredictionIO...

[INFO] [Console$] PredictionIO 0.10.0-incubating is installed at /home/aml/apache-predictionio-0.10.0-incubating/PredictionIO-0.10.0-incubating

[INFO] [Console$] Inspecting Apache Spark...

[INFO] [Console$] Apache Spark is installed at /home/aml/apache-predictionio-0.10.0-incubating/PredictionIO-0.10.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6

[INFO] [Console$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)

[INFO] [Console$] Inspecting storage backend connections...

[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...

[INFO] [Storage$] Verifying Model Data Backend (Source: PGSQL)...

[INFO] [Storage$] Verifying Event Data Backend (Source: PGSQL)...

[INFO] [Storage$] Test writing to Event Store (App Id 0)...

[INFO] [Console$] (sleeping 5 seconds for all messages to show up...)

[INFO] [Console$] Your system is all ready to go.

[INFO] [App$] Initialized Event Store for this app ID: 3.

[INFO] [App$] Created new app:

[INFO] [App$]       Name: handmade

[INFO] [App$]         ID: 3

[INFO] [App$] Access Key: 6hmon40RPmwJNKRbe5_2D6lptjmZP1OgWBLW20w6DsTRYHf1AyF-NUzO2GcBQMMn

Checking to see if handmade app exists, should exit if not.

[INFO] [App$]     App Name: handmade

[INFO] [App$]       App ID: 3

[INFO] [App$]  Description:

[INFO] [App$]   Access Key: 6hmon40RPmwJNKRbe5_2D6lptjmZP1OgWBLW20w6DsTRYHf1AyF-NUzO2GcBQMMn | (all)

Moving engine.json to user-engine.json

Moving examples/handmade-engine.json to engine.json for integration test.

Deleting handmade app data since the test is date dependent

[INFO] [App$] Data of the following app (default channel only) will be deleted. Are you sure?

[INFO] [App$]     App Name: handmade

[INFO] [App$]       App ID: 3

[INFO] [App$]  Description: None

[INFO] [App$] Removed Event Store for this app ID: 3

[INFO] [App$] Initialized Event Store for this app ID: 3.

[INFO] [App$] Done.

Importing data for integration test

Access key: 6hmon40RPmwJNKRbe5_2D6lptjmZP1OgWBLW20w6DsTRYHf1AyF-NUzO2GcBQMMn

Namespace(access_key='6hmon40RPmwJNKRbe5_2D6lptjmZP1OgWBLW20w6DsTRYHf1AyF-NUzO2GcBQMMn', file='./data/sample-handmade-data.txt', url='http://localhost:7070')

Importing data...

Traceback (most recent call last):

 File "examples/import_handmade.py", line 114, in <module>

   import_events(client, args.file)

 File "examples/import_handmade.py", line 43, in import_events

   event_time = current_date

 File "/usr/lib/python2.7/site-packages/predictionio/__init__.py", line 247, in create_event

   event_time).get_response()

 File "/usr/lib/python2.7/site-packages/predictionio/connection.py", line 111, in get_response

   self._response = self.rfunc(tmp_response)

 File "/usr/lib/python2.7/site-packages/predictionio/__init__.py", line 122, in _acreate_resp

   response.body))

predictionio.NotCreatedError: request: POST /events.json?accessKey=6hmon40RPmwJNKRbe5_2D6lptjmZP1OgWBLW20w6DsTRYHf1AyF-NUzO2GcBQMMn {'eventTime': '2017-02-09T13:31:39.144+0000', 'entityType': 'user', 'targetEntityType': 'item', 'targetEntityId': 'Iphone 6', 'entityId': 'u1', 'event': 'purchase'} /events.json?accessKey=6hmon40RPmwJNKRbe5_2D6lptjmZP1OgWBLW20w6DsTRYHf1AyF-NUzO2GcBQMMn?eventTime=2017-02-09T13%3A31%3A39.144%2B0000&entityType=user&targetEntityType=item&targetEntityId=Iphone+6&entityId=u1&event=purchase status: 401 body: {"message":"Invalid accessKey."}

[Vaulstein]

Here is the error for another user:
https://stackoverflow.com/questions/36913924/error-in-importing-sample-events-for-predictioio-universal-recommender

--
You received this message because you are subscribed to a topic in the Google Groups "actionml-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/actionml-user/d2LamYVJytk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to actionml-user+unsubscribe@googlegroups.com.

To post to this group, send email to action...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/8ecf047a-e199-49eb-ad77-c1d07eb2b9b0%40googlegroups.com.

[Vaulstein]

Hi Pat,

I was able to solve my issue.

I had to do the following.
https://stackoverflow.com/questions/36913924/error-in-importing-sample-events-for-predictioio-universal-recommender/42141187#42141187

Access key was not being added to DB.
Thank you for your help.