pio train error , possibly in Preparator params

[an...@normativedesign.com]

Hi there,

I've been working through some of the prediction.io tutorials and started using a modified version of the PredictionIO/template-scala-parallel-similarproduct template. I am stumped by an issue that its cropping up while running 'pio train' .   I've used the sanitycheck methods on my trainingData class , but the process never gets to that point, it fails with the following error.

[INFO] [Console$] Using existing engine manifest JSON at /vagrant/movie_rating/manifest.json
[INFO] [Runner$] Submission command: /home/vagrant/PredictionIO/vendors/spark-1.4.1/bin/spark-submit --class io.prediction.workflow.CreateWorkflow --jars file:/vagrant/movie_rating/target/scala-2.10/template-scala-parallel-similarproduct-assembly-0.1-SNAPSHOT-deps.jar,file:/vagrant/movie_rating/target/scala-2.10/template-scala-parallel-similarproduct_2.10-0.1-SNAPSHOT.jar --files file:/home/vagrant/PredictionIO/conf/log4j.properties,file:/home/vagrant/PredictionIO/vendors/hbase-1.0.0/conf/hbase-site.xml --driver-class-path /home/vagrant/PredictionIO/conf:/home/vagrant/PredictionIO/vendors/hbase-1.0.0/conf file:/home/vagrant/PredictionIO/lib/pio-assembly-0.9.4.jar --engine-id WiuHmxbgAS7z2sy9T92Y0YdGUd1MC0uX --engine-version 9cc966187ccdf9346b36d0998b859f2c3ff60d41 --engine-variant file:/vagrant/movie_rating/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/vagrant/.pio_store,PIO_STORAGE_SOURCES_HBASE_HOME=/home/vagrant/PredictionIO/vendors/hbase-1.0.0,PIO_HOME=/home/vagrant/PredictionIO,PIO_FS_ENGINESDIR=/home/vagrant/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/vagrant/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/home/vagrant/PredictionIO/vendors/elasticsearch-1.4.4,PIO_FS_TMPDIR=/home/vagrant/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/home/vagrant/PredictionIO/conf,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(movie_rating,List(rate)))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
Exception in thread "main" java.lang.RuntimeException: Failed to invoke public scala.Option() with no args
 at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:107)
 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:162)
 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
 at com.google.gson.Gson.fromJson(Gson.java:803)
 at com.google.gson.Gson.fromJson(Gson.java:768)
 at com.google.gson.Gson.fromJson(Gson.java:717)
 at com.google.gson.Gson.fromJson(Gson.java:689)
 at io.prediction.workflow.JsonExtractor$.extractWithGson(JsonExtractor.scala:152)
 at io.prediction.workflow.JsonExtractor$.extract(JsonExtractor.scala:70)
 at io.prediction.workflow.WorkflowUtils$.extractParams(WorkflowUtils.scala:150)
 at io.prediction.controller.Engine$$anonfun$10$$anonfun$apply$6.apply(Engine.scala:390)
 at io.prediction.controller.Engine$$anonfun$10$$anonfun$apply$6.apply(Engine.scala:386)
 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at scala.collection.immutable.List.foreach(List.scala:318)
 at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
 at scala.collection.AbstractTraversable.map(Traversable.scala:105)
 at io.prediction.controller.Engine$$anonfun$10.apply(Engine.scala:386)
 at io.prediction.controller.Engine$$anonfun$10.apply(Engine.scala:383)
 at scala.Option.map(Option.scala:145)
 at io.prediction.controller.Engine.jValueToEngineParams(Engine.scala:383)
 at io.prediction.workflow.CreateWorkflow$.main(CreateWorkflow.scala:218)
 at io.prediction.workflow.CreateWorkflow.main(CreateWorkflow.scala)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.InstantiationException
 at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:104)
 ... 32 more

  

Can anyone point me in a certain direction ?

Thanks

Andy

[Paul Li]

Hi Andy,

Do you mind sharing your engine.json configuration please?

Paul

[an...@normativedesign.com]

Hi Paul,

Of course, engine.json is below

{

  "id": "default",

  "description": "Default settings",

  "engineFactory": "com.movie_rating.SimilarProductEngine",

  "datasource": {

    "params" : {

      "appName": "movie_rating",

      "eventNames": ["rate"]

    }

  }

  "algorithms": [

    {

      "name": "als",

      "params": {

        "appName": "movie_rating",

        "typeName": "items",

        "num": 5,

        "eventNames": ["rate"],

        "maxEventsPerEventType": 100,

        "maxCorrelatorsPerEventType": 50,

        "maxQueryEvents": 500,

        "num": 20,

        "seed": 3,

      }

    }

  ]

}

Thanks!

Andy

...

[Pat Ferrel]

This seems to have mixed parameters from different engines. Go back to the original and get the default engine.json then modify only what you need to change. For instance “maxCorrelatorsPerEventType” is not a similarproduct template param.

[Hemant Agrawal]

My engine.json is

{
  "id": "default",
  "description": "Default settings",

  "engineFactory": "com.readwhere.SimilarProductEngine",
  "datasource": {
    "params" : {
      "appName": "readwhere_store",
      "eventNames": ["view"]
    }
  },

  "algorithms": [
    {
      "name": "als",
      "params": {

        "eventNames": ["view"],
        "rank": 10,
        "numIterations" : 20,
        "lambda": 0.01,
        "seed": 3
      }
    }
  ]
}

I am getting this error
Stack Trace:
java.lang.RuntimeException: Failed to invoke public scala.collection.immutable.List() with no args

    at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:107)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:162)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
    at com.google.gson.Gson.fromJson(Gson.java:803)
    at com.google.gson.Gson.fromJson(Gson.java:768)
    at com.google.gson.Gson.fromJson(Gson.java:717)
    at com.google.gson.Gson.fromJson(Gson.java:689)
    at io.prediction.workflow.JsonExtractor$.extractWithGson(JsonExtractor.scala:152)
    at io.prediction.workflow.JsonExtractor$.extract(JsonExtractor.scala:70)

    at io.prediction.workflow.ServerActor$$anonfun$24.apply(CreateServer.scala:519)
    at io.prediction.workflow.ServerActor$$anonfun$24.apply(CreateServer.scala:513)
    at spray.routing.ApplyConverterInstances$$anon$22$$anonfun$apply$1.apply(ApplyConverterInstances.scala:25)
    at spray.routing.ApplyConverterInstances$$anon$22$$anonfun$apply$1.apply(ApplyConverterInstances.scala:24)
    at spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:38)
    at spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:37)
    at spray.routing.directives.BasicDirectives$$anon$1.happly(BasicDirectives.scala:26)
    at spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1.apply(Directive.scala:37)
    at spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1.apply(Directive.scala:36)
    at spray.routing.directives.BasicDirectives$$anon$2.happly(BasicDirectives.scala:79)
    at spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)
    at spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)
    at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
    at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)
    at spray.routing.directives.ExecutionDirectives$$anonfun$detach$1$$anonfun$apply$7$$anonfun$apply$3.apply$mcV$sp(ExecutionDirectives.scala:89)
    at spray.routing.directives.ExecutionDirectives$$anonfun$detach$1$$anonfun$apply$7$$anonfun$apply$3.apply(ExecutionDirectives.scala:89)
    at spray.routing.directives.ExecutionDirectives$$anonfun$detach$1$$anonfun$apply$7$$anonfun$apply$3.apply(ExecutionDirectives.scala:89)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: java.lang.InstantiationException
    at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.google.gson.internal.ConstructorConstructor$3.construct(ConstructorConstructor.java:104)
    ... 33 more

[Pat Ferrel]

Not too familiar with this template but does it need the appName in the algorithm params too?

BTW the maintainers here don’t seem to be responding. You might have better luck here: https://groups.google.com/forum/#!forum/actionml-user