Using Scalamock with Scalatest in a case required serialization.

[Kane Ho]

I am working on a project that has dependency on Apache Spark, I am trying to do some mocking in the parallel model which required serializable instances. Correct me if I am wrong, but it looks like Scalamock doesn't support serialization in general.

Here is my use case, basically I want to mock a class which has method that takes RDD object.

val algoMock = mock[Algorithm[RDD[HashMap[String, Double]], RDD[Double]]]
val server = new Server(algoMock)
val result = server.predict(rddM, rddQ).collect()

predict method has one call to Apache Spark library which cause the serialization of algoMock:

rddModel.zipPartitions(reInput)(algo.predict)

Basically it requires to serialize algoMock so that it can distribute algo to all partitions for parallel processing. And here is the exception:

org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.scalamock.MockFunction2
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:772)
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:715)
    at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:699)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1203)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
    at akka.actor.ActorCell.invoke(ActorCell.scala:456)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Does anyone know if ScalaMock support serialization? Could you shed some light here?

Thank you very much!

Regards,

Kane

[Paul Butcher]

I've never used Spark, but it sounds like Spark is trying to serialise a mock so that it can deserialise it somewhere else (presumably on some other machine)? That certainly isn't supported by ScalaMock as things currently stand, and it's hard to see how it could be (because mocks depend on various bits of state - among other things, the expectations that have been set - that won't be in place wherever the mock is deserialised). 

Does Spark not provide a means for algorithms to be tested without actually being serialised/deserialised?

--
paul.butcher->msgCount++

Silverstone, Brands Hatch, Donington Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
Skype: paulrabutcher

Author of Seven Concurrency Models in Seven Weeks: When Threads Unravel

http://pragprog.com/book/pb7con

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.