Support for Apache Spark

[Rado Buranský]

Spark is incredibly growing but trying to "unit" test it is still a big issue. By unit testing I mean minimalistic integration testing where you try to mock as much as possible. That means two things that need to be done:

1) Use mocks which are serializable so that Spark can send them around.

2) Use accumulators and other Spark features to manage distributed state of all mocks so that it can be verified.

Is anybody aware of an attempt to implement this? Any other ideas how to use mocking in distributed applications where (de)serialization is used?

[Brice Dutheil]

Hi Rado,

1) Use mocks which are serializable so that Spark can send them around.
2) Use accumulators and other Spark features to manage distributed state of all mocks so that it can be verified.

1. Mocks can already be made serializable. However note that mocks are big objects, there’s now two serialization mode :
   a. the regular serialization

    mock(Book.class, withSettings().serializable());
   

   b. the serialization that works across class loaders

    mock(Book.class, withSettings().serializable(ACROSS_CLASSLOADERS));
   

   I don’t know much spark but I believe this mode is the one needed. As mocks maybe deserialized on another VM. Note though that the other VM / classloader need the same set of libraries that was used to craft the mock, including mockito !

   Note my feeling is that it may be wrong to test a distributed environment with Mockito, but yet again I don’t know much spark.

2. I don’t know much about these, I’m not sure it should be included in mockito, it should however be in a some spark / mockito lib. Anyway from what I understand I believe those accumulators could be implemented with a custom Answer.

Don’t hesitate to let us know how that did pan out.

HTH
Cheers

​

-- Brice

--
You received this message because you are subscribed to the Google Groups "mockito" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.
To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/be3b67b6-9d94-4f81-a9fe-cdefa9a1386d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.