Hi All,
I`m trying to figure out how one should use Sources within Spark Jobs keeping all benefits of Spark.
Consider following snippet:
val items: Seq[String] = Seq("a", "b", "c")
sparkSession.sparkContext.parallelize(items, 10).flatMap { item =>
val subItems: Source[String, _] = f(item)
// (1)
???
}.map { subItem =>
f1(subItem)
}.reduce(_ + _)
I also created an IteratorSinkStage which materialized into Iterator similar as InputStreamSinkStage which is materialized into InputStream, but i`m not sure if this is best solution for this problem.
What would be the best way to work with akka Source within Spark environment?
Regards,
Kyrylo