|
Thats a good point. I guess we could also save it to HDFS and read it back but that requires knowing the HDFS path etc. The main problem is not in the first step when `parallelize` is called, but since we serialize the lineage for every task, the collection is shipped for all tasks on that RDD chain.
Lets leave this open for now – We could also try to fix the Scala implementation to avoid this as this problem happens in all languages
|