[JIRA] (SPARKR-234) Improve performance for parallelize

17 views
Skip to first unread message

Shivaram Venkataraman (JIRA)

unread,
Mar 16, 2015, 11:38:02 PM3/16/15
to spark...@googlegroups.com
Shivaram Venkataraman created an issue
 
SparkR / Improvement SPARKR-234
Improve performance for parallelize
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 16/Mar/15 8:37 PM
Priority: Minor Minor
Reporter: Shivaram Venkataraman

The parallelize operation in SparkR creates a ParallelCollectionRDD which ships part of the collection in its partition information. This creates large tasks which throw a warning and slow down execution. Instead we could follow a trick similar to PySpark where we create a local file and then call `objectFile` on the local file to create the RDD.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Davies (JIRA)

unread,
Mar 19, 2015, 1:45:03 PM3/19/15
to spark...@googlegroups.com
Davies commented on Improvement SPARKR-234
 
Re: Improve performance for parallelize

The reason we use local file to pass the data into JVM is that Py4j has very pool performance with large data (because of text protocol). After that, we still use parallelize() in JVM.

The local file can not be accessed by other workers in cluster, so we can not avoid the ParallelCollectionRDD without a distributed file system.

Shivaram Venkataraman (JIRA)

unread,
Mar 19, 2015, 2:03:01 PM3/19/15
to spark...@googlegroups.com

Thats a good point. I guess we could also save it to HDFS and read it back but that requires knowing the HDFS path etc. The main problem is not in the first step when `parallelize` is called, but since we serialize the lineage for every task, the collection is shipped for all tasks on that RDD chain.

Lets leave this open for now – We could also try to fix the Scala implementation to avoid this as this problem happens in all languages

Davies (JIRA)

unread,
Mar 19, 2015, 3:21:01 PM3/19/15
to spark...@googlegroups.com
Davies commented on Improvement SPARKR-234

I think the data only come with tasks which need it. After a shuffle, it will not be included in the lineage.

Actually, putting the data into tasks is the cheapest way to deliver it. If you store it in DFS or block storage of Spark, it should be have at least two copies.

Reply all
Reply to author
Forward
0 new messages