[JIRA] (SPARKR-203) Load and Save DataFrames

16 views
Skip to first unread message

Shivaram Venkataraman (JIRA)

unread,
Feb 17, 2015, 9:25:03 PM2/17/15
to spark...@googlegroups.com
Shivaram Venkataraman created an issue
 
SparkR / Sub-task SPARKR-203
Load and Save DataFrames
Issue Type: Sub-task Sub-task
Assignee: Unassigned
Created: 17/Feb/15 6:24 PM
Priority: Major Major
Reporter: Shivaram Venkataraman

We should add support to load data frames in SQLContext given just a datasource name (as specified in https://issues.apache.org/jira/browse/SPARK-5420)

Similarly we should add support to save out DataFrames. Relevant PySpark code

Load
https://github.com/apache/spark/blob/e50934f11e1e3ded21a631e5ab69db3c79467137/python/pyspark/sql/context.py#L573

Save
https://github.com/apache/spark/blob/e50934f11e1e3ded21a631e5ab69db3c79467137/python/pyspark/sql/dataframe.py#L192

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-14-082#64012-sha1:d36fff0)
Atlassian logo

Davies (JIRA)

unread,
Feb 24, 2015, 6:12:02 PM2/24/15
to spark...@googlegroups.com
Davies assigned an issue to Davies
Change By: Davies
Assignee: Davies
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Davies (JIRA)

unread,
Feb 25, 2015, 1:24:02 AM2/25/15
to spark...@googlegroups.com
Davies commented on Sub-task SPARKR-203
 
Re: Load and Save DataFrames

This depends on latest version of Spark, how to use spark 1.3.0-rc1 to build and test?

Shivaram Venkataraman (JIRA)

unread,
Feb 25, 2015, 1:31:02 AM2/25/15
to spark...@googlegroups.com

If you check out the `sparkr-sql` branch in github it depends on Spark 1.3.0 by default. We also include the Apache Staging repository[1] so SBT should pull in 1.3.0-rc1 – Let me know if this doesn't work

[1] https://github.com/amplab-extras/SparkR-pkg/blob/sparkr-sql/pkg/src/build.sbt#L56

Davies (JIRA)

unread,
Mar 2, 2015, 2:50:03 PM3/2/15
to spark...@googlegroups.com
Davies commented on Sub-task SPARKR-203

This jar pulled from Apache staging repository is published on 2015/02/11, but the latest save/load API are only available after 2/16 :

 	<info organisation="org.apache.spark"
		module="spark-sql_2.10"
		revision="1.3.0"
		status="release"
		publication="20150211002748"
	>

I'm currently blocked by this.

Shivaram Venkataraman (JIRA)

unread,
Mar 3, 2015, 5:18:02 PM3/3/15
to spark...@googlegroups.com

This should be unblocked now that we are using a later snapshot

Shivaram Venkataraman (JIRA)

unread,
Mar 4, 2015, 8:02:02 PM3/4/15
to spark...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages