We should add support to load data frames in SQLContext given just a datasource name (as specified in https://issues.apache.org/jira/browse/SPARK-5420)
Similarly we should add support to save out DataFrames. Relevant PySpark code
Load https://github.com/apache/spark/blob/e50934f11e1e3ded21a631e5ab69db3c79467137/python/pyspark/sql/context.py#L573
Save https://github.com/apache/spark/blob/e50934f11e1e3ded21a631e5ab69db3c79467137/python/pyspark/sql/dataframe.py#L192
This depends on latest version of Spark, how to use spark 1.3.0-rc1 to build and test?
If you check out the `sparkr-sql` branch in github it depends on Spark 1.3.0 by default. We also include the Apache Staging repository[1] so SBT should pull in 1.3.0-rc1 – Let me know if this doesn't work
[1] https://github.com/amplab-extras/SparkR-pkg/blob/sparkr-sql/pkg/src/build.sbt#L56
This jar pulled from Apache staging repository is published on 2015/02/11, but the latest save/load API are only available after 2/16 :
<info organisation="org.apache.spark" module="spark-sql_2.10" revision="1.3.0" status="release" publication="20150211002748" >
I'm currently blocked by this.
This should be unblocked now that we are using a later snapshot
Fixed by https://github.com/amplab-extras/SparkR-pkg/pull/199