saveAsTextFile

[David Cabanillas]

Hi,

I have a problem with saveAsTextFile on Windows 10. The function saveAsTextFile creates a set of folders but the files are empty.

SparkR:::parallelize(sc, 1:3)->rdd

SparkR:::saveAsTextFile(rdd, "dirtest")

And I'm getting the following error.

Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : 

  org.apache.spark.SparkException: Job aborted.

at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:149)

at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:115)

at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:115)

at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)

at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:115)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:57)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:55)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:69)

at org.apache.spark.

[Shivaram Venkataraman]

For Hadoop operations to work on Windows you might need the Hadoop
binary package installed (See the unit tests section of
https://github.com/apache/spark/blob/master/R/WINDOWS.md).

Also this mailing list is no longer active -- please email the Spark
user mailing list (http://spark.apache.org/community.html) for further
questions about SparkR.

Thanks
Shivaram

> --
> You received this message because you are subscribed to the Google Groups
> "SparkR Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sparkr-dev+...@googlegroups.com.
> To post to this group, send email to spark...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sparkr-dev/d441ca9a-7bcf-44da-a757-f959cf473d52%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.