Can I convert sparkStreamRDD to RDD?

[3504...@qq.com]

When I recieve streamRDD,I want to use RDD's function and method in it,So is there a method that convert streamRDD to RDD?

[Tathagata Das]

I am not sure what you mean by streamRDD. In SparkStreaming the basic abstraction is DStream, which is basically a continuous stream of RDDs. Besides the usual operations (map, filter, reduce),  if you want to do some custom RDD operation on every RDD in a DStream, there are a couple of options. 

1. DStream.transform()    This operation uses a RDD-to-RDD function to transform each RDD in the original DStream to new RDDs, thus creating a new DStream. The RDD-to-RDD function can use any of the RDD operations. Take a look at the Spark and Spark Streaming documentations.

2. DStream.foreach()       This operation uses a RDD-to-Unit function to that is execute on each RDD in the original DStream. You can use this to do things like take the final processed data from a dstream and push the data to some external databased, filesystem, etc. 

Spark Streaming docs: http://spark-project.org/docs/latest/api/streaming/index.html#spark.streaming.DStream

On Tue, May 21, 2013 at 5:34 AM, <3504...@qq.com> wrote:

When I recieve streamRDD,I want to use RDD's function and method in it,So is there a method that convert streamRDD to RDD?

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.