Does Dataproc support stream processing?

[Paresh Sahare]

Does Dataproc support stream processing?

[Richard Holowczak]

Short answer is Yes through Spark.  I have used examples with Spark Structured Streaming.

Some configuration is required, for example, if you want to consume a Kafka stream.

If you are interested in an example written in PySpark, please let me know.

Rich H.

[Alexander Goida]

Hi,

Dataproc by itself is an infrastructure component. Streaming might be incorporated using different approaches. As mentioned before, it could be Spark Streaming. Or it might be polling messages from some messaging system and then utilizing Spark to process received data. For example, in our scenario, we are reading Pub/Sub messages and then processing them with Spark (https://medium.com/p/0e33b046372d).

@Richard Holowczak

It would be very interesting to see your approach. If this won't take to much of your time. Thank you.

Regards,