Pub/Sub Connector for Dataproc

[Prakhar Gautam]

Hi,

Is there any connector available to have Pub/Sub talk to Dataproc directly? I believe there's integration available for Spark streaming with pub/sub (https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/tree/master/scala/spark-pubsub). Is there any such connector extended to Dataproc as well? Pls advise.

Thanks,

Prakhar

[James Malone]

Hi Prakhar,

Apologies for the delay in responding to your message.

At present, the Spark-Cloud Pubsub connector detailed in the Bigtable example you mention should work with Spark, whether you run it on Dataproc, bdutil, or a different environment. With Dataproc, we recently updated the account scopes so PubSub should work out of the box. 

Have you run into a specific issue with that solution? Are you looking for something different? 

Best,

James

[Prakhar Gautam]

Hi James,

Many thanks for your response.

I was working with a client that was using Kafka along with Spark. As part of their migration to GCP, we proposed replacing Kafka with PubSub and Spark with Dataproc. This connector seems to be working well for them.

On the recently updated account scopes, do we have any relevant documentation in place that I can refer to? 

Thanks,

Prakhar