Gobblin support for kafka 10

[Krupa Varughese]

Hi Team,

Does Gobblin currently support kafka 10 , or is such a plan under progress. Since we at Pubmatic wants to move away from (kafka 0.8.2 +camus) combination for ingestion to (kafka10 + gobblin).

Thanks in advance for response,
krupa

[sh...@helpshift.com]

On Wednesday, April 19, 2017 at 12:51:45 AM UTC+5:30, sh...@helpshift.com wrote:

HI Team,

I have same question which Krupa has asked. Thanks in advance

[Shirshanka Das]

Hi Krupa, Shyam: 

   We're actually migrating to Kafka 10 internally at LinkedIn currently. We should be able to release something in open-source shortly thereafter. 

   What sort of timeframe are you looking for? 

Shirshanka

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/fbe3c5ec-4570-49ff-8581-da9c7a539d50%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Krupa Varughese]

Hi Shirshanka,

It would be great if its available to us in next 2 months.

Thanks for responding ,
krupa

On Wednesday, April 19, 2017 at 10:18:06 AM UTC+5:30, Shirshanka Das wrote:

Hi Krupa, Shyam: 

   We're actually migrating to Kafka 10 internally at LinkedIn currently. We should be able to release something in open-source shortly thereafter. 

   What sort of timeframe are you looking for? 

Shirshanka

On Tue, Apr 18, 2017 at 12:22 PM, <sh...@helpshift.com> wrote:

On Wednesday, April 19, 2017 at 12:51:45 AM UTC+5:30, sh...@helpshift.com wrote:

HI Team,

I have same question which Krupa has asked. Thanks in advance

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.

To post to this group, send email to gobbli...@googlegroups.com.

[Shyam Shinde]

Hi Shirshanka / Krupa ,

I tried sample KAFKA-10 --> HDFS gobblin job. I used job config from here 

http://gobblin.readthedocs.io/en/latest/case-studies/Kafka-HDFS-Ingestion/#standalone.

And surprisingly, It worked and writing data to local directory. I tried this in standalone deployment and using LocalJobLauncher. I hope same will work with MR deployment as well.

@ Shirshanka -  I think, There is no change required to in Gobblin-0.9 to support kafka-10. please reply suggestion/explanation.

Thanks,

shyam

[Shirshanka Das]

That sounds right. You should be able to read from Kafka 10 cluster using Gobblin 0.9. 

We will upgrade the dependency in a month or so. 

Shirshanka

To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/CAA3qLidyPFO6x%2BZ9Bh%3DCFTUWMiWnHDTwrW_7rfyerJebzX1x4Q%40mail.gmail.com.

[Krupa Varughese]

Thanks guys for helping me out. It works fine for my requirements !!!