Kafka Integration with Django Application

[Sampath Reddy]

Hey All,

We have developed REST back end API services using Django Rest Framework & mongodb as back end database and these end points have been working fine so far. Now we want to consume data, by subscribing to a Kafka Topic through Kafka consumer.

Now I tried consuming data using django-logpipe service but could not able to read data using this service. Not sure how many are using this service or weather this service is stable version or not.

We need to have continuous listening service which will subscribe to Kakfa topic using KafkaConsumer and throw the data at Django Application, where the read data is being processed.

Tried couple of approaches like django-crontabs which are in nature scheduled jobs but we want a continuous listener, django-logpipe service was of no use.

Can anyone please redirect me at the correct resource or point me in the path of the Kafka Integration with Django Application ( both should run in parallel)

Any help is much appreciated

Many thanks in advance,,

Sampath

[David Andrade]

Hey  Sampath,

Once I thought about it and I found this framework Faust: ttps://github.com/robinhood/faust/tree/master/examples/django. I think it has django integration, but I haven't tried it yet.

Updated me with any news!

Best,

David Andrade

Co Founder - instaviagem.com
(11) 9 7559 9540

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAErixQ3wLFNNS2XMvPSeEe-eyxtUS5mqHxHi%3Ds%3DSpGCn8%2B9cOw%40mail.gmail.com.

[ktrkumarbablu kumar]

I have faced same problem with Kafka.do you have example for the same

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMUtp1wgE9R_K_JKBKDRzaE6Cm9%2BmQkgW%2BRDJ-2AzUewBwCFsA%40mail.gmail.com.

[Sampath Reddy]

Thanks guys, will try out  the above mentione Faust and let you know.

[Zoobie Doobie]

Hey! Is it necessary for you to use only the #kafka MQ, I mean you can opt for others like #rabbit or #zero MQ's if you can, that may solve the issue. Personally I didn't came across any such issues, still I'll try & share any necessary solution as I come across. BOL!!

--

[Sampath Reddy]

Yes want to go with Kafka MQ and that too with CONSUMER API running along with Django application to avoid network calls like post data to django application. Stream is not perhaps the best solution as of now, so want to try with kafka consumer If anyone used any python service or comeacross some library please share, will try out the approach.

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALTqLVd5G0TT1O5Xejv0ERaO7koKDtv4A%3DwPTwWPkum8ggXaSQ%40mail.gmail.com.