Using Django Channels to subscribe to a Redis channel

[Jochen Breuer]

Hi!

I'm not entirely sure where to start and I hope you can help me. I'm using SaltStack to execute tasks on minions. The results of those jobs (that's the term used in Salt) are then published to a job cache. In this case its a Redis server (Redis Pubsub). Now I would like to subscribe to a specific Redis channel, where the job results are published, with my Django (Channels) application. Every time a job result is pushed, I'd also like to push a message to a channel in Django.

Where to begin? Do I need to write a new protocol server or just a custom channel? Even after reading the docs I'm still lacking overview. Perhaps someone can push me into the right direction. Thanks!

Jochen

[Andrew Godwin]

Hi Jochen,

Your problem is that if you want to listen to the pubsub channel you will need a dedicated process to do so, as you can't just poll something like that. Given that restriction, you're going to have to write something like a management command that opens a connection to Redis and listens for messages, and then whenever it gets one, sends the message onto a channel in the Django channel system.

You can send to channels from anywhere, so it's just a basic listener with one line of channel send added. You're likely to want some sort of logic on these messages, I suspect, so I would send onto a single custom channel and then, now your messages are in the evented channel system, you can then write a consumer for that channel and tie it into the routing and handle any distribution/storage there.

This keeps the body of your logic inside the Django consumer code, and the management command nice and simple - it just needs to send onto a custom channel and not worry about what to do with that data, you can hook that up later. If you have fields in the message you split out into different dictionary keys, you can even use the Channels routing options on those fields - so, for example, you could route everything with a certain task name to one consumer, and everything else to a catch-all consumer.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c933a5b5-7b72-4d7d-985a-a311c0f16b26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jochen Breuer]

Hi Andrew,

thanks for your very detailed answer! This is much simpler that anticipated. You are absolutely right, a management command is the most simple approach here. That was the missing piece in my picture. Thank you very much!

Jochen

Am Mittwoch, 22. Februar 2017 19:55:33 UTC+1 schrieb Andrew Godwin:

Hi Jochen,

Your problem is that if you want to listen to the pubsub channel you will need a dedicated process to do so, as you can't just poll something like that. Given that restriction, you're going to have to write something like a management command that opens a connection to Redis and listens for messages, and then whenever it gets one, sends the message onto a channel in the Django channel system.

You can send to channels from anywhere, so it's just a basic listener with one line of channel send added. You're likely to want some sort of logic on these messages, I suspect, so I would send onto a single custom channel and then, now your messages are in the evented channel system, you can then write a consumer for that channel and tie it into the routing and handle any distribution/storage there.

This keeps the body of your logic inside the Django consumer code, and the management command nice and simple - it just needs to send onto a custom channel and not worry about what to do with that data, you can hook that up later. If you have fields in the message you split out into different dictionary keys, you can even use the Channels routing options on those fields - so, for example, you could route everything with a certain task name to one consumer, and everything else to a catch-all consumer.

Andrew

On Wed, Feb 22, 2017 at 6:55 AM, Jochen Breuer <bre...@gmail.com> wrote:

Hi!

I'm not entirely sure where to start and I hope you can help me. I'm using SaltStack to execute tasks on minions. The results of those jobs (that's the term used in Salt) are then published to a job cache. In this case its a Redis server (Redis Pubsub). Now I would like to subscribe to a specific Redis channel, where the job results are published, with my Django (Channels) application. Every time a job result is pushed, I'd also like to push a message to a channel in Django.

Where to begin? Do I need to write a new protocol server or just a custom channel? Even after reading the docs I'm still lacking overview. Perhaps someone can push me into the right direction. Thanks!

Jochen

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Michael]

Did you have luck with this. I've tried the same thing, but it seems like if no one is consuming on the other side I get a "channels.exceptions.ChannelFull" exception. 

I'm trying to provide a real-time feed for an exchange. There might be zero people listing or 1 million listening. The high level design should be the same. 

[morlandi]

I posted a working snippet here

https://gist.github.com/morlandi/bb915db7acef0ee0e4cb070921208610

It's a Django management command which subscribes a Redis channel, and upon receiving a new message sends it to django-channel.

You can skip the logging and redis connection boilerplate, and take a look at the loop() and broadcast() method which are the relevant part.

[Nasir Sh]

Thanks Andrew this helps me as well. There is a missing piece I don't understand yet. In my management command, then I probably will have to use `async_to_sync(channel_layer.group_send)` to send the message to consumers (right?). The only problem is that whenever I use async_to_sync it creates a new connection to redis every single time. I now use channels_redis v2.3.1 and in this version I don't get excessive connections but still it opens a new connection on every send. I was wondering if that is intentional or not in a sync context and if so, how can it be solved.

Thanks again for your great work

On Wednesday, February 22, 2017 at 7:55:33 PM UTC+1, Andrew Godwin wrote:

Hi Jochen,

Your problem is that if you want to listen to the pubsub channel you will need a dedicated process to do so, as you can't just poll something like that. Given that restriction, you're going to have to write something like a management command that opens a connection to Redis and listens for messages, and then whenever it gets one, sends the message onto a channel in the Django channel system.

You can send to channels from anywhere, so it's just a basic listener with one line of channel send added. You're likely to want some sort of logic on these messages, I suspect, so I would send onto a single custom channel and then, now your messages are in the evented channel system, you can then write a consumer for that channel and tie it into the routing and handle any distribution/storage there.

This keeps the body of your logic inside the Django consumer code, and the management command nice and simple - it just needs to send onto a custom channel and not worry about what to do with that data, you can hook that up later. If you have fields in the message you split out into different dictionary keys, you can even use the Channels routing options on those fields - so, for example, you could route everything with a certain task name to one consumer, and everything else to a catch-all consumer.

Andrew

On Wed, Feb 22, 2017 at 6:55 AM, Jochen Breuer <bre...@gmail.com> wrote:

Hi!

I'm not entirely sure where to start and I hope you can help me. I'm using SaltStack to execute tasks on minions. The results of those jobs (that's the term used in Salt) are then published to a job cache. In this case its a Redis server (Redis Pubsub). Now I would like to subscribe to a specific Redis channel, where the job results are published, with my Django (Channels) application. Every time a job result is pushed, I'd also like to push a message to a channel in Django.

Where to begin? Do I need to write a new protocol server or just a custom channel? Even after reading the docs I'm still lacking overview. Perhaps someone can push me into the right direction. Thanks!

Jochen

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Yavin Aalto Arba]

try with  self.channel_layer.group_send ?

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/800a3278-ba4b-402d-b8ee-341b999879ba%40googlegroups.com.

[Nasir Shadravan]

Thanks Yyavin, the problem is that in my management command I don't have access to `self` to send it. Is it possible to get access to a consumer from outside or how should my data producer relay the messages to channel in an async way?

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2B%2Be-ZVtt3kw9g-t2_D44aVvjRtBebbUTDbCdnKSDUWYBDRcBQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards
       Nasir Shadravan

[Nasir Sh]

Thanks Yyavin, the problem is that in my management command I don't have access to `self` to send it. Is it possible to get access to a consumer from outside or how should my data producer relay the messages to channel in an async way?

[Yavin Aalto Arba]

You need to set up the consumer to perform this task.

cf. https://github.com/justdjango/justchat/blob/master/src/chat/consumers.py

this corresponds to the following video tutorial which might be useful to you:

https://www.youtube.com/watch?v=Wv5jlmJs2sU

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/730e9a11-4226-4c7e-8252-e33518634418%40googlegroups.com.