Feedback on Django Channels

[Jacob Kaplan-Moss]

Hi folks (and especially Andrew):

I've just completed writing an example Channels app [1] for an article about Channels [2]. Overall it was a super-pleasant experience: Channels seems pretty solid, the APIs make sense to me, and I couldn't be more excited about the new things this'll let me do! 

In the interests of making this thing as solid as possible before we merge it into Django, I do have some feedback on some of the hiccups I encountered. Roughly in order of severity (as I perceive it), they are:

1. Debugging errors:

I found debugging errors that happen in a consumer to be *really* difficult -- errors mostly presented as things just silently not working. It took a ton of messing around with logging setups before I could get anything of use dumped to the console. This isn't strictly a Channels issue (I've noted similar problems with AJAX views and errors in Celery tasks), but I think Channels sorta brings the issue to a head. 

I think we need some better defaults, and simple, clear documentation, to make sure that exceptions go somewhere useful.

2. Static files:

I had trouble getting static files served. I'm used to using Whitenoise (http://whitenoise.evans.io/en/stable/) for small-to-medium-ish sites that don't need a proper static server, but of course it doesn't work with Channels since Channels doesn't use WSGI! I found the (undocumented) StaticFilesConsumer (https://github.com/jacobian/channels-example/blob/master/chat/routing.py#L5-L8), but that feels less than ideal.

I think this might be an opportunity, however. If Daphne learns how to serve static files (perhaps via optional integration with Whitenoise?), this would actually make static media in Django a bit easier by default.

[I would be happy to work on this if I get a thumbsup.]

3. WebSocket routing:

Channels routes all WebSocket connections to a single set of consumers (the `websocket.*` consumers). This means that if you want multiple WebSocket URLs in a single app you need to manually parse the path. And, to make things more complicated, you only get the WebSocket path passed in the connection message, so you have to also use a channel session to keep track.

This feels like a distinct step back from the URL routing we already have in Django, and it was surprising to me to have to do this by hand. It definitely felt like Channels is missing some sort of WebSocket URL router. 

I had a brief chat with Andrew, who indicates that he'd planned for this to be a post-1.0 feature. I'm not sure I agree - it feels pretty fundamental - but I'd like to hear other thoughts. 

[This is another thing I'd be interested in working on, assuming a thumbs.]

---

Has anyone else here played with Channels? Are there other things I'm missing that might need to be included before we merge this?

Jacob

[1] https://github.com/jacobian/channels-example

[2] https://blog.heroku.com/archives/2016/3/17/in_deep_with_django_channels_the_future_of_real_time_apps_in_django?

[Florian Apolloner]

On Thursday, March 17, 2016 at 5:42:05 PM UTC+1, Jacob Kaplan-Moss wrote:

Channels routes all WebSocket connections to a single set of consumers (the `websocket.*` consumers). This means that if you want multiple WebSocket URLs in a single app you need to manually parse the path. And, to make things more complicated, you only get the WebSocket path passed in the connection message, so you have to also use a channel session to keep track.

Yes, this seems like a major pain point, especially since the routing does not scale if you add another app, ie you need to add a wrapper which then dispatches to the individual connect routines. In a best case scenario I'd just have to add another router in the settings and get everything dispatched to the correct handlers (in that sense, the ROUTING setting should allow for multiple entries). Basically I think there needs to be a way to scope websocket.* to a single session(type) [And we should have this from the start]. Maybe something along the lines of:
```
channel_routing = {
  '^chat/(?P<room>\w+)': {
    'websocket.connect': …,
    'websocket.receive': …,
  },
  '^notifies/…': {
    'websocket.connect': …,
    'websocket.receive': …,
  },
  '^.*': {
    'http.request': my_handler
  }
}
```

So in essence: If the URL starts with "chat", the handlers for this "scope" should get called with room as argument (the data is already there, ) -- just like normal views. Same goes for notifications… I realize that this definition seems overly complicated, but this is the feature set I'd expect personally. Theoretically, the frontend server should have this information over the whole (websocket)session anyways and there shouldn't be any need for channel_session at all (That said, channel_session does not work with signed cookies ;)).

Cheers,
Florian

[Jacob Kaplan-Moss]

On Thu, Mar 17, 2016 at 1:44 PM, Florian Apolloner <f.apo...@gmail.com> wrote:

Yes, this seems like a major pain point, especially since the routing does not scale if you add another app, ie you need to add a wrapper which then dispatches to the individual connect routines. In a best case scenario I'd just have to add another router in the settings and get everything dispatched to the correct handlers (in that sense, the ROUTING setting should allow for multiple entries).

Ah yes, this was a point I forgot -- thanks for making it! Yes, if we want reusable apps that use websockets, we'll need some sort of routing to make that not painful.

 

Maybe something along the lines of:

[snip]

I'd considered an API like this, and it's certainly clean and straightforward. However, we've already got a URL routing library in Django, so I think I'd like to try to find a way to re-use it for websockets. It'd be a shame to have two different URL-routing-things in Django (especially since they'd almost certainly have subtlety different semantics).

Jacob

[Florian Apolloner]

On Thursday, March 17, 2016 at 9:56:40 PM UTC+1, Jacob Kaplan-Moss wrote:

I'd considered an API like this, and it's certainly clean and straightforward. However, we've already got a URL routing library in Django, so I think I'd like to try to find a way to re-use it for websockets. It'd be a shame to have two different URL-routing-things in Django (especially since they'd almost certainly have subtlety different semantics).

Absolutely, in my head I was thinking of putting url around those, or just refering to a URLConf (probably overkill), but I then opted for the most straight forward example. We should strive for the simplicity the current approach (in channels as of today) provides without making it too hard to decouple and split the handlers if needed.

[Andrew Godwin]

On Thu, Mar 17, 2016 at 1:41 PM, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:

1. Debugging errors:

I found debugging errors that happen in a consumer to be *really* difficult -- errors mostly presented as things just silently not working. It took a ton of messing around with logging setups before I could get anything of use dumped to the console. This isn't strictly a Channels issue (I've noted similar problems with AJAX views and errors in Celery tasks), but I think Channels sorta brings the issue to a head. 

I think we need some better defaults, and simple, clear documentation, to make sure that exceptions go somewhere useful.

Yes, this is definitely an issue - right now, they pretty much only log to stdout. I would like to at least:

a) Tie consumer errors into the same email-dispatch system for ADMINS as view errors, by default

b) Write better docs about how to handle errors in a channels-like scenario, and what they present as (as you said, generally just a lack of response, like most networking code errors)

 

2. Static files:

I had trouble getting static files served. I'm used to using Whitenoise (http://whitenoise.evans.io/en/stable/) for small-to-medium-ish sites that don't need a proper static server, but of course it doesn't work with Channels since Channels doesn't use WSGI! I found the (undocumented) StaticFilesConsumer (https://github.com/jacobian/channels-example/blob/master/chat/routing.py#L5-L8), but that feels less than ideal.

I think this might be an opportunity, however. If Daphne learns how to serve static files (perhaps via optional integration with Whitenoise?), this would actually make static media in Django a bit easier by default.

[I would be happy to work on this if I get a thumbsup.]

The static files consumer thing is merely for runserver, to match the capabilities of the existing WSGI one. We could take it further and make it an official public API, or work on something more reusable, or use whitenoise.

It's worth noting that you can wrap ASGI consumers in something approaching middleware inside Django - that's what StaticFilesConsumer does to ViewConsumer - so maybe we should consider making this a more formal thing and having a nice API for applying them.

 

3. WebSocket routing:

Channels routes all WebSocket connections to a single set of consumers (the `websocket.*` consumers). This means that if you want multiple WebSocket URLs in a single app you need to manually parse the path. And, to make things more complicated, you only get the WebSocket path passed in the connection message, so you have to also use a channel session to keep track.

This feels like a distinct step back from the URL routing we already have in Django, and it was surprising to me to have to do this by hand. It definitely felt like Channels is missing some sort of WebSocket URL router. 

I had a brief chat with Andrew, who indicates that he'd planned for this to be a post-1.0 feature. I'm not sure I agree - it feels pretty fundamental - but I'd like to hear other thoughts. 

 So, Florian's idea above is interesting, but unfortunately puts the HTTP-like parts at the outer level, which means it makes no sense for someone doing routing from something like a WAMP or SMS protocol server (to pick two protocols people might implement at some point).

The idea I had in my head looked more like this:

consumers = {

    'sms.receive': incoming_sms,

    'websocket.connect': UrlRouter(

        url("^$", root_ws_connect),

        url("^chat/([^/]+)/$", chat_ws_connect),

    ),

    'websocket.receive': UrlRouter(

        url("^$", root_ws_receive),

        url("^chat/", chat_ws_receive),

    ),

}

Where UrlRouter is a class that you configure with URLs the same way views are done, and then its __call__ merely passes the message down to the right consumer.

But yes, we do need this.

Andrew

[James Pic]

Perhaps it is a bit early for this but Is there anywhere origin is checked against ALLOWED_HOSTS ?

Middleware support would be nice to but I guess you'll come to that when implementing user authentication.

Keep up the great work !

[Vincent]

Jacob, Florian, Andrew,

I've spent the last 200 minutes thinking this through and writing, and here's what I've come up with:

https://gist.github.com/orokusaki/c67d46965a4ebeb3035a

Below are the full contents of that Gist (but I recommend the Gist for formatting).

I also created https://github.com/andrewgodwin/channels/issues/87 last weekend (re: your static files point above, Jacob).

### Problem

  1. Channel / URL routing is 2-dimensional (compared with 1D URL handling in Django)
  2. This creates a chicken / egg problem between channel name and URL path
  2. That's illustrated by the discrepancy between Florian's URL -> channel and Andrew's channel -> URL
  3. Put a Channel on the Y axis and URL are on the X axis, and the intersection is where a Consumer comes into play

### Considerations

Here are some design considerations for my API proposal:

  1. Includes - because nobody wants to all of their channels / URLs in a project-level module
  2. URL generation for channels with URLs
  3. The following duplicate include functionality works perfectly in Django currently with URLs
  4. `urlpatterns` are kepts in `urls.py`
  5. So, I've renamed `routing.py` to `channels.py` - after all, we're defining `channelpatterns`
  6. Either channel name OR URL has to come first, so that we don't need a 2D graph in Python for routing consumers

Project-level channels:

    # channels.py
    channelpatterns = [
        channel(r'^websocket\.', include('chat.channels', namespace='chat')),
        channel(r'^websocket\.', include('game.channels', namespace='game')),
        channel(
            r'^websocket',
            name='active-visitors',
            urls=[
                url(r'^active-visitors/$', VisitorCount.as_consumer()),
            ]
        ),
    ]

App-level channels:

    # game/channels.py
    channelpatterns = [
        channel(
            r'^receive',
            name='receive',
            urls=[
                url(r'^game/moves/up/$', Move.as_consumer(direction='up'), name='move-up'),
                url(r'^game/moves/down/$', Move.as_consumer(direction='down'), name='move-down'),
            ]
        ),
    ]

Channel routing would be handled similar to URLs in Django.

Given the above, getting the Channel URL for "moving up" in game could be: `{% channel 'game:receive:move-up' %}`

Here's an example of `websocket.connect` @ `/game/moves/up/`

  1. Encounter first match (the WebSocket channel named `chat`)
  2. Include `chat.channels`
  3. Determine there is no match
  4. Encounter second match (the WebSocket channel named `game`)
  5. Include `game.channels`
  6. Encounter first URL match (the `/game/moves/up/` pattern)
  7. Delegate to `Move` consumer with default `direction='up'`

But wait, there's more :)

Since Channel + URL is 2-dimensional, I propose we allow `include` at the `url` level and at the `channel` level.

    # channels.py
    channelpatterns = [
        channel(
            r'^websocket\.receive$',
            urls=[
                url(r'^game/', include('chat.channels', namespace='game')),
            ]
        ),
    ]

Then:

    # game/channels.py
    channelpatterns = [
        channel(
            r'^$',
            name='moves',
            urls=[
                url(r'^moves/up$', Move.as_consumer(direction='up'), name='move-up'),
                url(r'^moves/down$', Move.as_consumer(direction='down'), name='move-down'),
            ]
        )
    ]

Since I'm allowing `include` in a `channel` or any of its `urls`, either breadth-first search (giving URls / URL includes precedence) or depth-first search (giving channel includes precedence) could be used. I'd argue for breadth-first.

[Andrew Godwin]

Hi Vincent,

I think you make some good points - in particular, I think includes are probably a necessity, as you say. However, I disagree we should let people have either channel->url or url->channel; I'd like us not to try and give people multiple ways to do things if we can help it, especially in an ecosystem of third-party apps.

You can't rename routing.py to channels.py, however, as then the import names will clash with the `channels` third-party app if the user hasn't turned on absolute imports.

I'd build on your proposal and say the following:

---

Routing should be a list of pattern matches rather than a dict, so you can spread it across multiple includes sensibly. Each entry in the list will be a route() match object that takes a channel name (it has to be a name, no regular expressions here) as a required argument, and an optional `path` argument that's a regular expression matched against the incoming message `path`.

In fact, the routing will be generic, so you can regex against any primary message attribute you know will be present - for example, `method` for HTTP, or in the SMS example, by country code. It'll work like URL routing too and let you capture groups to be passed to the consumer (but only by name, to avoid confusion when a user matches against multiple message attributes and there's no defined ordering).

The include() function will take any keyword argument and do prefixing like it does now for URLs.

Example:

routing = [

    route("http.request", ViewConsumer),

    route("websocket.connect", path="^chat/(?P<room>[^/]+)/$", ChatConnect),

    route("sms.receive", sender="+44(?P<local_number>[0-9]+)$", UkSmsConsumer),

    include(path="^notifications", "notification.routing.routing"),

]

It also means that the simple example case stays quite readable:

routing = [

    route("websocket.connect", ws_connect),

    route("websocket.receive", ws_receive),

    route("websocket.disconnect", ws_disconnect),

]

We can also have channels upconvert the old dict format into this trivially to preserve all the existing code and examples, like Jacob's article.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/747fcf54-ff68-470c-ba0d-9e52b4cb85f6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ryan Hiebert]

On Mar 18, 2016, at 9:58 AM, Andrew Godwin <and...@aeracode.org> wrote:

routing = [

    route("http.request", ViewConsumer),

    route("websocket.connect", path="^chat/(?P<room>[^/]+)/$", ChatConnect),

    route("sms.receive", sender="+44(?P<local_number>[0-9]+)$", UkSmsConsumer),

    include(path="^notifications", "notification.routing.routing"),

]

Something's up with this example, it has postional arguments after keyword arguments, which is invalid Python. Is there supposed to be a keyword used for the final parameters?

[Andrew Godwin]

You're right. Assume I put the keyword arguments at the end, sorry - the two positional arguments are the channel name and the consumer.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/68C930D0-1C2E-4CF4-B3B8-DA8450C468AB%40ryanhiebert.com.

[Vincent]

Hey Andrew,

Thanks for looking through all that, and for the reply.

I like the simplicity of your updated examples. I started to make a counter-example to suggest that `include` be inside of a `route` (https://gist.github.com/orokusaki/c0c934013ee7911071ef).

But then, as I thought through this, I think I like your example almost* exactly like it is, but I'm afraid there might be a problem:

Given your example, any message how would a 'websocket.connect' message at the path `notifications/foo/` be routed, giving this example:

routing = [

    route("websocket.connect", ChatConnect),

    include(path="^notifications", "notification.routing.routing"),

]

Given that the chat route is the first match, wouldn't the notifications route never be used? Would path need to be required, so that the matching would be similar to `urlpatterns`? Otherwise, we're allowing routing based on channel name or path again? Maybe I'm misunderstanding.

[Vincent]

Also note, I just copy-pasted the same SyntaxError (kwarg before arg).

[Andrew Godwin]

On Fri, Mar 18, 2016 at 4:40 PM, Vincent <thebe...@gmail.com> wrote:

Hey Andrew,

Thanks for looking through all that, and for the reply.

I like the simplicity of your updated examples. I started to make a counter-example to suggest that `include` be inside of a `route` (https://gist.github.com/orokusaki/c0c934013ee7911071ef).

But then, as I thought through this, I think I like your example almost* exactly like it is, but I'm afraid there might be a problem:

Given your example, any message how would a 'websocket.connect' message at the path `notifications/foo/` be routed, giving this example:

routing = [

    route("websocket.connect", ChatConnect),

    include(path="^notifications", "notification.routing.routing"),

]

Given that the chat route is the first match, wouldn't the notifications route never be used? Would path need to be required, so that the matching would be similar to `urlpatterns`? Otherwise, we're allowing routing based on channel name or path again? Maybe I'm misunderstanding.

My idea there was that, like urlpatterns, it would go down the list in order, so if you did it like this: 

routing = [

    include("notification.routing.routing", path="^notifications"),

    route("websocket.connect", ChatConnect),

]

Then it would check the include first, before hitting the catchall. This is what you have to do with catchall URLs now, so I don't think it will be too unusual.

Andrew

[Vincent]

Andrew,

Ah, excellent. I just took a short break (too many LOC at work today), and while I was away I was thinking about all of this.

Here's what I came up with:

https://gist.github.com/orokusaki/17b4cf734b4d2f2af117

[Andrew Godwin]

I like most of it apart from the fact you can set a consumer to consume ANY channel, which seems incredibly dangerous - when a channel is wrongly consumed the only visible error is usually just a lack of response to the end client, and no two channels have messages that are similar in any useful way.

I think include should never need a channel, and route should always need one - which means the router always knows that every entry has exactly one channel defined that can be matched against.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9ce52a05-88d6-412f-9cfe-59c600c35df7%40googlegroups.com.

[Vincent]

Andrew,

Thanks for the explanation.

(re: including based only on path, routing based on channel (and optionally path?)), I really really like that simplicity, the more I think about it.

Expanding on that to include your prior examples, I'm assuming `path` is just incidental for http / websocket, and that other kwargs would be relevant for `include`:

routing = [
    # Catches any US phone number and delegates to myapp.us.routing.routing
    include('my_us_app.routing', sender=r'\+1(?P<local_number>[0-9]+)$'),

    # Catches any UK phone number and routes to UkSmsConsumer
    route('sms.receive', sender=r'\+44(?P<local_number>[0-9]+)$', UkSmsConsumer),
]

Is that right?

[Andrew Godwin]

Yes, my intent is that "path" is just a stand-in for "any string key in a message", thus you could route other ways (for example, `method` in http.request, though that's probably not super useful inside Django).

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/67476f8c-6cce-428d-93bd-562ffaabe2ff%40googlegroups.com.

[David Evans]

On the static files question, I'm about to release v3 of WhiteNoise (http://whitenoise.evans.io/en/latest/changelog.html) which provides the option to integrate via Django middlware, rather than wsgi middleware. (It uses the FileResponse class, which didn't exist when I first wrote WhiteNoise.) I'm hoping (though I haven't tested it yet) that this will work out of the box with Channels, as it's only using standard Django APIs.

Dave

[Sean Brant]

How does the new channels model handle requests with sensitive information? Say you have a login view and the user is submitting a username/password. Does the password get serialized into the message queue as plain text? If so is that a security concern users should be aware of?

Sean

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ba29cc7a-90fc-4cf2-a65d-da782e83b3ca%40googlegroups.com.

[Andrew Godwin]

On Mon, Mar 21, 2016 at 9:55 AM, David Evans <drh....@gmail.com> wrote:

On the static files question, I'm about to release v3 of WhiteNoise (http://whitenoise.evans.io/en/latest/changelog.html) which provides the option to integrate via Django middlware, rather than wsgi middleware. (It uses the FileResponse class, which didn't exist when I first wrote WhiteNoise.) I'm hoping (though I haven't tested it yet) that this will work out of the box with Channels, as it's only using standard Django APIs.

Yes, as long as you use the standard Django file response stuff it should work just fine (I made sure of that so staticfiles' serving class worked).

The only difference is that Channels prefers a chunking size of around 256KB, whereas the default is 2KB, which causes a whole lot of messages, but I can move the code that adjusts it up for FileResponses to the main handling code rather than just Staticfiles, so it should be fine.

Andrew

[Andrew Godwin]

On Mon, Mar 21, 2016 at 12:11 PM, Sean Brant <brant...@gmail.com> wrote:

How does the new channels model handle requests with sensitive information? Say you have a login view and the user is submitting a username/password. Does the password get serialized into the message queue as plain text? If so is that a security concern users should be aware of?

With the default backend (redis), yes, it goes over the wire and into Redis as plain text - the overhead of both on-the-wire and at-rest encryption would be reasonably high, so I didn't include it by default; however, I very much hope we'll get at least one channel layer implementation that does do this.

For Redis, at least, you can run a TLS tunnel between machines and point Channels at the local end of the tunnel to get transport security (as it has none of its own). At-rest is a bit harder - I plan to just turn my redis nodes into memory-only mode so that the data is never persisted, at least (channels doesn't need persistance beyond about a minute), but I can imagine a channel layer where you pass a symmetric encryption key in its configuration or similar.

Andrew

[Josh Smeaton]

Assuming the frontend has access to DJANGO_SETTINGS_MODULE, couldn't it use the SECRET_KEY to encrypt the message before passing it to the message broker? On message receipt it could then use the SECRET_KEY to decrypt the message. It'd be nice if encryption were an option encoded within the message or route somehow, so it could be optionally turned on for sensitive data. I'm guessing that there's probably an issue with my assumptions though.

[Andrew Godwin]

The channel layer could indeed use SECRET_KEY, since it's loaded in via a Django codepath in both the protocol server and the worker server.

It's better done as a channel layer feature rather than in Channels/Daphne directly, as then it requires no extra supporting code in anything else that does ASGI (like the WSGI adapter). The layer could also have options to turn it on only for certain channels (e.g. only http.request) - it would be harder to make deeper encryption choices, though, as things like the body content and headers both come through as a single message key. Maybe let people provide a regex to match against e.g. http.request and websocket.connect's `path` key.

It's probably something we could pay for someone to work on for the "main" (Redis-backed) layer implementation? I could add it in too, but it's not as high priority as some other stuff I have to get to.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3002448c-f36d-4f70-a2fd-c4d5e9bb2c34%40googlegroups.com.

[Jacob Kaplan-Moss]

I do think encrypting the Redis channel layer is something we should offer: Redis out of the box doesn't do transport-layer encryption, which is going to make Channels a hard sell to anyone with any for of regulatory/compliance requirements. [1]

I think probably Fernet [2] is the right way to do this -- since we've got a key we can share between Daphne and Django -- though I'd welcome the input of people who know more about crypto than me!

Jacob

[1] I do view it more as a compliance problem than a security one. You really shouldn't be routing things like channels over public networks, and if someone's snooping on your private network you've probably got bigger problems than unencrypted channels. Nonetheless, encryption-in-transit is a basic requirement of nearly every regulatory regime, and we should do what we can to help.

[2] https://cryptography.io/en/latest/fernet/

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urDr0LWvgK%2BRx9maEPezoFxqyFXCzNQbHedJwwvJM9pgw%40mail.gmail.com.

[Donald Stufft]

On Mar 22, 2016, at 1:24 PM, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:

I do think encrypting the Redis channel layer is something we should offer: Redis out of the box doesn't do transport-layer encryption, which is going to make Channels a hard sell to anyone with any for of regulatory/compliance requirements. [1]

I don’t know a whole lot about the design of channels, so ymmv but I tend to be a bit nervous on trying to implement a “mini” replacement to something like TLS because it’s fairly tricky to get right. Even using something like fernet, you still have things like replay attacks and such that may be possible (unless channels design prevents it) that TLS already has baked in protection for.

It is true that reds doesn’t do TLS out of the box, but adding it is not really *that* hard, you typically just bind redis to a localhost only port (which I think it does by default) and then install stunnel bound to a public port to “unwrap” the TLS before passing it on to redis. On the client side you may need to also run stunnel to “wrap” the stream with TLS if your client does not support it, but redis-py supports TLS’d connections natively so, at least from Python, it’s not any harder to connect to a TLS’d Redis server.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Andrew Godwin]

Indeed, we run Redis over TLS tunnels at work to fulfill this requirement, so I know transport security is required, but at the same time reinventing it might be more work than we need - would you trust our internal symmetric encryption system, or go for TLS tunnels instead?

Still, if we want to do it, Fernet seems like a sensible choice, given that it looks like we can just run through a key derivation function with the SECRET_KEY and then use that in place. Key rotation is likely not necessary as a feature since messages expire very quickly and are designed to be lost occasionally.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJEMx6Mk4_O1cfYECxechKXMViNyLXXSoaK4eGRQjpEdxw%40mail.gmail.com.

[Michael Manfre]

On Tue, Mar 22, 2016 at 1:44 PM, Andrew Godwin <and...@aeracode.org> wrote:

Indeed, we run Redis over TLS tunnels at work to fulfill this requirement, so I know transport security is required, but at the same time reinventing it might be more work than we need - would you trust our internal symmetric encryption system, or go for TLS tunnels instead?

If not provided out of the box, there needs to be a supported way of wiring in encryption. The security/compliance person at my job stated that only securing the transport was not good enough for our compliance requirements when I was dealing with HIPAA (and some other compliance regulations) a few months ago.

Regards,

Michael Manfre

[Andrew Godwin]

On Tue, Mar 22, 2016 at 4:04 PM, Michael Manfre <mma...@gmail.com> wrote:

If not provided out of the box, there needs to be a supported way of wiring in encryption. The security/compliance person at my job stated that only securing the transport was not good enough for our compliance requirements when I was dealing with HIPAA (and some other compliance regulations) a few months ago.

Alright, good to know. I'll put a task down in my notes for message encryption and we can add it to the list of things people can work on and get paid for when we put that up.

Andrew 

[Shai Berger]

Hi,

Finally found the time to go through this discussion.

The first note that comes to mind is:

Although it has already been pointed out more than once that positional
arguments cannot follow keyword arguments, you both (Andrew and Vincent) keep
giving examples such as

# SYNTAX ERROR
route('sms.receive', sender=r'\+44(?P<local_number>[0-9]+)$', Consumer)

I believe the reason for that is the clash between the will to preserve the
current `route(channel, consumer)` syntax, which is like the parallel
`url(path_regex, view, **kw)` syntax, on one hand; and the instinct that the
identifying parameters should be together, on the other hand.

I think that the latter consideration prevails; unlike the kw parameters in
url(), which are essentially parameters to the view, the kw parameters
proposed for channel help select the consumer; and so they must be adjacent to
the channel name -- and hence, the consumer must be the first argument.

A second thought is about multiple keyword arguments:

It isn't clear if more than one keyword argument is allowed, and if so, what
should be the semantics of the combination. When thinking about routing, it is
almost obvious that if there are many kw arguments, they must all match for
the whole route to match. But when thinking about includes, a different
consideration arises: How do I specify routing in an app which deals with
several incompatible channels?

For argument's sake, let's say I have an app which can handle SMS's, mails and
websocket.connect's. I want it to handle all of these if they come from within
my company -- so, based on partial phone number, sender domain and client IP
addresses, respectively. Would I need to have the app's routing.py module
include()'ed three times? Or would it make more sense to give include() kw
parameters an "or" semantics -- that is, match as soon as one argument
matches?

I think both of these solutions are bad, and the only solution that makes
sense is to allow a more complex structure of argument -- so that multiple kw
args are all required to match, and for a disjunction of channels we use
positional channel-spec args -- something like,

include('my_app.routing',
chan_spec('sms.receive', sender=r'\1212555(?:[0-9]{4})$'),
chan_spec('mail.receive', sender=r'^(?:\W+)@my_org.com$),
chan_spec('websocket.connect', remote_ip='^172.10'),
)

where a set of kw parameters are considered to all belong to one implicit
chan_spec.

My 2 cents,
Shai.

[Andrew Godwin]

On Fri, Apr 1, 2016 at 11:32 PM, Shai Berger <sh...@platonix.com> wrote:

Hi,

Finally found the time to go through this discussion.

The first note that comes to mind is:

Although it has already been pointed out more than once that positional
arguments cannot follow keyword arguments, you both (Andrew and Vincent) keep
giving examples such as

        # SYNTAX ERROR
        route('sms.receive', sender=r'\+44(?P<local_number>[0-9]+)$', Consumer)

I believe the reason for that is the clash between the will to preserve the
current `route(channel, consumer)` syntax, which is like the parallel
`url(path_regex, view, **kw)` syntax, on one hand; and the instinct that the
identifying parameters should be together, on the other hand.

I think that the latter consideration prevails; unlike the kw parameters in
url(), which are essentially parameters to the view, the kw parameters
proposed for channel help select the consumer; and so they must be adjacent to
the channel name -- and hence, the consumer must be the first argument.

Ah, that's an interesting ordering I hadn't considered before - the problem is, I would have it (channel, kwargs, consumer) if I could, as generally when I'm reading the file I want to identify things by route. For that reason, I still like having the channel at the beginning as it makes it obvious what the rest of the line is dealing with (websocket? http? etc.)

 

A second thought is about multiple keyword arguments:

It isn't clear if more than one keyword argument is allowed, and if so, what
should be the semantics of the combination. When thinking about routing, it is
almost obvious that if there are many kw arguments, they must all match for
the whole route to match. But when thinking about includes, a different
consideration arises: How do I specify routing in an app which deals with
several incompatible channels?

For argument's sake, let's say I have an app which can handle SMS's, mails and
websocket.connect's. I want it to handle all of these if they come from within
my company -- so, based on partial phone number, sender domain and client IP
addresses, respectively. Would I need to have the app's routing.py module
include()'ed three times? Or would it make more sense to give include() kw
parameters an "or" semantics -- that is, match as soon as one argument
matches?

The way I've done this in the chat example is two includes:

https://github.com/andrewgodwin/channels-examples/blob/master/multichat/multichat/routing.py

https://github.com/andrewgodwin/channels-examples/blob/master/multichat/chat/routing.py

 

I think both of these solutions are bad, and the only solution that makes
sense is to allow a more complex structure of argument -- so that multiple kw
args are all required to match, and for a disjunction of channels we use
positional channel-spec args -- something like,

        include('my_app.routing',
                chan_spec('sms.receive', sender=r'\1212555(?:[0-9]{4})$'),
                chan_spec('mail.receive', sender=r'^(?:\W+)@my_org.com$),
                chan_spec('websocket.connect', remote_ip='^172.10'),
        )

where a set of kw parameters are considered to all belong to one implicit
chan_spec.

I'm not entirely convinced this is better than just having three includes; it would be basically the same number of lines, and would work just as you suggest, with the added benefit of no extra thing to import.

Is there something about this method you think is better than three includes? Remember, routing specifies the variable inside the module (unlike url includes, which only specify the module) so you can still keep the three routing sets in the one routing.py inside the app.

Andrew 

[Shai Berger]

Well, in that case I would consider defining the consumer as a required, but
keyword (and keyword-only in Python3) argument, specified in the end by
convention. Putting it between the channel and channel parameters is ugly IMO.

> > A second thought is about multiple keyword arguments:
> >
>

> Is there something about this method you think is better than three
> includes? Remember, routing specifies the variable inside the module
> (unlike url includes, which only specify the module) so you can still keep
> the three routing sets in the one routing.py inside the app.
>

I somehow missed this, it makes my entire argument moot. Agreed.

Thanks,
Shai.

[Andrew Godwin]

Well, in that case I would consider defining the consumer as a required, but
keyword (and keyword-only in Python3) argument, specified in the end by
convention. Putting it between the channel and channel parameters is ugly IMO.

It would be easy enough to change all the docs and examples to work this way - specifying "consumer=" as a kwarg will already work. I'm slightly concerned about forcing keyword-only, though.

Andrew