Testing Channels 2.0 consumers - problem with routing

861 views
Skip to first unread message

Azamat Galimzhanov

unread,
Feb 7, 2018, 4:46:19 PM2/7/18
to Django users
Hello,

I'm setting up an auctions application using channels and websockets. When I'm running the server the code works just fine, troubles arise when I'm trying to test my app.

asgiref==2.1.3
daphne==2.0.2
Django==2.0.2
channels==2.0.1
channels-redis==2.0.2
pytest==3.4.0
pytest-asyncio==0.8.0

a bunch of other stuff, but I think that's it for relevant libraries.

Here is my consumer:

class AuctionConsumer(WebsocketConsumer):
    @property
    @lru_cache(maxsize=None)
    def group_name(self):
        auction_id = self.scope['url_route']['kwargs']['auction_id']
        return 'auction_{}'.format(auction_id)

    def connect(self):
        AsyncToSync(self.channel_layer.group_add)(self.group_name, self.channel_name)
        self.accept()

Here is my routing:

application = ProtocolTypeRouter({
    'websocket': URLRouter([
        path("auctions/<auction_id>/", AuctionConsumer),
    ]),
})

And here is my test:


@pytest.mark.asyncio
async def test_auction_consumer(mocker):
    communicator = WebsocketCommunicator(AuctionConsumer, 'auctions/1/')
    connected, subprotocol = await communicator.connect()
    assert connected
    # Test sending text
    await communicator.send_to(text_data='1')
    response = await communicator.receive_from()
    assert response == '1'
    # Close
    await communicator.disconnect()


I'm running into the following problem;

    @property
    @lru_cache(maxsize=None)
    def group_name(self):
>       auction_id = self.scope['url_route']['kwargs']['auction_id']
E       KeyError: 'url_route'

I tried mocking at the top of my test:

    group_name = mocker.patch('auctions.consumers.AuctionConsumer.group_name')
    group_name.return_value = 'auction'

But then I run into:

    def valid_group_name(self, name):
        if self.match_type_and_length(name):
            if bool(self.group_name_regex.match(name)):
                return True
        raise TypeError(
            "Group name must be a valid unicode string containing only ASCII " +
>           "alphanumerics, hyphens, or periods."
        )
E       TypeError: Group name must be a valid unicode string containing only ASCII alphanumerics, hyphens, or periods.


I also tried:

scope = mocker.patch('auctions.consumers.AuctionConsumer.scope')
scope.return_value = {'url_route': {}}

But that got me:

        if not self.create and original is DEFAULT:
            raise AttributeError(
>               "%s does not have the attribute %r" % (target, name)
            )
E           AttributeError: <class 'auctions.consumers.AuctionConsumer'> does not have the attribute 'scope'


At this point I'm out of ideas. Anything I should try?

Thanks in advance.

Andrew Godwin

unread,
Feb 7, 2018, 5:07:38 PM2/7/18
to django...@googlegroups.com
I'm not going to address the mocking, as there's subtle details there (e.g. you mocked group_name as a non-property, I think), but look at the main issue.

Can you make your code print out what _is_ in `self.scope`? url_route should definitely be in there, but I am curious what it looks like if it is not.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/528e93c5-dfb1-4cc1-a599-b99404c59533%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Azamat Galimzhanov

unread,
Feb 9, 2018, 6:58:21 AM2/9/18
to Django users
Thanks for reply!

self.scope looks like this:

{'type': 'websocket', 'path': 'auctions/1/', 'headers': [], 'subprotocols': []}

I am not sure why url_route is not there. I rewrote my group_name code to use path instead and now this part works.

There is a new error though, during disconnect, when calling this part of the test:

await communicator.disconnect()

Here is disconnect method of my consumer, but I think the problem isnt here:

def disconnect(self):
    AsyncToSync(self.channel_layer.group_discard)(self.group_name, self.channel_name)

This is the pytest output:

>       await communicator.disconnect()

auctions/tests/test_consumers.py:18: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/channels/testing/websocket.py:100: in disconnect
    await self.future
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/channels/consumer.py:53: in __call__
    await await_many_dispatch([receive, self.channel_receive], self.dispatch)
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/channels/utils.py:48: in await_many_dispatch
    await dispatch(result)
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/asgiref/sync.py:103: in inner
    return await async_func(*args, **kwargs)
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/asgiref/sync.py:83: in __call__
    return await asyncio.wait_for(future, timeout=None)
/usr/lib/python3.6/asyncio/tasks.py:339: in wait_for
    return (yield from fut)
/usr/lib/python3.6/concurrent/futures/thread.py:56: in run
    result = self.fn(*self.args, **self.kwargs)
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/asgiref/sync.py:95: in thread_handler
    raise e
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/asgiref/sync.py:93: in thread_handler
    self.func(*args, **kwargs)
/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/channels/consumer.py:98: in dispatch
    handler(message)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <auctions.consumers.AuctionConsumer object at 0x7f098d2a4198>, message = {'code': 1000, 'type': 'websocket.disconnect'}

    def websocket_disconnect(self, message):
        """
            Called when a WebSocket connection is closed. Base level so you don't
            need to call super() all the time.
            """
        # TODO: group leaving
>       self.disconnect(message["code"])
E       TypeError: disconnect() takes 1 positional argument but 2 were given

/home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/channels/generic/websocket.py:86: TypeError
============================================================================================ 1 failed in 0.36 seconds =============================================================================================
Task was destroyed but it is pending!
task: <Task pending coro=<RedisConnection._read_data() running at /home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/aioredis/connection.py:181> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f098d344ac8>()]> cb=[Future.set_result()]>



P.S. I have a couple more problems with testing this particular project related to channels. For example I tried ignoring this disconnect problem, this test passed with 99% coverage, but when I run the whole test suite it breaks and I figured out what other tests break this one, but not sure why. Would you mind looking at this too? If yes, should I create a new thread or continue in this one?

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

Andrew Godwin

unread,
Feb 9, 2018, 1:03:53 PM2/9/18
to django...@googlegroups.com
A disconnect() method in your generic consumer takes one argument, a `close_code` - add that in to your function and it will fix it.

Feel free to continue problems in this thread, or you can also use GitHub issues on the channels project if you think they're actual bugs!

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c8bcb0e6-9792-4762-b031-a48b7c5e8389%40googlegroups.com.

Azamat Galimzhanov

unread,
Feb 11, 2018, 11:54:08 AM2/11/18
to Django users
After fixing this issue, here is the updated consumer:


from functools import lru_cache

from asgiref.sync import AsyncToSync
from channels.generic.websocket import WebsocketConsumer



class AuctionConsumer(WebsocketConsumer):
    @property
    @lru_cache(maxsize=None)
    def group_name(self):
        path = self.scope['path']
        auction_id = path[len('auctions/'):-1]

        return 'auction_{}'.format(auction_id)

    def connect(self):
        AsyncToSync(self.channel_layer.group_add)(self.group_name, self.channel_name)
        self.accept()

    def disconnect(self, close_code):
        AsyncToSync(self.channel_layer.group_discard)(self.group_name, self.channel_name)

    def receive(self, text_data):
        AsyncToSync(self.channel_layer.group_send)(
            self.group_name,
            {
                'type': 'auction.bid',
                'text': text_data,
            },
        )

    def auction_bid(self, event):
        self.send(text_data=event['text'])

 some of my other tests seem to have influence test_consumers, so I set up a simplified test case, here it is:

from channels.layers import get_channel_layer
from asgiref.sync import AsyncToSync

import pytest
from channels.testing import WebsocketCommunicator
from auctions.consumers import AuctionConsumer


def test_A():
    channel_layer = get_channel_layer()
    AsyncToSync(channel_layer.group_send)(
        'auction_1',
        {
            'type': 'auction.bid',
            'text': '1'
        }
    )


@pytest.mark.asyncio
async def test_B():
    communicator = WebsocketCommunicator(AuctionConsumer, 'auctions/1/')
    connected, subprotocol = await communicator.connect()
    assert connected
    # Test sending text
    await communicator.send_to(text_data='1')
    response = await communicator.receive_from()
    assert response == '1'
    # Close
    await communicator.disconnect()

And individually they pass:

 !  ◰³ django_root-FXsg3jFU  /m/k/D/d/auction   41/bidding_websockets *…  django_root  pytest -v -s auctions/tests/test_test.py::test_A                                5.8s  Sun 11 Feb 2018 10:46:12 PM +06
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.6.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0 -- /home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/bin/python3.6m
cachedir: .pytest_cache
Django settings: core.settings (from ini file)
rootdir: /media/khazidhea/Data/dev/auction/django_root, inifile: setup.cfg
plugins: mock-1.6.3, flake8-0.9.1, django-3.1.2, cov-2.5.1, asyncio-0.8.0, celery-4.1.0
collected 1 item                                                                                                                                                                                                  

auctions/tests/test_test.py::test_A PASSED

============================================================================================ 1 passed in 0.07 seconds =============================================================================================
 ◰³ django_root-FXsg3jFU  /m/k/D/d/auction   41/bidding_websockets *…  django_root  pytest -v -s auctions/tests/test_test.py::test_B                                  4053ms  Sun 11 Feb 2018 10:46:20 PM +06
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.6.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0 -- /home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/bin/python3.6m
cachedir: .pytest_cache
Django settings: core.settings (from ini file)
rootdir: /media/khazidhea/Data/dev/auction/django_root, inifile: setup.cfg
plugins: mock-1.6.3, flake8-0.9.1, django-3.1.2, cov-2.5.1, asyncio-0.8.0, celery-4.1.0
collected 1 item                                                                                                                                                                                                  

auctions/tests/test_test.py::test_B PASSED

============================================================================================ 1 passed in 0.09 seconds =============================================================================================
Task was destroyed but it is pending!
task: <Task pending coro=<RedisConnection._read_data() done, defined at /home/khazidhea/.local/share/virtualenvs/django_root-FXsg3jFU/lib/python3.6/site-packages/aioredis/connection.py:175> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f19b9b6bbe8>()]> cb=[Future.set_result()]>




But when I run both (I think specifically when B is run after A) I get a huge traceback with this in the end:

    def _check_closed(self):
        if self._closed:
>           raise RuntimeError('Event loop is closed')
E           RuntimeError: Event loop is closed


Full stack trace is here: https://pastebin.com/ZeiMVZD7

Why could it be?

Andrew Godwin

unread,
Feb 11, 2018, 6:57:00 PM2/11/18
to django...@googlegroups.com
Hi,

This is a known issue with the Redis backend - https://github.com/django/channels/issues/859 

I am looking at it today.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a7ad793c-fa01-452a-b03d-5084d67ba353%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages