How to pass data between Django module/app functions without using database in asynchronous web service

[Ken McDonald]

I've got a web service under development that uses Django and Django Channels to send data across websockets to a remote application. The arrangement is asynchronous and I pass information between the 2 by sending JSON formatted commands across websockets and then receive replies back on the same websocket.

The problem I'm having is figuring out how to get the replies back to a Javascript call from a Django template that invokes a Python function to initiate the JSON websocket question. Since the command question & data reply happen in different Django areas and the originating Javascript/Python functions call does not have a blocking statement, the Q&A are basically disconnected and I can't figure out how to get the results back to the browser.

Right now, my idea is to use Django global variables or store the results in the Django models. I can get either to work, but I beleive the Django global variables would not scale beyond multiple workers from runserver or if the system was eventually spread across multiple servers.

But since the reply data is for different purposes (for example, list of users waiting in a remote lobby, current debugging levels in remote system, etc), the database option seems unworkable because the reply data is varying structure. That, plus the replies are temporal and don't need to be permanently stored in the database.

Here's some code showing the flow. I'm open to different implementation recommendations or a direct answer to the question of how to share information between the 2 Django functions.

In the template, for testing, I just have a button defined like this:

    <button id="request_lobby">Request Lobby</button>

With a Javascript function. This function is incomplete as I've yet to do anything about the response (because I can't figure out how to connect it):

            $("#request_lobby").click(function(){
                $.ajax({
                    type: "POST",
                    url: "{% url 'test_panel_function' %}",
                    data: { csrfmiddlewaretoken: '{{ csrf_token }}', button:"request_lobby" },
                    success: function(response){
                    }
                });
            });

This is the Django/Python function in views.py . The return channel for the remote application is pre-stored in the database as srv.server_channel when the websocket is initially connected (not shown):

    @login_required
    def test_panel_function(request):

        button = request.POST.get('button', '')

        if button == "request_lobby" :
            srv = Server.objects.get(server_key="1234567890")
            json_res = []
            json_res.append({"COMMAND": "REQUESTLOBBY"})
            message = ({
                "text": json.dumps(json_res)
            })
            Channel(srv.server_channel).send(message)
            return HttpResponse(button)

Later, the remote application sends the reply back on the websocket and it's received by a Django Channels demultiplexer in routing.py :

    class RemoteDemultiplexer(WebsocketDemultiplexer):
        mapping = {
            "gLOBBY"   : "gLOBBY.receive",
        }
        http_user = True
        slight_ordering = True
         
       
    channel_routing = [
        route_class(RemoteDemultiplexer, path=r"^/server/(?P<server_key>[a-zA-Z0-9]+)$"),
        route("gLOBBY.receive"   , command_LOBBY),
    ]

And the consumer.py :

    @channel_session    
    def command_LOBBY(message):
        skey = message.channel_session["server_key"]
        for x in range(int(message.content['LOBBY'])):
            logger.info("USERNAME:  " + message.content[str(x)]["USERNAME"])
            logger.info("LOBBY_ID:  " + message.content[str(x)]["LOBBY_ID"])
            logger.info("OWNER_ID:  " + message.content[str(x)]["IPADDRESS"])
            logger.info("DATETIME:  " + message.content[str(x)]["DATETIME"])

So I need to figure out how to get the reply data in command_LOBBY to the Javascript/Python function call in test_panel_function

Current ideas, both of which seem bad and why I think I need to ask this question for SO:

1) Use Django global variables:

Define in globals.py:

    global_async_result = {}

And include in all relevant Django modules:

    from test.globals import global_async_result

In order to make this work, when I originate the initial command in test_panel_function to send to the remote application (the REQUESTLOBBY), I'll include a randomized key in the JSON message which would be round-tripped back to command_LOBBY and then global_async_result dictionary would be indexed with the randomized key.

In test_panel_function , I would wait in a loop checking a flag for the results to be ready in global_async_result and then retrieve them from the randomized key and delete the entry in global_async_result.

Then the reply can be given back to the Javascript in the Django template.

That all makes sense to me, but uses global variables (bad), and seems that it wouldn't scale as the web service is spread across servers.

2) Store replies in Django mySQL model.py table

I could create a table in models.py to hold the replies temporarily. Since Django doesn't allow for dynamic or temporary table creations on the fly, this would have to be a pre-defined table.

Also, because the websocket replies would be different formats for different questions, I could not know in advance all the fields ever needed and even if so, most fields would not be used for differing replies.

My workable idea here is to create the reply tables using a field for the randomized key (which is still routed back round-trip through the websocket) and another large field to just store the JSON reply entirely.

Then in test_panel_function which is blocking in a loop waiting for the results, pull the JSON from the table, delete the row, and decode. Then the reply can be given back to the Javascript in the Django template.

3) Use Django signals

Django has a signals capability, but the response function doesn't seem to be able to be embedded (like inside test_panel_function) and there seems to be no wait() function available for an arbitrary function to just wait for the signal. If this were available, it would be very helpful

[Andrew Godwin]

Hi Ken,

I'm mostly clear on what's happening here (you should definitely not use global variables for the reasons you mention), but I have a few questions to try and clear things up:

- Are the websocket and the AJAX calls going to different clients/pages entirely, rather than them both being in the same page context? It sounds like the websocket is going to a server somewhere that's allocating resources and not going to a webpage/browser at all.

- Could it potentially take the websocket client that is sending the LOBBY data back that needs to then be conveyed over AJAX some time to reply? Say, more than ten seconds?

I think I have an idea of what you're trying to do here and how to solve it, but I just need to be a bit more clear on the architecture first.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/728d41d3-2fbf-405f-bf5a-b861deaf42be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ken McDonald]

On Thursday, December 29, 2016 at 6:00:03 AM UTC-5, Andrew Godwin wrote:

Q) Are the websocket and the AJAX calls going to different clients/pages entirely, rather than them both being in the same page context? It sounds like the websocket is going to a server somewhere that's allocating resources and not going to a webpage/browser at all.

A) Currently the websockets are used in a background system running on Django and most of the use for websockets on this service are so the remote system we are interfacing with can pull a list of subscribers from a Django model and then the remote system does uncoordinated stuff with that. Occasionally, the remote system updates data to Django models for debugging information or sending statistics back.

In the normal course of the application, the web server web page interface is between user and Django model and has no need for a direct connection to the remote system through websockets; they are just setting up subscription types as a seller or buying subscriptions as a buyer. Later, when they are using the remote system, which happens with a completely different client native compiled application, not Django or web interface, they make use of the real purpose of our system which is enhanced by the subscriptions we sell. Basically, we
are using the Django web service to sell subscriptions which are then periodically download through websockets to a remote native compiled application.
 
On Thursday, December 29, 2016 at 6:00:03 AM UTC-5, Andrew Godwin wrote:

Q) - Could it potentially take the websocket client that is sending the LOBBY data back that needs to then be conveyed over AJAX some time to reply? Say, more than ten seconds?

A) The return replies are just about instantaneous and very small in size. Although ultimately the system will interface with potentially 1000's of distinct remote systems, the Django web service is only used for a very, very short period of time by a user, and then maybe never again if they just continue to renew their subscription. The users are at the remote systems. We just sell stuff that makes that experience better. The need for displaying return data as envisioned in my original question is a rare occurannce almost exclusively used by sellers and more so, just by administrators of the system.
 
On Thursday, December 29, 2016 at 6:00:03 AM UTC-5, Andrew Godwin wrote:

Q) I think I have an idea of what you're trying to do here and how to solve it, but I just need to be a bit more clear on the architecture first.

A) My primary purpose for the django-channels to remote system is working great. I can download from a model and upload to a model through websockets (just for reference: the interface at the remote system is a C++ compiled application (w/ websockets) connected to a C# final product).

There are various need, by this example can best describe a single feature:

1) We have a debugging system on the remote system that will upload transactions from there to our Django model through websockets.
2) In that debugging system, we can set various levels of verbosity to control how much detail is returned.
3) I have a working Django web page now that can send commands that change the debugging level through a websocket-routed command in a JSON message with a particular debugging level (like 1-7). This web page uses a Django template with a radio button form for the 1-7 choices.
4) When a choice is changed on the form, Javascript catches it and calls a Django function that send the websocket command to the remote system to change the debugging (we track websocket instances of channel names in a database, so I can always contact a particular remote system by knowing our own internally-assign db id to lookup the Django channel name for that websocket connection).

In the above example, what I am looking for is that when the web page is initially rendered (url'd in browser), fetch the current debugging level from the remote system and pre-set the radio buttons. Right now of course, the page just renders with a default layout of the radio button without reference to what the current debugging level actually is at the remote site.

In addition to the websocket JSON command to tell the remote system to set a particular debugging level, I also have a command that requests the current debugging level. It's connecting the reply back data from that request command to the webpage when it's initially rendered that is proving difficult to understand.

Thanks for any help you or anyone can provide. I'm dealing with a lot of technologies and a lot of languages that I'm not totally expert or experienced with, but I will complete this project in one form or another. My system engineering background leads me to want a design that is more push and event-driven, than pull and polling. I can find a solution that is pull and polling (store in table, Javascript timed refresh from table), but I find the lack of immediacy of results and slushiness of current data with that strategy to be less than ideal.

[Andrew Godwin]

It sounds like the only true way of solving this "cleanly" is to be able to use Channels to communicate with the user's web browser as well, which right now means WebSockets (one of the things I want to look at this year is Channels support for something like socket.io/sock.js which has a polling fallback as well).

Given you can't do that as you're using AJAX, you'll have to do your own buffering and queuing, and so a database table is probably the least bad way. I would suggest re-using parts of django sessions (e.g. session key) for identification of the users rather than re-inventing that.

It's not going to be hugely scalable, but it should do well enough and it sounds like your Django interactions are not super common anyway. Just make sure you plan around what to do when a message does not come back (retry? show failure?) and that you clear old messages out from the storage that were not consumed, e.g. if the user closed the browser before the response was read.

You could also try using the low-level ASGI API and make your own channels to store and buffer these messages and then use non-blocking channel_layer.receive_many(), but that's a bit more complex and I don't want to recommend it if it's basically just adding complexity without much benefit (it would save you doing database stuff and expiry, not sure if that's worth it for you or not)

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/df5d9d35-ac42-4e74-b948-23d4e9badb52%40googlegroups.com.