Async / long running background task

[Kevin Keller]

I found this on Stackoverflow:

https://stackoverflow.com/questions/24896632/bottle-execute-a-long-running-function-asynchronously-and-send-an-early-respons

I was wondering if anyone has already done something like this in py4web?

Appreciate your help

Powered by cloudHQ

[Alexander Beskopilny]

for long running tasks I try to use tornado

some prototype 

def tornadoSioWsServer():

    # py4web.py run -s tornadoSioWsServer apps

    import tornado.websocket

    import time, urllib

    from tornado.httputil import url_concat

    import tornado.httpclient

    import asyncio

    import aioredis

    ws_debug = True

    connections = []

    async def consumer(channel):

        while await channel.wait_message():

            msg = await channel.get(encoding='utf-8')

            for connection in connections:

                await connection.write_message(msg)

    async def setup():

        connection = await aioredis.create_redis('redis://localhost')

        channel = await connection.subscribe('notifications')

        asyncio.ensure_future(consumer(channel))

    class web_socket_handler(tornado.websocket.WebSocketHandler):

        # This class handles the websocket channel

        @classmethod

        def route_urls(cls):

            return (r'/',cls, {})

        def simple_init(self):

            self.last = time.time()

            self.stop = False

        def open(self):

            #    client opens a connection

            self.simple_init()

            ws_debug and print(f"ws: {time.time() - self.last:.1f}: New client connected")

            self.write_message(f"ws: {time.time() - self.last:.1f}: You are connected")

            connections.append(self)

        def on_message(self, message):

            #    Message received on the handler

            ws_debug and print(f"ws: {time.time() - self.last:.1f}: received message {message}")

            self.write_message(f"ws: {time.time() - self.last:.1f}: You said - {message}")

            self.last = time.time()

        def on_close(self):

            #    Channel is closed

            ws_debug and print(f"ws: {time.time() - self.last:.1f}: connection is closed")

            self.stop= True

            connections.remove(self)

        def check_origin(self, origin):

            return True

    # socketio    pip install python-socketio

    import socketio

    sio_debug = False

    sio = socketio.AsyncServer(async_mode='tornado')

    def handle_request( response):

        pass

    @sio.event

    async def connect(sid, environ):

        sio_debug and print('sio: connect ', sid)

    @sio.event

    async def disconnect(sid):

         sio_debug and print('sio: disconnect ', sid)

    @sio.on('to_py4web')

    async def echo(sid, data):

         sio_debug and  print('sio: from client: ', data)

         http_client = tornado.httpclient.AsyncHTTPClient()

         request = "http://localhost:8000/_socketio/echo/xx/yy/zz"

         http_client.fetch(request, handle_request)

         await sio.emit("py4web_echo", data)

    class TornadoSioWsServer(ServerAdapter):

        def run(self, handler): # pragma: no cover

            import tornado.wsgi, tornado.httpserver,  tornado.web,  tornado.ioloop

            container = tornado.wsgi.WSGIContainer(handler)

            app= tornado.web.Application([

                    web_socket_handler.route_urls(),

                    (r"/socket.io/", socketio.get_tornado_handler(sio) ),

                    (r".*", tornado.web.FallbackHandler, dict(fallback=container) ),

                 ])

            server = tornado.httpserver.HTTPServer(app)

            server.listen(port=self.port,address=self.host)

            tornado.ioloop.IOLoop.instance().start()

    return TornadoSioWsServer

[stifan kristi]

think everything works in bottle will work in py4web

e.g. base on stackoverflow link (not tested)

from py4web import action, HTTP

from threading import Thread

@action('task', method = ['GET'] )

def f():

    task_thread = Thread(target=longRunningTask) # create a thread that will execute your longRunningTask() function

    task_thread.setDaemon(True) # setDaemon to True so it terminates when the function returns

    task_thread.start() # launch the thread

    return HTTP(202)

best regards,

stifan

[Val K]

Spawn thread per request? You have unlimited resources?

среда, 17 февраля 2021 г. в 20:21:04 UTC+3, stifan kristi:

[Kevin Keller]

Thanks guys, I ll try with threading and tornado. 

Powered by cloudHQ

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/57a2e4a2-60de-4d3b-8745-d52a02fdcff1n%40googlegroups.com.

[Massimo]

I expect a possible problem here. pydal done some weird stuff with threads. It basically makes sure that every thread has one database connection/session/cursor. In that when different threads serve different clients, database connections do not get mixed up. If one action creates an additional thread I do not know what db will be pointing to. Also the logic that wraps a db to commit/rollback will break because the thread may continue after the action returns.

So bottom line do not share a db object between threads you create. If you do, maybe try insider the thread body

with db.single_transaction():

     # use db only inside here

[Alexander Beskopilny]

interesting pattern for long running background  tasks
https://matthewminer.com/2015/02/21/pattern-for-async-task-queue-results

here is the py4web version with pydal
https://github.com/ali96343/lvsio