blocking calls in aiohttp

[h h]

Hi! First of all I want to thank you for awesome framework aiohttp and libs!

I'm rewriting my app from flask to aiohttp and I need to call synchronous functions (cx_Oracle) in my handlers.

With stackoverflow's help I've figured out how to call single function:

def test_one():

    time.sleep(5)

    return 'test_one'

    

a = await app.loop.run_in_executor(None, test_one)
print(a)

But now I need to call in two different oracle databases in one handler, which way is best to do this?

like

def test_two():

    time.sleep(5)

    return 'test_two'

a = await app.loop.run_in_executor(None, test_one)

b = await app.loop.run_in_executor(None, test_two)

print(a, b)

Thanks!

[imbolc]

If your blocking function just uses python sockets, you can use ThreadPoolExecutor, otherwise you have to use ProcessPoolExecutor. And if your first function does not depend on the results of second one, you can run them parallel:

a, b = await asyncio.gather(

    app.loop.run_in_executor(None, test_one),

    app.loop.run_in_executor(None, test_two)

)

[Artem Malyshev]

If test_one and test_two functions doesn't depend on each other then I is possible to run this queries in parallel. In example above your coroutine will be suspended until test_one is done. Then your coroutine will schedule test_two execution and suspend second time. If test_two call doesn't need test_one result then use asyncio gather to make this parallel (https://docs.python.org/3/library/asyncio-task.html#asyncio.gather):

coros = [
    app.loop.run_in_executor(None, test_one),
    app.loop.run_in_executor(None, test_two),
]

results = await asyncio.gather(*coros)
print(*results)



On Saturday, July 30, 2016 at 9:55:58 PM UTC+3, h h wrote:

[h h]

Thanks for explanation!

Now it's getting more complicated, I need some function wait for return value of other function and process that value then, so I wrote working example, but I'm not sure how good it is

def test_one(future):

    time.sleep(3)

    b = future.result()

    print('P test_one' + b)

    return 'test_one' + b

def test_two():

    time.sleep(5)

    print('P test_two')

    return 'test_two'

def test_three():

    time.sleep(2)

    print('P test_three')

    return 'test_three'

a, b = await asyncio.gather(

    app.loop.run_in_executor(None, test_one, app.loop.run_in_executor(None, test_three)),

    app.loop.run_in_executor(None, test_two)

)

It works for 5 seconds as expected, but I don't like future in test_one and construction on asyncio.gather seems like complicated, can you please enlighten me?

воскресенье, 31 июля 2016 г., 11:17:50 UTC+3 пользователь Artem Malyshev написал:

[Artem Malyshev]

I prefer to write simple coroutine function which will await test_three then pass in result into test one. Then use this function with independent test_two in the gather furtion.

async def coro1():
    result = await app.loop.run_in_executor(None, test_three)
    return await app.loop.run_in_executor(None, test_one, result)

a, b = gather(
    coro1(),
    app.loop.run_in_executor(None, test_two),
)


[imbolc]

def test_one():

    time.sleep(3)

    b = test_three()

    print('P test_one' + b)

    return 'test_one' + b

a, b = await asyncio.gather(

    app.loop.run_in_executor(None, test_one),

    app.loop.run_in_executor(None, test_two)

)

[h h]

I thought about it, yes. But I can't call test_three in test_one because it's different databases, I can pass app in test_one tho instead of pool of connections to dabatase, but I don't think that's good way.

I can explain better. I've two databases and I'm expecting either IP or contract id from user input. But database1 can search by ip and id, but database2 can't search by ip. So I need to ask database1 convert ip to id and pass to database2. That's why I don't want call functions of database1 in database2 function.

воскресенье, 31 июля 2016 г., 13:06:11 UTC+3 пользователь imbolc написал: