Nesting concurrent.futures.ThreadPoolExecutor

[Andrew McLean]

I have a program where I am currently using a
concurrent.futures.ThreadPoolExecutor to run multiple tasks
concurrently. These tasks are typically I/O bound, involving access to
local databases and remote REST APIs. However, these tasks could
themselves be split into subtasks, which would also benefit from
concurrency.

What I am hoping is that it is safe to use a
concurrent.futures.ThreadPoolExecutor within the tasks. I have coded up
a toy example, which seems to work. However, I'd like some confidence
that this is intentional. Concurrency is notoriously tricky.

I very much hope this is safe, because otherwise it would not be safe to
use a ThreadPoolExecutor to execute arbitrary code, in case it also used
concurrent.futures to exploit concurrency.

Here is the toy example:

> |importconcurrent.futures definner(i,j):returni,j,i**j
> defouter(i):withconcurrent.futures.ThreadPoolExecutor(max_workers=5)asexecutor:futures
> ={executor.submit(inner,i,j):j forj inrange(5)}results =[]forfuture
> inconcurrent.futures.as_completed(futures):results.append(future.result())returnresults
> defmain():withconcurrent.futures.ThreadPoolExecutor(max_workers=5)asexecutor:futures
> ={executor.submit(outer,i):i fori inrange(10)}results =[]forfuture
> inconcurrent.futures.as_completed(futures):results.extend(future.result())print(results)if__name__
> =="__main__":main()|
I have previously posted this on Stack Overflow, but didn't get any
replies. Apologies if you are seeing this twice.

https://stackoverflow.com/questions/44989473/nesting-concurrent-futures-threadpoolexecutor

[Rob Gaddi]

On 07/20/2017 12:44 PM, Andrew McLean wrote:
> I have a program where I am currently using a
> concurrent.futures.ThreadPoolExecutor to run multiple tasks
> concurrently. These tasks are typically I/O bound, involving access to
> local databases and remote REST APIs. However, these tasks could
> themselves be split into subtasks, which would also benefit from
> concurrency.
>
> What I am hoping is that it is safe to use a
> concurrent.futures.ThreadPoolExecutor within the tasks. I have coded up
> a toy example, which seems to work. However, I'd like some confidence
> that this is intentional. Concurrency is notoriously tricky.
>
> I very much hope this is safe, because otherwise it would not be safe to
> use a ThreadPoolExecutor to execute arbitrary code, in case it also used
> concurrent.futures to exploit concurrency.
>

Well that last statement is clearly false. It's not safe to use any
multiple access mechanism (threading, processes, async stuff) to execute
arbitrary code; so it's by definition not safe to use a ThreadPoolExecutor.

I'm not being cute and using "safe" in some Turing sense, I'm talking
specifically about multiple accesses. Whenever you have multiple access
you have interlock issues, which you resolve with mutexes or message
queues or however you decide to do so. Those issues don't go away when
you use the ThreadPoolExecutor. There is every possibility, especially
if you start recursively spawning threads, that A spawns B, A blocks on
something that B is supposed to do (such as completing a Future), but
due to the thread limit of the pool, the mere existence of A is
preventing B from being executed, and you have a deadlock.

--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.