scoop is slow in Windows

[Chris]

Is anyone else finding that scoop runs a lot slower in Windows than in Linux? I experienced this on multiple systems, so I wrote a short test script to confirm, and for me scoop actually slows things down compared to a single process in Windows, whereas it results in a speedup under Linux. Any ideas on why this is happening?

Here is the test script, I tried to keep the amount of data that needs to be transferred between processes to a minimum:

import sys
import time
import numpy as np
import scoop.futures
import concurrent.futures

mode = sys.argv[1]

M = np.random.rand(1000, 1000)
a = list(np.random.rand(1e3))

def f(x):
    return np.sum(x * M * M)

if __name__ == '__main__':
    print('start')
    start_time = time.time()
    if mode == 'scoop':
        out = list(scoop.futures.map(f, a))
    elif mode == 'mp':
        with concurrent.futures.ProcessPoolExecutor() as executor:
            out = list(executor.map(f, a))
    elif mode == 'single':
        out = list(map(f, a))
    else:
        raise ValueError
    print(time.time() - start_time)

And here are the run times in seconds, on a 2-core, 4-thread system. As you can see, both python's multiprocessing and scoop are faster than a single process in Linux, but scoop is way slower in Windows.

                                  linux   windows
python -m scoop test.py scoop     4.3     16.5
python test.py mp                 4.0     5.0
python test.py single             5.2     7.3

[Yannick Hold-Geoffroy]

Hello,

I assure you that there are no check done to actively slow down SCOOP on Windows.

The thing is that we mainly developed SCOOP on Linux and mostly debugged and benchrmaked it on Linux clusters.

It would be interesting to profile the run on Windows to see what is actually slower on this OS... Launching multiple python executables or setting up the networking? I don't know.
I don't have access to a Windows machine these days, though. If anyone could perform the profiling and post back the results, that could be handy.

Have a nice day,
Yannick