Python multiprocessing under JEP does not work

[Marius Seiceanu]

Hello,

To avoid GIL issues, I'm trying to use the multiprocessing module to speedup my script.

When running the script from a shell I see 8 Python processes in task manager (as expected).

When I'm running the script from Java 11 using JEP there are no new processes; Java process is using 100% of one core and execution time is about 8 time slower.

I'm running Pyhon 3.7 and tried Jep 3.8.2 and 3.9.0.

My script is doing something like:

import multiprocessing
...
cores = multiprocessing.cpu_count()
pool = multiprocessing.Pool(cores)
parallelized_results = pool.map(compute, verticals)
...


Any advice how to solve the problem?

Thank you,
Marius

[Muhammad Bin Aqeel]

Have you solved this issue, I'm also facing the same problem and no help found

[Marius Seiceanu]

Hi,
Unfortunately I could not solve the problem using JEP.

I solved the problem by implementing my own protocol for data exchange using stdin/stdout and an adapter which starts separate Python process for each script execution. I can start as many processes as needed and the performance is very good. Finally CPU usage is 100% when there are multiple scripts running in parallel.

It was pretty easy to do the data exchange using JSON which is easy to parse in both Java and Python.

I hope this helps.
Best Regards,
Marius