RpyC and Joblib

17 views
Skip to first unread message

Fula Toro

unread,
Mar 31, 2016, 9:16:01 PM3/31/16
to rpyc
I am using rpyc to spawn a python process that uses joblib on a remote server. I am running:

#rpyc_classic -m forking

This is the only way I have been able to get the remote process to use multiple CPUs. However after running for a while, I only get 2 or 3 CPUs active whereas initially I was using 48 CPUs. Looking at the list processes on the server, I do see about 200 processes like the one below

 69976 67657  0 12:23 pts/1    00:00:00 /usr/bin/python /usr/bin/rpyc_classic.py -m forking


Despite that only 2 or 3 CPUs are active and  my job basically slows down. It is almost as if once the RPYC process is spawned it does not return. How do I get the forking processes to return on finish  Is this the best way to do use rpyc for remotely executing my job.  Note that when running the python program locally, it uses joblib likes it is supposed to and uses up as many CPU's as available. It is only when running through rpyc that it plateaus after about 200 rpyc_classic processes.

Thank you


Reply all
Reply to author
Forward
0 new messages