Disable/limit parallel processing in cvxopt

[Daan Christiaens]

Dear cvxopt-users and developers,

I've been very happy with cvxopt on my local machine, and I'm porting my code to our Linux cluster. Everything runs fine, except that the QP solver is using >4000% CPU power (on a 64-core system), seriously annoying my fellow sge-users on the system. I am therefore looking for a way to limit the number of threads being used behind the scenes.

Does cvxopt provide an environment variable to set the maximum number of threads or even disable multithreading altogether? If not, where is the parallelism coming from? I'm guessing from within the BLAS or ATLAS libraries, but isn't the default setup.py linking against the single-threaded versions of those libraries (e.g., cblas instead of ptcblas)?

I would expect that many users have had to disable multithreading for use with a job scheduler. Any help is much appreciated.

Thanks,

Daan

[Julian Manzano]

Hi I am having the same problem.
The CPU usage is mostly system CPU:


CPU%  MEM%  VIRT   RES   PID USER        NI S    TIME+ IOR/s IOW/s Command
1354.9   0.6       1.51G  197M  4036 julian         0 R  3:12.89     0     0 /usr/bin/python2.7 /home/julian/pyt...

CPU     100.0% 
user:    17.3%
system:  82.7%

>  sudo strace -c -p 4036
 Process 4036 attached

^CProcess 4036 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 95.50    2.028718          11    183407           sched_yield
  1.34    0.028563         117       244           futex
  1.25    0.026524         510        52           brk
  0.95    0.020129        1258        16           sendto
  0.58    0.012410        1551         8           write
  0.37    0.007805        1561         5           mmap
  0.00    0.000077          10         8           recvfrom
------ ----------- ----------- --------- --------- ----------------
100.00    2.124226                183740           total


Note the high number of calls to sched_yield taking almost all the system CPU time.
I have Ubuntu 15.04, cxvopt 1.1.7

Any help is highly appreciated!
Julian

[Joachim Dahl]

CVXOPT uses whatever BLAS library linked to in the build process.  You should be able to control the number of threads by setting an environment variable, for example

OMP_NUM_THREADS. 

--
You received this message because you are subscribed to the Google Groups "CVXOPT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cvxopt+un...@googlegroups.com.
To post to this group, send email to cvx...@googlegroups.com.
Visit this group at http://groups.google.com/group/cvxopt.
For more options, visit https://groups.google.com/d/optout.

[Julian Manzano]

Hi Joachim,

Thanks for the tip. You are in the right track.
I added
export OMP_NUM_THREADS=2
in ~/.bashrc
rebooted the machine and now I observe the following behaviour@

a) If I run my script from the bash shell the problem is solved (thanks!)
b) If I run my script from pycham the problem is still there :( seems that pycharm has its own  enviroment?)

One last comment. I did not have this issue on Ubuntu 14.10. Therefore I have the feeling that something should be changed upstream that should not require this setting from the user...
If you have any tips for pycharm that would be great.

Julian

[Joachim Dahl]

Searching google for "pycharm environment variable" gives several answer on how to set environment variables from pycharm.

[Julian Manzano]

One last commet:

export OMP_NUM_THREADS=1 
is actually the right value that supress the system CPU usage (with OMP_NUM_THREADS=2 I still have some 10% residual)

Julian

[Julian Manzano]

Yes, I've tried:

File | Settings | Build, Execution, Deployment | Console | Python Console | Environmental Variables ...

Setting:
Name
OMP_NUM_THREADS
Value
1

But I still see the same degradation, also restarted pycharm but still the same...

Julian

[Julian Manzano]

Ok, this is interesting. I found a way to make it work on pycharm.

I go to:
File | Settings | Project: python | Project Interpreter

There If I switch to:
Python 2.7.9 (~/anaconda/bin/python): Python 2.7.9 (/home/julian/anaconda/bin/python)

Then it works!

But if I switch back to my Ubuntu mate 15.04 version:
Python 2.7.9 (/usr/bin/python2.7): Python 2.7.9 (/usr/bin/python2.7)

Then I see the degradation again...

So I do not know why but I will have to use the anaconda interpreter from pycharm (on the shell I use the system one without problems).

Conclusion: Now it works both from the shell and from pycharm but honestly all looks very hackish to me.

[Daan Christiaens]

Thanks Joachim,

Setting OMP_NUM_THREADS works great. I didn't know Blas relies on OpenMP.

Best,

Daan