How to restrict number of cores used?

[Ed Mountjoy]

Hi,

I looking to find a way to restrict the number of cores used when doing OLS regression. At the moment statsmodels is using all available cores. This is a problem as I'm using a trying to run jobs on a cluster with nodes that are shared with other users.

Thanks,

Ed

[josef...@gmail.com]

I have no idea how to do that. It isn't statsmodels doing it.

I think it's the underlying Blas/Lapack libraries in numpy/scipy that
are using mutliple cores. I saw the same mutli processor usage with
MKL but didn't try to figure out yet how to turn it off and restrict
to one processor.

So this is a question for numpy/scipy or the linalg libraries. But I
would also like to know.


Statsmodels is using joblib in a few places for parallel processing
where it's under our control but mainly for bootstrap and not in the
models.


Josef


>
> Thanks,
> Ed

[Nathaniel Smith]

On Jan 11, 2016 5:05 AM, "Ed Mountjoy" <emou...@gmail.com> wrote:
>
> Hi,
>
> I looking to find a way to restrict the number of cores used when doing OLS regression. At the moment statsmodels is using all available cores. This is a problem as I'm using a trying to run jobs on a cluster with nodes that are shared with other users.

Yeah, this is your BLAS library, so it depends on which BLAS you're using.

For openblas, try setting the environment variable OMP_NUM_THREADS=1. (Technically this is how you tell OpenMP to be not-threaded, so watch out for that in case you're using OpenMP on purpose in your own code. But openblas also pays attention to it in all configurations. If for some reason you want OpenMP code to use multiple threads but openblas not to, then check the openblas docs, I think it's possible.)

If you're using MKL then check the MKL docs.

AFAIK those are the only popular multi-threaded BLAS libraries.

-n

[Skipper Seabold]

FWIW, in Accelerate on OSX, you have VECLIB_MAXIMUM_THREADS.

Skipper

[Ed Mountjoy]

I figured out that it was because I was using Anaconda which includes MKL optimisations. I can restrict the number of cores using e.g.:

import mkl
mkl.set_num_threads(1)

Thanks

[josef...@gmail.com]

Thanks,

I made a FAQ issue to collect this information

https://github.com/statsmodels/statsmodels/issues/2914

Josef