Jupyter notebook CPU load makes no sense

[Glenn Tarbox]

I have a CPU bound optimization which takes about 15 seconds to run and I'm looping over a range of values.  Takes a couple hours overall, but there are no threads, nothing fancy... just a scipy / numpy optimization running in a loop

According to htop (and top), Jupyter is taking 800% CPU

So, to be clear, this is a single loop which should be running 100% but somehow takes 800%.  

If I had to guess, I'd say that Jupyter notebook has implemented some kind of multiprocess job control system and it has gone horribly wrong.

-glenn

[Glenn Tarbox]

Oh, I should say, this is a very recently updated Anaconda install on Ubuntu 15.04

[HadiM]

Does some of your code use Pandas ? 

They release the GIL recently : http://pandas.pydata.org/pandas-docs/version/0.17.1/whatsnew.html#performance-improvements

--
HadiM

On Fri, Dec 4, 2015 at 9:42 PM, Glenn Tarbox <gl...@tarbox.org> wrote:

Oh, I should say, this is a very recently updated Anaconda install on Ubuntu 15.04

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/3444041c-790b-42bc-966b-97814204162f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Thomas Kluyver]

It looks like there are 7 other Python processes (PID 9023-9029), each keeping one CPU busy. I'd guess something is using multiprocessing, and the 800% is the total for the lead process and its children. Maybe some of the optimisation routines you're using do this internally?

--

You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/112ea993-a33f-4c63-857c-3b313629e7fc%40googlegroups.com.

[Brian Granger]

Also, some processors report virtual and physical cores differently (by default). In that case, you can easily see %800 CPU on a 4 core system.

Cheers,

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qhQqb47OFZ2-0i3t9EwPrRiqaaKcODjpYLCVM40mOyeuA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgra...@calpoly.edu and elli...@gmail.com

[Brian Granger]

Also, you may also be using a thread enabled BLAS/LAPACK?

[Glenn Tarbox]

On Friday, December 4, 2015 at 12:59:25 PM UTC-8, ellisonbg wrote:

Also, you may also be using a thread enabled BLAS/LAPACK?

On Fri, Dec 4, 2015 at 12:58 PM, Brian Granger <elli...@gmail.com> wrote:

Also, some processors report virtual and physical cores differently (by default). In that case, you can easily see %800 CPU on a 4 core system.

Interesting.  I guess it could be (never occurred to me... doh :-)

There are only 2 calls into anything non-trivial.  One is numpy.inv() for an  order 100s x 100s matrix, which is only slightly above non-trivial, and this:

            r = scipy.optimize.fmin_l_bfgs_b(self.objective, 

                                x0, approx_grad=True, bounds=bounds, 

                                maxiter = 1e6, maxfun = 1e6,

                                pgtol=1e-12)

self.objective calls the numpy.inv() (hence why its kinda slow)

I'll drill down

-glenn

-- 

Glenn H. Tarbox, PhD

 =]|[=

[Michael Milligan]

Yes, that's exactly what's going on. scipy.optimize.fmin_l_bfgs_b eventually calls into scipy.optimize._lbfgsb, which is an f2py wrapper around various math libraries including BLAS and LAPACK. These days, many implementations of these (including OpenBLAS, which I believe is the default in Ubuntu) use OpenMP multithreading by default. What usually happens with load monitoring of multithreaded code is that all the CPU time is credited to the master process, so if you have 8 threads running full speed you would indeed expect to see 800% CPU load.

To confirm that's what is happening, try exporting OMP_NUM_THREADS=1 in the environment before starting python (I don't *think* setting it via os.environ after everything is loaded will work). If it's OpenMP at work, that should limit it to one thread and result in only 100% CPU usage.

Michael