Why Ubuntu 16.04 version runs only on one CPU?

[K leo]

Prior to running Ubuntu 16.04, I get Julia from the PPA, and run it simply like:

> julia

Then when I run julia code, "top" shows CPU usage of Julia as something like 200% (I have two cores).

Now on 16.04, julia only runs upto 100% of CPU.  The version of julia is said to maintained by "Ubuntu Developers <ubuntu-dev...@lists.ubuntu.com>"

What is happening?

[Yichao Yu]

On Thu, Apr 21, 2016 at 10:39 AM, K leo <cnbi...@gmail.com> wrote:
> Prior to running Ubuntu 16.04, I get Julia from the PPA, and run it simply
> like:
>
>> julia
>
> Then when I run julia code, "top" shows CPU usage of Julia as something like
> 200% (I have two cores).

What code are you running. Any pre-build version should run julia code
only on one thread. Any other thread are created by libraries like
fftw or openblas.

[K leo]

On Thursday, April 21, 2016 at 8:21:14 PM UTC+5:30, Yichao Yu wrote:

On Thu, Apr 21, 2016 at 10:39 AM, K leo <cnbi...@gmail.com> wrote:
> Prior to running Ubuntu 16.04, I get Julia from the PPA, and run it simply
> like:
>
>> julia
>
> Then when I run julia code, "top" shows CPU usage of Julia as something like
> 200% (I have two cores).

What code are you running. Any pre-build version should run julia code
only on one thread. Any other thread are created by libraries like
fftw or openblas.

 

I run my own code which does not directly require fftw or openblas.  The same code runs on the version on Ubuntu 15.10, it uses 200% CPU, but on the version on 16.04, it only uses 100%.  Note the versions of Julia on the two systems are likely from different builds.

[K leo]

I did some timing measures running the same code.  On Ubuntu 15.10 where Julia uses 200% of CPU, the code runs in 2500 seconds.  On Ubuntu 16.04 where Julia takes 100% CPU, the code runs in 2700 seconds.  I don't know what causes Julia to use less CPU on 16.04, whether being the Linux version or the Julia version.  So given the little difference in time, perhaps the conclusion is Ubuntu 16.04 is more efficient using hardware resources.  Any comments?

[Milan Bouchet-Valat]

Le samedi 23 avril 2016 à 04:52 -0700, K leo a écrit :
> I did some timing measures running the same code.  On Ubuntu 15.10
> where Julia uses 200% of CPU, the code runs in 2500 seconds.  On
> Ubuntu 16.04 where Julia takes 100% CPU, the code runs in 2700
> seconds.  I don't know what causes Julia to use less CPU on 16.04,
> whether being the Linux version or the Julia version.  So given the
> little difference in time, perhaps the conclusion is Ubuntu 16.04 is
> more efficient using hardware resources.  Any comments?

This is probably due to a newer OpenBLAS being more efficient. Anyway,
the Ubuntu PPA is no longer maintained. The recommended solution is to
use generic Linux binaries from the Julia website.


Regards

[K leo]

On Saturday, April 23, 2016 at 7:01:45 PM UTC+5:30, Milan Bouchet-Valat wrote:

Le samedi 23 avril 2016 à 04:52 -0700, K leo a écrit :
Anyway,
the Ubuntu PPA is no longer maintained. The recommended solution is to
use generic Linux binaries from the Julia website.

 

On Ubuntu 16.04, Julia 0.4.5 is available on multiverse repository for some reason.  Not sure who maintains it.

[Milan Bouchet-Valat]

Le samedi 23 avril 2016 à 18:10 -0700, K leo a écrit :
> > Le samedi 23 avril 2016 à 04:52 -0700, K leo a écrit : 
> > Anyway, 
> > the Ubuntu PPA is no longer maintained. The recommended solution is to 
> > use generic Linux binaries from the Julia website. 
> >
>  
> On Ubuntu 16.04, Julia 0.4.5 is available on multiverse repository
> for some reason.  Not sure who maintains it.

Ah, OK, I thought you were using the PPA package. Anyway, Ubuntu 16.04
includes the latest OpenBLAS and the Julia package depends on it, so it
should be good. That said, maybe that OpenBLAS library doesn't use
threading.

Can you confirm that running 
x=rand(10000,10000);
x*x
does not make Julia use more than 100% CPU?

If not, could you post the output of versioninfo()? Finally, please run
'ls -l /usr/lib/julia/', 'ls -l /usr/lib64/julia' from a shell and copy
the result.

I'm Ccing Graham Inggs, who maintains the Debian package on which the
Ubuntu package is based.


Regards

[K leo]

See below.

On Sunday, April 24, 2016 at 4:01:49 PM UTC+5:30, Milan Bouchet-Valat wrote:

Le samedi 23 avril 2016 à 18:10 -0700, K leo a écrit :
> > Le samedi 23 avril 2016 à 04:52 -0700, K leo a écrit : 
> > Anyway, 
> > the Ubuntu PPA is no longer maintained. The recommended solution is to 
> > use generic Linux binaries from the Julia website. 
> >
>  
> On Ubuntu 16.04, Julia 0.4.5 is available on multiverse repository
> for some reason.  Not sure who maintains it.
Ah, OK, I thought you were using the PPA package. Anyway, Ubuntu 16.04
includes the latest OpenBLAS and the Julia package depends on it, so it
should be good. That said, maybe that OpenBLAS library doesn't use
threading.

Can you confirm that running 
x=rand(10000,10000);
x*x
does not make Julia use more than 100% CPU?

It actually uses close to 200% of CPU.  

 

If not, could you post the output of versioninfo()? Finally, please run
'ls -l /usr/lib/julia/', 'ls -l /usr/lib64/julia' from a shell and copy
the result.

 julia> versioninfo()

Julia Version 0.4.5

Commit 2ac304d (2016-03-18 00:58 UTC)

Platform Info:

  System: Linux (x86_64-linux-gnu)

  CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz

  WORD_SIZE: 64

  BLAS: libopenblas (NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)

  LAPACK: libopenblas

  LIBM: libopenlibm

  LLVM: libLLVM-3.8

$ ls -l /usr/lib/julia/

ls: cannot access '/usr/lib/julia/': No such file or directory

$ ls -l /usr/lib/x86_64-linux-gnu/julia

total 26440

lrwxrwxrwx 1 root root       20 Apr 18 17:45 libarpack.so -> ../../libarpack.so.2

-rw-r--r-- 1 root root    15608 Apr 18 17:45 libccalltest.so

lrwxrwxrwx 1 root root       22 Apr 18 17:45 libcholmod.so -> ../libcholmod.so.3.0.6

lrwxrwxrwx 1 root root       22 Apr 18 17:45 libdSFMT.so -> ../libdSFMT-19937.so.1

lrwxrwxrwx 1 root root       25 Apr 18 17:45 libfftw3f_threads.so -> ../libfftw3f_threads.so.3

lrwxrwxrwx 1 root root       24 Apr 18 17:45 libfftw3_threads.so -> ../libfftw3_threads.so.3

lrwxrwxrwx 1 root root       15 Apr 18 17:45 libgmp.so -> ../libgmp.so.10

-rw-r--r-- 1 root root  1175608 Apr 18 17:45 libjulia.so

lrwxrwxrwx 1 root root       15 Apr 18 17:45 libmpfr.so -> ../libmpfr.so.4

lrwxrwxrwx 1 root root       22 Apr 18 17:45 libopenblas.so -> ../../libopenblas.so.0

lrwxrwxrwx 1 root root       19 Apr 18 17:45 libopenlibm.so -> ../libopenlibm.so.2

lrwxrwxrwx 1 root root       22 Apr 18 17:45 libopenspecfun.so -> ../libopenspecfun.so.1

lrwxrwxrwx 1 root root       18 Apr 18 17:45 libpcre2-8.so -> ../libpcre2-8.so.0

-rw-r--r-- 1 root root   203048 Apr 18 17:45 libRmath-julia.so

lrwxrwxrwx 1 root root       19 Apr 18 17:45 libspqr.so -> ../libspqr.so.2.0.2

lrwxrwxrwx 1 root root       32 Apr 18 17:45 libsuitesparseconfig.so -> ../libsuitesparseconfig.so.4.4.6

-rw-r--r-- 1 root root     6000 Apr 18 17:45 libsuitesparse_wrapper.so

lrwxrwxrwx 1 root root       22 Apr 18 17:45 libumfpack.so -> ../libumfpack.so.5.7.1

-rw-r--r-- 1 root root 25664120 Apr 18 17:45 sys.so

$ ls -l /usr/lib64/julia

ls: cannot access '/usr/lib64/julia': No such file or directory

[Milan Bouchet-Valat]

Le dimanche 24 avril 2016 à 05:38 -0700, K leo a écrit :
>
> See below.

>
> > Le samedi 23 avril 2016 à 18:10 -0700, K leo a écrit : 
> > > > Le samedi 23 avril 2016 à 04:52 -0700, K leo a écrit :  
> > > > Anyway,  
> > > > the Ubuntu PPA is no longer maintained. The recommended solution is to  
> > > > use generic Linux binaries from the Julia website.  
> > > > 
> > >   
> > > On Ubuntu 16.04, Julia 0.4.5 is available on multiverse repository 
> > > for some reason.  Not sure who maintains it. 
> > Ah, OK, I thought you were using the PPA package. Anyway, Ubuntu 16.04 
> > includes the latest OpenBLAS and the Julia package depends on it, so it 
> > should be good. That said, maybe that OpenBLAS library doesn't use 
> > threading. 
> >
> > Can you confirm that running  
> > x=rand(10000,10000); 
> > x*x 
> > does not make Julia use more than 100% CPU? 
> It actually uses close to 200% of CPU.  

OK, so no problem in this area.

I think you'll have to give more details about the code you mentioned
in your first post for us to be able to help. Also, please confirm that
you see the same problem (i.e. code using only one core) with the
generic Linux binaries.


Regards

[K leo]

It is hard to know how to describe my code.  So I tried to run it with the generic version which does use 200% of CPU.  So there seems something different with the Ubuntu version of Julia.

I installed under my home folder.  I ran it in the following way hoping that the shared libs are not mixed with the system ones.

$ ~/Software/julia-2ac304dfba/bin/julia

               _

   _       _ _(_)_     |  A fresh approach to technical computing

  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org

   _ _   _| |_  __ _   |  Type "?help" for help.

  | | | | | | |/ _` |  |

  | | |_| | | | (_| |  |  Version 0.4.5 (2016-03-18 00:58 UTC)

 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release

|__/                   |  x86_64-unknown-linux-gnu

[Milan Bouchet-Valat]

Le dimanche 24 avril 2016 à 08:44 -0700, K leo a écrit :
> It is hard to know how to describe my code.  So I tried to run it
> with the generic version which does use 200% of CPU.  So there seems
> something different with the Ubuntu version of Julia.

OK. Could you at least make a list of linear algebra functions that you
are using? It could be that Ubuntu's OpenBLAS doesn't include optimized
LAPACK functions.


Regards

> I installed under my home folder.  I ran it in the following way
> hoping that the shared libs are not mixed with the system ones.
>
> $ ~/Software/julia-2ac304dfba/bin/julia
>                _
>    _       _ _(_)_     |  A fresh approach to technical computing
>   (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
>    _ _   _| |_  __ _   |  Type "?help" for help.
>   | | | | | | |/ _` |  |
>   | | |_| | | | (_| |  |  Version 0.4.5 (2016-03-18 00:58 UTC)
>  _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
> |__/                   |  x86_64-unknown-linux-gnu
>
>

[K leo]

Looking through the linear standard functions list in the documentation, I think perhaps the only function used is linreg.  I don't directly use LAPACK.

[Milan Bouchet-Valat]

Le dimanche 24 avril 2016 à 09:43 -0700, K leo a écrit :
> Looking through the linear standard functions list in the
> documentation, I think perhaps the only function used is linreg.  I
> don't directly use LAPACK.
>

Hmm, doesn't sound like the best candidate. I've made a few attempts,
and I couldn't create a vector big enough so that linreg() took more
than one or two seconds. So either you have a lot of RAM, or this isn't
the function which makes your CPU go to 200% for a significant time. 

Did you consider operators when looking for linear algebra functions?
Anyway, the only way to be certain a function exhibits the problem is
to run it in isolation in both Julia versions.


Regards