Building Numba without NumPy

[Marcus Ottosson]

Hi Numba community!

I've got an uneducated question that I haven't been able to find an answer to, but in short I'm very impressed with the ability to JIT-compile Python but have been searching high and low for a way to do that without the NumPy dependency. My goal is to compile and distribute Numba to Windows, Linux and MacOS, for use in a number of software frameworks, including Autodesk Maya, 3ds Max, and SideFX Houdini for improved geometry and animation performance and NumPy is currently somewhat of a bottleneck due to its size (500 mb+) and number of additional build steps and build requirements.

Is it possible to build and use Numba without NumPy? If not as-is, then what kind of features can one expect to go missing if it were to be stripped out? I'd be happy to start with support for bare essentials like compiling plain-old-data and basic math operations, though it would be interesting to include support for CUDA even though my guess it that the support comes from NumPy. The optimist in me hopes to get Numba down to < 5 mb in size, but what would be a realistic final size?

Best,

Marcus

[Jim Pivarski]

I'm going to speak as a user, not as a developer of Numba, but I don't think this will be possible. For one thing, you can't usefully JIT-compile functions of scalars, without arrays, because the border from Python to compiled code must be crossed infrequently, which means that you want to pass arrays and loop over the arrays inside the function. For another, Numba depends on LLVM, and LLVM versions 3.4, 3.5, and 3.8 on my system are 65, 80, and 108 MB— all more than your 5 MB limit and growing with time.

-- Jim

--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/1d4e5e6c-a51c-42de-b855-9598db723f82%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[Kevin Sheppard]

I’m not sure where you go your size for NumPy, but here it is 16.7MiB. Compressed on PyPi it is around 10MiB. For comparison numba is 10MiB on disk.  

 

Kevin

--

You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.

To unsubscribe from this group and stop receiving emails from it, send an email to numba-users...@continuum.io.

[Marcus Ottosson]

Thanks guys, the LLVM size does indeed sound problematic, but not unbearable.

I’m not sure where you go your size for NumPy, but here it is 16.7MiB. Compressed on PyPi it is around 10MiB. For comparison numba is 10MiB on disk.

That is encouraging! I have a copy here at 501 mb, and looking closer I found an “mkl” directory at 492 mb. Perhaps mkl is optional? Can Numba do without it? That would put things within a reasonable scope.
​

​

[Stanley Seibert]

If you invoke linear algebra routines in Numba from numpy.linalg, Numba will call the BLAS/LAPACK implementation that SciPy was linked against (not NumPy, because NumPy does not export its BLAS/LAPACK function symbols, but they are always the same).  So if you have a build of NumPy/SciPy which is built with a smaller linear algebra library, Numba will use that automatically.  If you don't need any numpy.linalg functions, you can also leave out SciPy entirely and build NumPy with the smallest BLAS/LAPACK you can find.

The MKL libraries are big because they contain implementations for every single architecture that Intel has produced in the past 5-10 years (not sure exactly where the cutoff is).  

--

You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.

To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.

To post to this group, send email to numba...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/CAFRtmOAFbxbB2%3D5veB0Rds%3D9CF5fcO8SVgTjaJGQXaP_6_m91A%40mail.gmail.com.

[Chris Bartak]

mkl is optional - it is an optimized BLAS/LAPACK that numpy uses by default from Anaconda, but another BLAS can be used.  I believe the numpy wheels on PyPI use OpenBLAS, and the same can be installed through conda with the `nomkl` meta-package, see https://www.anaconda.com/blog/developer-blog/anaconda-25-release-now-mkl-optimizations/

On Mon, Apr 2, 2018 at 11:56 AM, Marcus Ottosson <konstr...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.

To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.

To post to this group, send email to numba...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/CAFRtmOAFbxbB2%3D5veB0Rds%3D9CF5fcO8SVgTjaJGQXaP_6_m91A%40mail.gmail.com.

[Marcus Ottosson]

Thanks Chris and Stanley, that resolves my original issue of distribution size.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/CAJtmDoC9%2BM-r_8YKyQbv405kfd9oYXKpwKP0m7mbGz_Grx3PCg%40mail.gmail.com.