Moore Foundation Funding Announcement for Numba and Dask

[Stanley Seibert]

Hi Numba users,

For those of you who haven't already seen the announcement:

https://www.continuum.io/blog/developer-blog/gordon-and-betty-moore-foundation-grant-numba-and-dask

This means we can accelerate our drive toward a stable, production-ready Numba 1.0 release.  We don't have a precise ETA on Numba 1.0, but we will be tracking our progress in the Numba issue tracker and soliciting feedback on priorities.

On behalf of the Numba team, I want to thank all of you for the support, feature requests, and bug reports.  It really helps us enormously, and we look forward to more as we head for Numba 1.0!

[Tem Pl]

Great! Do you have a rough list of high level concepts and features for 1.0 and post? 

[kas...@gmail.com]

Great news, congratulations Stanley and the rest of the team.

Besides the benefits of performance and scaling up, another big advantage of Numba is that is often allows more 'lean' implementations compared to a Numpy approach. Its much easier to read a scientific paper and implement an algorithm if you don't have to think in/about array's or vectorizing etc. It sort of better separates your code in a algorithm part and 'data managing' part.

Keep up the good work!

Regards,
Rutger

[Stanley Seibert]

Before 1.0 we need to address these 5 major areas:

• Core capabilities of the Python compiler: This includes improving our type inference to handle more of Python's dynamism efficiently, as well as support for temporarily jumping out of nopython mode back into Python to call uncompiled functions.  We also want to improve support for more Python language features like list comprehensions.  Improvements to basic Python data type support (statically typed dictionaries perhaps?) also fall into this category.
  
  
• NumPy support: Expanding coverage of NumPy to cover more of the API that is commonly used.  We know we are missing some linear algebra support, as well as various array manipulation methods related to stacking, concatentation, etc.
  
  
• Extension API: We would like Numba's extensibility API to allow it to be customized to support new data types, or entirely new compilation use cases, without having to modify the Numba core directly.  We have some of this now, but we're hoping an external project (perhaps xarray??) will add support for Numba to their package and highlight places where the API needs improvement.
  
  
• Improving usability and debugging: Numba needs to communicate more clearly why it cannot compile certain kinds of code, and also facilitate debugging of compiled code.  There are several solutions now that can inspect the Python and C level call stacks at the same time, so we want to make sure that Numba can emit whatever symbol information is required for those tools to work properly.
  
  
• "Cookbook" style resources: People learn a new tool best by example, so we need the documentation to include a repository of realistic Numba examples.  This is especially important when dealing with unfamiliar targets, like the GPU.  We also want to have a lot of custom ufunc and gufunc examples, because we feel those abstractions are underutilized, and they enable Numba to do things like automatic multi-threading and GPU offload which are not possible with the @jit decorator.
  

--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users...@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/cb653952-309b-4fae-bdef-463340c4d7a9%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[Nathaniel Smith]

This is really cool, congratulations to the team.

Is the proposal available anywhere?

--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users...@continuum.io.
To post to this group, send email to numba...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/CADv3RKSP2wzuR1ea2Fiy_zif%3DwsTyKePU-SahwwCmRzH2wXqRQ%40mail.gmail.com.

[Tem Pl]

Stanley- Very cool.

I'm interested in integration with Dynd for things like custom arrays and types, missing data, ragged arrays, string processing. Is that on the roadmap or is it a separate effort? 

What about improvements to JIT classes (immutable, iter method etc).?

[Stanley Seibert]

Full string support is out of scope for the Moore grant (we specifically wanted to stay focused on the numerical case in the push for 1.0), but could certainly be added by interested third parties or even other projects at Continuum if needed.

I think the best approach for dealing with ragged arrays (and probably also missing data) is DyND, for which Numba support is in our list of goals.  This also out of scope for the Moore grant (though the extension API is precisely how we want to build that support decoupled from the Numba), but I am very eager to find resources to make that happen sooner rather than later.  DyND has a very advanced container format that should still be very friendly to JIT compilers.

Basic JIT class fixes (we know there is a long list already) will be part of the 1.0 release.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/4210b2fd-8fad-445a-956c-e1554932bf3e%40continuum.io.

[Tem Pl]

Sounds exciting.

But as you know, Dynd also has very different function semantics (multimethods, pattern matching etc) and type system (generics etc), aside from the container. Wouldn't this all require deeper surgery on numba?