Julia programming language

[Philippe Gras]

Hello,

Do we have any group who is looking or have looked at Julia [1] ?

This language reconciles Python's easiness of code writing and c++-like
performance. It looks very appropriate for HEP.

Philippe.

[1] https://julialang.org/

[Pere Mato Vila]

Hi Philippe,

Have a look at this article: https://link.springer.com/article/10.1007/s41781-021-00053-3
Julia is available in SWAN (LCG releases) in case you are interested.

Cheers,

Pere

> --
> This is the discussion forum of the HEP Software Foundation, http://hepsoftwarefoundation.org
> --- You received this message because you are subscribed to the Google Groups "HEP Software Foundation Open Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hsf-forum+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/hsf-forum/02c7fd34-a9b5-f895-cead-fa2e58a3a8f8%40cern.ch.

[Jim Pivarski]

It comes up from time to time, such as this DIANA/HEP meeting in July 2016: https://indico.cern.ch/event/545738/ (Despite my "technical issues in Julia" talk, I was trying to make it work at that time: https://github.com/JuliaHEP/ROOT.jl/issues/4)

In the PyHEP planning, the idea of having a short session on Julia came up, particularly to highlight Python-Julia interoperability.

Nowadays, I often advocate using Numba to include JIT-compilation in Python with the caveat that only a subset of the Python language can be compiled by Numba, though that can be difficult if you don't apply it to small functions and work outward. (If you apply it to a large codebase all at once, there's a good chance it will hit something it can't compile, and the error messages are verbose.) Julia would solve that problem—as a language designed to be JIT-compiled, it lacks the dynamic features that makes static analysis and compilation difficult. Any legal Julia code can be compiled.

But whereas the community is moving toward Python on its own, they are not moving to Julia in similar numbers. These are the responses to the "What language do you use?" question at PyHEP 2020: it's a steep cliff!

There is a trend toward using more Python in physics analysis, but not toward other languages in general. Moreover, I don't think that it happened strictly because people in HEP advocated Python (e.g. Wim Lavrijsen had been advocating Python since at least 2003, but it didn't take off until the last 5 years or so). I think that external factors—like language choice in university CS classes, Python's reputation as a data analysis language, and the prospect of careers in data science—played a much bigger role. For instance, Matlab and R certainly have a lot of data analysis functionality, and R even has ROOT bindings, but a similar trend didn't happen there. Julia can use all of Python's machine learning tools, but a movement to Julia hasn't happened yet. The difference is that this distributed community of physicists is hearing about Python from a variety of sources a lot more than they're hearing about these other languages, independent of their technical merits.

So I think it's great that we're investigating Julia in HEP, since it would thoroughly solve "The Numba problem" if it catches on. (I just saw Pere's response. I'll check out that paper!) My point is that we influence the HEP user community less than we might think. Despite its technical merits, Julia will be a hard sell.

-- Jim

[Philippe Gras]

Hello,

I've played with Julia since my last message. It has convinced me it's extremely well suited for HEP.

I'm more optimistic than Jim and Andy for the adoption, because it fulfills an immediate need, which is writing python-like code that runs fast.

It will be very good, if we can discuss the matter at PyHEP 2021. From the links Jim sent I understand it was studied in 2016, while the language was young (launched in 2012 according to wikipedia). The Language has been developing since then and it is worth to revisit it 5 years after.

I can give a talk at PyHEP to present the language including performance comparison with python for our usage.

Cheers,

Philippe.

On 17/05/2021 15:27, Jim Pivarski wrote:

It comes up from time to time, such as this DIANA/HEP meeting in July 2016: https://indico.cern.ch/event/545738/ (Despite my "technical issues in Julia" talk, I was trying to make it work at that time: https://github.com/JuliaHEP/ROOT.jl/issues/4)

In the PyHEP planning, the idea of having a short session on Julia came up, particularly to highlight Python-Julia interoperability.

Nowadays, I often advocate using Numba to include JIT-compilation in Python with the caveat that only a subset of the Python language can be compiled by Numba, though that can be difficult if you don't apply it to small functions and work outward. (If you apply it to a large codebase all at once, there's a good chance it will hit something it can't compile, and the error messages are verbose.) Julia would solve that problem—as a language designed to be JIT-compiled, it lacks the dynamic features that makes static analysis and compilation difficult. Any legal Julia code can be compiled.

But whereas the community is moving toward Python on its own, they are not moving to Julia in similar numbers. These are the responses to the "What language do you use?" question at PyHEP 2020: it's a steep cliff!

There is a trend toward using more Python in physics analysis, but not toward other languages in general. Moreover, I don't think that it happened strictly because people in HEP advocated Python (e.g. Wim Lavrijsen had been advocating Python since at least 2003, but it didn't take off until the last 5 years or so). I think that external factors—like language choice in university CS classes, Python's reputation as a data analysis language, and the prospect of careers in data science—played a much bigger role. For instance, Matlab and R certainly have a lot of data analysis functionality, and R even has ROOT bindings, but a similar trend didn't happen there. Julia can use all of Python's machine learning tools, but a movement to Julia hasn't happened yet. The difference is that this distributed community of physicists is hearing about Python from a variety of sources a lot more than they're hearing about these other languages, independent of their technical merits.

So I think it's great that we're investigating Julia in HEP, since it would thoroughly solve "The Numba problem" if it catches on. (I just saw Pere's response. I'll check out that paper!) My point is that we influence the HEP user community less than we might think. Despite its technical merits, Julia will be a hard sell.

-- Jim

On Mon, May 17, 2021 at 6:53 AM Philippe Gras <philip...@cern.ch> wrote:

--
This is the discussion forum of the HEP Software Foundation, http://hepsoftwarefoundation.org
---
You received this message because you are subscribed to the Google Groups "HEP Software Foundation Open Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hsf-forum+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hsf-forum/02c7fd34-a9b5-f895-cead-fa2e58a3a8f8%40cern.ch.

--
This is the discussion forum of the HEP Software Foundation, http://hepsoftwarefoundation.org
---
You received this message because you are subscribed to the Google Groups "HEP Software Foundation Open Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hsf-forum+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hsf-forum/CADXTekKKNv4Ds-tqLtVcD9Xon2eiZYEzd5p3HXN-NjjZWVJ-Bw%40mail.gmail.com.

[Eduardo Rodrigues]

Hello Philippe, Jim, cc PyHEP 2021 organisers,

We should then take this offline for a possible presentation at PyHEP 2021 (https://indico.cern.ch/e/PyHEP2021) ...

Note that others have looked at Julia for HEP, see in particular the paper https://link.springer.com/article/10.1007/s41781-021-00053-3.

Thanks for the interest and the suggestion,

Eduardo

| Eduardo Rodrigues, PhD (he/him) | University of Liverpool | LHCb Experiment @ CERN | Tel. +41 (0) 22 76 72088 |

---

From: hsf-...@googlegroups.com [hsf-...@googlegroups.com] on behalf of Philippe Gras [philip...@cern.ch]
Sent: 26 May 2021 08:56
To: hsf-...@googlegroups.com
Subject: Re: Julia programming language

[Nathan Simpson]

Hi Philippe,

Just throwing in my 2 cents — Julia may well be the language of the future (native JIT and auto diff support is really nice), but to echo Jim’s thoughts, Python is very much the language of the present.

“Pythonic code that runs fast” can also be Python itself; language forks like pypy, tools like numba, and frameworks like jax give Python C-like speeds (since it’s usually C or CUDA underneath anyway), especially for things like linear algebra.

This being said, if there was to be an immediate future for Julia in HEP, I think it would be as a backend replacement that ultimately still provides a Python interface — see PySR as an example of such a package https://github.com/MilesCranmer/PySR

Best, Nathan

On 26 May 2021, at 10:23, Eduardo Rodrigues <eduardo....@cern.ch> wrote:



Hello Philippe, Jim, cc PyHEP 2021 organisers,

We should then take this offline for a possible presentation at PyHEP 2021 (https://indico.cern.ch/e/PyHEP2021) ...

Note that others have looked at Julia for HEP, see in particular the paper https://link.springer.com/article/10.1007/s41781-021-00053-3.

Thanks for the interest and the suggestion,

Eduardo

| Eduardo Rodrigues, PhD (he/him) | University of Liverpool | LHCb Experiment @ CERN | Tel. +41 (0) 22 76 72088 |

---

From: hsf-...@googlegroups.com [hsf-...@googlegroups.com] on behalf of Philippe Gras [philip...@cern.ch]
Sent: 26 May 2021 08:56
To: hsf-...@googlegroups.com
Subject: Re: Julia programming language

Hello,

I've played with Julia since my last message. It has convinced me it's extremely well suited for HEP.

I'm more optimistic than Jim and Andy for the adoption, because it fulfills an immediate need, which is writing python-like code that runs fast.

It will be very good, if we can discuss the matter at PyHEP 2021. From the links Jim sent I understand it was studied in 2016, while the language was young (launched in 2012 according to wikipedia). The Language has been developing since then and it is worth to revisit it 5 years after.

I can give a talk at PyHEP to present the language including performance comparison with python for our usage.

Cheers,

Philippe.

On 17/05/2021 15:27, Jim Pivarski wrote:

It comes up from time to time, such as this DIANA/HEP meeting in July 2016: https://indico.cern.ch/event/545738/ (Despite my "technical issues in Julia" talk, I was trying to make it work at that time: https://github.com/JuliaHEP/ROOT.jl/issues/4)

In the PyHEP planning, the idea of having a short session on Julia came up, particularly to highlight Python-Julia interoperability.

Nowadays, I often advocate using Numba to include JIT-compilation in Python with the caveat that only a subset of the Python language can be compiled by Numba, though that can be difficult if you don't apply it to small functions and work outward. (If you apply it to a large codebase all at once, there's a good chance it will hit something it can't compile, and the error messages are verbose.) Julia would solve that problem—as a language designed to be JIT-compiled, it lacks the dynamic features that makes static analysis and compilation difficult. Any legal Julia code can be compiled.

But whereas the community is moving toward Python on its own, they are not moving to Julia in similar numbers. These are the responses to the "What language do you use?" question at PyHEP 2020: it's a steep cliff!

<image.png>

There is a trend toward using more Python in physics analysis, but not toward other languages in general. Moreover, I don't think that it happened strictly because people in HEP advocated Python (e.g. Wim Lavrijsen had been advocating Python since at least 2003, but it didn't take off until the last 5 years or so). I think that external factors—like language choice in university CS classes, Python's reputation as a data analysis language, and the prospect of careers in data science—played a much bigger role. For instance, Matlab and R certainly have a lot of data analysis functionality, and R even has ROOT bindings, but a similar trend didn't happen there. Julia can use all of Python's machine learning tools, but a movement to Julia hasn't happened yet. The difference is that this distributed community of physicists is hearing about Python from a variety of sources a lot more than they're hearing about these other languages, independent of their technical merits.

So I think it's great that we're investigating Julia in HEP, since it would thoroughly solve "The Numba problem" if it catches on. (I just saw Pere's response. I'll check out that paper!) My point is that we influence the HEP user community less than we might think. Despite its technical merits, Julia will be a hard sell.

-- Jim

On Mon, May 17, 2021 at 6:53 AM Philippe Gras <philip...@cern.ch> wrote:

Hello,

Do we have any group who is looking or have looked at Julia [1] ?

This language reconciles Python's easiness of code writing and c++-like
performance. It looks very appropriate for HEP.

Philippe.

[1] https://julialang.org/
 

--

This is the discussion forum of the HEP Software Foundation, http://hepsoftwarefoundation.org
---
You received this message because you are subscribed to the Google Groups "HEP Software Foundation Open Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hsf-forum+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hsf-forum/159237B2AD3ED44EA3FD8BE591B05CC60259FC2ED3%40CERNXCHG73.cern.ch.

[Eduardo Rodrigues]

Hi all,

A quick message for those of you interested in discussing Python and Julia further - at the PyHEP 2021 workshop  (https://indico.cern.ch/e/PyHEP2021) we will have a session on Julia for HEP and interoperability with Python, so feel invited to register if you fancy to participate and contribute. We had such a session in mind already and this thread just makes it clear that it is welcomed and timely.

Best wishes,

Eduardo, for the PyHEP 2021 organisers

| Eduardo Rodrigues, PhD (he/him) | University of Liverpool | LHCb Experiment @ CERN | Tel. +41 (0) 22 76 72088 |

---

From: hsf-...@googlegroups.com [hsf-...@googlegroups.com] on behalf of Nathan Simpson [email.n...@gmail.com]
Sent: 26 May 2021 14:17
To: Philippe Gras
Cc: hsf-...@googlegroups.com

[Jerry Ling]

Hi All,

It's very exciting to see Julia gets mentioned here out of the blue! I (and many, including Jan Strube whose paper was cited above) am looking forward to a more in-depth discussion during the PyHEP

sessions but just want to make a comment here to facillitate the topics for discussion later.

>language forks like pypy, tools like numba, and frameworks like Jax give Python C-like speeds (since it’s usually C or CUDA underneath anyway)

This certainly has been a source of success ("python is the 2nd best language for everything")[1] but this state of python ecosystem also creates difficulties. For example, I can only imagine how much effort Jim had to put into awkward-array1.0 to make it work with Numba (no doubt, uproot and friends have saved countless souls). In Julia, a performant jagged array is free, and you don't have to trade high-level language flexibility with fast loops, and libraries don't need to know about CUDA to work with GPU/TPU arrays etc ann thanks to pure code base.

I would only imagine it's more difficult to make things like auto-diff work when every function is across the language barrier (because everything is implemented in something but python). Working in research, we're destined to do new things (i.e. someone hasn't written a Python and made it work with Numpy, Jax etc. yet) at a rapid pace and it seems unreasonable to ask an end scientific user to know CUDA/C++/Python at the same time so they can test out their algorithm alongside their existing code.

I'm happy to learn what people are most curious about Julia (or doubts), I think many of us are happy to prepare some material for this session since we know we're pitching something new and making changes is hard in any industry.

Best regards,

Jerry

[1]: thus, Julia is at least 3rd best language for everything: http://uaf-7.t2.ucsd.edu/~namin/dump/slides/SNT_julia_20Dec15.pdf#page=10 (see page 10)