Seeking Support for Lena: An Architectural Framework for Data Analysis

[Yaroslav Nikitenko]

Dear HSF organisers and community,

I have created an architectural framework for data analysis in Python called Lena. I presented it at PyHEP.dev 2024 [1].

It is based on functional programming and has as its core features:

- lazy evaluation. This prevents loading all data into memory and enables optimizations before execution,

- metadata. A scientist does not have to track all metadata in a large analysis manually (existing tools don't allow that),

- computational pipelines as first-class citizens.

For a user, a framework takes many accounting and architectural decisions, leading to a more structured, maintainable and reusable code. It also provides tools and optimisations a scientist wouldn't write themselves. There is no existing framework for data analysis (in the sense of inversion of control); see terminology in [2]. 

I have learnt that recently NVIDIA introduced lazy evaluation as a core part of their Python libraries. That allows efficient kernel fusion. I have run a benchmark that a global optimization could increase performance up to 50% [3]. I have also recently contacted AMD [4], maybe they get more interested.

Functional programming is closely connected with parallel processing; that is why I'm reaching out to organisations supporting accelerated computing.

Could someone recommend organisations or specific teams/individuals who would be interested to support or integrate the framework?

I'm based in Germany if that is important.

Thanks for your support.

Links:

1. My talk on Lena at PyHEP.dev 2024: https://www.youtube.com/watch?v=uNYoR6Y7708

2. Framework vs library (not very well known in C++ and data analysis): https://www.sencha.com/blog/difference-between-framework-vs-library-snc/

3. Benchmarking global performance for CUDA, 50% improvement with a framework: https://github.com/NVIDIA/cccl/discussions/7685#discussioncomment-16291929

4. My post for ROCm (in more details) https://github.com/ROCm/ROCm/discussions/6079

--

Best regards,
Yaroslav Nikitenko

[Ianna Osborne]

Hi Yaroslav,

Thanks for sharing! Indeed this is a direction we are very interested in because lazy evaluation and kernel fusion gives us a significant performance boost. We've been working with Nvidia on migrating awkward to cuda.compute (e.g. CCCL) and were looking into the kernel fusion alternatives. Rabbit, for example, is one of the possible directions. They operate on RabbitArrays and there was a discussion about to_awkward/from_awkward.

It would be nice to have a dedicated meeting about Lena to see how we can use it in conjunction with awkward.

Kind regards,

Ianna

--
This is the discussion forum of the HEP Software Foundation, http://hepsoftwarefoundation.org
---
You received this message because you are subscribed to the Google Groups "HEP Software Foundation Open Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hsf-forum+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hsf-forum/CA%2BRLzGAVrusyT6KnbSZDqEXcyXLOvRQqX8s8QNzV%2BrT706OBUQ%40mail.gmail.com.