[PROPOSAL] In-memory Arrow Data Cache Solution for Distributed Training on Kubernetes
47 views
Skip to first unread message
Akshay Chitneni
unread,
Mar 27, 2025, 11:47:47 AMMar 27
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to kubeflow-discuss
Hi Kubeflow community,
We are working on an in-memory cache solution based on Apache Arrow and DataFusion for ML workloads optimizing data loading of tabular data (e.g. Apache Iceberg tables) for distirbuted training on Kubernetes. We have an intitial proposal on the design of cache and integration with kubeflow trainer v2. Please share your feedback, and we will soon transfer this to a KEP.
We are also speaking at Kubeflow Summit 2025 in London next week and would love to meet and discuss if you are at the conference -https://sched.co/1u5iQ