Bring your own backend

Denish Patel

unread,

Dec 27, 2021, 5:02:23 PM12/27/21

to modin-dev

Hi,

Is there any more documentation around the bring your own backend feature and what sorts of things it will allow?

We are trying to see if modin provides a data load interface that we can implement to define our own custom way of loading the data as computation happen.

My understanding is that modin provides user API that are well known such as panda's API and then has a separate backend API that can be implemented for our own backend for how to load data. Please correct me if my understanding is wrong.

Thanks,

Denish

Devin Petersohn

unread,

Dec 28, 2021, 10:06:55 AM12/28/21

to Denish Patel, modin-dev

Hi Denish, thanks for emailing the developer mailing list!

Is there any more documentation around the bring your own backend feature and what sorts of things it will allow?

We have Developer documentation for the components of the architecture that you would need to override here: https://modin.readthedocs.io/en/latest/developer/index.html (these are latest docs, rather than the stable docs so if you've been to the docs before it will look different).

My understanding is that modin provides user API that are well known such as panda's API and then has a separate backend API that can be implemented for our own backend for how to load data. Please correct me if my understanding is wrong.

Yes, this is correct. You can actually control more than just data loading. You can implement a full query processing engine to handle queries that come in as pandas.

What level you choose to implement will likely depend on whether you want to control the full query execution or just want to be able to implement a way to submit python tasks (e.g. like Dask and Ray). Did you have an idea of what level you'd like to implement?

Thanks again for the email!

Devin

--
You received this message because you are subscribed to the Google Groups "modin-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modin-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modin-dev/ca5dff5a-fb8b-41d8-a2a6-880b45b8678fn%40googlegroups.com.

Denish Patel

unread,

Dec 28, 2021, 2:24:23 PM12/28/21

to modin-dev

We are mostly interested in the data loading part rather than the full query execution. I will take a look through those new docs but just wanted to confirm that it was possible to control the the data loading part. Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to modin-dev+...@googlegroups.com.

Reply all

Reply to author

Forward