What issue is cause of experimental of librpmem?

[y-g...@fujitsu.com]

Hello,

I have a question about librpmem which is RDMA library.

Though librpmem document says it is experimental status yet, what issues must be resolved to remove "experimental" ?

In my understanding, Filesystem-DAX had a problem against RMDA.
If the target area of DMA/RDMA was mmaped file in userspace, and kernel must be removed the area
due to truncate() or any other event, kernel could not negotiate between RDMA and truncate.
As a result, the data may be lost.

If the area was inside of kernel or driver, then kernel could wait truncate() until completion of
transfer of RDMA, but kernel could not wait it if the target area was in userspace.
Because the application could not know when user would stop transferring data.

IIRC it was one of the cause of experimental, but I'm not sure this issue was solved or not.

Is PMDK still waiting for solve the above issues ?
Otherwize, does librpmem itself have other critical issues?

Thanks,

[Janek Mi]

Hi

The librpmem library is marked experimental because the PMDK team has not seen any interest from potential customers to use it.

On the contrary, we got a lot of feedback pointing out issues with this API. Having all of this input we decided to propose a new library which is currently under development.

If you are interested in using it feel free to get in touch with us on our GitHub repo: https://github.com/pmem/rpma

Yesterday I and our architect had a presentation describing this story during OFA 2020. I think it is worth a while:
https://www.openfabrics.org/wp-content/uploads/2020-workshop-presentations/202.-gromadzki-ofa-workshop-2020.pdf

Note: I am not saying the librpmem library has any issues with the idea itself. The only issue is how we initially decided to provide this functionality.

Which caused the librpmem API to not be easily generally applicable to various workload scenarios.

I hope it helps.

Jan

PS If you find out the librpmem API is good enough for your applications, feel free to reach out. So we can discuss removing the "experimental" mark.

PPS The Filesystem-DAX vs RDMA issue is still with us. A general solution to this is still work in progress. Nonetheless, it is possible to work around it using e.g. On-Demand-Paging extension from Mellanox:

https://docs.mellanox.com/display/MLNXOFEDv451010/Optimized+Memory+Access#OptimizedMemoryAccess-On-Demand-Paging(ODP)

[Yasunori Goto]

On 2020/06/10 22:42, Janek Mi wrote:
> Hi
>
> The librpmem library is marked experimental because the PMDK team has
> not seen any interest from potential customers to use it.
> On the contrary, we got a lot of feedback pointing out issues with
> this API. Having all of this input we decided to propose a new library
> which is currently under development.
> If you are interested in using it feel free to get in touch with us on
> our GitHub repo: https://github.com/pmem/rpma
>
> Yesterday I and our architect had a presentation describing this story
> during OFA 2020. I think it is worth a while:
> https://www.openfabrics.org/wp-content/uploads/2020-workshop-presentations/202.-gromadzki-ofa-workshop-2020.pdf

Thank you for your information. It is quite interesting.
Now, I would like to know future plan of rpma,
do you have any current issues of rpma or any roadmap for its enhancement?

>
> Note: I am not saying the librpmem library has any issues with the
> idea itself. The only issue is how we initially decided to provide
> this functionality.
> Which caused the librpmem API to not be easily generally applicable to
> various workload scenarios.

Ok.

>
> I hope it helps.
> Jan
>
> PS If you find out the librpmem API is good enough for your
> applications, feel free to reach out. So we can discuss removing the
> "experimental" mark.
> PPS The Filesystem-DAX vs RDMA issue is still with us. A general
> solution to this is still work in progress. Nonetheless, it is
> possible to work around it using e.g. On-Demand-Paging extension from
> Mellanox:
> https://docs.mellanox.com/display/MLNXOFEDv451010/Optimized+Memory+Access#OptimizedMemoryAccess-On-Demand-Paging(ODP)

Here is my understanding about it.

In ODP, driver/kernel can invalidate the target page under data translation,
and it validates the page when application need to access the page.
When kernel validate the page, filesystem can coordinate
between metadata update like truncate() and RDMA data translation.
This is why ODP can be work around.

Is it correct?

Thanks,

> On Wednesday, 10 June 2020 09:23:13 UTC+2, y-g...@fujitsu.com wrote:
>
> Hello,
>
> I have a question about librpmem which is RDMA library.
>
> Though librpmem document says it is experimental status yet, what
> issues must be resolved to remove "experimental" ?
>
> In my understanding, Filesystem-DAX had a problem against RMDA.
> If the target area of DMA/RDMA was mmaped file in userspace, and
> kernel must be removed the area
> due to truncate() or any other event, kernel could not negotiate
> between RDMA and truncate.
> As a result, the data may be lost.
>
> If the area was inside of kernel or driver, then kernel could wait
> truncate() until completion of
> transfer of RDMA, but kernel could not wait it if the target area
> was in userspace.
> Because the application could not know when user would stop
> transferring data.
>
> IIRC it was one of the cause of experimental, but I'm not sure
> this issue was solved or not.
>
> Is PMDK still waiting for solve the above issues ?
> Otherwize, does librpmem itself have other critical issues?
>
> Thanks,
>

> --
> You received this message because you are subscribed to the Google
> Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pmem+uns...@googlegroups.com
> <mailto:pmem+uns...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/cb8dc2bb-4134-4f21-8782-03452a25b8d0o%40googlegroups.com
> <https://groups.google.com/d/msgid/pmem/cb8dc2bb-4134-4f21-8782-03452a25b8d0o%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Yasunori Goto

[Janek Mi]

On Thu, 11 Jun 2020 at 09:46, Yasunori Goto <y-g...@fujitsu.com> wrote:

On 2020/06/10 22:42, Janek Mi wrote:
> Hi
>
> The librpmem library is marked experimental because the PMDK team has
> not seen any interest from potential customers to use it.
> On the contrary, we got a lot of feedback pointing out issues with
> this API. Having all of this input we decided to propose a new library
> which is currently under development.
> If you are interested in using it feel free to get in touch with us on
> our GitHub repo: https://github.com/pmem/rpma
>
> Yesterday I and our architect had a presentation describing this story
> during OFA 2020. I think it is worth a while:
> https://www.openfabrics.org/wp-content/uploads/2020-workshop-presentations/202.-gromadzki-ofa-workshop-2020.pdf

Thank you for your information. It is quite interesting.
Now, I would like to know future plan of rpma,
do you have any current issues of rpma or any roadmap for its enhancement?

We are currently preparing a functional developer code drop which should be ready in a couple of weeks.
It should be a good enough example for somebody to start its own development if it is capable of developing based on libibverbs/librdmacm libraries.

Going further, in a couple of months we are planning a first official release.
I will check if I can make public our roadmap on GitHub. It should be useful for you and for others who are waiting for the library.

Having that said, you are more than welcome to let us know if you have any particular needs, so we can prioritize differently to deliver things that are wanted the most as soon as possible.
Please send such requests directly to me: jan.m.m...@intel.com or, if you don't mind its public, just post an issue on GitHub.

 

>
> Note: I am not saying the librpmem library has any issues with the
> idea itself. The only issue is how we initially decided to provide
> this functionality.
> Which caused the librpmem API to not be easily generally applicable to
> various workload scenarios.
Ok.

>
> I hope it helps.
> Jan
>
> PS If you find out the librpmem API is good enough for your
> applications, feel free to reach out. So we can discuss removing the
> "experimental" mark.
> PPS The Filesystem-DAX vs RDMA issue is still with us. A general
> solution to this is still work in progress. Nonetheless, it is
> possible to work around it using e.g. On-Demand-Paging extension from
> Mellanox:
> https://docs.mellanox.com/display/MLNXOFEDv451010/Optimized+Memory+Access#OptimizedMemoryAccess-On-Demand-Paging(ODP)

Here is my understanding about it.

In ODP, driver/kernel can invalidate the target page under data translation,
and it validates the page when application need to access the page.
When kernel validate the page, filesystem can coordinate
between metadata update like truncate() and RDMA data translation.
This is why ODP can be work around.

Is it correct?

Yes. You can solve this issue in two possible ways:

- forcing kernel/filesystem and other interested parties to not move the page as long as it is registered for RDMA use or
- the kernel has to let to know RNIC each time the page is migrated for whatever reason

The Mellanox ODP is applying the second strategy.

[Yasunori Goto]

> <mailto:jan.m.m...@intel.com> or, if you don't mind its public,

Thank you for your explanation.
It is very helpful for me.

Thanks a lot.

---

Yasunori Goto

[Janek Mi]

I'm pleased to let you know that we have published the RPMA roadmap:
https://github.com/pmem/rpma/blob/master/ROADMAP.md

In case you will have any comments or requests. Feel free to let us know.

Jan