Atomicity of RDMA verbs in DiSNI

[Mojtaba Eslahi Kelorazi]

Hi,

We are using DiSNI for direct access to remote memory. We have some concerns about the atomicity of the RDMA verbs, that we have not found any source that could clear the doubt. The scenarios are as:

(1) - If a sender A writes something in the shared memory of receiver B, for instant an integer number of 4 bytes, and B try to read that memory at the same time. As my understanding, RDMA NIC writes from left to right, so B should read from left to write too, but I couldn't find any information of the atomicity of the write. I'm not sure what is the minimum size of a memory chunk that B could be sure that will be atomically written to that shared memory (1 bytes, 4 bytes, etc.) to start reading. Could there be the case that B reads only 2 bytes from that memory and incorrectly parse that integer value?

(2) - The second scenario would be similar to the first one, but with another server C. A writes to B's memory, and C reads from B memory, also via RDMA. What is the minimum size of memory chunk should C read to ensure that it that memory chunk has been completely written.

High Regards,

Mojtaba

[Jonas Pfefferle1]

Hi Mojtaba

To my knowledge RDMA reads/write are atomic to cache line size. However, this requires your access to be cache-line size aligned. This is not a property of RDMA but of the underlying DMA transfers. A few papers that talk about this:
https://www.usenix.org/system/files/conference/nsdi14/nsdi14-paper-dragojevic.pdf
https://infoscience.epfl.ch/record/220614/files/sabres_micro16_1.pdf

"One-sided operations have semantic limitations, which is a direct consequence of their DMA-based implementation on the remote end: while each individual cache block can be accessed atomically, there are no guarantees for larger accesses straddling multiple cache block"

This should be true in respect to the CPU and other remote operations.

Regards,
Jonas

"Mojtaba Eslahi Kelorazi" ---03/07/2019 09:08:23 AM---Hi, We are using DiSNI for direct access to remote memory. We have some

--
You received this message because you are subscribed to the Google Groups "zrlio-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zrlio-users...@googlegroups.com.
To post to this group, send email to zrlio...@googlegroups.com.
Visit this group at https://groups.google.com/group/zrlio-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/zrlio-users/7605d60c-0c24-4cb1-86f1-b981e4255094%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Patrick Stuedi]

Hi, I think there is no assumption that can be made beyond a single byte being written atomically. This also not specific to DiSNi.

--

[Jonas Pfefferle1]

Generally, yes. No RDMA protocol specification guarantees any form of atomicity (except for atomic operations). As stated in my other email the per-cache-line atomicity is a property of the underlying DMA transfers. However, the vendor can choose to implement it in a way that this is no longer guaranteed. All software RDMA devices for example do not have this property, Omnipath <64KB transfer (below direct placement threshold), etc.

Regards,
Jonas

"Patrick Stuedi" ---03/07/2019 09:40:19 AM---Hi, I think there is no assumption that can be made beyond a single byte being written atomically. T

From: "Patrick Stuedi" <pst...@gmail.com>
To: "Mojtaba Eslahi Kelorazi" <m.esl...@gmail.com>
Cc: "zrlio-users" <zrlio...@googlegroups.com>
Date: 03/07/2019 09:40 AM
Subject: Re: [zrlio-users] Atomicity of RDMA verbs in DiSNI
Sent by: zrlio...@googlegroups.com

---

To view this discussion on the web visit https://groups.google.com/d/msgid/zrlio-users/CAFFT%2BgyLTmcS32xbFco5iw9ndJFrKF3tVq74QyjN1nwNEsRVHA%40mail.gmail.com.