DMA address space encoding idea

[Luke Gorrie]

Howdy!

Following on from the linear packet buffers idea...

Here is an idea to make it simple and efficient to share packet buffers between Snabb Switch processes, resolve physical addresses on demand, and check physical addresses for validity (to avoid unfortunate accidents with DMA).

(Note that in the proposed simplified packet struct we don't explicitly store the physical address of the packet in the struct, and it would be annoying for app authors if resolving physical addresses were slow or complex.)

The idea is to encode two pieces of information into the virtual address of each packet buffer:

1. The fact that this is a valid packet buffer and not some random pointer.

2. The physical address of the packet buffer.

The scheme could be that all packet buffers have a virtual address of the pattern 0x50xxxxxxxxxx where xxxxxxxxxx is the physical address and the 0x50 prefix is a tag that indicates this is a valid packet buffer address (and keeps the address in a range that won't clash with what Linux is using for malloc and shared libraries and so on).

So if you have a packet buffer with physical address 0xFA000000 then in every Snabb Switch process it would have the virtual address 0x5000FA000000. You could push that address onto a 'struct link' in shared memory and whoever received it could use it directly. When you use the address for DMA you would first check that it's valid (address & 0xFF0000000000 == 0x500000000000) and then strip off the tag bits to get the physical address (physical = address & 0x00FFFFFFFFFF).

For extra safety on DMA we could also squirrel away a flag or cookie value somewhere to indicate that the address belongs to a huge page that was allocated for packet buffers e.g. in the first word of the page, or in the packet struct, or in the fact that the address is actually mapped in our virtual address space. (This is just wanting a packet address being screwed up to cause an assertion failure rather than memory corruption.)

So to allocate a new huge page the process would be something more elaborate like (assuming a multiprocess environment):

1. mmap(MAP_HUGETLB|MAP_FILE) to get a huge page in file-backed shared memory.

2. Resolve the physical address.

3. mremap() the page onto its 0x50xxxxxxxxxx address.

4. Rename the shared memory file to indicate the physical address (e.g. /var/snabbswitch/buffers/FA000000.dat) so that other processes can find it on demand.

Sound reasonable, or mad?

There would be extra wrinkles in a multiprocess environment...

When would a Snabb Switch process map the packet buffer memory files? (On startup? Or on demand via a SIGSEGV handler that recognises the address as a packet buffer and tries to map the right page?)

How would you free packets? (put them onto a 'struct link' where their owner can find and recycle them?)

How would you free whole pages? (how to detect when they have fallen out of use?)

(The 0x50 is inspired by this semi-dubious information: http://stackoverflow.com/questions/7729048/elf64-x86-64-and-start-address-of-the-memory-mapping-segment-for-shared-objects)

[Luke Gorrie]

On 5 January 2015 at 21:18, Luke Gorrie <lu...@snabb.co> wrote:

Here is an idea to make it simple and efficient to share packet buffers between Snabb Switch processes, resolve physical addresses on demand, and check physical addresses for validity (to avoid unfortunate accidents with DMA).

This idea turned out to be simple to implement and to work well so far.

Code: https://github.com/lukego/snabbswitch/commit/a889d0b1cad220cfae9bbefc5637ce1c667968ce

A question is whether this part of the scheme needs to be modified to be reliable:

The scheme could be that all packet buffers have a virtual address of the pattern 0x50xxxxxxxxxx where xxxxxxxxxx is the physical address and the 0x50 prefix is a tag that indicates this is a valid packet buffer address (and keeps the address in a range that won't clash with what Linux is using for malloc and shared libraries and so on).

There was one wrinkle in this scheme:

So to allocate a new huge page the process would be something more elaborate like (assuming a multiprocess environment):

1. mmap(MAP_HUGETLB|MAP_FILE) to get a huge page in file-backed shared memory.

2. Resolve the physical address.

3. mremap() the page onto its 0x50xxxxxxxxxx address.

4. Rename the shared memory file to indicate the physical address (e.g. /var/snabbswitch/buffers/FA000000.dat) so that other processes can find it on demand.

which is that mremap() doesn't seem to actually work on HugeTLB pages on Linux. So instead of mmap+mremap I use mmap+mmap+munmap to create the initial binding and replace it with the new one. See http://stackoverflow.com/questions/27997934/mremap2-with-hugetlb-to-change-virtual-address/28012671#28012671

I also have one passing idea for managing packet memory in a multiprocess environment:

There would be extra wrinkles in a multiprocess environment...

When would a Snabb Switch process map the packet buffer memory files? (On startup? Or on demand via a SIGSEGV handler that recognises the address as a packet buffer and tries to map the right page?)

How would you free packets? (put them onto a 'struct link' where their owner can find and recycle them?)

How would you free whole pages? (how to detect when they have fallen out of use?)

which is that packets sent from one process to another could be copied by link.receive(). Then if a process restarts it can "garbage collect" all the packets that were allocated by its previous instance that are not waiting on a link somewhere. (Just a braindump.. more thought/whiteboard needed to be sure.)

The whole concept of a "copy" is very interesting in the context of a complex distributed system like a modern PC. Braindump/whiteboard on that topic needed too :).

Cheers,

-Luke