Huge page support on Linux

[Qing Chen]

Hi, 

Does the library support huge page when mmap( 2MB or 1GB) ? thanks.  

[Andy Rudoff]

Hi,

Since you mention mmap, I assume you're asking about Linux, so I'll provide the Linux-specific answer:

Yes, the PMDK libraries support huge pages, but mostly by trying to make things align so that the OS can choose to use them.  Since persistent memory is allocated by the file system, whether or not the allocations are aligned and large enough for huge pages depends on the specific allocator.  When using mmap() to create the mapping, the application doesn't need to provide any special flags for huge pages, the OS will use them if possible.

Your best bet to get large pages is to start with a clean file system and allocate the space using posix_fallocate() (the PMDK libraries do this if you use them to create the file).  Then, mmap() is likely to use 2MB pages when possible (1GB mappings are not yet used because there are far fewer TLB entries for 1GB entries in current processors -- in the future, this may change).

You can use Device DAX instead of a file system and you will be sure to get large pages because now there's no file system allocator in the mix, but using Device DAX means you give up the ability to manage your pmem like files (names, permissions, mv, cp, etc).

-andy

[Qing Chen]

Thanks for your detail answer , one thing i am not sure i understand correctly, 

so you mean if we create a file using posix_fallocate , say 24MB, then we mmap it , system will atomically map it with 2MB page , is this right ?  Which linux kernel version supporting this ? 

 

So what's difference between "Device dax" and "mmap "  ? From my understanding, when mmap file on pmem disk, it has device dax atomically ?

Does it mean the first one need special system API other than mmap on Linux? 

Could you help clarify this a little bit ? 

Thank you very much. 

[Ross Zwisler]

Device DAX and filesystem DAX are different.  For Filesystem DAX you have a normal block device on which you can put a filesystem, and that filesystem can choose to use DAX via a mount option.  "Device DAX" in this case refers to a character device that does not implement a normal block I/O path, but that you can interact with using mmap().

More details on these namespace modes can be found in the "ndctl create-namespace" man page:

http://pmem.io/ndctl/ndctl-create-namespace.html

Filesystem DAX is built upon "fsdax" mode namespaces, and device DAX is used with "devdax" mode namespaces.

Filesystem DAX PMD support has been enabled and working in it's current form since v4.10.  Device DAX has had PMD support since it was introduced in v4.7, and gained PUD (1 GiB page) support in v4.11.

For further reading I suggest our man pages online:

http://pmem.io/ndctl/

and our wiki, which also links to other reading:

https://nvdimm.wiki.kernel.org/

Specifically for 2 MiB pages on filesystem DAX, you'll probably find this useful:

https://nvdimm.wiki.kernel.org/2mib_fs_dax

Thanks,

- Ross