Mounting a DAX file system on linear device mapper

[Rohan Kadekodi]

Hello,

I am trying to mount the ext4 DAX file system on top of a linear mapped device on Intel Optane DC PM machines, by following the instructions posted here: https://pmem.io/2018/05/15/using_persistent_memory_devices_with_the_linux_device_mapper.html

The linear-mapped device seems to support DAX only for linux kernel versions before v5.1. I tried on v4.13 and it worked, but doesn't seem to work on the recent kernels. Could I get any pointers on why this is happening, and why the DAX support seems to have been removed in the recent kernels? Are there any kernel config options I need to enable to make the linear-mapped devices work on dax?

Thanks!

- Rohan

[Steve Scargall]

Hi Rohan,

What error(s) were reported?

I documented how to create contact, stripe, and mirrored LVM volumes using PMEM on my blog - https://stevescargall.com/2019/05/21/using-linux-volume-manager-lvm-with-persistent-memory/ as it's easier to use.

I verified the Device-Mapper procedure on Fedora 31 running kernel v5.6.13 still works.

// There are two FSDAX Namespaces defined:
# ndctl list -N

[

  {

    "dev":"namespace1.0",

    "mode":"fsdax",

    "map":"dev",

    "size":1598128390144,

    "uuid":"05660492-8ae9-4942-8523-efbdf73bea19",

    "sector_size":512,

    "align":2097152,

    "blockdev":"pmem1"

  },

  {

    "dev":"namespace0.0",

    "mode":"fsdax",

    "map":"dev",

    "size":1598128390144,

    "uuid":"157217f9-d9f1-43f9-b1bb-1e75f308182a",

    "sector_size":512,

    "align":2097152,

    "blockdev":"pmem0"

  }

]

// Both namespaces show up in lsbkl

# lsblk /dev/pmem*

NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

pmem1 259:2    0  1.5T  0 disk

pmem0 259:3    0  1.5T  0 disk

// Create a dm-linear device called 'linear-pmem':

$ echo -e "0 `blockdev --getsz /dev/pmem0` linear /dev/pmem0 0 "\\n"`blockdev --getsz /dev/pmem0` `blockdev --getsz /dev/pmem1` linear /dev/pmem1 0" | sudo dmsetup create linear-pmem

// The device is successfully created:

# dmsetup ls --tree

....

linear-pmem (253:5)

 ├─ (259:2)

 └─ (259:3)

....

# lsblk /dev/pmem*

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

pmem1         259:2    0  1.5T  0 disk

└─linear-pmem 253:5    0  2.9T  0 dm

pmem0         259:3    0  1.5T  0 disk

└─linear-pmem 253:5    0  2.9T  0 dm

// Create a GPT labeled partition on 'linear-pmem'

# fdisk /dev/mapper/linear-pmem

Welcome to fdisk (util-linux 2.34).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Device does not contain a recognized partition table.

The size of this disk is 2.9 TiB (3196256780288 bytes). DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors. Use GUID partition table format (GPT).

Created a new DOS disklabel with disk identifier 0x9dce496c.

Command (m for help): g

Created a new GPT disklabel (GUID: 5553DDB9-C381-8C4E-9980-AE64A1094B7E).

Command (m for help): n

Partition number (1-128, default 1):

First sector (2048-6242688990, default 2048): 4096

Last sector, +/-sectors or +/-size{K,M,G,T,P} (4096-6242688990, default 6242688990):

Created a new partition 1 of type 'Linux filesystem' and of size 2.9 TiB.

Command (m for help): w

The partition table has been altered.

Failed to add partition 1 to system: Invalid argument

The kernel still uses the old partitions. The new table will be used at the next reboot.

Syncing disks.

// Create an ext4 filesystem:

# mkfs.ext4 -b 4096 -E stride=512 -F /dev/mapper/linear-pmem

mke2fs 1.45.5 (07-Jan-2020)

Found a gpt partition table in /dev/mapper/linear-pmem

Creating filesystem with 780336128 4k blocks and 195084288 inodes

Filesystem UUID: a58d02c2-ce1f-4cf1-a62a-b0f5c4e0512e

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,

        102400000, 214990848, 512000000, 550731776, 644972544

Allocating group tables: done

Writing inode tables: done

Creating journal (262144 blocks): done

Writing superblocks and filesystem accounting information: done

// Mount the linear-pmem device using the '-o dax' option:

# mount -o dax /dev/mapper/linear-pmem /pmem

# df -h /pmem

Filesystem               Size  Used Avail Use% Mounted on

/dev/mapper/linear-pmem  2.9T   89M  2.8T   1% /pmem

[root@pmemdev1 /]#

// Confirm the device is mounted with the DAX option:

# mount -v | grep -i pmem

/dev/mapper/fedora_pmemdev1-root on / type ext4 (rw,relatime,seclabel)

/dev/mapper/linear-pmem on /pmem type ext4 (rw,relatime,seclabel,dax,stripe=512)

# dmesg | tail | grep -i ext4

...

[499596.843967] EXT4-fs (dm-5): DAX enabled. Warning: EXPERIMENTAL, use at your own risk

[499596.887342] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: dax

HTH

   Steve