pmempool returns bus error

[Yue Li]

hi everyone, 

In my machine, I can see

[root@hilo mnt]# ndctl list

{

  "dev":"namespace0.0",

  "mode":"fsdax",

  "size":34359738368,

  "sector_size":512,

  "blockdev":"pmem0",

  "numa_node":0

}

And as I tested, I can successfully mount ext4-dax on top of it. 

After unmount it, I tried to create a device dax by

[root@hilo mnt]# ndctl create-namespace --mode devdax -e namespace0.0 -f

{

  "dev":"namespace0.0",

  "mode":"devdax",

  "size":"31.50 GiB (33.82 GB)",

  "uuid":"72730c40-8cf2-4bca-a31e-c9f99b93a5c3",

  "raw_uuid":"00000000-0000-0000-0000-000000000000",

  "daxregion":{

    "id":0,

    "devices":[

      {

        "chardev":"dax0.0",

        "size":"31.50 GiB (33.82 GB)"

      }

    ]

  },

  "numa_node":0

}

However, pmempool on this device dax always give me bus error:

[root@hilo mnt]# pmempool info /dev/dax0.0

Bus error (core dumped)

I tried different version of pmempool as well as ndctl (all from master branches), and the problem remains... have you seen this problem before? 

I'm using Kernel 4.9.41 with all the compilation flags recommended here:

https://nvdimm.wiki.kernel.org/

Thank you!

Best, 

Yue

[Yue Li]

Removing the pool file using pmempool gives the following error:

[root@hilo mnt]# pmempool rm /dev/dax0.0

pmempool:41549 terminated with signal 7 at PC=7f15a2507814 SP=7ffd9c20e1b8.  Backtrace:

/usr/lib64/libc.so.6(+0x89814)[0x7f15a2507814]

pmempool[0x442fc0]

pmempool[0x443ab4]

pmempool[0x40c094]

pmempool[0x40c52e]

pmempool[0x4055e7]

/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f15a249fc05]

[Andy Rudoff]

Hi Yue,

You only mention creating the namespace.  Did you also create a pool?  Something like:

# pmempool create obj --layout=mylayout /dev/dax0.0

I suspect what is happening is that there is not a valid pool there, and pmempool has a bug that causes it to dump core instead of just telling you it cannot find a valid pool.

We'll look into the core dump issue.

Thanks.

-andy

[Yue Li]

hi Andy, 

Yes, I creating pool as well, which also gives me bus error...

It's kind of strange as my other two machines with the same kernel work quite well with device dax...

Thanks, 

Yue

[Steve Scargall]

Hello Yue,

I've been unable to reproduce it using emulated NVDIMMs on Fedora 27 (Kernel 4.16), Fedora 28 (Kernel 4.17), or Fedora RawHide (Kernel 4.18).  Can you provide more information about your config please.  

• What Linux Distro are you using? (Fedora, SUSE, CentOS, etc)
• What version of the Linux Distro are you using?
• Are you using emulated NVDIMMS?  If so how:
• The memmap grub option?
• QEMU?
• Other (please provide details)?

If you can attach the core dumps we can take a look.

Thanks,

    Steve

[Dan Williams]

The usual problem with unexpected SIGBUS is a mismatch between the
device-dax alignment and the memory-mapping alignment assumed by the
application. Does the failure reproduce with a 4K aligned device-dax
instance? That is the most compatible alignment:

ndctl create-namespace --mode devdax -e namespace0.0 -f -a 4K

[Yue Li]

Thank you Dan, using 4K aligned works!

So do 4k and 2M alignment depend on the hardware we use? The other two
machines with exactly the same kernel config have no issue with 2M
alignment...

Thanks again,

Yue

[Dan Williams]

No, there's no hardware dependency. Perhaps you have updated versions of
pmdk on those other systems that know how to honor the device-dax
alignment? That's my only idea at the moment.

[Yue Li]

hmm... this is strange.. I verified the pmempool are using exactly the
same distribution

pmempool 1.4-rc4-360-g14296ee

built from an earlier pmdk master branch...

Best,

Yue

[Dan Williams]

Weird. You can try:

echo "module dax +fp" > /sys/kernel/debug/dynamic_debug/control

...to see if that gives you more information about why the device-dax
driver is failing the mapping. You need to be root to do that and the
messages will show up in the kernel log.

[Yue Li]

Thanks for helping Dan!

I just tried this on the machine. Here's what dmesg prints when I do

pmempool info /dev/dax0.0

which gave me bus error:

[312905.140977] dax_open: dax dax0.0: dax_open
[312905.141082] dax_release: dax dax0.0: dax_release
[312905.141087] dax_open: dax dax0.0: dax_open
[312905.141096] dax_open: dax dax0.0: dax_open
[312905.141142] dax_release: dax dax0.0: dax_release
[312905.141146] dax_open: dax dax0.0: dax_open
[312905.141150] dax_open: dax dax0.0: dax_open
[312905.141193] dax_release: dax dax0.0: dax_release
[312905.141239] dax_mmap: dax dax0.0: dax_mmap
[312905.141249] dax_dev_fault: dax dax0.0: dax_dev_fault: pmempool:
read (0x7f2380000000 - 0x7f2577e00000)
[312905.141251] __dax_dev_fault: dax dax0.0: __dax_dev_fault:
alignment > fault size
[312905.172797] dax_release: dax dax0.0: dax_release
[312905.172811] dax_release: dax dax0.0: dax_release

dax0.0 is created using

ndctl create-namespace --mode devdax -e namespace0.0 -f

Best,

Yue

[Dan Williams]

Yes, this confirms that this application is triggering a 4K fault on a
2M aligned device.

[Yue Li]

Do you know why in my other machines there was no such problem? All my
machines installed exactly the same self compiled kernel...

Best,

Yue