nvdimm access using "raw" mode + libpmem apis

[Rukhsana Ansari]

Hi,

I am trying to confirm whether the following is a valid mode of access and usage of the nvml that ensures persistence for a NVDIMM in the event of AC power loss.

Is the raw block device mode a valid mode for using the libpmem apis of nvml (v 1.2) ?

or

 Does the device need to be in the "dax" mode as  detailed in: https://nvdimm.wiki.kernel.org/ (dax - /dev/daxN.M character device supporting DAX)

A legacy NVDIMM, detected as type persistent (type 12) and presented by default  as a block device in the raw access mode.

ndctl list --namespaces

[

  {

    "dev":"namespace1.0",

    "mode":"raw",

    "size":8589934592,

    "blockdev":"pmem1"

  },

The app opens the device in direct io mode and mmaps the device.

if ((fd = open(“/dev/pmem0”, O_RDWR | O_DIRECT))< 0) {

perror(“open failed.”);

exit(1);

mmapp = mmap(NULL,length,PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);

// Write to the device using NVML api

pmem_memcpy_persist(mmap, buf, buf_length)

i.e use the nvml  library for byte access the  device, as shown in slide 28 of  https://www.snia.org/sites/default/files/SDC15_presentations/persistant_mem/JeffChang-ArthurSainio_NVDIMM_Cookbook.pdf

where the application uses load/store via NVML.

The examples at src/examples/libpmem that use  pmem_map_file() expect a file as the src argument and fail if a raw block device is specified.

Thanks,

[Dan Williams]

On Fri, Jul 21, 2017 at 11:08 AM, Rukhsana Ansari <ruk.a...@gmail.com> wrote:
> Hi,
>
> I am trying to confirm whether the following is a valid mode of access and
> usage of the nvml that ensures persistence for a NVDIMM in the event of AC
> power loss.
>
> Is the raw block device mode a valid mode for using the libpmem apis of nvml
> (v 1.2) ?
> or
> Does the device need to be in the "dax" mode as detailed in:
> https://nvdimm.wiki.kernel.org/ (dax - /dev/daxN.M character device
> supporting DAX)
>
> A legacy NVDIMM, detected as type persistent (type 12) and presented by
> default as a block device in the raw access mode.
>
> ndctl list --namespaces
> [
> {
> "dev":"namespace1.0",
> "mode":"raw",
> "size":8589934592,
> "blockdev":"pmem1"
> },

For proper DAX operation you'll want the mode to say either "memory"
or "dax". Otherwise, direct-I/O with dax mapped buffer will fail. The
kernel tries use "memory" mode by default for legacy / type-12
persistent memory. Does your kernel have CONFIG_ZONE_DEVICE=y? That
would explain why it's falling into raw mode. What kernel version is
being used?

> The app opens the device in direct io mode and mmaps the device.
> if ((fd = open(“/dev/pmem0”, O_RDWR | O_DIRECT))< 0) {

DAX is not supported on the block device directly. You would need to
use the dax facilities of an ext4 or xfs filesystem mounted on
/dev/pmem0.

[Rukhsana Ansari]

Thanks Dan.

Kernel version is 4.9.11. I will get back regarding the confirmation of kernel config for CONFIG_ZONE_DEVICE.

Can you clarify if  a filesystem isnt used , which of  the following is the right option for byte access via: device open (direct io option), followed by mmap and use if libpmem nvml apis.

1. Dax mode 

 2. Memory mode

[Dan Williams]

If you want to just map a device file and then use the nvml apis "dax"
mode is what you want. The following command:

ndctl create-namespace --reconfig=namespace0.0 --mode=dax --force

...will destroy /dev/pmem0 and establish a character device named
/dev/dax0.0 that nvml can consume. Note that the command will fail if
CONFIG_ZONE_DEVICE is disabled.

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/CAPu0trqSr0Q_RGjQf1Ovy%3DzmHC_KXCpfiWMqwNP9izU8kSwZ3g%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Krzysztof Czurylo]

On Friday, July 21, 2017 at 8:08:06 PM UTC+2, Rukhsana Ansari wrote:

(...)

The examples at src/examples/libpmem that use  pmem_map_file() expect a file as the src argument and fail if a raw block device is specified. 

The pmem_map_file() can be used with Device DAX (i.e. /dev/dax1.0), but you need to pass 0 as the second argument (len), as for mapping the already existing file.

This would result in mapping the entire device. Currently, NVML does not support partial mapping of Device DAX devices.

Same applies to other functions like pmemobj_create(), pmemblk_create(), ...

We'll update the example(s) to make it clear.

Thanks,

K.

[Rukhsana Ansari]

Thanks Dan.

> email to pmem+unsubscribe@googlegroups.com.

[Rukhsana Ansari]

Hi,

Following up on the replies, I re-compiled 4.9.11 kernel to have the following config:

CONFIG_ZONE_DEVICE=y
CONFIG_X86_PMEM_LEGACY_DEVICE=y
CONFIG_X86_PMEM_LEGACY=y
# CONFIG_BLK_DEV_RAM_DAX is not set
CONFIG_LIBNVDIMM=y
CONFIG_BLK_DEV_PMEM=m
CONFIG_ND_PFN=m
CONFIG_NVDIMM_PFN=y
CONFIG_NVDIMM_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_NR_DEV_DAX=32768
CONFIG_FS_DAX=y
CONFIG_ARCH_HAS_PMEM_API=y

With a kernel cmd line including a pmem device: memmap=530M!4G

ndctl (version 57.1)  doesn't list any namespaces. Howeverm under device tree I do see dax and pfn config.

Appreciate help in resolving this:

Here are some details:

1. dmesg | grep persistent
[    0.000000] user: [mem 0x0000000100000000-0x00000001211fffff] persistent (type 12)

2. ndctl list

// No output

tree /sys/bus/nd/devices/region0/ -I power
/sys/bus/nd/devices/region0/
 btt0.0
    devtype
    modalias
    namespace
    numa_node
    sector_size
    size
    subsystem -> ../../../../../../bus/nd
    uevent
    uuid
 btt_seed
 dax0.0
    align
    devtype
    modalias
    mode
    namespace
    numa_node
    resource
    size
    subsystem -> ../../../../../../bus/nd
    uevent
    uuid
 dax_seed
 devtype
 driver -> ../../../../../bus/nd/drivers/nd_region
 init_namespaces
 mappings
 modalias
 namespace0.0
    devtype
    force_raw
    holder
    modalias
    mode
    nstype
    numa_node
    resource
    size
    subsystem -> ../../../../../../bus/nd
    uevent
 namespace_seed
 nstype
 pfn0.0
    align
    devtype
    modalias
    mode
    namespace
    numa_node
    resource

tree /sys/bus/nd/devices/region0/ -L 1
/sys/bus/nd/devices/region0/
 btt0.0
 btt_seed
 dax0.0
 dax_seed
 devtype
 driver -> ../../../../../bus/nd/drivers/nd_region
 init_namespaces
 mappings
 modalias
 namespace0.0
 namespace_seed
 nstype
 pfn0.0
 pfn_seed
 power
 read_only
 size
 subsystem -> ../../../../../bus/nd
 uevent

[Rukhsana Ansari]

The  disappearing namespace issue seems to be related to: section-unaligned ZONE_DEVICE memory ranges (https://lwn.net/Articles/708627/)

as I see a kernel stack trace: 

WARNING: CPU: 4 PID: 315 at kernel/memremap.c:302 devm_memremap_pages+0x3da/0x3f0
[    0.851773] devm_memremap_pages attempted on mixed region [mem 0x100000000-0x1211fffff flags 0x200]

[    0.851790] Call Trace:
[    0.851795]  [<ffffffff8136883f>] dump_stack+0x63/0x84
[    0.851798]  [<ffffffff81082701>] __warn+0xd1/0xf0
[    0.851799]  [<ffffffff8108277f>] warn_slowpath_fmt+0x5f/0x80
[    0.851802]  [<ffffffff8108918a>] ? region_intersects+0x2a/0x100
[    0.851804]  [<ffffffff8119ad2a>] devm_memremap_pages+0x3da/0x3f0
[    0.851807]  [<ffffffffa01423fb>] pmem_attach_disk+0x19b/0x450 [nd_pmem]
[    0.851810]  [<ffffffff811c0042>] ? kfree_const+0x22/0x30
[    0.851812]  [<ffffffff8136afe5>] ? kobject_cleanup+0x75/0x170
[    0.851814]  [<ffffffff8136b115>] ? kobject_put+0x25/0x50
[    0.851816]  [<ffffffff814c8d37>] ? put_device+0x17/0x20
[    0.851818]  [<ffffffffa014272e>] nd_pmem_probe+0x7e/0xa0 [nd_pmem]
[    0.851821]  [<ffffffff814f3547>] nvdimm_bus_probe+0x67/0x110

===

Is the recommendation to move to 4.11.3 kernel?

Thanks,

Rukhsana

[Dan Williams]

On Mon, Jul 31, 2017 at 7:30 PM, Rukhsana Ansari <ruk.a...@gmail.com> wrote:
>
> The disappearing namespace issue seems to be related to: section-unaligned
> ZONE_DEVICE memory ranges (https://lwn.net/Articles/708627/)
> as I see a kernel stack trace:
>
> WARNING: CPU: 4 PID: 315 at kernel/memremap.c:302
> devm_memremap_pages+0x3da/0x3f0
> [ 0.851773] devm_memremap_pages attempted on mixed region [mem
> 0x100000000-0x1211fffff flags 0x200]
> [ 0.851790] Call Trace:
> [ 0.851795] [<ffffffff8136883f>] dump_stack+0x63/0x84
> [ 0.851798] [<ffffffff81082701>] __warn+0xd1/0xf0
> [ 0.851799] [<ffffffff8108277f>] warn_slowpath_fmt+0x5f/0x80
> [ 0.851802] [<ffffffff8108918a>] ? region_intersects+0x2a/0x100
> [ 0.851804] [<ffffffff8119ad2a>] devm_memremap_pages+0x3da/0x3f0
> [ 0.851807] [<ffffffffa01423fb>] pmem_attach_disk+0x19b/0x450 [nd_pmem]
> [ 0.851810] [<ffffffff811c0042>] ? kfree_const+0x22/0x30
> [ 0.851812] [<ffffffff8136afe5>] ? kobject_cleanup+0x75/0x170
> [ 0.851814] [<ffffffff8136b115>] ? kobject_put+0x25/0x50
> [ 0.851816] [<ffffffff814c8d37>] ? put_device+0x17/0x20
> [ 0.851818] [<ffffffffa014272e>] nd_pmem_probe+0x7e/0xa0 [nd_pmem]
> [ 0.851821] [<ffffffff814f3547>] nvdimm_bus_probe+0x67/0x110
> ===
>
> Is the recommendation to move to 4.11.3 kernel?

No, that is not supported in any kernel. You need to make your memmap
range 128MB aligned.

[Rukhsana Ansari]

Thanks Dan.

With a 128M aligned size, namespace detection and conversion to dax mode using ndctl worked fine.

I've tried with 2 options:

 memmap=640M!4G  

ndctl list
{
  "dev":"namespace0.0",
  "mode":"dax",
  "size":658505728,
  "uuid":"84b63087-4473-4cf6-8a70-0e09b4c3dad0"
}

and memmap=4G!4G

{
    "dev":"namespace0.0",
    "mode":"dax",
    "size":4225761280,
    "uuid":"838d39de-7d6d-4f9d-9bde-8d7d509424c5",
    "daxdevs":[
      {
        "chardev":"dax0.0",
        "size":4225761280
      }

I modified simple_copy and full_copy along the lines suggested by Krzysztof  to make it work with a dax device.

This was also detailed in another email thread: "How to specify a file, that is a dax device file".

nvml version used is:  tag version 1.2

kernel version: 4.9.11

-- simple_copy.c (revision 37308)
+++ simple_copy.c (working copy)
@@ -77,8 +77,8 @@
  }
 
  /* create a pmem file and memory map it */
- if ((pmemaddr = pmem_map_file(argv[2], BUF_LEN,
-    PMEM_FILE_CREATE|PMEM_FILE_EXCL,
+ if ((pmemaddr = pmem_map_file(argv[2], 0,
+    PMEM_FILE_CREATE,
     0666, &mapped_len, &is_pmem)) == NULL) {

With this change, the simple_copy/full_copy   programs terminate with a bus error in memmove_nodrain_movnt: _mm_stream_si128(d, xmm0);

gdb output shows:

Program received signal SIGBUS, Bus error.
0x00007ffff7bd5357 in _mm_stream_si128 (__B=..., __A=0x7ffec0000000) at /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/emmintrin.h:1447
1447   __builtin_ia32_movntdq ((__v2di *)__A, (__v2di)__B);

Is there a further modification required to simple_copy/full_copy?

Thanks,

Rukhsana

[Dan Williams]

[ response inline below ]

On Tue, Aug 1, 2017 at 11:32 AM, Rukhsana Ansari <ruk.a...@gmail.com> wrote:
[..]

> -- simple_copy.c (revision 37308)
> +++ simple_copy.c (working copy)
> @@ -77,8 +77,8 @@
> }
>
> /* create a pmem file and memory map it */
> - if ((pmemaddr = pmem_map_file(argv[2], BUF_LEN,
> - PMEM_FILE_CREATE|PMEM_FILE_EXCL,
> + if ((pmemaddr = pmem_map_file(argv[2], 0,
> + PMEM_FILE_CREATE,
> 0666, &mapped_len, &is_pmem)) == NULL) {
>
> With this change, the simple_copy/full_copy programs terminate with a bus
> error in memmove_nodrain_movnt: _mm_stream_si128(d, xmm0);
>
> gdb output shows:
>
> Program received signal SIGBUS, Bus error.
> 0x00007ffff7bd5357 in _mm_stream_si128 (__B=..., __A=0x7ffec0000000) at
> /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/emmintrin.h:1447
> 1447 __builtin_ia32_movntdq ((__v2di *)__A, (__v2di)__B);
>
> Is there a further modification required to simple_copy/full_copy?

Device-dax is very particular about the arguments to mmap. 'addr', if
specified, 'size', 'length', and 'offset' must all be multiples of the
device-dax alignment size which is 2MiB by default. Alignment
violations result in SIGBUS. You can change this alignment to 4K with
the command below if you suspect libpmem is not honoring that
alignment. However, recent versions of libpmem should autodetect this
device-dax alignment constraint by default.

ndctl create-namespace -e namespace0.0 -f -a 4K

[Krzysztof Czurylo]

I don't think this is a problem with device-dax alignment.

Assuming the above are the only changes in simple_copy.c, it should work fine

on device dax, both with 4K and 2M alignment.

At least, it runs w/o any issues on my machine (kernel 4.13.0-rc2).

Rukhsana,

If you run the example from the nvml tree, could you try this:

$ PMEM_LOG_LEVEL=15 LD_LIBRARY_PATH=../../debug ./simple_copy <src_file> /dev/dax0.0

and send me the log?

--

K.

 

[Rukhsana Ansari]

Thanks  Krzysztof, Dan for the  help.

Based on Dan's  feedback, I did run "sudo ndctl create-namespace -e namespace0.0 -f -a 4K"

 and with this change  both simple_copy and full_copy succeeded. So, it does appear to be related with device-dax alignment.

I also switched back to the default 2M via: "sudo ndctl create-namespace -e namespace0.0 -f -a 2M" and  the programs failed, reporting the bus error.

Krzysztof - Where do I find the output log file? I don't see any output on stdout.

Also:

Do all writes  using the pmem_memcpy_persists() api require the device-dax destaddr to be 4K aligned (this is after creating the namespace with 4K alignment)? It doesn't appear to be so based on some experiments I did, but I wanted  to confirm.

--

-Rukhsana

[Rukhsana Ansari]

Is there a recommended kernel version  and pmem library version to use for device dax mode?

I am using  kernel - 4.9.11 and nvml tag 1.2. Is this a supported version for device dax mode?

--

-Rukhsana

[Dan Williams]

The 4.9-stable kernel series is still receiving device-dax fixes, but
you need to move on past 4.9.11 to at least 4.9.25 to pick up all the
known fixes.

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to pmem+uns...@googlegroups.com.

> To post to this group, send email to pm...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pmem/CAPu0trrjUFu3Nr6iao1m-acGVXNtZBUMGWBhAsFdpfZZ-yYPhg%40mail.gmail.com.