rpmem basic example doesn't work

Robert Gong

unread,

Jun 10, 2019, 5:33:57 AM6/10/19

to pmem

Hi all,

Can someone give some clue for this issue? I plan to use PMoF, so first thing I did try the librpmem example code -- basic.

my environment:

the pmdk version : 1.6

the libfabric version :

fi_info: 1.7.2a

libfabric: 1.7.2a

libfabric api: 1.7

the client and target server installed with the NIC: Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

and both server installed the SuSE OS : cat /etc/os-release

NAME="SLES"

VERSION="12-SP4"

VERSION_ID="12.4"

PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"

ID="sles"

ANSI_COLOR="0;32"

CPE_NAME="cpe:/o:suse:sles:12:sp4"

pool set file in the target server:

cat /home/admin/poolsets/pool.set

PMEMPOOLSET

64M /mnt/nvdimm/file1.dat

1. run the basic program in the client :

./basic create ro...@192.168.219.7 pool.set

2. the client log:

<librpmem>: <1> [out.c:234 out_init] pid 128578: program: /tmp/nvdimm_build/pmdk-1.6/src/examples/librpmem/basic

<librpmem>: <1> [out.c:236 out_init] librpmem version 1.2

<librpmem>: <1> [out.c:240 out_init] src version: 1.6

<librpmem>: <1> [out.c:248 out_init] compiled with support for Valgrind pmemcheck

<librpmem>: <1> [out.c:253 out_init] compiled with support for Valgrind helgrind

<librpmem>: <1> [out.c:258 out_init] compiled with support for Valgrind memcheck

<librpmem>: <1> [out.c:263 out_init] compiled with support for Valgrind drd

<librpmem>: <3> [librpmem.c:61 librpmem_init]

<librpmem>: <3> [librpmem.c:67 librpmem_init] Libfabric is fork safe

<librpmem>: <3> [rpmem.c:454 rpmem_create] target ro...@192.168.219.7, pool_set_name pool.set, pool_addr 0x7f37d446d000, pool_size 33554432, nlanes 0x7ffcae4ee6f4, create_attr 0x7ffcae4ee680

<librpmem>: <3> [rpmem.c:361 rpmem_log_args] req create, target ro...@192.168.219.7, pool_set_name pool.set, pool_addr 0x7f37d446d000, pool_size 33554432, nlanes 64

<librpmem>: <3> [rpmem.c:363 rpmem_log_args] create request:

<librpmem>: <3> [rpmem.c:364 rpmem_log_args] target: ro...@192.168.219.7

<librpmem>: <3> [rpmem.c:365 rpmem_log_args] pool set: pool.set

<librpmem>: <4> [rpmem.c:366 rpmem_log_args] pool addr: 0x7f37d446d000

<librpmem>: <4> [rpmem.c:367 rpmem_log_args] pool size: 33554432

<librpmem>: <3> [rpmem.c:368 rpmem_log_args] nlanes: 64

<librpmem>: <3> [rpmem.c:394 rpmem_check_args] pool_addr 0x7f37d446d000, pool_size 33554432, nlanes 0x7ffcae4ee6f4

<librpmem>: <3> [rpmem.c:196 rpmem_common_init] target ro...@192.168.219.7

<librpmem>: <3> [rpmem.c:128 rpmem_get_provider] node 192.168.219.7

<librpmem>: <3> [rpmem.c:99 env_get_bool] name RPMEM_ENABLE_VERBS, valp 0x7ffcae4ee4d8

<librpmem>: <3> [rpmem.c:219 rpmem_common_init] provider: verbs

<librpmem>: <4> [rpmem.c:233 rpmem_common_init] establishing out-of-band connection

<librpmem>: <4> [rpmem_cmd.c:147 rpmem_cmd_log] executing command 'ssh -T -oBatchMode=yes ro...@192.168.219.7 rpmemd'

<librpmem>: <4> [rpmem_ssh.c:320 rpmem_ssh_open] received status: 0

<librpmem>: <3> [rpmem.c:241 rpmem_common_init] out-of-band connection established

<librpmem>: <4> [rpmem_obc.c:497 rpmem_obc_create] sending create request message

<librpmem>: <3> [rpmem_obc.c:505 rpmem_obc_create] create request message sent

<librpmem>: <4> [rpmem_obc.c:506 rpmem_obc_create] receiving create request response

<librpmem>: <3> [rpmem_obc.c:515 rpmem_obc_create] create request response received

<librpmem>: <1> [rpmem_obc.c:253 rpmem_obc_check_hdr_resp] Fatal error

<librpmem>: <1> [rpmem.c:480 rpmem_create] create request failed: Remote I/O error

<librpmem>: <3> [rpmem.c:261 rpmem_common_fini] rpp 0xa9faa0, join 0

<librpmem>: <3> [librpmem.c:79 librpmem_fini]

3. target server log:

rpmemd version 1.6

ssh connection: 192.168.219.8 58974 192.168.219.7 22

user: root

configuration

pool set directory: '/home/admin/poolsets/'

persist method: Appliance Persistency Method

number of threads: 0

create request:

pool descriptor: 'pool.set'

pool size: 33554432

nlanes: 64

provider: verbs

buff_size: 8192

pool attributes:

signature: 'EXAMPLE'

major: 1

compat_features: 0x0

incompat_features: 0x0

ro_compat_features: 0x0

poolset_uuid: 01010101-0101-0101-0101-010101010101

uuid: 02020202-0202-0202-0202-020202020202

next_uuid: 03030303-0303-0303-0303-030303030303

prev_uuid: 04040404-0404-0404-0404-040404040404

persistency policy:

persist method: Appliance Persistency Method

persist flush: none

libfabric version: 1.7.2a

libfabric interface info:

fi_info:

caps: [ FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_LOCAL_COMM, FI_REMOTE_COMM ]

mode: [ FI_RX_CQ_DATA, FI_LOCAL_MR ]

addr_format: FI_SOCKADDR_IN

src_addrlen: 16

dest_addrlen: 0

src_addr: fi_sockaddr_in://192.168.219.7:0

dest_addr: (null)

handle: (nil)

fi_tx_attr:

caps: [ FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_LOCAL_COMM, FI_REMOTE_COMM ]

mode: [ ]

op_flags: [ ]

msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS ]

comp_order: [ FI_ORDER_STRICT ]

inject_size: 256

size: 384

iov_limit: 4

rma_iov_limit: 1

fi_rx_attr:

caps: [ FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_LOCAL_COMM, FI_REMOTE_COMM ]

mode: [ FI_RX_CQ_DATA ]

op_flags: [ ]

msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS ]

comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]

total_buffered_recv: 0

size: 384

iov_limit: 4

fi_ep_attr:

type: FI_EP_MSG

protocol: FI_PROTO_RDMA_CM_IB_RC

protocol_version: 1

max_msg_size: 1073741824

msg_prefix_size: 0

max_order_raw_size: 1073741824

max_order_war_size: 0

max_order_waw_size: 1073741824

mem_tag_format: 0x0000000000000000

tx_ctx_cnt: 1

rx_ctx_cnt: 1

auth_key_size: 0

fi_domain_attr:

domain: 0x0

name: mlx5_bond_0

threading: FI_THREAD_SAFE

control_progress: FI_PROGRESS_AUTO

data_progress: FI_PROGRESS_AUTO

resource_mgmt: FI_RM_ENABLED

av_type: FI_AV_UNSPEC

mr_mode: [ FI_MR_BASIC ]

mr_key_size: 4

cq_data_size: 4

cq_cnt: 16777216

ep_cnt: 262144

tx_ctx_cnt: 1024

rx_ctx_cnt: 1024

max_ep_tx_ctx: 1024

max_ep_rx_ctx: 1024

max_ep_stx_ctx: 0

max_ep_srx_ctx: 8388608

cntr_cnt: 0

mr_iov_limit: 1

caps: [ FI_LOCAL_COMM, FI_REMOTE_COMM ]

mode: [ ]

auth_key_size: 0

max_err_data: 255

mr_cnt: 16777216

fi_fabric_attr:

name: IB-0xfe80000000000000

prov_name: verbs

prov_version: 1.0

api_version: 1.4

nic_fid: (nil)

registering memory: Unknown error -95

4. the failure code :

It seems there is issue when registering memory region:

/*

* rpmemd_fip_init_memory -- initialize memory pool's resources

*/

static int

rpmemd_fip_init_memory(struct rpmemd_fip *fip)

{

int ret;

/*

* Register memory region with appropriate access bits:

* - FI_REMOTE_READ - remote peer can issue READ operation,

* - FI_REMOTE_WRITE - remote peer can issue WRITE operation,

*/

ret = fi_mr_reg(fip->domain, fip->addr, fip->size,

FI_REMOTE_READ | FI_REMOTE_WRITE, 0, 0, 0,

&fip->mr, NULL);

if (ret) {

RPMEMD_FI_ERR(ret, "registering memory");

return -1;

}

return 0;

}

Andy Rudoff

unread,

Jun 10, 2019, 10:42:19 AM6/10/19

to pmem

Hi Robert,

The long-lived memory registrations required for RDMA are currently not supported on Linux file system DAX (hence the EOPNOTSUPP error you're getting back). There are proposed changes to Linux to fix this, currently under review by the Linux community, but it may be a while before those are accepted and propagate into your favorite Linux distro.

In the meantime, the workaround is to use device DAX instead of file system DAX. Start by using ndctl to change the namespace type to devdax. Then, in your poolset file, use the AUTO keyword that tells the library to determine the size automatically, like this:

PMEMPOOLSET

AUTO /dev/daxX.Y

where /dev/daxX.Y is the name of the device DAX device you created. Note that if you aren't running as root, you'll need to set the permissions on /dev/daxX.Y to allow you to read/write to it.

We're also in the process of writing up all the steps for using rpmem in more detail. Hopefully we'll post that document pretty soon.

-andy

Wayne Gao

unread,

Jun 10, 2019, 9:37:58 PM6/10/19

to pmem

buddy, I am colleague of Robert.

do we have any work around that for us the play the librpmem? we are only to get some performance data.

is SPDK can work around this kernel file system dax issue?

Andy Rudoff

unread,

Jun 10, 2019, 9:45:43 PM6/10/19

to pmem

Hi Wayne,

The workaround is in the reply you quoted below. Device DAX will work with librpmem and you can get whatever performance data you want that way.

-andy

Robert Gong

unread,

Jun 12, 2019, 4:50:02 AM6/12/19

to pmem

Hi Andy,

I did try your workaround:

In the target server:

1) ndctl info after creating the namespace with the commmand: ndctl create-namespace --type=pmem --mode=devdax

admin@lehi-dirt:~> ndctl list -BiRN

{

"provider":"ACPI.NFIT",

"dev":"ndbus0",

"scrub_state":"idle",

"regions":[

{

"dev":"region0",

"size":17179869184,

"available_size":0,

"type":"pmem",

"numa_node":0,

"iset_id":7622173260169589529,

"persistence_domain":"unknown",

"namespaces":[

{

"dev":"namespace0.0",

"mode":"devdax",

"map":"dev",

"size":16909336576,

"uuid":"8166f8ee-6056-49d8-bbd5-ef02813f9098",

"raw_uuid":"1cc0d826-762a-4369-8c7b-0bc2bbfa71d5",

"chardev":"dax0.0",

"numa_node":0

},

{

"dev":"namespace0.1",

"mode":"raw",

"size":0,

"uuid":"00000000-0000-0000-0000-000000000000",

"sector_size":512,

"state":"disabled",

"numa_node":0

}

]

}

]

}

poolset file:

lehi-dirt:~ # cat /home/admin/poolsets/pool.set

PMEMPOOLSET

AUTO /dev/dax0.0

After running the basic program in the client side:

./basic create ro...@192.168.219.7 pool.set

the client side log:

<librpmem>: <1> [out.c:234 out_init] pid 121595: program: /tmp/nvdimm_build/pmdk-1.6/src/examples/librpmem/basic

<librpmem>: <1> [out.c:236 out_init] librpmem version 1.2

<librpmem>: <1> [out.c:240 out_init] src version: 1.6

<librpmem>: <1> [out.c:248 out_init] compiled with support for Valgrind pmemcheck

<librpmem>: <1> [out.c:253 out_init] compiled with support for Valgrind helgrind

<librpmem>: <1> [out.c:258 out_init] compiled with support for Valgrind memcheck

<librpmem>: <1> [out.c:263 out_init] compiled with support for Valgrind drd

<librpmem>: <3> [librpmem.c:61 librpmem_init]

<librpmem>: <3> [librpmem.c:67 librpmem_init] Libfabric is fork safe

<librpmem>: <3> [rpmem.c:454 rpmem_create] target ro...@192.168.219.7, pool_set_name pool.set, pool_addr 0x7f60dccd0000, pool_size 33554432, nlanes 0x7fffc7935fc4, create_attr 0x7fffc7935f50

<librpmem>: <3> [rpmem.c:361 rpmem_log_args] req create, target ro...@192.168.219.7, pool_set_name pool.set, pool_addr 0x7f60dccd0000, pool_size 33554432, nlanes 64

<librpmem>: <3> [rpmem.c:363 rpmem_log_args] create request:

<librpmem>: <3> [rpmem.c:364 rpmem_log_args] target: ro...@192.168.219.7

<librpmem>: <3> [rpmem.c:365 rpmem_log_args] pool set: pool.set

<librpmem>: <4> [rpmem.c:366 rpmem_log_args] pool addr: 0x7f60dccd0000

<librpmem>: <4> [rpmem.c:367 rpmem_log_args] pool size: 33554432

<librpmem>: <3> [rpmem.c:368 rpmem_log_args] nlanes: 64

<librpmem>: <3> [rpmem.c:394 rpmem_check_args] pool_addr 0x7f60dccd0000, pool_size 33554432, nlanes 0x7fffc7935fc4

<librpmem>: <3> [rpmem.c:196 rpmem_common_init] target ro...@192.168.219.7

<librpmem>: <3> [rpmem.c:128 rpmem_get_provider] node 192.168.219.7

<librpmem>: <3> [rpmem.c:99 env_get_bool] name RPMEM_ENABLE_VERBS, valp 0x7fffc7935da8