What 's the status/plan for RDMA support of pmdk?

[Yong Yang]

I am not sure librpmem is actually updated. When will it be production-ready for RDMA support of pmdk?

Jerry

[Piotr Balcer]

Hi,

librpmem is intended more as an example of combining persistent memory with RDMA than a fully-fledged solution. It's current version is marked as experimental and that's unlikely to ever change.

However, we do have ongoing efforts to create a new generic solution for remote persistent memory access, but I can't speak about the timelines just yet.

For more information please see our article series about this topic:

https://software.intel.com/en-us/articles/persistent-memory-replication-over-traditional-rdma-part-1-understanding-remote-persistent

Piotr

[Yong Yang]

@Piotr, Thanks for reply.

Will it need libfabric support as well speaking of the 'new solution'? 

Jerry

[Piotr Balcer]

Yes, libfabric provides essential functionality that would be difficult to re-implement from scratch.

[Yong Yang]

Can I also run experiments on emulated pmem with librpmem? 

Jerry

[Piotr Balcer]

Yes, that should be fine.

[Yong Yang]

Hi @Piotr,

I run the librpmem example of single matchine test following https://software.intel.com/en-us/articles/persistent-memory-replication-over-traditional-rdma-part-2-remote-persistent-memory-101
It seems not working as expected that the data seems not persistent and flushed into target.

1. rpmem_close() invoke doens't trigger flush on target.

2. explicitly rpmem_flush or rpmem_deep_persistent doesn't work, which reports "waiting for RECV completion failed".

jerry@u19:~$ cc rpmem101.c -o rpmem101 -lrpmem && rm /mnt/mem/* -rf && ./rpmem101 localhost target.poolset && hexdump -C /mnt/mem/myfile.part0

pool: DATA_STARTS_HERE

rpmem_flush: waiting for RECV completion failed

rpmem_deep_persist: waiting for RECV completion failed

rpmem_close: closing fabric access domain: Unknown error -16

jerry@u19:~$ echo $RPMEM_ENABLE_SOCKETS

1
# memory pool consists of three files

100M /mnt/mem/myfile.part0

200M /mnt/mem/myfile.part1

400M /mnt/mem/myfile.part2

jerry@u19:~$ !cat

cat rpmem101.c

#include <assert.h>

#include <unistd.h>

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <librpmem.h>

#define POOL_SIGNATURE          "LIBRPMEM"

#define POOL_SIZE         (32 * 1024 * 1024)

#define NLANES                               4

#define DATA_OFF          4096

#define DATA_SIZE         (POOL_SIZE - DATA_OFF)

#define MAGIC_DATA_HEADER          "DATA_STARTS_HERE"

static void

parse_args(int argc, char *argv[], const char **target, const char **poolset)

{

        if (argc < 3) {

                fprintf(stderr, "usage:\t%s <target> <poolset>\n", argv[0]);

                exit(1);

        }

        *target = argv[1];

        *poolset = argv[2];

}

static void *

alloc_memory()

{

        long pagesize = sysconf(_SC_PAGESIZE);

        if (pagesize < 0) {

                perror("sysconf");

                exit(1);

        }

        /* allocate a page size aligned local memory pool */

        void *mem;

        int ret = posix_memalign(&mem, pagesize, POOL_SIZE);

        if (ret) {

                fprintf(stderr, "posix_memalign: %s\n", strerror(ret));

                exit(1);

        }

        assert(mem != NULL);

        return mem;

}

int main(int argc, char *argv[])

{

        const char *target, *poolset;

        parse_args(argc, argv, &target, &poolset);

        unsigned nlanes = NLANES;

        void *pool = alloc_memory();

        int ret;

        /* fill pool_attributes */

        struct rpmem_pool_attr pool_attr;

        memset(&pool_attr, 0, sizeof(pool_attr));

        strncpy(pool_attr.signature, POOL_SIGNATURE, RPMEM_POOL_HDR_SIG_LEN);

        /* create a remote pool */

        RPMEMpool *rpp = rpmem_create(target, poolset, pool, POOL_SIZE,

                                        &nlanes, &pool_attr);

        if (!rpp) {

                        fprintf(stderr, "rpmem_create: %s\n", rpmem_errormsg());

                        return 1;

        }

        /* store data on local pool */

        memset(pool, 0xee, POOL_SIZE);

        strncpy(pool, MAGIC_DATA_HEADER, strlen(MAGIC_DATA_HEADER) + 1);

        fprintf(stdout, "pool: %s\n", (char*)pool);

        /* make local data persistent on remote node */

        ret = rpmem_persist(rpp, DATA_OFF, DATA_SIZE, 0, 0);// RPMEM_PERSIST_RELAXED

        if (ret) {

                        fprintf(stderr, "rpmem_persist: %s\n", rpmem_errormsg());

                        return 1;

        }

        /* flush. NOT WORKING AT ALL*/

        ret = rpmem_flush(rpp, DATA_OFF, DATA_SIZE, 0, 0);

        if (ret) {

                        fprintf(stderr, "rpmem_flush: %s\n", rpmem_errormsg());

                        //return 1;

        }

        /* deep flush on target. NOT WORKING AT ALL!*/

        ret = rpmem_deep_persist(rpp, DATA_OFF, DATA_SIZE, 0);

        if (ret) {

                        fprintf(stderr, "rpmem_deep_persist: %s\n", rpmem_errormsg());

                        //return 1;

        }

        /* close the remote pool */

        ret = rpmem_close(rpp);

        if (ret) {

                        fprintf(stderr, "rpmem_close: %s\n", rpmem_errormsg());

                        return 1;

        }

        free(pool);

        return 0;

}

hexdump -C /mnt/mem/myfile.part1

00000000  4c 49 42 52 50 4d 45 4d  00 00 00 00 00 00 00 00  |LIBRPMEM........|

00000010  00 00 00 00 00 00 00 00  01 a4 19 66 b1 98 a7 4b  |...........f...K|

00000020  a2 64 dc 1b 70 da cf af  98 ac 58 7e bf a3 2d 46  |.d..p.....X~..-F|

00000030  a8 ca 0a 72 8a 0a 7c a1  3e 04 45 8c 41 c7 a1 4f  |...r..|.>.E.A..O|

00000040  8b 7c 7d fd 1e f1 82 bc  de 16 08 e9 2d bd 6f 4f  |.|}.........-.oO|

00000050  83 a0 71 83 62 eb ff 87  3e 04 45 8c 41 c7 a1 4f  |..q.b...>.E.A..O|

00000060  8b 7c 7d fd 1e f1 82 bc  3e 04 45 8c 41 c7 a1 4f  |.|}.....>.E.A..O|

00000070  8b 7c 7d fd 1e f1 82 bc  9e 91 09 5e 00 00 00 00  |.|}........^....|

00000080  10 73 77 37 f7 07 00 00  02 01 00 00 00 00 3e 00  |.sw7..........>.|

00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*

00000ff0  00 00 00 00 00 00 00 00  f8 50 60 90 31 3d ae b1  |.........P`.1=..|

00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*

0c800000

[Tomasz Gromadzki]

Hi @Jerry,

you could try to get more diagnostic information using following setup:

$ export RPMEM_LOG_FILE=/dev/shm/librpmem.log

$ export RPMEM_LOG_LEVEL=4

...

and later see results in:

$ cat /dev/shm/librpmem.log

Tomasz

--

You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/4510799f-4fad-42bb-9161-a030e7f306ac%40googlegroups.com.

[Yong Yang]

I got below :

<librpmem>: <1> [out.c:236 out_init] pid 13147: program: /home/jerry/rpmem101

<librpmem>: <1> [out.c:238 out_init] librpmem version 1.3

<librpmem>: <1> [out.c:242 out_init] src version: 1.7+git10.g8c533ff77

<librpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck

<librpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind

<librpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck

<librpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd

<librpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state

<librpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+

<librpmem>: <3> [librpmem.c:61 librpmem_init]

<librpmem>: <3> [rpmem.c:459 rpmem_create] target localhost, pool_set_name target.poolset, pool_addr 0x7fd3f50cc000, pool_size 33554432, nlanes 0x7ffc17aa2ff8, create_attr 0x7ffc17aa3020

<librpmem>: <3> [rpmem.c:366 rpmem_log_args] req create, target localhost, pool_set_name target.poolset, pool_addr 0x7fd3f50cc000, pool_size 33554432, nlanes 4

<librpmem>: <3> [rpmem.c:368 rpmem_log_args] create request:

<librpmem>: <3> [rpmem.c:369 rpmem_log_args]    target: localhost

<librpmem>: <3> [rpmem.c:370 rpmem_log_args]    pool set: target.poolset

<librpmem>: <4> [rpmem.c:371 rpmem_log_args]    pool addr: 0x7fd3f50cc000

<librpmem>: <4> [rpmem.c:372 rpmem_log_args]    pool size: 33554432

<librpmem>: <3> [rpmem.c:373 rpmem_log_args]    nlanes: 4

<librpmem>: <3> [rpmem.c:399 rpmem_check_args] pool_addr 0x7fd3f50cc000, pool_size 33554432, nlanes 0x7ffc17aa2ff8

<librpmem>: <3> [rpmem.c:200 rpmem_common_init] target localhost

<librpmem>: <3> [rpmem.c:131 rpmem_set_provider] rpp 0x560045d2d410, node localhost

<librpmem>: <3> [rpmem.c:102 env_get_bool] name RPMEM_ENABLE_SOCKETS, valp 0x7ffc17aa2eec

<librpmem>: <3> [rpmem.c:223 rpmem_common_init] provider: sockets

<librpmem>: <3> [rpmem.c:227 rpmem_common_init] forcing using IPv4

<librpmem>: <4> [rpmem.c:237 rpmem_common_init] establishing out-of-band connection

<librpmem>: <4> [rpmem_cmd.c:148 rpmem_cmd_log] executing command 'ssh -T -4 -oBatchMode=yes localhost rpmemd'

<librpmem>: <4> [rpmem_ssh.c:320 rpmem_ssh_open] received status: 0

<librpmem>: <3> [rpmem.c:245 rpmem_common_init] out-of-band connection established

<librpmem>: <4> [rpmem_obc.c:497 rpmem_obc_create] sending create request message

<librpmem>: <3> [rpmem_obc.c:505 rpmem_obc_create] create request message sent

<librpmem>: <4> [rpmem_obc.c:506 rpmem_obc_create] receiving create request response

<librpmem>: <3> [rpmem_obc.c:515 rpmem_obc_create] create request response received

<librpmem>: <3> [rpmem.c:382 rpmem_log_resp] req create, resp 0x7ffc17aa2f50

<librpmem>: <3> [rpmem.c:384 rpmem_log_resp] create request response:

<librpmem>: <3> [rpmem.c:385 rpmem_log_resp]    nlanes: 4

<librpmem>: <3> [rpmem.c:386 rpmem_log_resp]    port: 44169

<librpmem>: <3> [rpmem.c:388 rpmem_log_resp]    persist method: General Purpose Server Persistency Method

<librpmem>: <3> [rpmem.c:389 rpmem_log_resp]    remote addr: 0x7f7c79a00000

<librpmem>: <3> [rpmem.c:292 rpmem_common_fip_init] rpp 0x560045d2d410, req 0x7ffc17aa2f70, resp 0x7ffc17aa2f50, pool_addr 0x7fd3f50cc000, pool_size 33554432, nlanes 0x7ffc17aa2ff8

<librpmem>: <4> [rpmem_fip_common.c:334 rpmem_fip_print_info] libfabric version: 1.6.2

<librpmem>: <4> [rpmem_fip_common.c:344 rpmem_fip_print_info] libfabric interface info:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info] fi_info:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     caps: [ FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE, FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     mode: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     addr_format: FI_SOCKADDR_IN

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     src_addrlen: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     dest_addrlen: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     src_addr: fi_sockaddr_in://127.0.0.1:0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     dest_addr: fi_sockaddr_in://127.0.0.1:44169

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     handle: (null)

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     fi_tx_attr:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         caps: [ FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE, FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mode: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         op_flags: [ FI_TRANSMIT_COMPLETE ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         comp_order: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         inject_size: 255

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         size: 376

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         iov_limit: 8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         rma_iov_limit: 8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     fi_rx_attr:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         caps: [ FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE, FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mode: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         op_flags: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         comp_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT, FI_ORDER_DATA ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         total_buffered_recv: 67108864

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         size: 376

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         iov_limit: 8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     fi_ep_attr:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         type: FI_EP_MSG

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         protocol: FI_PROTO_SOCK_TCP

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         protocol_version: 1

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_msg_size: -4097

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         msg_prefix_size: 0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_order_raw_size: -4097

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_order_war_size: -4097

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_order_waw_size: -4097

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mem_tag_format: 0xaaaaaaaaaaaaaaaa

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         tx_ctx_cnt: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         rx_ctx_cnt: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         auth_key_size: 0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     fi_domain_attr:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         domain: 0x0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         name: lo

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         threading: FI_THREAD_SAFE

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         control_progress: FI_PROGRESS_AUTO

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         data_progress: FI_PROGRESS_AUTO

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         resource_mgmt: FI_RM_ENABLED

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         av_type: FI_AV_UNSPEC

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mr_mode: [ FI_MR_BASIC ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mr_key_size: 8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         cq_data_size: 8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         cq_cnt: 32

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         ep_cnt: 128

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         tx_ctx_cnt: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         rx_ctx_cnt: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_ep_tx_ctx: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_ep_rx_ctx: 16

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_ep_stx_ctx: 128

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_ep_srx_ctx: 128

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         cntr_cnt: 128

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mr_iov_limit: 8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     caps: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     mode: [  ]

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         auth_key_size: 0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         max_err_data: 0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         mr_cnt: 0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]     fi_fabric_attr:

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         name: 127.0.0.0/8

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         prov_name: sockets

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         prov_version: 2.0

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]         api_version: 1.4

<librpmem>: <4> [rpmem_fip_common.c:355 rpmem_fip_print_info]

<librpmem>: <3> [rpmem.c:323 rpmem_common_fip_init] final nlanes: 4

<librpmem>: <4> [rpmem.c:324 rpmem_common_fip_init] establishing in-band connection

<librpmem>: <3> [rpmem.c:332 rpmem_common_fip_init] in-band connection established

<librpmem>: <3> [rpmem.c:181 rpmem_monitor_thread] arg 0x560045d2d410

<librpmem>: <3> [rpmem.c:722 rpmem_persist] rpp 0x560045d2d410, offset 4096, length 33550336, lane 0, flags 0x0

<librpmem>: <3> [rpmem.c:636 rpmem_flush] rpp 0x560045d2d410, offset 4096, length 33550336, lane 0, flags 0x0

<librpmem>: <1> [rpmem_fip.c:681 rpmem_fip_monitor_thread] event queue got FI_SHUTDOWN

<librpmem>: <1> [rpmem_fip.c:360 rpmem_fip_lane_wait] error reading from completion queue: cannot read error from event queue: Unknown error -11

<librpmem>: <1> [rpmem_fip.c:1261 rpmem_fip_persist_saw] waiting for RECV completion failed

<librpmem>: <1> [rpmem_fip.c:1786 rpmem_fip_flush] flush operation failed

<librpmem>: <2> [rpmem.c:666 rpmem_flush] flush operation failed

<librpmem>: <3> [rpmem.c:602 rpmem_close] rpp 0x560045d2d410

<librpmem>: <4> [rpmem.c:604 rpmem_close] closing out-of-band connection

<librpmem>: <4> [rpmem_obc.c:681 rpmem_obc_close] sending close request message

<librpmem>: <3> [rpmem_obc.c:689 rpmem_obc_close] close request message sent

<librpmem>: <4> [rpmem_obc.c:690 rpmem_obc_close] receiving close request response

<librpmem>: <1> [rpmem_obc.c:695 rpmem_obc_close] receiving close request response failed: Connection reset by peer

<librpmem>: <1> [rpmem.c:612 rpmem_close] close request failed: Connection reset by peer

<librpmem>: <3> [rpmem.c:614 rpmem_close] out-of-band connection closed

<librpmem>: <3> [rpmem.c:348 rpmem_common_fip_fini] rpp 0x560045d2d410

<librpmem>: <4> [rpmem.c:350 rpmem_common_fip_fini] closing in-band connection

<librpmem>: <1> [rpmem_fip.c:479 rpmem_fip_fini_fabric_res] closing fabric access domain: Unknown error -16

<librpmem>: <3> [rpmem.c:354 rpmem_common_fip_fini] in-band connection closed

<librpmem>: <3> [rpmem.c:265 rpmem_common_fini] rpp 0x560045d2d410, join 1

<librpmem>: <3> [librpmem.c:77 librpmem_fini]

On Monday, December 30, 2019 at 6:05:53 PM UTC+8, Tomasz Gromadzki wrote:

Hi @Jerry,

you could try to get more diagnostic information using following setup:

$ export RPMEM_LOG_FILE=/dev/shm/librpmem.log

$ export RPMEM_LOG_LEVEL=4

...

and later see results in:

$ cat /dev/shm/librpmem.log

Tomasz

To unsubscribe from this group and stop receiving emails from it, send an email to pm...@googlegroups.com.

[Janek Mi]

Hi Jerry

It seems the connection is closed by rpmemd just after its establishing so all other rpmem_* operations do not reach the target.

<librpmem>: <1> [rpmem_fip.c:681 rpmem_fip_monitor_thread] event queue got FI_SHUTDOWN

In this case, you should collect a rpmemd log by adding two lines to rpmemd config file (e.g. /home/user/.rpmemd.conf):

log-file = /dev/shm/rpmemd.log
log-level = debug

For details please see rpmemd.1.

Regards

Jan

[Yong Yang]

Thanks Janek.  But it doesn't show error from rpmemd.log, as below :

jerry@u19:~$ cc rpmem101.c -o rpmem101 -lrpmem && rm /mnt/mem/* -rf && ./rpmem101 localhost target.poolset && hexdump -C /mnt/mem/myfile.part0

pool: DATA_STARTS_HERE

rpmem_flush: waiting for RECV completion failed

rpmem_close: closing fabric access domain: Unknown error -16

jerry@u19:~$ cat /dev/shm/rpmemd.log

rpmemd version 1.7+git10.g8c533ff77

ssh connection: 127.0.0.1 45548 127.0.0.1 22

user: jerry

configuration

    pool set directory: '/home/jerry'

    persist method: General Purpose Server Persistency Method

    number of threads: 0

create request:

    pool descriptor: 'target.poolset'

    pool size: 33554432

    nlanes: 4

    provider: sockets

    buff_size: 8192

pool attributes:

    signature: 'LIBRPMEM'

    major: 0

    compat_features: 0x0

    incompat_features: 0x0

    ro_compat_features: 0x0

    poolset_uuid: 00000000-0000-0000-0000-000000000000

    uuid: 00000000-0000-0000-0000-000000000000

    next_uuid: 00000000-0000-0000-0000-000000000000

    prev_uuid: 00000000-0000-0000-0000-000000000000

persistency policy:

    persist method: General Purpose Server Persistency Method

    persist flush: pmem_persist

libfabric version: 1.6.2

libfabric interface info:

fi_info:

    caps: [ FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE, FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM ]

    mode: [  ]

    addr_format: FI_SOCKADDR_IN

    src_addrlen: 16

    dest_addrlen: 0

    src_addr: fi_sockaddr_in://127.0.0.1:0

    dest_addr: (null)

    handle: (null)

    fi_tx_attr:

        caps: [ FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE, FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM ]

        mode: [  ]

        op_flags: [ FI_TRANSMIT_COMPLETE ]

        msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT ]

        comp_order: [  ]

        inject_size: 255

        size: 376

        iov_limit: 8

        rma_iov_limit: 8

    fi_rx_attr:

        caps: [ FI_RMA_PMEM, FI_SHARED_AV, FI_RMA_EVENT, FI_SOURCE, FI_MSG, FI_RMA, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_RMA_PMEM ]

        mode: [  ]

        op_flags: [  ]

        msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT ]

        comp_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT, FI_ORDER_DATA ]

        total_buffered_recv: 67108864

        size: 376

        iov_limit: 8

    fi_ep_attr:

        type: FI_EP_MSG

        protocol: FI_PROTO_SOCK_TCP

        protocol_version: 1

        max_msg_size: -4097

        msg_prefix_size: 0

        max_order_raw_size: -4097

        max_order_war_size: -4097

        max_order_waw_size: -4097

        mem_tag_format: 0xaaaaaaaaaaaaaaaa

        tx_ctx_cnt: 16

        rx_ctx_cnt: 16

        auth_key_size: 0

    fi_domain_attr:

        domain: 0x0

        name: lo

        threading: FI_THREAD_SAFE

        control_progress: FI_PROGRESS_AUTO

        data_progress: FI_PROGRESS_AUTO

        resource_mgmt: FI_RM_ENABLED

        av_type: FI_AV_UNSPEC

        mr_mode: [ FI_MR_BASIC ]

        mr_key_size: 8

        cq_data_size: 8

        cq_cnt: 32

        ep_cnt: 128

        tx_ctx_cnt: 16

        rx_ctx_cnt: 16

        max_ep_tx_ctx: 16

        max_ep_rx_ctx: 16

        max_ep_stx_ctx: 128

        max_ep_srx_ctx: 128

        cntr_cnt: 128

        mr_iov_limit: 8

    caps: [  ]

    mode: [  ]

        auth_key_size: 0

        max_err_data: 0

        mr_cnt: 0

    fi_fabric_attr:

        name: 127.0.0.0/8

        prov_name: sockets

        prov_version: 2.0

        api_version: 1.4

create request response: (status = 0)

    port: 44685

    rkey: 0x1

    raddr: 0x7fa253800000

    nlanes: 4

    persist method: General Purpose Server Persistency Method

waiting for in-band connection

in-band connection established

[Janek Mi]

Hi Jerry

It is a very weird issue. We are doing our best to find a root cause.

Can you please send your distribution and kernel version?

Best regards

Jan

[Yong Yang]

kernal is 5.0

(Linux u19 5.0.0-37-generic #40-Ubuntu SMP Thu Nov 14 00:14:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux)

and gcc version is 8.3.0

[Yong Yang]

The problem looks related to rpmem_flush  while rpmem_persist seems working.

Besides that, I notices below issue as well, which is the data is not persisted by strncpy/strcpy copy,  although memcpy works.

<br

[Janek Mi]

Hi Jerry

Yesterday Tomasz and I have found a root cause of your issue.

I have submitted a pull request fixing it: https://github.com/pmem/pmdk/pull/4424

You can use a branch from this pull request to verify if it works for you: https://github.com/janekmi/pmdk/tree/rpmem-gpspm-flush-fix

Please let me know in case you will have any further issues with this workload.