pmem_map_file report "cannot find a contiguous region of given size"

[chris....@gmail.com]

Hi experts of fsdax,

I have written code to implement a file cache with fsdax pmem. In the code, it would create and mmap files and use the space to load/store.  When the free space of the pmem is less than 15%, some least used files would be unmapped and removed to make rooms for new files. The files are in fixed size 4M+256 Bytes and the pmem is 500GB.

Then after a long time's test, I got mmap error from pmem_map_file in libpmem reports "cannot find a contiguous region of given size." 

 

I have two questions:

1. According to my understanding, when using fsdax, there should be no problem in creating files as inode would gather the scattered blocks together. So this problem should be problem reported in mmap means it can not find contiguous virtual memory for the file. Am I right? 

2. Since the files are in fixed size and the files are unmapped before they are removed, the removed files' virtual address could always be reused to mmap the new files. So in this case, there should be no problem in finding the contiguous region for the new files. Why would it still report this?

Anyone who understands this issue, please try to help. Thanks a lot.

Regards,

Chris

[Andy Rudoff]

Hi Chris,

If I'm understanding your description correctly then yes, it sounds like mmap is not finding virtual address space sufficient for the mapping.  I haven't seen this issue, but it might help if you show us the pmem_map_file() call you're using (i.e. the args you're passing).  Also, when your program encounters the error, before exiting, you could open "/proc/self/maps" and dump the contents to a file.  That would give us a look at how the virtual address space is being used.

One last question, is your program multi-threaded?  Is it possible you have multiple pmem_map_file() calls in-flight concurrently?

Thanks,

-andy

[chris....@gmail.com]

Hi Andy, 

Thanks for the reply.

The calling of pmem_map_file is pretty simple as follows:

1236     if ((pmemaddr = pmem_map_file(file_name, file_len, PMEM_FILE_CREATE,

1237         0666, &mapped_len, &is_pmem)) == NULL) {

1238         printf("pmem map error and msg %s.\n",pmem_errormsg());

1239         assert(0);

1240         return -4;

1241     }

Right now I am trying to reproduce it and then dump the "proc/self/maps" to a file.

Yes, the program is multi-threaded. And there might be multiple calling of pmem_map_file in-flight concurrently. Is it OK to use like that? or do I need to add a lock on the pmem_map_file?

Thanks,

Chris

在 2018年12月17日星期一 UTC+8上午12:27:05，Andy Rudoff写道：

[Andy Rudoff]

Hi Chris,

Yes, that could be the issue: pmem_map_file() will create the file (the way you call it), then look for an appropriate gap in the virtual address space, then call mmap() to map it there.  If you've got concurrent operations in-flight, one thread could find the gap and then have it taken up by another one.

I'm not 100% sure this is the issue you're seeing, but I would serialize the calls to pmem_map_file() with a lock.

I will also look into updating our docs to make it more clear which APIs are MT-safe and which ones are not.

Thanks,

-andy

[Marcin Ślusarz]

Hey guys,

Yes, the internal mapping algorithm used by pmem_map_file is not
thread-safe. However for this API the only thing affected by this bug
is performance - if there's a race between two threads (one calling
pmem_map_file and another calling regular mmap) the mmap from
pmem_map_file may ignore the hint address and create unaligned
mapping, which means huge pages won't be used.

Chris, I suspect the error you are seeing comes from your application
either not unmapping at all or calling pmem_unmap with wrong length.
If you are sure it's not true, then I think we'll need a reproducer.

Fixing the mapping algorithm bug for Linux should be relatively easy
(just do not unmap temporary VM space and then map with MAP_FIXED),
but our Windows implementation of mmap doesn't seem to handle
MAP_FIXED correctly, so it will require more work.
At the beginning I said that this bug doesn't affect correctness of
pmem_map_file, but it affects pmemobj correctness in one case -
allocations with alignment (introduced in 1.5) between 8kB and 2MB
(the max value) may not actually be aligned if pool opening raced with
another thread mapping something and OS decided to use non-2MB aligned
address. If application relies on this (for example by assuming lower
bits of PMEMoid are 0), it may lead to corruption. We'll have to fix
this somehow. I'll open bug report tomorrow if nobody will find any
hole in this theory.

Marcin

> --
> You received this message because you are subscribed to the Google Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c4759576-1191-499d-a7a8-047f83292969%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Chris Yu]

Ok, thanks for the info. The issue only happed once and I cannot reproduce it. It might be some problem in my management data structures. 

Anyway, thanks for the information that the pmem_map_file is not thread safe as I also checked the function my self too. I would add lock on it.

Marcin Ślusarz <marcin....@gmail.com> 于2018年12月18日周二 上午3:52写道：