Linux/C/mmap(): Avoiding the flushing of memory mapped region created using mmap to the underlying file

[Pavankumar S V]

Hello,
I am using mmap() to map device file "mtdblock0" into virtual address space of one process1. Some data whose value changes continuously for some time are stored in memory mapped region as well. I want to prevent these data changes to be flushed from memory mapped region to the device file for some time until the calculations are over(so that process2 which is mapping to the same file does not see these changes). Note: The same device file is mapped by another process also to read the data.

Assuming that mmap() syncing works for the device files in the same way as for regular files, I tried setting "dirty_writeback_centisecs" to 0 to disable the kernel threads that perform the flushing. Then I'm setting "dirty_writeback_centisecs" to default value after some time to again enable the flushing. This is working partially. But the data is still getting flushed to the device file occassionally even when "dirty_writeback_centisecs" is 0. Please help me to know why this is not working.

Does the mmap() syncing works for the device files in the same way as for regular files?

Are the same set of threads that are responsible for flushing the page cache are also responsible for flushing the memory mapped region to the underlying file?

Files mapped using mmap() are directly mapped from the disk to the virtual address space of process. They don't use intermediate page cache. So any flushing done through msync() happens directly between memory map and disk. Is my understanding right?

If this method is not possible, are there any other ways of achieving this?(I have tried mysnc() with INVALIDATE and it does not work)

Thanks in Advance

[Richard Kettlewell]

Pavankumar S V <pavanku...@gmail.com> writes:
> I am using mmap() to map device file "mtdblock0" into
> virtual address space of one process1. Some data whose value changes
> continuously for some time are stored in memory mapped region as
> well. I want to prevent these data changes to be flushed from memory
> mapped region to the device file for some time until the calculations
> are over(so that process2 which is mapping to the same file does not
> see these changes). Note: The same device file is mapped by another
> process also to read the data.
>
> Assuming that mmap() syncing works for the device files in the same
> way as for regular files, I tried setting "dirty_writeback_centisecs"
> to 0 to disable the kernel threads that perform the flushing. Then I'm
> setting "dirty_writeback_centisecs" to default value after some time
> to again enable the flushing. This is working partially. But the data
> is still getting flushed to the device file occassionally even when
> "dirty_writeback_centisecs" is 0. Please help me to know why this is
> not working.

The kernel will also flush if there is more than a certain number of
modified pages.

> Does the mmap() syncing works for the device files in the same way as
> for regular files?
>
> Are the same set of threads that are responsible for flushing the page
> cache are also responsible for flushing the memory mapped region to
> the underlying file?
>
> Files mapped using mmap() are directly mapped from the disk to the
> virtual address space of process. They don't use intermediate page
> cache. So any flushing done through msync() happens directly between
> memory map and disk. Is my understanding right?
>
> If this method is not possible, are there any other ways of achieving
> this?(I have tried mysnc() with INVALIDATE and it does not work)

I don’t think this is a viable strategy in its current form; you need
some explicit synchronization between the two processes.

--
https://www.greenend.org.uk/rjk/

[Pavankumar S V]

On Monday, 6 March 2023 at 22:50:28 UTC+5:30, Richard Kettlewell wrote:
> Pavankumar S V <pavanku...@gmail.com> writes:

Thanks for your inputs.

> The kernel will also flush if there is more than a certain number of
> modified pages.

Yes. In combination with "dirty_writeback_centisecs", I had also tried increasing the values of kernel parameters "dirty_background_ratio", "dirty_ratio" to large values to increase the number of dirty pages that can be kept in memory before flushing to disk.
But this also did not work.

> I don’t think this is a viable strategy in its current form; you need
> some explicit synchronization between the two processes.

I am trying now with explicit synchronization mechanism like this:
1) Created a shared memory object of size 1 byte in /dev/shm(let's consider its name as "prevent_read"). Memory mapped this to the virtual address space of both the processes.
2) When I want to prevent the process2 from reading "mtdblock0" region, then I'm writing "prevent_read" with '1' from process1.
3) process2 checks "prevent_read"(value is set as 1 by process1) before reading "mtdblock0" region and it should not read.
4) After some time, process1 writes '0' to "prevent_read".
5) process2 checks "prevent_read"(value is set as 0 by process1) before reading "mtdblock0" region and it should be able to read.

But above mechanism is also not working. Note: Both process1 and process2 are independent processes running on different cores.

As per my understanding, since shared memory object "prevent_read" uses the mmap() concept, again kernel has to do the flushing of data from the memory mapped region to the shared memory object "prevent_read".
So whenever process1 writes '1' to ""prevent_read", it actually writes to the memory mapped region of process1. So kernel has to do the flushing from memory mapped region to shared memory object "prevent_read". If this flushing is delayed, then process2 can read
'0' from "prevent_read" and can access "mtdblock0" until the flushing is completed. So this mechanism is failing sometimes.
Is there any way to do flushing immediately that will help in this case? (I tried msync, but msync is not syncing because msync's behavior is undefined if used on a shared memory object as per the man page.)

Thanks in Advance.

> --
> https://www.greenend.org.uk/rjk/

[Richard Kettlewell]

Pavankumar S V <pavanku...@gmail.com> writes:
> I am trying now with explicit synchronization mechanism like this:
> 1) Created a shared memory object of size 1 byte in /dev/shm(let's
> consider its name as "prevent_read"). Memory mapped this to the
> virtual address space of both the processes.
> 2) When I want to prevent the process2 from reading "mtdblock0"
> region, then I'm writing "prevent_read" with '1' from process1.
> 3) process2 checks "prevent_read"(value is set as 1 by process1)
> before reading "mtdblock0" region and it should not read.
> 4) After some time, process1 writes '0' to "prevent_read".
> 5) process2 checks "prevent_read"(value is set as 0 by process1)
> before reading "mtdblock0" region and it should be able to read.

That has a time-of-check/time-of-use bug even on a single core, and
there’s no chance it’ll work across multiple cores due to the lack of
memory synchronization.

> But above mechanism is also not working. Note: Both process1 and
> process2 are independent processes running on different cores.

By synchronization mechanism I mean something like fctnl with F_SETLK,
or pthread_mutex_lock (with a mutex initialized with
PTHREAD_PROCESS_SHARED).

--
https://www.greenend.org.uk/rjk/

[Pavankumar S V]

On Wednesday, 8 March 2023 at 14:23:03 UTC+5:30, Richard Kettlewell wrote:
> Pavankumar S V <pavanku...@gmail.com> writes:

> That has a time-of-check/time-of-use bug even on a single core, and
> there’s no chance it’ll work across multiple cores due to the lack of
> memory synchronization.

Ok understood. I was trying to use shared memory concept to achieve explicit synchronization (as Shared memory is the fastest way of IPC). I think intention of shared memory concept is just to transfer some data from one process to another quickly. We cannot use Shared memory concept for synchronization purpose. Because, even though shared memory is very fast, data written by one process will not be visible to another immediately. Is my understanding right?

> By synchronization mechanism I mean something like fctnl with F_SETLK,
> or pthread_mutex_lock (with a mutex initialized with
> PTHREAD_PROCESS_SHARED).

ok. I'll try pthread_mutex_lock or semaphores to achieve the synchronization.

Thank you very much.

> --
> https://www.greenend.org.uk/rjk/