mmap without automatic flush to disk

[abi.va...@gmail.com]

Hi,

have a use case in linux to keep frequently modified data in memory which is synced to disk at regular intervals (say once in a second). I am evaluating mmap but I am not sure if I can achieve the below with memory map.

1. Can I turn off automatic flush from memory mapped files to disk? I don't want to change the system level settings which could have an impact to the entire machine.
2. Is it possible to sync the data in memory to disk periodically say using msync? I just would like the sync to disk happen as per the configuration.
3. Is there a possibility of corruption of memory mapped files say in the event of a machine crash or an abrupt shutdown? I have only integer/long(64 bit) type data but in large numbers say 25000 integer values.

Kindly advice.

Thanks for the help.
Abi

[Lusotec]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

abi.va...@gmail.com wrote:
> have a use case in linux to keep frequently modified data in memory which
> is synced to disk at regular intervals (say once in a second). I am
> evaluating mmap but I am not sure if I can achieve the below with memory
> map.
>
> 1. Can I turn off automatic flush from memory mapped files to disk?

If you are memory mapping a file then Linux will write dirty pages
(changed/written memory pages) to disk when it wants to free memory or when
these pages are older than a certain configurable time. You can increase the
duration dirty pages are kept without being written to the file system but I
don't think there is any way to disable the first behavior.

> I don't want to change the system level settings which could have an
> impact to the entire machine. 2. Is it possible to sync the data in memory
> to disk periodically say using msync?

Yes, simply set a timer (setitimer, alarm) and on the timer handler call
msync (use probably want to the async flag).

> I just would like the sync to disk happen as per the configuration.
> 3. Is there a possibility of corruption of memory mapped files say in the
> event of a machine crash or an abrupt shutdown?

Yes, obviously, just like with any file system operation, an system crash
can cause data loss. If the mmap file is being written at the time of the
crash then the file may end up in a half-updated and inconsistent state.

> I have only integer/long(64 bit) type data but in large numbers
> say 25000 integer values.
>
> Kindly advice.

I recommend you don't use mmap in your case. Use plain fread/fwrite
operations and don't overwrite files.

Use the following method:
- - Create temporary file in the same directory as older file.
- - Write data to temporary file.
- - fsync temporary file.
- - close temporary file.
- - Rename older file to *.bak (or something similar).
- - Rename temporary file to final name.

This should give a good degree of safety even in the event of a crash.
Either the older, the newer or both versions should be in the file system.

Regards.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAlKmGz4ACgkQGQjO2ccW76qZMwD6At6S5x2X/fr8Ksz9iVdocmrk
YKDQdDjx3SxRDqx8hewA/j5dOD43ku/dI3xJUY7iXGHbMulN4iyQEqmvv8501gB7
=R0T5
-----END PGP SIGNATURE-----

[Pavankumar S V]

On Tuesday, 10 December 2013 at 01:04:22 UTC+5:30, Lusotec wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> abi.va...@gmail.com wrote:

> If you are memory mapping a file then Linux will write dirty pages
> (changed/written memory pages) to disk when it wants to free memory or when
> these pages are older than a certain configurable time. You can increase the
> duration dirty pages are kept without being written to the file system but I
> don't think there is any way to disable the first behavior.

Please let me know how to increase the duration dirty pages are kept without being written to the file system. I tried achieving this by changing the kernel paramters "dirty_writeback_centisecs" to 0 for some time and changed back to the default value to disable the flushing for some time. But it worked only partially and also the operations to used to write these parameters is taking too much time for execution.
Please let me know if there are any other methods to achieve this.

Best Regards,
Pavankumar