A problem about unmapping memory mapped files

[Peter Lawrey]

Yes. It is interesting that this is an issud which has been around at least back to Java 5.0 (2004) but is a real problem for a few people now. Possibly because they are using chronicle in long running applications.

The cleaner *used* to unmap this mapping but now calls through to code which does nothing.

I will look at using a cleaner or finaliser if I have to. I will use a resource counter for unmapping without a GC. I have to check this doesn't conflict with the current unmapping mechanism I.e. it doesn't crash the GC.

I have to check this gives the desired behaviour on windows and linux.

On 17 Jan 2014 06:05, "Yavuz Gökırmak" <yavuz.g...@intellica.net> wrote:

Hi Peter,

it is strange that we all on the same problem at the same time :)

So .. What will you do even you know there is no reference to the buffer, actually in our case we know that no one else is using the buffer.

İ think we can use cleaner from unsafe, what do you think?

On Jan 16, 2014 9:42 PM, "Peter Lawrey" <peter....@higherfrequencytrading.com> wrote:

This is common problem with Java's implementation of memory mapped files. It requires a stop the world collection to determine when a memory mapping can be unmapped.

This is high on my todo list to create a library or add to JavaLang to solve this, most likely by using reference counts to track when the buffer is not needed.

The risk is that if you unmap and use the address later the while JVM crashes (not just throw an error)

BTW I was thinking about this problem this morning.  😋

On 16 Jan 2014 19:25, "Yavuz Gökırmak" <yavuz.g...@intellica.net> wrote:

Hi Peter,

I have a problem about memory mapped files, I think you may hit the same problem and solved somehow  :)

I create randomAccessFile and then map this file to a bytebuffer

fileChannel = new RandomAccessFile("testbug." + i, "rw").raf.getChannel();

byteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, 0,

1 * 1024 * 1024 * 1024);

when I need to delete the file, I close the channel and delete the file

fileChannel.close();

File file = new File("testbug." + (i - 1));

deleted = file.delete();

although java thinks that the file is deleted, it still remains space on the disk. I can see the file as "DEL" at the lsof output

evam@HURBIGENG04:~/evamengine$ lsof | grep test

bash      16465         evam  cwd       DIR              252,0       4096  1974273 /home/evam/bugtest

java      20073         evam  cwd       DIR              252,0       4096  1974273 /home/evam/bugtest

java      20073         evam  mem       REG              252,0 1073741824  1973734 /home/evam/bugtest/testbug.9

java      20073         evam  DEL       REG              252,0             1973733 /home/evam/bugtest/testbug.8

java      20073         evam  DEL       REG              252,0             1973732 /home/evam/bugtest/testbug.7

java      20073         evam  DEL       REG              252,0             1973731 /home/evam/bugtest/testbug.6

java      20073         evam  DEL       REG              252,0             1973730 /home/evam/bugtest/testbug.5

java      20073         evam  DEL       REG              252,0             1973729 /home/evam/bugtest/testbug.4

java      20073         evam  DEL       REG              252,0             1973728 /home/evam/bugtest/testbug.3

java      20073         evam  DEL       REG              252,0             1973727 /home/evam/bugtest/testbug.2

java      20073         evam  DEL       REG              252,0             1973726 /home/evam/bugtest/testbug.1

java      20073         evam  DEL       REG              252,0             1973698 /home/evam/bugtest/testbug.0

The problem is A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected. So in order to actually delete the file I have to call 

    byteBuffer = null;

    System.gc();

this is the same problem ( http://stackoverflow.com/questions/2972986/how-to-unmap-a-file-from-memory-mapped-using-filechannel-in-java ) 

here is my code that reproduces the problem 

http://pastebin.com/akfR3ybC

and this is the solution

http://pastebin.com/59qvgjHt

do you have any advice without using system.gc()

regards

[Piero De Gol]

Hello,
since I am also experiencing this issue, I just wanted to share what I observed, hoping this can be helpful, somehow.
(version 2.0.2)
Adding the following cleaner call in acquireBuffer method of PrefetchingMappedFileCache (before line 92):

...
lastIndex = index;

// cleaner call
if(lastMBB != null) {
      Cleaner cleaner = ((DirectBuffer) lastMBB).cleaner();
      if (cleaner != null) {
          cleaner.clean();
      }
}

lastMBB = mappedByteBuffer;
...

it did solve for me the problem of memory not being released (and, in the long run, freezing the OS), due to the unmapping not taking place. Indeed, observing the memory usage through Windows task manager, I can see it being released every time the cleaner gets called for the last MappedByteBuffer (which is then, anyways, being overridden).
Though I don't know if and how this influences GC, by possibly having it crash, or anything else.