No proposals to implement 64bit mmap?

[Kevin Burton]

It's going to be 2017 soon. 

Have there been any proposals to add 64bit mmaps to Java?

I had to implement one for a project I was working on but found my implementation caused a very strange segfault which I couldn't duplicate.

The fact that the JVM is limited to only mapping 2GB at a time is really something that should be addressed.

The reason I'm bringing this up is that we just hit this internally due to spark suffering from this... it has a bug where its shuffle files can't be more than 2GB...

[Norman Maurer]

Maybe you should bring this to the openjdk dev mailing list ?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tomasz Kowalczewski]

This together with long indexed ByteBuffers is a must have nowadays. Looks like simple thing to do relatively to other changes proposed this year for Java/OpenJDK.

Actually there is simple workaround: FileChannel::map makes initial arguments check (including 2GB size limit) and calls map0 which is native. You can call it via reflection. There is even implementation of this in Martin's own Agrona (IoUtil class). We use this to mmap multi gigabyte files on Linux without crashes. 

How you wrap that long pointer is another story as Netty ByteBuf and Agrona buffers are both int indexed. We have our own buffers impl. for that (needs some docs love before open sourcing though).

P.S.: another story is how to do this mmap trick in post jigsaw JDK.

Regards,
Tomasz Kowalczewski

Sent from my phone

[Vitaly Davidovich]

I think long indexing is punted until Arrays 2.0, although I don't know where/how that's going (if at all - initial proposal is from circa 2012).  Otherwise, e.g., long indexing on a HeapByteBuffer is crufty.

So here's a workaround - stop using Java for such scenarios/apps :) j/k (... sort of).

On Thu, Dec 8, 2016 at 2:20 PM, Tomasz Kowalczewski <tomasz.ko...@gmail.com> wrote:

This together with long indexed ByteBuffers is a must have nowadays. Looks like simple thing to do relatively to other changes proposed this year for Java/OpenJDK.

Actually there is simple workaround: FileChannel::map makes initial arguments check (including 2GB size limit) and calls map0 which is native. You can call it via reflection. There is even implementation of this in Martin's own Agrona (IoUtil class). We use this to mmap multi gigabyte files on Linux without crashes. 

How you wrap that long pointer is another story as Netty ByteBuf and Agrona buffers are both int indexed. We have our own buffers impl. for that (needs some docs love before open sourcing though).

P.S.: another story is how to do this mmap trick in post jigsaw JDK.

Regards,
Tomasz Kowalczewski

Sent from my phone

On 8 Dec 2016, at 19:31, 'Norman Maurer' via mechanical-sympathy <mechanical-sympathy@googlegroups.com> wrote:

Maybe you should bring this to the openjdk dev mailing list ?

On 8 Dec 2016, at 19:30, Kevin Burton <burto...@gmail.com> wrote:

It's going to be 2017 soon. 

Have there been any proposals to add 64bit mmaps to Java?

I had to implement one for a project I was working on but found my implementation caused a very strange segfault which I couldn't duplicate.

The fact that the JVM is limited to only mapping 2GB at a time is really something that should be addressed.

The reason I'm bringing this up is that we just hit this internally due to spark suffering from this... it has a bug where its shuffle files can't be more than 2GB...

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Kevin Burton]

Been thinking about this more and I worked with the Netty gang on their own implementations back in the day.  

But their implementation is int based so maybe its best to just take their ByteBuf and port it to use long...