ByteBuffer limits ChannelBuffer and ByteBuf whether we want it to or not ...

[Kevin Burton]

So I've been working on a ByteSlab implementation which is similar to ChannelBuffer.  The difference is that I use mmap() directly for files and Unsafe.allocateMemory directly.

This way I can have memory ranges > 2GB and the same thing for files as well as bypassing a TON of limitations in with ByteBuffer and Java's memory allocation bugs (which there are tons).

However, I realized that since Netty is always using ChannelBuffers the only way I can efficiently perform IO on large files is to take a slice < 2GB from a larger file and send it over the network.

Of course that makes a ton of sense since you really want to be thinking in packets and not GB of data.

However, ByteBuffer can't be efficiently instantiated since it's package private.  I can't create a DirectByteBuffer easily.

So the only way I can think to do this is to implement the ChannelBuffer interface within my code.  This API has a LOT of surface area.  

I'm not sure there's an elegant way to bypass this.  

I can't really pool implementations of ByteBuffer because I'm not sure when netty is done with them.  And I generally don't like this approach ANYWAY.

I could create a new class that implements all the functionality of ChannelBuffer but that's a bit onerous of course.  This will probably be the strategy I go with but it's going to add a lot of code.

[Rajiv Kurian]

Does your implementation do the memory management itself or is there an explicit delete/free functionality? I don't know the unsafe api at all but in JNI direct byte buffers can wrap a pointer to your data in native heap. Can't a channel buffer wrap such a direct byte buffer without any copies?

Of course with any native buffer implementation you will need to know when exactly it is safe to free it, which is tricky because you'll need to know when Netty or any other external library is really done with it.

[Rajiv Kurian]

For the first question, I meant is do you have a delete function that is explicitly called to release memory?

[Kevin Burton]

Explicit delete/free... I just called it release() ... 

Technically you can allocate a byte[] and use Unsafe on it so I might do this with heap buffers but for now I don't want to as I don't need it... 

In my situation I'm handling all the memory directly within one thread so I know when it's time for free...

[Rajiv Kurian]

Have you then considered writing the buffer implementation in C/C++ and creating direct byte buffers in JNI land with pointers to native buffers? I am assuming you want to avoid some disadvantage with the current implementation of byte buffers in the JDK. If so this may or may not help you depending on where you have been bottlenecked by the byte buffer api/implementation.

[Norman Maurer]

In Netty 4 we pool them because of the problems with allocate and free. 

Maybe you want to move to netty 4 as I think it would fit better your needs?!?

--
 
---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Norman Maurer]

We do this in Netty 4. If Unsafe is present we make use of it to get most out of performance.

[Norman Maurer]

I started to work on using JNI for this in netty as using Cleaner for explicit release is a bottleneck (it use static synchronized in some methods)