Transfer of a large buffer through the JNI

[Антон Иванов]

In connection with the audio have to pass a very large buffer (byte). Now climbs this error:
Out of memory on a 1638416-byte allocation, and further huge DEBUG to 8000 lines ...
Can this problem as it solved? And then in my case, to transfer a smaller buffer problem ...

[fadden]

On Jun 6, 9:32 am, Антон Иванов <wkybo...@gmail.com> wrote:
> In connection with the audio have to pass a very large buffer (byte). Now
> climbs this error:
> Out of memory on a 1638416-byte allocation, and further huge DEBUG to 8000
> lines ...

Start here:

http://android.git.kernel.org/?p=platform/dalvik.git;a=blob_plain;f=docs/jni-tips.html;hb=HEAD#FAQSharing

Not sure what "further huge DEBUG to 8000 lines" means.

[Антон Иванов]

It does not matter. I want to address this bug, as the rest - a consequence. You understood wrong? If the decision is, then help.

2011/6/6 fadden <fad...@android.com>

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.

To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Антон Иванов]

Ie I want to know: Is it possible to transmit such a large byte [] and if so, how?

6 июня 2011 г. 20:27 пользователь Антон Иванов <wkyb...@gmail.com> написал:

[Антон Иванов]

Realized that you had in mind the link. Right now try to

6 июня 2011 г. 20:28 пользователь Антон Иванов <wkyb...@gmail.com> написал:

[Антон Иванов]

I used NewByteArray as follows:
jbyteArray bytearrayBuffer = (* env) -> NewByteArray (env, buffer_size); / / construct a new byte array

(* env) -> SetByteArrayRegion (env, bytearrayBuffer, 0, buffer_size, (jbyte *) buffer); / / copies the region of length nBytes starting from 0 from the char array to the new byte array
(* env) -> CallStaticVoidMethod (env, cls, mid, bytearrayBuffer, buffer_size, numbers);

   Is it possible to somehow solve the problem with conventional byteArray? I just do not understand how in this case, use NewDirectByteBuffer

6 июня 2011 г. 20:30 пользователь Антон Иванов <wkyb...@gmail.com> написал:

[Baodong Chen]

I use this to share data between java and native code:
In java:
ByteBuffer.allocateDirect(__size__);
In native code:
void* buf = (void*)env->GetDirectBufferAddress((jobject)jobj);
now i can use buf in my native code.

2011/6/7 Антон Иванов <wkyb...@gmail.com>:

[Stephen Williams]

You can allocate the byte array in Java or in C via JNI.  In Java, just do 'byte[] buf = new byte[1600000];  Then pass it as a parameter to C.  You have to lock (pin) it before accessing from C, then unlock (unpin) when you're done.  There is little or no reason to use direct byte buffers, especially with current versions of Android.  My code has used 20MB or more this way with no problems.

Stephen

2011/6/6 Антон Иванов <wkyb...@gmail.com>

--
--
Stephen D. Williams s...@lig.net scie...@gmail.com LinkedIn: http://sdw.st/in
V:650-450-UNIX (8649) V:866.SDW.UNIX V:703.371.9362 F:703.995.0407
AIM:sdw Skype:StephenDWilliams Resume: http://sdw.st/gres
Personal: sdw.st facebook.com/sdwlig twitter.com/scienteer

[Baodong Chen]

yes, i think that works.
but when you need to consider the efficiency of your program,
DirectBuffer is better than a lot of new calls in java and data copies
through JNI.

2011/6/7 Stephen Williams <scie...@gmail.com>:

[Stephen Williams]

Generally, you can't get away from a copy for things like IO.  However, if you are thinking of efficiency, then you normally should do all of the new()s in Java.  If you use the lock / read/write / unlock sequence, there are no "data copies through JNI": your native code is writing directly to the Java data buffer.  (Technically, a JVM implementation could do a copy with that interface, but neither Dalvik or the JVM do in any case that I've found.)  If, for instance, you have a byte buffer, you can interleave Java and C/C++ code reading/writing the same data efficiently.  The pin/unpin just prevents GC from moving the data, which it shouldn't be doing for large buffers anyway.

DirectBuffers incur an extra copy in some cases and are either more expensive or the same as a Java-owned buffer.  Whatever extra benefit they had in the past doesn't seem to exist now.  Is there a key case (OpenGL maybe?) where they still have an advantage?

The one extra thing to keep in mind is the jnigraphics optimization: Normally, you can't get directly at the byte[] in a Bitmap.  jnigraphics uses a private API to allow this from native code.  Bitmap could have just had a method to return byte[] and it could have been used the same way, but the interface allows the implementation to do something non-obvious.  jnigraphics prevents actually doing something other than byte[]...  Without jnigraphics, you have to create bitmap from byte [].  While fast, and almost certainly using a native method, it is a copy.

Stephen

2011/6/6 Baodong Chen <chenbd...@gmail.com>

[Baodong Chen]

at the beginning of my develop I used flows to share data between java and c:
at program start:
sg_jImage = (jintArray)env->NewGlobalRef(env->NewIntArray( __size__));

when write data in c:
env->SetIntArrayRegion( sg_jImage,
0,
__size__,
pixel_data );
and in java, i can use sg_jImage in java.

but i find it too slow for me, maybe because of SetIntArrayRegion call.

so i try to use DirectBuffer as flows:

ByteBuffer.allocateDirect(__size__);
In native code:
void* buf = (void*)env->GetDirectBufferAddress((jobject)jobj);
now i can use buf in my native code.

and i find it much faster than the first one.

2011/6/8 Stephen Williams <stephend...@gmail.com>:

[Hitendrasinh Gohil]

Hey,

I am downloading mp3 file and write it to sdcard in encrypted format.Then while playing first i m decrypting this file to cache and then play it.To play mp3 it takes 4 to 5 secs in galaxy tab and htc.But its too slow(takes more than 30sec) in low config devices like dell zte blade.i have read that ndk is some what faster(20%) and we should use it when there is performance related issues with sdk.

so i just need to do it with ndk and not much aware of ndk.As i have made one proj that simply write text to file with ndk.

can anbody suggest me some code snippets for the same that  i can do the above.

2011/6/7 Baodong Chen <chenbd...@gmail.com>

--
Regards,
Hitendrasinh Gohil

[fadden]

On Jun 7, 11:41 am, Stephen Williams <stephendwilli...@gmail.com>
wrote:

> DirectBuffers incur an extra copy in some cases and are either more
> expensive or the same as a Java-owned buffer.  Whatever extra benefit they
> had in the past doesn't seem to exist now.

Advantages of "direct" ByteBuffers:

(1) The storage doesn't move, ever, until it's freed. You can keep
pointers to stuff inside the buffer. A byte[] might move around in
the managed heap.
(2) No JNI overhead to access it (the calls to Get/
ReleaseByteArrayElements are cheap but not free).
(3) Guaranteed copy behavior, i.e. you know access from native code
will never involve a copy (which can bite you on performance, but also
on correctness -- if the VM decides to hand you a buffer instead of a
pointer to the live data, your changes won't be visible to non-native
code until you Release(JNI_COMMIT)).

You can get (1) and (2) by Getting the buffer and never Releasing it,
but that makes GC developers sad. You have no control over (3), and
no guarantees are made about how future versions of the VM will work.

There's also:

(4) You can allocate your own storage to back a "direct" ByteBuffer.
For example, you can mmap() a large bitmap or audio file and ease the
physical memory burden.

This is probably more interesting for API design. For example, when
calling an API that takes ByteBuffer you can easily wrap a byte[], but
if the API takes a byte[] you can't get it easily out of a "direct"
ByteBuffer (unless it happens to be backed by memory that lives on the
managed heap, in which case ByteBuffer.array() will work fine).

FWIW, you can force Dalvik to copy data returned by
GetByteArrayElements with the "forcecopy" CheckJNI mode. This is
meant as a debugging tool to identify incorrect code (e.g. code that
continues to reference data after ReleaseByteArrayElements has been
called). It has a noticeable impact on performance.

[Stephen Williams]

Right, I meant from the Java side.  And the cases I encountered probably involved needing byte[], which is what you can't get from a direct ByteBuffer.  (Non-Android file/network IO and other APIs.)  Then, you are forced to copy on the Java side, which is potentially much slower.  Bitmap.copyPixelsFromBuffer(Buffer) however does just the right thing to support both cases.

I expect that the garbage collector doesn't move large objects excessively.  This seems likely unless other large objects are being allocated/deallocated, in which case you may be much better off with the ability to defragment memory to succeed in a new allocation.

But yes, from the native side, it's much better in a lot of cases.  The absolute best case for using direct buffers is for mmapping / shared memory.

sdw

--

[Stephen Williams]

And then there's the deallocation problem.  With byte[] and similar Java objects, you can null all references and space will definitely be reused on the next GC.  (But beware circular references in objects.  Even in 3.1.)

With direct buffers, at least in some implementations at some point, it was/is not so nice.  I was told not long ago that Android's implementation still has to do multiple passes before it can deallocate a direct buffer.  Plus, native heap space, although you can call a method to gauge it, doesn't show up conveniently in the DDMS heap view.

http://www.javamex.com/tutorials/io/nio_buffer_direct.shtml

How do you deallocate/destroy a direct buffer?

There's actually no explicit method you can call from Java to destroy or deallocate a direct buffer. When you allocate a direct buffer, the VM effectively registers a "cleanup" method with the garbage collector which will should be called at some point once the ByteBuffer object itself is no longer reachable (informally, when it is "ready to be garbage collected")¹, in order to deallocate the underlying memory that was reserved for that buffer.

Usually, this "automatic" deallocation works well enough. But it's important to bear in mind that there's no immediate link between the last time you access a direct buffer and the point at which the memory is actually deallocated. 

sdw

[Stephen Williams]

Just one more thing...:

http://www.evanjones.ca/software/java-bytebuffers.html

All of that copying makes you want to write your own native methods to read and write directly from pinned byte[] or direct buffers or similar.  The details (like the 8K+ malloc "feature") probably vary a lot between JDK and Android.

sdw