JNI fails creating too much objects

[Alexis ROBERT]

Hi,

I'm currently looking for using the NDK to parse my dictionary index
file in a more efficient way.

I've done all the parsing stuff, and implemented the JNI glue around,
but I have an issue which I don't know how to handle :

W/dalvikvm( 1318): ReferenceTable overflow (max=512)
W/dalvikvm( 1318): Last 10 entries in JNI local reference table:
W/dalvikvm( 1318): 502: 0x435ee3d8 cls=Ljava/lang/String; (28 bytes)
[...]
W/dalvikvm( 1318): 511: 0x435ee660 cls=Ljava/lang/String; (28 bytes)
W/dalvikvm( 1318): JNI local reference table summary (512 entries):
W/dalvikvm( 1318): 1 of Ljava/lang/Class; 164B
W/dalvikvm( 1318): 510 of Ljava/lang/String; 28B (510 unique)
W/dalvikvm( 1318): 1 of [Ljava/lang/String; 100596B
W/dalvikvm( 1318): Memory held directly by native code is 115040 bytes
E/dalvikvm( 1318): Failed adding to JNI local ref table (has 512
entries)

Apparently, the JNI interface doesn't want me to build more than 512
objects by default. I don't really know how I can give to my Java
program more than 512 words except than using String[].

A huge string with \0 as a delimiter ?!

A limit of 512 objects seems a little bit too low, isn't it ?

Thanks

Alexis ROBERT

[Anders Hasselqvist]

Hi,

Have a look at PushLocalFrame()/PopLocalFrame() or EnsureLocalCapacity().
But actually, I would try to change to not have so many local
references around in at once in the code if possible...

Regards,
Anders

2009/6/28 Alexis ROBERT <alexis...@gmail.com>:

[John Ripley]

It's a limit of 512 *local* references. What you need to do is convert them to global references using NewGlobalRef(), or to clean up the locals shortly after creating them if they're not needed.

This is normal behavior for a VM. You'll find the same happens on desktop JVMs, only the limit will be much higher.

2009/6/27 Alexis ROBERT <alexis...@gmail.com>

[Alexis ROBERT]

Thanks!

I'm pretty new to JNI, and I didn't knew about local/global
references. That answers all my questions about JNI's memory
management ! :)

I've solved the issue by freeing the local references before return;
(just after SetObjectArrayElement), and that seems to work.

Thanks again.

Alexis

On 28 juin, 02:33, John Ripley <jrip...@google.com> wrote:
> It's a limit of 512 *local* references. What you need to do is convert them
> to global references using NewGlobalRef(), or to clean up the locals shortly
> after creating them if they're not needed.
>
> This is normal behavior for a VM. You'll find the same happens on desktop
> JVMs, only the limit will be much higher.
>

> 2009/6/27 Alexis ROBERT <alexis.rob...@gmail.com>

[Alexis ROBERT]

I have an other issue which concern UTF-8 parsing.

The strangest thing is that it runs perfectly on the emulator, but not
on my ADP1 (firmware 1.5).

On my ADP1, I get that :

W/dalvikvm( 3899): JNI WARNING: illegal continuation byte 0x45
W/dalvikvm( 3899): string: 'EMPYRÉAL, ALE �EMMAIGRI'
W/dalvikvm( 3899): in Lorg/alexis/libstardict/
Index;.getRawWords (Ljava/lang/String;)[Ljava/lang/String;
(NewStringUTF)

I've checked my file, it's a well-formed UTF-8 file. On this segment,
the string is 45 4d 50 59 52 c3 89 41 4c 2c 20 41 4c 45 00 (which
seems to be a valid UTF-8 string).

Here, the 0x45 seems not to be a continuation byte (and iconv didn't
hit me :) ).

On the same time, EMMAIGRI is around the position 0x000bab60 in my
file, where EMPYRÉAL is around 0x000bd2d0 (so it's not a missing '\0'
delimiter). The parser reads the data in a linear way, so EMMAIGRI was
already in memory.

On the emulator, I get perfect results (SDK : 1.5.0-r2).

Thanks :)

Alexis

[fadden]

On Jun 27, 5:33 pm, John Ripley <jrip...@google.com> wrote:
> It's a limit of 512 *local* references. What you need to do is convert them
> to global references using NewGlobalRef(), or to clean up the locals shortly
> after creating them if they're not needed.
>
> This is normal behavior for a VM. You'll find the same happens on desktop
> JVMs, only the limit will be much higher.

The VM is required to ensure that 16 local reference slots are
available. Anything beyond that must be requested with
EnsureLocalCapacity. Dalvik is arguably too lenient here.

Converting them to global references just means they'll never go away,
and you'll leak memory.

This bit of the JNI manual (5.2.1 "Freeing Local References"):
http://java.sun.com/docs/books/jni/html/refs.html#27567

shows a simple example using DeleteLocalRef when processing an array
of objects with GetObjectArrayElement.

[fadden]

On Jun 28, 5:22 am, Alexis ROBERT <alexis.rob...@gmail.com> wrote:
> I have an other issue which concern UTF-8 parsing.
>
> The strangest thing is that it runs perfectly on the emulator, but not
> on my ADP1 (firmware 1.5).

That seems backward -- CheckJNI is enabled by default on the emulator,
but not on the device.

> W/dalvikvm( 3899): JNI WARNING: illegal continuation byte 0x45
> W/dalvikvm( 3899):              string: 'EMPYRÉAL, ALE EMMAIGRI'
> W/dalvikvm( 3899):              in Lorg/alexis/libstardict/
> Index;.getRawWords (Ljava/lang/String;)[Ljava/lang/String;
> (NewStringUTF)
>
> I've checked my file, it's a well-formed UTF-8 file. On this segment,
> the string is 45 4d 50 59 52 c3 89 41 4c 2c 20 41 4c 45 00 (which
> seems to be a valid UTF-8 string).

Something isn't quite right -- the hex string you gave is "EMPYR<E>AL,
ALE\0", but the string printed by CheckJNI continues on past that. So
the VM isn't choking on what you think it's choking on.

> On the same time, EMMAIGRI is around the position 0x000bab60 in my
> file, where EMPYRÉAL is around 0x000bd2d0 (so it's not a missing '\0'
> delimiter). The parser reads the data in a linear way, so EMMAIGRI was
> already in memory.

Do you have gdb/gdbserver set up? If so, you can attach gdb and catch
it at the point of the break. Move up the stack to your code and hex-
dump the string that's being passed into NewStringUTF.

Alternatively, extract checkUtfString() from dalvik/vm/CheckJni.c, add
it to your code, and call it right before the NewStringUTF call. Then
you can just add printfs to get additional diagnostics.

[John Ripley]

Beware that Java uses a modified UTF-8 encoding... although I can't see a fault with your string.

I try not to use the JNI UTF-8 functions and instead only pass UTF-16 strings to/from the VM. Use a library such as ICU to do the conversion.

2009/6/28 Alexis ROBERT <alexis...@gmail.com>

[Dianne Hackborn]

Fyi, something to watch out for -- the ARM CPU can't access mis-aligned ints and shorts.  Last time I worked with this, such things would actually work fine in the emulator, but you would get garbage data when running on a real device.

I don't know if that is what is going on here, but it sounds very similar to problems I would have for example in the resource code.  It will all run fine on the emulator, but then on the device you see strange garbage values.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support, and so won't reply to such e-mails.  All such questions should be posted on public forums, where I and others can see and answer them.

[John Ripley]

The core used in the G1 and other similar device does unaligned access. Have a look at /proc/cpu/alignment - you'll see some stats on hits there. Still, support is sketchy on ARMs in general, and in any case unaligned access is slower than aligned (and that's even true on x86). So avoid it and beware of it if you can :)

2009/6/29 Dianne Hackborn <hac...@android.com>