embedding redis via JNI

[Jak Sprats]

Hi All,

I am working w/ a Java expert to embed redis into Java via JNI.

Basically all redis calls are made to a single Java thread to make
sure that redis remains single threaded, these calls then reach inside
of redis (via hacking an API into redis' C code) via JNI .

We have been having some problems:
1.) jemalloc may corrupt the JVM's memory
2.) there are functions like "zmalloc_enable_thread_safeness" that I
dont understand.

Has anyone tried anything like this, any tips would be appreciated,
this is a nightmare to debug (I am a Java weakling)

- jak

[Colin Vipurs]

Jak,

Why are you trying to do this? Is there something you're trying to do
that can't be achieved with the native Java collections?

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>

--
Maybe she awoke to see the roommate's boyfriend swinging from the
chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Jak Sprats]

HI Colin,

I dont know much about Java. JNI is being used to call C from Java.
The app is in Java, redis is in C. That is all I know.

What is the advantage of using Java collections in this case?

- jak

On Feb 21, 7:10 am, Colin Vipurs <zodiac...@gmail.com> wrote:
> Jak,
>
> Why are you trying to do this? Is there something you're trying to do
> that can't be achieved with the native Java collections?
>
>
>
>
>
>
>
>
>
> On Mon, Feb 20, 2012 at 5:12 PM, Jak Sprats <jakspr...@gmail.com> wrote:
> > Hi All,
>
> > I am working w/ a Java expert to embed redis into Java via JNI.
>
> > Basically all redis calls are made to a single Java thread to make
> > sure that redis remains single threaded, these calls then reach inside
> > of redis (via hacking an API into redis' C code) via JNI .
>
> > We have been having some problems:
> > 1.) jemalloc may corrupt the JVM's memory
> > 2.) there are functions like "zmalloc_enable_thread_safeness" that I
> > dont understand.
>
> > Has anyone tried anything like this, any tips would be appreciated,
> > this is a nightmare to debug (I am a Java weakling)
>
> > - jak
>
> > --
> > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

[Colin Vipurs]

It's more of what is the advantage of going to all the effort of
calling out to an embedded Redis. Redis gives you sets, sorted sets,
lists and hashes, all of which exist natively in Java. It seems like
an awful lot of effort to go to for no foreseeable benefit

> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Jak Sprats]

Hi Colin,

I more or less agree w/ you, but this is not my call.

This guy has data already in redis and wants the extra little
performance boost that embedding provides.

Personally I just recommended having a local instance serving via
UnixDomainSockets, but again, not my call

- jak

[Colin Vipurs]

It sounds like you're in a corner on this, but have you done any
profiling to see if the extra performance boost is required? I would
think your suggestion sounds like the most sane option

> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Josiah Carlson]

Sane is relative. Not having to copy memory to the OS and back to the
process, not having to pack and unpack commands, etc., can make a huge
difference in performance.

Jak, I'm sorry I can't offer any advice on this front, I've not used
JNI. Though it does seem strange that jemalloc and the Java memory
allocator are getting overlapping memory regions from the system. That
seems really strange to me. Maybe strace will be able to offer memory
alloc/free information that can help tell you what's going on?

- Josiah

[Jak Sprats]

Hi Colin & Josiah,

thanks for the advice ... this has been more than frustrating.

so the developments of today were it just magically works, re-built
and works, no clue what changed :). I am not in charge of building and
suck at Java, so this is all 2nd hand.

so it may or may not have been jemalloc, and it was definitely a
result of an overly complicated build process, which is not a surprise
at all, because these pieces are not meant to fit together like this,
so getting them to work harmoniously requires some very invasive
hacking ...

this gets marked as cant-be-reproduced-and-good-riddance.

if anyone wants to embed redis, I can tell them how, but I can also
advise them that it is only useful in use cases where extreme
performance (>300K TPS) is needed and where the calling application
takes care of data distribution (because the embedded version is a
single thread w/ no TCP-abilities) and funneling requests to a single
thread.

- jak

On Feb 21, 1:32 pm, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> Sane is relative. Not having to copy memory to the OS and back to the
> process, not having to pack and unpack commands, etc., can make a huge
> difference in performance.
>
> Jak, I'm sorry I can't offer any advice on this front, I've not used
> JNI. Though it does seem strange that jemalloc and the Java memory
> allocator are getting overlapping memory regions from the system. That
> seems really strange to me. Maybe strace will be able to offer memory
> alloc/free information that can help tell you what's going on?
>
>  - Josiah
>

> On Tue, Feb 21, 2012 at 6:38 AM, Colin Vipurs <zodiac...@gmail.com> wrote:
> > It sounds like you're in a corner on this, but have you done any
> > profiling to see if the extra performance boost is required? I would
> > think your suggestion sounds like the most sane option
>

[Jak Sprats]

and for the record, the magical secret ingredient to doing this was to
NOT use jemalloc.

jemalloc can have problems w/ the JVM's memory allocator ... I dont
know what they are, but w/ plain old malloc this works

I am just adding this in, in case someone google's this thread later.