Custom SQLite Tokenizer in Android

[Jason Workman]

Hello All,

I am currently writing an android app that using sqlite. I have a
virtual fts table that stores the content I would like to search.
Unfortunately, the default tokenizers don't handle the format by data
is stored in, so I need to write a custom tokenizer. Here is a link to
the documentation:

http://www.sqlite.org/fts3.html#section_5_1

I am assuming that this possible, but I was wondering if anyone has
done this before. I plan on compiling a module written in C, using NDK
and accessing it through JNI. I plan to install it using

SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module
ptr>);

Is this even possible to do with android (going through the java
interface)?

Thanks in advance for your help!

[Olivier Guilyardi]

Hello Jason,

I haven't done that, but as long as you compile and bundle your own version of
SQLite, you can modify it, add so add your tokenizer. Then just write your
own/minimal Java/JNI wrapper. I see no reason why this wouldn't work.

What is impossible is to add a tokenizer to the sqlite lib which is
pre-installed on the system. So, in any case, you can't use the default SDK Java
SQLite bindings, unless you duplicate that in your app. But I've looked at the
source and I think that it's not really reusable, plus it may noticeably
increase your app size. A minimal/specific wrapper around your bundled version
of SQLite is better IMO.

Olivier

[Olivier Guilyardi]

Hmm, I have looked again at this statement:

SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>);

An so IIUC you would like to execute that type of statement through the SDK Java
SQLite API. That is, you want to pass a native pointer as a blob to SQLite
through JNI, and then expect SQLite to call back some functions in your native
code...

I guess that since all of this happens within a single process, it could work,
provided that such things are not disabled in the default SQLite build for
security reasons. But it sounds like a lot of trouble to me, and if it ever
works today, I am not sure how that would behave in future Android releases.

Olivier

[Jason Workman]

Olivier,

Thank you for your responses. I was leaning towards passing a pointer
through, but I agree I don't know if that will work in the future.

For now, I'll do as you first suggested. I have compiled/bundled the
sqlite code into my app, and I'll just write a thin JNI wrapper to
register my tokenizer. I just hope I don't run into compatibility
issues. I want to target all android versions 1.5+, so I hope that the
versions of SQLite bundled with previous versions of Android will work
correctly. I guess a little investigation is in order to see what
compatibility issues I may run into doing this custom work.

I'll post more info as I work through this.

Thanks again,

Jason

> >> Thanks in advance for your help!- Hide quoted text -
>
> - Show quoted text -

[Mihai Uricaru]

I will be very interested in having an example of a JNI wrapper. 

I need very soon to write my own and any point of reference will help.

Thank you,

Mike Uricaru

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Olivier Guilyardi]

Jason,

yes, although it may increase you app size, bundling your own SQLite lib is
safer. This way, you have control over the entire database "pipeline", and you
know that it won't break in the future.

I don't understand what you mean about compatibility issues and "SQLite bundled
with previous versions of Android". As long as you bundle your own SQLite lib
and only use that instead of the default SQLite Java API, then there should be
no compatibility issue.

Your app uses its own version of SQLite to read and write to a file, that's all.
You basically don't rely on anything from the platform apart from file I/O...
It's very safe, down to Android 1.5. But still, the increased app size can be a
problem in this context sometimes. For this reason, your original idea could
still make sense, although it seems dangerous IMO.

Olivier