FTS unknown tokenizer

[Brendan Duddridge]

Hi Jens,

One of my beta testers is in Holland and he has his language set to Dutch. Today he said the full text searching wasn't working. So I had him send me the log output and this is what we saw:

Error calling sqlite3_step (1: unknown tokenizer) SQLITE_ERROR

DB Query:         CREATE VIRTUAL TABLE IF NOT EXISTS fulltext             USING fts4(content, tokenize=unicodesn "stemmer=dutch")

It works for me in Canada in English.

I'm guessing that I haven't compiled SQLCipher properly to support all these different stemmers. Normally when I compile SQLCipher I just use the following which covered all the bases for me in my old version of my app:

-DSQLITE_HAS_CODEC -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61

Is there another thing I should be adding to the compile options?

Thanks,

Brendan

[Jens Alfke]

By coincidence I just discovered the source of this a few days ago — the tokenizer stemmers for the non-English languages are being compiled into the code, but there are some missing #defines so they don’t get called. I fixed it on the branch I was working on, but I should get it into 1.2 too.

(You can try out the fix by going to the submodule in vendor/CBForest/vendor/sqlite3-unicodesn/ and pulling the latest commit, i.e. 48bcf0e.)

Time to file an issue on this so we can track it for 1.2...

—Jens

[Brendan Duddridge]

Awesome. Ok. I'll file an issue.

Thanks!

Brendan