Full Text Search and CBLTableViewDataSource

[Todd Freese]

Can CBLTableViewDataSource be used with Full Text Searching? Can't seem to get it working and thought it might have something to do with the fact that the results are CBLFullTextQueryRows.

Here is how I am creating the view:

+ (CBLQuery *)searchJobsInDatabase:(CBLDatabase* )db

{

    CBLView *view = [db viewNamed:@"jobSearchResults"];

    if (!view.mapBlock) {

        [view setMapBlock: MAPBLOCK({

        if ([doc[@"type"] isEqualToString:kListDocType]) {

            NSString *body = stripHTMLTags(doc[@"jobNumber"]);

            emit(CBLTextKey(body), doc[@"jobNumber"]);

        }

        }) reduceBlock:NULL version: @"1"];

    }

    return [view createQuery];

}

And here is now I am setting the data source:

self.dataSource.query = [Job searchJobsInDatabase:database].asLiveQuery;

self.dataSource.query.fullTextQuery = @"999";

Todd

[Jens Alfke]

On Feb 22, 2014, at 11:23 PM, Todd Freese <to...@filmworkers.com> wrote:

Can CBLTableViewDataSource be used with Full Text Searching? Can't seem to get it working and thought it might have something to do with the fact that the results are CBLFullTextQueryRows.

It should work fine, as far as I know.  What problems are you seeing? No rows, or no data showing up in the rows, or…?

—Jens

[Todd Freese]

It always returns all the docs. If I break the full text query code out into a test method, it shows that it is in fact returning the correct docs. But the tableview is showing the wrong docs.

Todd

[Todd Freese]

After a little more research, I'm thinking it might have something to do with the run method on CBLQuery. In my test code, I am using the run method and it works in my test method. My test method does not work with using rows. 

Todd

[Jens Alfke]

On Feb 23, 2014, at 1:25 PM, Todd Freese <to...@filmworkers.com> wrote:

It always returns all the docs. If I break the full text query code out into a test method, it shows that it is in fact returning the correct docs. But the tableview is showing the wrong docs.

I think the way you initialize the query is triggering a race condition:

self.dataSource.query = [Job searchJobsInDatabase:database].asLiveQuery;

self.dataSource.query.fullTextQuery = @"999";

The datasource will start the query as soon as you set it (in the first line). So by the time you set the .fullTextQuery property, the query’s already running. There isn’t currently any support in CBLLiveQuery for changing query options in mid-stream, so it’ll just keep running with the original options.

This code should work:

CBLLiveQuery* query = [Job searchJobsInDatabase:database].asLiveQuery;

query.fullTextQuery = @"999”;

self.dataSource.query = query;

—Jens

[Todd Freese]

That works. Thanks.

Also, do I need to import something for stripHTMLTags? It that a defined in a category on NSString?

Todd

[Jens Alfke]

On Feb 23, 2014, at 4:36 PM, Todd Freese <to...@filmworkers.com> wrote:

> Also, do I need to import something for stripHTMLTags? It that a defined in a category on NSString?

I don’t know anything about that method; I assumed it was something in your code. If you saw it in an example in our docs, it’s probably just a made-up method in that example.

—Jens

[Todd Freese]

It was in your example docs on Github.

Thanks again.

t

[Todd Freese]

One more issue with Full Text Query. It seems that it can't handle indexing on a field if one of the docs contain a property that is null if that property is being queried. For example, you have a master/detail interface and you create a new doc and all the fields are null at the beginning. 

I get the following error as soon as I create a new doc:

*** ASSERTION FAILED: text != nil

2014-02-24 11:07:27.411 ShotBOT[3636:340b] *** Assertion failure in -[CBLSpecialKey initWithText:](), /Users/couchbase/jenkins/workspace/build_cblite_ios_stable/couchbase-lite-ios/Source/CBLView+Internal.m:383

11:07:27.412‖ WARNING*** : Exception caught in map block of view 'jobSearchResults':

text != nil

5   ShotBOT                             0x00019ef9 -[CBLSpecialKey initWithPoint:] + 0

6   ShotBOT                             0x00017d40 CBLTextKey + 80

7   ShotBOT                             0x0000cc60 __28+[Job searchJobsInDatabase:]_block_invoke + 256

8   ShotBOT                             0x00019877 __32-[CBLView(Internal) updateIndex]_block_invoke + 3344

9   ShotBOT                             0x00010cba -[CBLDatabase(Internal) _inTransaction:] + 172

10  ShotBOT                             0x00018a74 -[CBLView(Internal) updateIndex] + 239

11  ShotBOT                             0x000550ad -[CBLDatabase(Views) queryViewNamed:options:lastSequence:status:] + 161

12  ShotBOT                             0x000528d5 __21-[CBLQuery runAsync:]_block_invoke + 140

13  ShotBOT                             0x0001e1b8 __35-[CBL_Server tellDatabaseNamed:to:]_block_invoke + 85

14  libobjc.A.dylib                     0x01ab481f -[NSObject performSelector:withObject:] + 70

15  Foundation                          0x016f79d8 __NSThreadPerformPerform + 285

16  CoreFoundation                      0x01ca883f __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 15

17  CoreFoundation                      0x01ca81cb __CFRunLoopDoSources0 + 235

18  CoreFoundation                      0x01cc529e __CFRunLoopRun + 910

19  CoreFoundation                      0x01cc4ac3 CFRunLoopRunSpecific + 467

...

11:07:27.420‖ WARNING*** : CouchbaseLite: Failed to rebuild view 'jobSearchResults': 593

11:07:27.420‖ WARNING*** : Failed to update view index: 593

11:07:27.420| WARNING*** : <CBLLiveQuery: 0x8a8aab0>: Error updating rows: Error Domain=CBLHTTP Code=500 "500 Application callback block failed" UserInfo=0x9e58e10 {NSLocalizedFailureReason=Application callback block failed, NSLocalizedDescription=500 Application callback block failed}

Is this an issue with SQLite or the framework?

Todd

[Jens Alfke]

On Feb 24, 2014, at 9:10 AM, Todd Freese <to...@filmworkers.com> wrote:

One more issue with Full Text Query. It seems that it can't handle indexing on a field if one of the docs contain a property that is null if that property is being queried.

CBLTextKey doesn’t allow nil as a parameter. (Note that the backtrace is inside your call to CBLTextKey, and the assertion that failed is “text != nil”.) You’ll need to preflight the text property and not emit anything if it’s nil.

—Jens

[Todd Freese]

Is there no way to do a contains search? *test* does not seem to work. The docs mention doing a prefix like test* which works.

Also, I need to search across multiple properties. Is the correct way to do this is combine them into one string before sending them to CBLTextKey?

Todd

[Jens Alfke]

On Feb 24, 2014, at 10:56 AM, Todd Freese <to...@filmworkers.com> wrote:

Is there no way to do a contains search? *test* does not seem to work. The docs mention doing a prefix like test* which works.

Apparently you can’t search on substrings within words; I think this is a side effect of the way the SQLite FTS indexing works (it’s probably building a sorted list of all the words found in all the documents.) Doing a full substring match would probably require a brute-force linear search. (Couchbase Lite could implement that at the SQL level using a LIKE comparison, but it’d probably be quite slow.)

Also, I need to search across multiple properties. Is the correct way to do this is combine them into one string before sending them to CBLTextKey?

Or you can call emit() multiple times, which is slightly faster.

—Jens

[Todd Freese]

Hmmm.. Thats a problem for me. I have a tableview (using CBLUITableSource) and need to add a search bar that looks at several properties. Searching needs to work on substrings. My dataset is small, so I'm not concerned about performance.

Rather than use FTS for this, maybe a NSPredicate on the datasource might be better? Do you think that would work? Not sure what the predicate string would be to drill down into the rows to get at the data….

Todd

[Jens Alfke]

On Feb 24, 2014, at 1:50 PM, Todd Freese <to...@filmworkers.com> wrote:

Rather than use FTS for this, maybe a NSPredicate on the datasource might be better? Do you think that would work? Not sure what the predicate string would be to drill down into the rows to get at the data….

You mean doing some in-memory filtering of the result set from the CBLQuery? That should work, and NSPredicate would be a pretty clean way to implement it. I think it has a built-in operator for substring matching.

—Jens

[Todd Freese]

Yes. Thinking that since it is in memory, performance should be pretty good on a reasonable about of docs.

Do you think this would work as a subclass of CBLUITableSource and override reloadFromQuery? Although I figured CBLUITableView is observing rows, so if I run the predicate on it, I'm thinking it may cause problems…

Todd

[Todd Freese]

Update: creating my own datasource based on CBIUITableSource works really well.

Todd