Filter a large list of data

[Raffi Senerol]

Hello!

I'm trying to filter a large list of guests (about 15K) with a combination of different criteria (VIP, Checked-In, Waitlist, etc.) but YapDatabaseFilteredView does not give great performances. I tried to use Full Text Search feature but I had some non solved issues (see post: https://groups.google.com/forum/#!topic/yapdatabase/jMqkTgcmyL4).

So, I've been testing with YapDatabaseSecondaryIndex and YapDatabaseQuery, the performances are much better! However, I could not find an easy way to use the results returned by the query with a table view. Seems like it's not possible to use YapDatabaseMappings in this case, right?

Am I going in the right direction? Is there a different way to filter a large list of data efficiently?

Thanks!

Raffi

[Chris Ballinger]

I think if you could make a generic version of YapDatabaseSearchResultsView that could be applied to YapDatabaseSecondaryIndex and possibly YapDatabaseRTreeIndex it would be helpful for a lot of people. We use filtered views extensively in iBurn for event filtering (removing expired events, filtering by multiple event type selections) and the performance is still a bit slow on older devices (~ 7-10k events). We sped it up by ensuring that you're using a Whitelist for the proper collection, and grouping events by day, so the final filtering is only over a smaller subset of data. 

We also use RTreeIndex w/ GPS coords to find objects nearby, and then manually create an array for use in a table view:

https://github.com/Burning-Man-Earth/iBurn-iOS/blob/master/iBurn/BRCNearbyViewController.swift#L50

https://github.com/Burning-Man-Earth/iBurn-iOS/blob/master/iBurn/BRCSortedViewController.swift#L144

I don't think the FilteredView tricks are applicable in your case, so exploring a YapDatabaseSecondaryIndexView extension based on YapDatabaseSearchResultsView would probably be your best bet if you're looking for a similar interface as far as Mappings. If you do make it, please submit it upstream! :)

--
You received this message because you are subscribed to the Google Groups "YapDatabase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yapdatabase+unsubscribe@googlegroups.com.
To post to this group, send email to yapda...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raffi Senerol]

I ended up using a forked version of YapDatabase that implements a generic version of YapDatabaseSearchResultsView that can be applied to YapDatabaseSecondaryIndex: https://github.com/ngocluu/YapDatabase

However, I found out few bugs in there that I've fixed and also wrote the unit tests related to this new feature. You can find the forked project here: https://github.com/SplashThat/YapDatabase  (branch named secondaryindex-search-2.7.3)   

I'm using YapDatabaseSearchResultsView with YapDatabaseSecondaryIndex for filtering a large list of data now and it is so much faster!

To unsubscribe from this group and stop receiving emails from it, send an email to yapdatabase...@googlegroups.com.

[Chris Ballinger]

Can you submit a pull request?

To unsubscribe from this group and stop receiving emails from it, send an email to yapdatabase+unsubscribe@googlegroups.com.