MySQL-style Full Text Search with Objectify and GAE

[Nermin B.]

Dear Group,

 

I have found this article about Full Text Search using MySQL?

http://blog.marceloaltmann.com/en-using-the-mysql-fulltext-index-search-pt-utilizando-mysql-fulltext/

I am wondering if Objectify provides a similar functionality?

 

What I need is something like this:

 

mysql> SELECT * FROM articles

    -> WHERE MATCH (title,body) AGAINST ('database');

+----+-------------------+------------------------------------------+

| id | title             | body                                     |

+----+-------------------+------------------------------------------+

|  5 | MySQL vs. YourSQL | In the following database comparison ... |

|  1 | MySQL Tutorial    | DBMS stands for DataBase ...             |

+----+-------------------+------------------------------------------+

 

 

Few questions:

1. Currently I use two separate Queries: (Q1) filter on title, (Q2) filter for body, and merge them afterwards. - Is this the right way to go?

2. Is there a way to handle small and capital letters with objectify/GAE? (e.g. database, DataBase, Database, data-base) ... I need to find all of them.

3. Is there a way to handle different word endings? (database, databases, etc.)

4. Is there a way to handle synonyms? (DB, MySql, Oracle, etc.) .... these are all synonyms for the database. Probably I need to store synonyms in a separate entity etc. ... what would be an appropriate strategy?

 

Thank you for your help!

 

Nermin

 

[Alejandro Gonzalez]

Hi Nermin,

Objectify has nothing to do with full text search, because datastore itself don't provide full text search capabilities.

I highly recommend you to use the Search API for this kind of things.

https://developers.google.com/appengine/docs/python/search/ (python)

https://developers.google.com/appengine/docs/java/search/ (java)

With the Search API you can do what you want in just one query

Regards,

Alejandro

[Nermin B.]

Dear Aejandro,

thank you for your fast answer to my question. Yes you are right. The Search API is the solution to go for.
However, I have very small documents which I need to look at.
Search API is probably too large for me and it seems to have only boolean results (String found/not found). I have not found if it can respond with probabilities.

I decided to stick with my own solution on Datastore:
I have implemented a simple keyword indexing using a list of keywords stored in a dedicated entity. (this is similar to what search API does.)
I apply text augmentation (put similar words that are missing in the original text). I use my own implementation here which is very domain specific.
Finally I put everything on small characters and filter special characters and stop-words.

For the search purpose I use query that implements "%Like%" pattern:
http://code.google.com/p/objectify-appengine/wiki/FrequentlyAskedQuestions#How_do_I_do_a_like_query_%28LIKE_%22foo%%22%29

Thank you for your help.
I hope this post will help someone else searching for a similar solution.

Best regards:

Nermin