TouchDB-Lucene is here finally!

[Sameer Segal]

Hi, I just ported CouchDB-Lucene to work with TouchDB-Android. GitHub repo: https://github.com/ArtooTrills/TouchDB-Android-Lucene. 

Thanks Marty and Jens for all the help!

Welcome feedback, comments and pull requests!

Best, Sameer

[Marty Schoch]

This is great! I look forward to giving it a try.

Also, I added the implementation of POST to _all_docs that you
mentioned in the README. So that should simplify the instructions for
new users.

marty

[Marty Schoch]

I think one thing we should add is support for external handlers (at
the global, db, and design doc levels). This would allow your plugin
to register at "_fti" just like CouchDB Lucene. That way CouchApp
style apps have access to the same functionality. Also it makes
integration with higher-level libraries a bit easier. One I found
after a quick search is:

https://github.com/ldriscoll/ektorplucene

marty

[Sameer Segal]

Hi Marty, I have limited time at the moment. I have created an issue: https://github.com/ArtooTrills/TouchDB-Android-Lucene/issues/1 and will get down to solving it in the coming weeks. Do write your thoughts on how this can be done -- using URLStreamHandlerFactory?

- Sameer

[Marty Schoch]

On Mon, Aug 27, 2012 at 1:51 PM, Tom Schuller <t...@schuller.lu> wrote:
> Hy.
> can anybody help me to get a TouchDB-Android-Lucene application running?
>
> I have problems to create the design document to index documents.
> If I got it running, I can write a step-by-step quide to help other users to
> start using TouchDB-Android-Lucene.
>
> Thanks,
> Tom

Can you describe the problems you are having? Are there errors in
Eclipse or in Logcat?

marty

[Sameer Segal]

Hi Tom,

Sorry I haven't gotten down to doing this.  Can you paste your design document here and let me see if I can help you. Just keep the following points in mind:

• Ensure you create a design document before you make a Lucene request
• Verify that your ddoc actually exists and its content is right
• There is a difference between ddocName ("test") and ddocId ("_design/test")
  
• Try your ddoc again couchdb-lucene (server) and see if you are getting the right results
• When you are making a query, ensure your query string looks like "name:Tom" where name is the field name of the index.

As Marty mentioned, please provide us logcat and other details for us to help you.

Best, Sameer

[Tom Schuller]

The problem is the creation of the dDoc.
Sorry, I'm sure it's a stupid question.

Could you post an example how to create a DesignDoc with the fulltext attribute?
I tryed with the DesignDoc class without success.

Thanks,
Tom

[Sameer Segal]

On Wed, Aug 29, 2012 at 2:44 PM, Tom Schuller <t...@schuller.lu> wrote:

The problem is the creation of the dDoc.
Sorry, I'm sure it's a stupid question.

Could you post an example how to create a DesignDoc with the fulltext attribute?
I tryed with the DesignDoc class without success.

Thanks,
Tom

Tom,

Supply the ddoc content (in javascript) through a PUT request using the TouchDBHttpClient as follows:

httpClient.put("/grocery-sync/_design/common", "{\"fulltext\": {\"byContent\": {\"analyzer\":\"NGRAM\",\"index\": \"function(doc) { var ret = new Document(); ret.add(doc.text,{'field':'text'});  return ret; }\"}}}");

You can use this with the TouchDB-Android-Lucene instructions on AndroidGrocerySync.

 Best, Sameer

[Sameer Segal]

Hi Angshuman,

Quick thoughts:

• Design Doc
• Don't specify a value for analyser in your design doc and use defaults. In your query, supply Chris*.
• Can you GET the "Chris" record and confirm the JSON structure should look like { data : { name: "Chris" } } (thats what you are indexing)
• Don't mention mydb in the design doc id (It automatically gets added in the TDConnector class
• Use Luke (http://code.google.com/p/luke/) when you are debugging Lucene indexes (use adb pull command to get the files)

Try these changes and let me know

Best, Sameer

On Tue, Nov 6, 2012 at 3:19 PM, Angshuman Sarkar <angshu...@gmail.com> wrote:

Hi,

I am trying to get TouchDB-Lucene working in Android. While I can get the Couch+Lucene query working in my machine independently, I can't get it to work on Android. The search result returns zero documents always.

Maybe I am missing something obvious! Any help would be greatly appreciated. 

This is what I did:

0) create Database: "mydb".

 

1) Created a design document

<pre>

TDServer localServer = new TDServer(<filesDir>);

TouchDBHttpClient touchDBHttpClient = new TouchDBHttpClient(localServer);

CouchDbInstance dbInstance = new StdCouchDbInstance(touchDBHttpClient);

mydb = dbInstance.createConnector("mydb", true);

touchDBHttpClient.put(

 "/mydb/_design/records", "{\"fulltext\": " +

 "{\"byName\": { \"analyzer\":\"NGRAM\"," +

 "\"index\": \"function(doc) { if (doc.data) { var ret=new Document(); ret.add(doc.data.name); return ret; } return null; } \"}}}"

); 

</pre>

2) Created a few documents

<pre>

mydb.create( <someGUID>, new Record("Chris") )

..

..

</pre>

3) Tried to query using lucene

<pre>

  lucene = new TDLucene(localServer);

  TDLuceneRequest req = new TDLuceneRequest();

  req.setUrl("/local/mydb/_design/records/byName").addParam("q", "Chris").addParam("include_docs", "true").addParam("highlights", "5");

  lucene.fetch(req, new Callback() { ... } );

</pre>

Unfortunately, the response object on the callback's onSuccess method always return 0 rows! 

I also tried to use "standard analyzer" while querying without success. I have tried to debug the DatabaseIndexer.search(TDLuceneRequest) method, and found that the IndexSearcher returns TopDocs.totalhits always 0. 

I am using Eclipse JUNO and have tried using both Android 2.2 (api 8) and 4.1 (api 16)! I noticed that on DDMS view, "/mnt/sdcard/indexes" have sub directories, corresponding to the directories "DatabaseIndexer" creates but does not contain any lucene index files. I wonder why? I have given WRITE_EXTERNAL_STORAGE user permission to the app. Below is some log that look strange from logcat.

***************************************************************************************************************

11-06 14:58:12.550: I/com.github.rnewson.couchdb.lucene.Config(338): Index output goes to: /mnt/sdcard/indexes

11-06 14:58:18.170: D/dalvikvm(338): GC_FOR_MALLOC freed 3398 objects / 436936 bytes in 146ms

11-06 14:58:21.480: D/dalvikvm(338): GC_FOR_MALLOC freed 4856 objects / 464536 bytes in 117ms

11-06 14:58:21.900: D/com.github.rnewson.couchdb.lucene.DatabaseIndexer.events(338): org.apache.lucene.store.NIOFSDirectory@/mnt/sdcard/indexes/774d1ad6-c751-41db-818d-42b088833d8d/8dmlqqj38mht0oas0xqabylqv lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@44f84be8 bumped since to start

11-06 14:58:23.510: I/dalvikvm(338): Could not find method java.lang.management.ManagementFactory.getPlatformMBeanServer, referenced from method org.apache.lucene.util.RamUsageEstimator.<clinit>

11-06 14:58:23.520: W/dalvikvm(338): VFY: unable to resolve static method 5764: Ljava/lang/management/ManagementFactory;.getPlatformMBeanServer ()Ljavax/management/MBeanServer;

11-06 14:58:23.520: D/dalvikvm(338): VFY: replacing opcode 0x71 at 0x01ef

11-06 14:58:23.540: D/dalvikvm(338): VFY: dead code 0x01f2-025b in Lorg/apache/lucene/util/RamUsageEstimator;.<clinit> ()V

11-06 14:58:24.090: D/com.github.rnewson.couchdb.lucene.DatabaseIndexer.events(338): paths: {_design/records/byName=View[name=_design/records/byName, digest=8dmlqqj38mht0oas0xqabylqv]}

11-06 14:58:25.340: D/dalvikvm(338): GC_FOR_MALLOC freed 5011 objects / 325440 bytes in 143ms

11-06 14:58:28.940: W/com.github.rnewson.couchdb.lucene.DatabaseIndexer.events(338): c0b6700c-ecf0-4e1a-9d45-7c561d814f7a caused No Context associated with current Thread  

***************************************************************************************************************

 

--
 
 

[Angshuman Sarkar]

Hi Sameer,
Thanks for your reply.

1. Tried without "analyzer" in design document. Also tried with NGRAM, standard, keyword etc. Also tried with "chris*" query. Didn't have any effect. :(

2. Queried "Chris" record and checked structure. Tried using a simpler document structure. Also, I know that the document structure works. I synched this with a couchdb server from local touchdb and ran a temp view. Even tried with couchdb-lucene with the synced design document. The design document works fine when queried via couch-lucene. 

3. Tried with not putting the "mydb" name in the design document url. I am not sure. The "TouchDBHttpClient.urlFromUri( )"  simply appends "touchdb://" as prefix to the supplied uri. The touchDBHttpClient.put() method just does a "PUT" command to the requested uri with the content. I tried several ways (e.g. "_design/.." etc) , and I was getting a "no such view: .. 404" response when I was trying to do a lucene query. Eventually I settled on touchDBHttpClient.put("/mydb/_design/record/", "{ .... }");

- btw, I know that the design document is processed because if I deliberately put a wrong code in the "index" function (like d.data) then I get an exception.

What I feel is not happening is creation of the index files. I can see the "/mnt/sdcard/indexes" folder and I can see  directories created by the DatabaseIndexer. But I do not see any index file being
written in those lucene index directories (for example: under /mnt/sdcard/indexes/e2a5d616-8ae5-48ec-9546-54905e8792c0/22r6ptpnoj1toxpvu3pwjf07k/)

My test AVD is created with "SD card support with 512M ram and size". I also came across several threads regarding files not being written to sdcard and followed advice of creating a sdcard (using mksdcard command) and using that sdcard but haven't had any success.

As I mentioned earlier, my logcat shows a warning like "11-07 15:27:24.363: I/dalvikvm(1704): Could not find method java.lang.management.ManagementFactory.getPlatformMBeanServer,
referenced from method org.apache.lucene.util.RamUsageEstimator.<clinit>". 

Googling it led me to this link

>  http://mail-archives.apache.org/mod_mbox/lucene-dev/201012.mbox/%3c001c01cb9465$730c1310$59243930$@thetaphi.de%3e

But lucene used in touchdb-android-lucene is 3.6, so this should not be the case!

Any thoughts or suggestions?

thanks and regards
~angshu