Search from Couchbase Lite to Couchbase Server

[CouchbaseLover]

Full Text Search - https://github.com/couchbase/couchbase-lite-ios/wiki/Full-Text-Search

Am i correct when Full Text Search is only for searching on the Device's local DB ( Couchbase Lite Database) , if so..how do you search correctly on Couchbase Server directly?

We search on the Couchbase Server database directly and get everything except attachments with ALOT of our own written logic (filtering _sync/id rev, same stuff as if we would REST API to our "shadowing bucket" directly  that already does that job for us), if we're already doing the "best solution" then we have to write a bunch of code for search.

What's the right way to go Couchbase Gods? Please guide us :)

Best Regards

CouchbaseLover 

[Jens Alfke]

On May 26, 2014, at 6:23 AM, CouchbaseLover <sharess...@gmail.com> wrote:

Am i correct when Full Text Search is only for searching on the Device's local DB ( Couchbase Lite Database)

Yes.

if so..how do you search correctly on Couchbase Server directly?

Hm. I believe there is a Lucene plugin available for Couchbase Server. I’m not an expert on that area, though.

—Jens

[Marty Schoch]

There is an adapter to integrate Couchbase Server with Elasticsearch.  You can find more about this here: http://www.couchbase.com/couchbase-server/connectors/elasticsearch

marty

[CouchbaseLover]

Thanks for the response and yes elasticsearch is the way to go i think, but we can still not come up with any solution to handle attachments when you search :(

[Marty Schoch]

On Sun, Jun 1, 2014 at 6:34 PM, CouchbaseLover <sharess...@gmail.com> wrote:

Thanks for the response and yes elasticsearch is the way to go i think, but we can still not come up with any solution to handle attachments when you search :(

I'm not sure I fully understand.  Could you explain more about how you would want it to work? 

Thanks,

marty

[CouchbaseLover]

Hey! Thanks for caring, appreciate it! :)

We have installed the Couchbase Elasticsearch Plugin and are up and running with Elasticsearch. Still talking to the REST API to Couchbase SDK though (can't remember if we should do the rest api to couchbase sdk or elasticsearch (and then XDCR handles the rest, but that's fine cause you can just read about that (good documentation on Elasticsearch and Couchbase). 

My Question is however, how do i make our iOS app compatible with search that has attachment's in them (Documents with attachments), we got searching for documents to work a long time ago but with attachment it's another story. 

We are using shadowing and the other bucket do indeed get attachments as well (if we want to search on the other bucket instead).

I belive we have understood something fundamentally wrong with the whole attachment stuff when it comes to searching. This is what we know: 

1. The attachment linked to a document is in another document within the database. (100%)

2. The shadowing bucket does not receive the attachment documents but only the documents that has a link to the attachment documents. (50%)

3. You simply can't search and get attachments cause the attachment filesystem is not reachable within the Couchbase SDK even though we search in the non shadowing bucket(the bucket Couchbase Lite talks to) (10 %)

I am sorry if this question is badly structured, it's very early here. We can't wait to finally release our app that we have worked on for a long time with Couchbase (since alpha) and congratulations again on 1.0, know how hard you all have worked and great support.

[Marty Schoch]

I guess what I don't understand is how attachments relate to search here.  Generally you search through documents, and get back matching document IDs.  Those documents could have attachments, or not, it doesn't really matter.

Are you saying you want to search through the content of the attachments?  If so, then unless the attachments are also JSON you are right, it may not be easy to accomplish this.

I think it may be that you are saying you can't accomplish what you want in a single call, because the documents, attachments, and search index are stored separately.  That may be the case.  I was assuming these would be done separately.  First find the document IDs satisfying your search.  Then separately acquire the documents and their attachments.  Its possible a single API could consolidate this in the future, but often times you want to perform these steps separately anyway.  For example, you might want to show the user search results, but not yet download attachments because they are large.  Or maybe you want to lazily download them while the user is browsing the results.  Having separate APIs gives you more flexibility here.

Or possibly I'm still not getting what you're saying, in which case feel free to try and explain again.

marty

--

You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/c2d730fe-eda3-4100-8ed5-52316b6dd795%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[CouchbaseLover]

You just made everything much more clearer and answered a lot of questions with that we should search seperately instead of "a single API".

But thing is, we still don't know how you can reach an attachments on a "shadowed" bucket.

Ok il try to make this clear with an example.

1. We create a document with an attachment attached to it, that document get's saved on Bucket A (the bucket that has all the sync gateway meta data, "_sync" channels etc)

2. Trough shadowing we get the same document on Bucket B but without the sync gateway meta data and still the attachment looking like this:

{

  "_attachments": {

    "xxxxx": {

      "content_type": "image/jpeg",

      "digest": "sha1-m0sikESS4UzfSh77N9yltJrjyRI=",

      "length": 32419,

      "revpos": 1,

      "stub": true

    }

3. Now we want to search for the "shadowed" document on Bucket B and find attachments connected to it, how is this possible when shadowing does not "replicate" the attachment connected to the document we are searching for, we only see the attachment in Bucket A still with a "_sync:att:sha1-m0sikESS4UzfSh77N9yltJrjyRI=" connected to it. Should we not see the attachment in horseweb also and without the _sync? 

Question: What's the point searching on Bucket B if it indirects searches in Bucket A for the attachment, cause that's the only place it is stored?

CouchbaseLover

[Marty Schoch]

So in your example I would expect that you would configure bucket B to be sent to the couchbase-elasticsearch plugin.

I would then expect searches to be sent directly to the elasticsearch index.

The results of this search would be document IDs, and any other fields you chose to "store" in the Elasticsearch index.  These stored fields could include attachment names.

Now that you have the matching document IDs (and possibly also attachment filenames) you could make requests to the gateway to retrieve the documents and/or their attachments.

Some of your wording still confuses me.  You would choose to index bucket B, because it contains the data you want to search.  To actually retrieve the attachment content, you have to make a request to bucket A (or the gateway) because bucket A is the only place the attachments are stored.

marty

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/4b010b2b-be0e-4eab-a4f6-a68838e702a9%40googlegroups.com.

[Jens Alfke]

On Jun 4, 2014, at 12:14 AM, CouchbaseLover <sharess...@gmail.com> wrote:

1. The attachment linked to a document is in another document within the database. (100%)

2. The shadowing bucket does not receive the attachment documents but only the documents that has a link to the attachment documents. (50%)

Right.

3. You simply can't search and get attachments cause the attachment filesystem is not reachable within the Couchbase SDK even though we search in the non shadowing bucket(the bucket Couchbase Lite talks to) (10 %)

I don’t know what you mean by “attachment filesystem”. The gateway doesn’t store attachments in the filesystem; they’re just extra docs in the bucket. The document ID starts with “_sync:att:” and is followed by the SHA-1 digest of the attachment data. (Ordinarily you shouldn’t mess with the docs in the gateway’s private bucket, but I agree that this is one case where you have to, in order to access the attachments directly from Couchbase.)

—Jens

[CouchbaseLover]

Thanks for the clarification.

Should we skip searching with attachment cause it seems like a hassle, and maybe wait for a feature that covers this from you guys? 

Regards CouchbaseLover

[J. Chris Anderson]

On Sunday, June 8, 2014 4:59:21 PM UTC-7, CouchbaseLover wrote:

Thanks for the clarification.

Should we skip searching with attachment cause it seems like a hassle, and maybe wait for a feature that covers this from you guys? 

If I wanted to do full text search on attachments in the cloud I would create a channel of documents to index, and write a little node.js script to index the documents it emits. I guess my script would load the document and then put it in a full text indexer.

There's also an Elastic Search plugin for Couchbase Server that might be able to handle the attachments, though I'm not sure what APIs it has for binary documents.

Chris

[CouchbaseLover]

Bump!

I pretty much asked the same question some days ago and you responded that i should make an REST API request to 

server/db/attachment(http://xxxx:xxx/bucket/4CE32C17-43AF-4EC5-B422-551BE730AD72/AttachmentName)

but i get the error 401.

How can i be Unauthorized when i solved that issue months ago and it's been working forever since, (pushing and pulling with continuous)

basicAuthenticatorWithName: (NSString*)name

              password: (NSString*)password;

The response i get is also different from the docs "http://developer.couchbase.com/mobile/develop/references/couchbase-lite/rest-api/document/get---db---doc---attachment-/index.html#example

"

 "Content-Length" = 50;

    "Content-Type" = "application/json";

    Date = "Thu, 30 Oct 2014 13:01:22 GMT";

    Server = "Couchbase Sync Gateway/1.00";

    "Www-Authenticate" = "Basic realm=\"Couchbase Sync Gateway\"";

"

 --CouchbaseLover

[Jens Alfke]

On Oct 30, 2014, at 7:10 AM, CouchbaseLover <sharess...@gmail.com> wrote:

I pretty much asked the same question some days ago and you responded that i should make an REST API request to 

server/db/attachment(http://xxxx:xxx/bucket/4CE32C17-43AF-4EC5-B422-551BE730AD72/AttachmentName)

but i get the error 401.

How can i be Unauthorized when i solved that issue months ago and it's been working forever since, (pushing and pulling with continuous)

Setting the replicator's authenticator only tells the replicator how to authenticate. If you're making your own request using NSURLConnection, you'll need to handle authentication yourself as described in the NSURLConnection delegate documentation.

The response i get is also different from the docs "http://developer.couchbase.com/mobile/develop/references/couchbase-lite/rest-api/document/get---db---doc---attachment-/index.html#example

Well, it's different because it's an error, not a success.

—Jens

[CouchbaseLover]

Hey again, new week new question.. ;)

I will try to make this as clear as possible!

Example:

Let's say i have a user profile and that document has properties that connect to other documents. 

type: user

friend1:documentID. <- the friends document id.

Now this works perfectly and you can see all the related document's propertys and we have gotten this to work since alpha, but how do you achieve this when searching like we are, so when you search you not only search for a document but the related content as well. Like it would work with Node.JS + Couchbase Server..how do you achieve this result with Couchbase Lite?

Like it would work in a normal social app, do you have that option? Obviously not everything should sync down on your device? The best would be to use the gateway to pull and sync but don't store the 1000 documents with attachment that it could be in a "Instagram scenario".

Best Regards,

CouchbaseLover

[Jens Alfke]

On Nov 20, 2014, at 3:27 PM, CouchbaseLover <sharess...@gmail.com> wrote:

Now this works perfectly and you can see all the related document's propertys and we have gotten this to work since alpha, but how do you achieve this when searching like we are, so when you search you not only search for a document but the related content as well. Like it would work with Node.JS + Couchbase Server..how do you achieve this result with Couchbase Lite?

I'm sorry, I really don't understand what you're asking. What do you mean by "searching like we are", or "but the related content as well" or "Node.JS + Couchbase Server"?

—Jens

[CouchbaseLover]

As i told previously in this thread, we are searching directly to Couchbase Server. So when we search for users for example, we do not store the result (DOCUMENTS) on the device. <- This is fully working!

Now i also want to get all the RELATED data to that user(document) that i search for. Is that also unclear?

Of course i get the data that is "static" like 

--

type:user

age: 30 <- static

gender: male <- static

--

friend: 12ASDER431231AD <- Document ID that is also a user document. How do i get that document id's static info within that one search? This works perfectly in Couchbase Lite when they are both in the same channel (the user and the friend), how do you do it here?

Best Regards,

CouchbaseLover

[Jens Alfke]

On Nov 20, 2014, at 4:33 PM, CouchbaseLover <sharess...@gmail.com> wrote:

friend: 12ASDER431231AD <- Document ID that is also a user document. How do i get that document id's static info within that one search?

So this is a question about querying the server, not a Couchbase Lite question?

View rows can't combine data from multiple documents. You'll have to separately request the related document from the Sync Gateway using the REST API.

—Jens