Monitoring document changes

[Sebastien ARBOGAST]

I'm experimenting with CouchBase Server, CouchBase Sync Gateway and CouchBase Lite to build a no-backend style application. It's really awesome that data replication between my iOS app and CouchBase server allows me to skip developing a custom API. But there are still things I need to do from the server, like sending emails and iOS push notifications when documents change in certain ways.

Is there any API or technique to monitor data changes on the server side, listen to data change events and do something when certain changes happen? Is it something I should do in Sync Gateway's sync function? Is there a way to do that using one of the client SDKs on CouchBase Server?

[Jens Alfke]

On Jul 17, 2014, at 2:18 PM, Sebastien ARBOGAST <sebastien...@gmail.com> wrote:

Is there any API or technique to monitor data changes on the server side, listen to data change events and do something when certain changes happen? Is it something I should do in Sync Gateway's sync function? Is there a way to do that using one of the client SDKs on CouchBase Server?

Yes, you can do it using the Sync Gateway's “changes feed” API. It’s documented here (that’s actually documentation for Couchbase Lite, but the API is basically the same.)

In a nutshell, you can send

GET /dbname/_changes?feed=continuous

on the admin port and the Gateway will respond with a series of CRLF-delimited lines, each of which is a small JSON object describing a change. The feed will keep running until you close the connection.

If you want to watch only specific channels, add query parameters:

&filter=sync_gateway/bychannel&channels=channel1,channel2,channel3

JChris wrote an example that uses this in a node.js app server, but I don’t think it’s been updated yet for current versions of the gateway (right?)

—Jens

[Matt Ingenthron]

From: Sebastien ARBOGAST <sebastien...@gmail.com>
Reply-To: "mobile-c...@googlegroups.com" <mobile-c...@googlegroups.com>
Date: Thursday, July 17, 2014 at 2:18 PM
To: "mobile-c...@googlegroups.com" <mobile-c...@googlegroups.com>
Subject: Monitoring document changes

I'm experimenting with CouchBase Server, CouchBase Sync Gateway and CouchBase Lite to build a no-backend style application. It's really awesome that data replication between my iOS app and CouchBase server allows me to skip developing a custom API. But there are still things I need to do from the server, like sending emails and iOS push notifications when documents change in certain ways.

Is there any API or technique to monitor data changes on the server side, listen to data change events and do something when certain changes happen? Is it something I should do in Sync Gateway's sync function? Is there a way to do that using one of the client SDKs on CouchBase Server?

At the moment, there is no public interface for monitoring changes from Couchbase Server directly.  There are plans for this in the future and there are some private interfaces that do this now.  

Sync Gateway has an interface for getting changes.  JChris can speak to it better.  You can see some docs he and the team have written.

The private interface I speak of is TAP and there are a few tap clients which are listed as experimental because the interface at Couchbase Server is unsupported.  The things to be cautious of are that use of TAP can cause a lot of IO and there is a feature called checkpointing that you should probably avoid.  They work well, but they need to be used very carefully to not take too many other resources away from other parts of the system.

Hope that helps,

Matt

-- 

Matt Ingenthron

Couchbase, Inc.

[Traun Leyden]

As Jens mentioned, you can follow the _changes feed and react to events to do these things like sending push notifications.

I recently worked on something that watches the TodoLite Sync Gateway database _changes feed to run each image attachment through OCR, and then updates the document with the OCR text:

It's written in Go, and the source is available at github.com/tleyden/todolite-appserver:

In particular, this code block is responsible for registering a callback function that is called whenever there are new changes detected.

By the way, what is your preferred server side language?

On Thu, Jul 17, 2014 at 2:18 PM, Sebastien ARBOGAST <sebastien...@gmail.com> wrote:

I'm experimenting with CouchBase Server, CouchBase Sync Gateway and CouchBase Lite to build a no-backend style application. It's really awesome that data replication between my iOS app and CouchBase server allows me to skip developing a custom API. But there are still things I need to do from the server, like sending emails and iOS push notifications when documents change in certain ways.

Is there any API or technique to monitor data changes on the server side, listen to data change events and do something when certain changes happen? Is it something I should do in Sync Gateway's sync function? Is there a way to do that using one of the client SDKs on CouchBase Server?

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/f6656de8-ba42-4071-8a40-147416271fad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastien ARBOGAST]

My favorite server side languages are Java and Groovy. Groovy support for Couchbase is nowhere and I can use Java libraries in Groovy so I'd settle with Java for now. But since most implementations use NodeJS I'll try to get to know that tech better.

---
Sébastien Arbogast
http://sebastien-arbogast.com

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CACSSHCEfWqTsTxE5DuPscuRBJpX52iWspX_4u%2Bvvkm3ztWOHJA%40mail.gmail.com.

[Traun Leyden]

If you are monitoring the Sync Gateway _changes feed, then you'll need to use a CouchDB library, since the Sync Gateway has a CouchDB compatible API.

I believe Ektorp is currently the most popular option for Java:

https://github.com/helun/Ektorp

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CAG1DMa_%3DrLvwSiHniykXUF2Q2AsYV-EtM3Lh-C1MdwDVRK7E%2Bg%40mail.gmail.com.

[Sebastien ARBOGAST]

What do you mean by "CouchDB library"? I don't understand the link between CouchDB and CouchBase in that context. I mean I know that CouchBase is a combination of CouchDB and Membase. But Why do I need something like Ektorp?

---
Sébastien Arbogast
http://sebastien-arbogast.com

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CACSSHCETT1L%3D5gUVL_M219V%2Bxd3om4Gj_keK_Mdyn8OQMug_%2Bw%40mail.gmail.com.

[Traun Leyden]

The communications protocol that Sync Gateway uses is nearly identical to the one used by CouchDB.  So a library that knows how to talk to CouchDB also knows how to talk to Sync Gateway.

If you want to listen on the Sync Gateway _changes feed, a CouchDB client library should work fine for that.

Couchbase Server doesn't support a _changes feed, and has a (very) different API for tapping into changes, called TAP.  (Matt Ingenthron mentioned this).  I would highly recommend using the Sync Gateway _changes feed rather than trying to interface w/ Couchbase Server directly, especially for your use case of triggering push notifications.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CAG1DMa-06PSgJyuskF4zy06%3DdOS6b9DqQcJghZVjuBZTAhp7qg%40mail.gmail.com.

[Sebastien ARBOGAST]

OK I finally got around to learning the basics of NodeJS and set up a simple program to track changes using iriscouch/follow and it seems to work OK. Now in order to do something useful with it, where can I find more documentation about the API of the change object I get in Follow's callback?

---
Sébastien Arbogast
http://sebastien-arbogast.com

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CACSSHCHiOg_mL%3D7VvyiwcQHT0Ut3_Re7peAcWdxwDTY%2BwBfw9w%40mail.gmail.com.

[Matt Quinn]

On Thu, Jul 24, 2014 at 12:44:54AM +0200, Sebastien ARBOGAST wrote:
> OK I finally got around to learning the basics of NodeJS and set up a
> simple program to track changes using iriscouch/follow and it seems to
> work OK. Now in order to do something useful with it, where can I find
> more documentation about the API of the change object I get in
> Follow's callback?

Since the API is compatible with CouchDB, you can take a look at Couch's
docs for _changes:
http://docs.couchdb.org/en/latest/api/database/changes.html#get--db-_changes

(The various query parameters may or may not work depending on whether
Sync Gateway implements them).

The change object is just a plain Javascript object, deserialized from
the JSON that SG returns. It typically has 'id', 'seq', and 'changes'
fields (where 'changes' is an array of revision IDs).

For change processing, you'll probably want to add "include_docs: true"
to your follow options. With that set, the change object will also have
a 'doc' property containing (what else) the changed document. From there
you should be able to add whatever logic you need to take actions based
on the document that changed.