Replication Push and Pull

[Brian Hardy]

We have and android application which mostly pulls data but can push a small amount too.

I have one sync gateway url and I create replications for push and for pull. I have not done anything to indicate what data is for pull and which is for push. Is there way to do that?

I've tried to clear out some old unwanted data from our couchbase server. Afterwards unfortunately, when the client stars up and connects again it syncs all the old unwanted data back to my couchbase server.

How can I accomplish this?

What is a good way to remove unwanted data from a couchbase server using a sync gateway?

When our application is installed and then starts up for the first time, It takes a long time syncing data. There are about 1800 items in our couchbase server. Why does it take so long for data to get synced on application load after application install? What are some ways I can alleviate this?

Thank you.

[Traun Leyden]

On Tue, May 20, 2014 at 12:29 PM, Brian Hardy <brian...@gmail.com> wrote:

We have and android application which mostly pulls data but can push a small amount too.

I have one sync gateway url and I create replications for push and for pull. I have not done anything to indicate what data is for pull and which is for push. Is there way to do that?

Yes, you can filter on both the push and the pull side.

For the push side, you can call:

database.setFilter("myPushFIlter", new ReplicationFilter() {

            @Override

            public boolean filter(SavedRevision revision, Map<String, Object> params) {

                // .. decide if you want that document pushed or not

            }

        });

For the pull side, you can use Sync Gateway "channels".  See the Sync Gateway Guide for more info.

 

I've tried to clear out some old unwanted data from our couchbase server. Afterwards unfortunately, when the client stars up and connects again it syncs all the old unwanted data back to my couchbase server.

Yep, that's the expected behavior.  You should delete the data via the Sync Gateway REST API instead.

When our application is installed and then starts up for the first time, It takes a long time syncing data. There are about 1800 items in our couchbase server. Why does it take so long for data to get synced on application load after application install? What are some ways I can alleviate this?

As a first step, I would look into filtering the pull via channels.

Which version of Couchbase Lite are you using?  We've been actively working to improve the replication performance -- for example we just recently added the ability to use the _bulk_get sync gateway endpoint, which has the effect of batching several requests into a single request.

[Brian Hardy]

I have tried updating unwanted documents in the past and marked them as deleted = true via sync gateway. I guess you have to write a script to handle marking a lot of documents this way in order to handle large number of documents correct?

After I marked items as deleted my database did not get any smaller. I was under the impression that my queries still had to evaluate the "deleted" data and sill pull it down to the client. I was trying to alleviate some of the work being done evaluating and downloaded so much data. I tried to compact the database. I thought that would remove deleted items. I'm not sure that it did.

[Brian Hardy]

I have some live queries in place. 

Is it possible they are being evaluated as the just installed app is syncing. In the logs I see a handful of documents downloaded and my query evaluating each document, then a short pause, then more documents downloading. Would it be better for me to somehow not start live queries after app gets installed until after all data is synced? How could I do that?

Thanks

[Jens Alfke]

On May 20, 2014, at 5:38 PM, Brian Hardy <brian...@gmail.com> wrote:

After I marked items as deleted my database did not get any smaller. I was under the impression that my queries still had to evaluate the "deleted" data and sill pull it down to the client. I was trying to alleviate some of the work being done evaluating and downloaded so much data. I tried to compact the database. I thought that would remove deleted items. I'm not sure that it did.

No, it doesn't. Deletion is part of the state of a document and needs to be synced, so the database won’t normally forget about a deleted document.

On Couchbase Lite (and CouchDB) there is a “purge” command that completely wipes out a document, removing all traces of it from the database. This isn’t yet implemented in the Sync Gateway, though. (I hate to say this, but you can emulate a purge by using the Couchbase Server’s admin API to go into the gateway’s bucket and delete the documents. Don’t do this while the gateway is running, though.)

—Jens

[Brian Hardy]

Thanks Jens,

I had deleted items from the couchbase server admin like that in the past. I don't recall if I shut the sync gateway down, but I was happy I was able to delete the data. Of course when I reconnected a client with all that same data on it, it replicated it right back to the server. Would I have to ensure all that data is off the client as well? If I added a filter for push I would assume the client would stop syncing everything back to the server. Does that sound like a good approach?

In order to handle deleting lots of data from the couchbase server efficiently, I guess you'd have to write a program to delete the items rather than delete them one at a time from the web app with a refresh after each delete. Is there any other way to purge just some items from the couchbase server? 

As I test our app and keep modifying document contents our database grows and grows and we have long start up times for initial data sync when the app is installed. As I mentioned I was trying to cut down some of that data or speed the initial sync. Our sync gateway is also logging most information do you think that is slowing down the initial data sync much? 

We have live queries. On initial data synchronization, do you think our queries running as data is downloaded is slowing the initial data synchronization as well?

Thanks again