Slow replication speeds when changing database from Walrus to Couchbase

[Eric Levine]

I've been trying to capture some data on how long it takes to synchronize document changes between CBL clients.  In my setup, I've created a consumer and producer app, where the producer will continually create and update documents.  In each document, I record a timestamp for when the producer creates it and the consumer receives it.  Here is what I've observed so far:

* SG with Walrus, and both apps, on my laptop, I'm seeing around 200ms for a document to get from the consumer to producer

* SG with Walrus on Azure, and both apps on laptop, I'm seeing around 500ms.  This seems reasonable given there is now network latency involved

* SG on my laptop using Couchbase on AWS, and both apps on laptop, I'm seeing around 6-8 seconds.  

* SG using Couchbase both on the same Azure VM, and both apps on laptop, I'm seeing around 6-12 seconds.  

I'm very concerned about the large increase in time when I switch from Walrus to Couchbase.  Is this normal?  Is there something I should look into changing with regards to my setup?

--

Eric

[Jens Alfke]

On Jan 29, 2015, at 8:00 PM, Eric Levine <levin...@gmail.com> wrote:

I'm very concerned about the large increase in time when I switch from Walrus to Couchbase.  Is this normal?  Is there something I should look into changing with regards to my setup?

That's not normal — a Couchbase Server should be very fast at returning data, unless it's under heavy load.

I'm not knowledgeable on deploying cloud-hosted Sync Gateways, so I don't have any idea how to troubleshoot this. Traun's done the most work on that, I think, so hopefully he can chime in.

—Jens

[Eric Levine]

I wound up installing Couchbase on my development machine to try running everything in one place.  At first, I was getting the same results.  Then, I changed my producer code to wait a little longer in between creating documents.  I'm now seeing those new documents take between 750ms and 1.5s to go from the producer to the consumer.  That is definitely a huge improvement.  Is this what you would expect, or should I be seeing even better times?

I'm using the C# version of CBL, and I think I need to get a better understanding of how to properly use it in a multi-threaded application.  

--

Eric Levune

[Jens Alfke]

On Jan 30, 2015, at 7:32 AM, Eric Levine <levin...@gmail.com> wrote:

I wound up installing Couchbase on my development machine to try running everything in one place.  At first, I was getting the same results.  Then, I changed my producer code to wait a little longer in between creating documents.  I'm now seeing those new documents take between 750ms and 1.5s to go from the producer to the consumer.

That makes it sound like the lag is on the client side. Have you used logging to check when the new documents arrive at the Gateway and on the destination device?

—Jens

[Eric Levine]

I'll have to look closer at the gateway logs to try and track the document there.  I currently have all logging enabled, but what would be the right ones for focusing on this information?  On the destination side, I'm waiting for the database changed event to know when the document has been received.  Does that seem reasonable, or would something else be more accurate?

--

Eric 

[Traun Leyden]

On Thu, Jan 29, 2015 at 9:33 PM, Jens Alfke <je...@couchbase.com> wrote:

I'm not knowledgeable on deploying cloud-hosted Sync Gateways, so I don't have any idea how to troubleshoot this. Traun's done the most work on that, I think, so hopefully he can chime in.

I don't have much to chime in with, because I haven't done any perf testing yet on the cloud-hosted Sync Gateways.

For our internal "perf runner" tests we have at Couchbase, we don't currently benchmark against Walrus, but I will suggest that we add that to our test matrix.

[Eric Levine]

Traun - What I'm really interested in trying to do is get an idea of the Sync Gateway's performance when using Couchbase as the database.  Can you share some of your results from the perf tests you run?  What I'd like to know is the time it takes to sync a document between clients, and from the REST API down to a client. Obviously, there area a lot of variables involved, but it would help to have something to set expectations.

[Traun Leyden]

Hey Eric,

Sorry for the late reply.  

If you look at this performance run result, which is being run against a recent snapshot of the master branch, it's running with 5K "pushers" and 8K "pulllers" (simulated clients pushing docs and doing continuous pull replications), and we're seeing lag times of approx 16.9 seconds -- meaning the time the doc was pushed to Sync Gateway by a pusher, versus the time it appeared on the changes feed of a puller.

That particular data point can be seen by looking at the gateload_expvar2.json file, and looking at this field:

• PushToSubscriberInteractive: 

  {

  • ...
  • p99: 16911357990

  },

which is in nanoseconds, so converting to seconds == 16.9 seconds.

HTH

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/39adaa99-ad87-45ad-ab12-27e5181f93e6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Levine]

Thanks Traun!  That is very interesting data. I've got a few follow up questions:

• Will the number of pushers and pullers directly impact the lag time? 
• If so, could we achieve lag times in the neighborhood of 2 seconds if the number of pushers and pullers are kept to a limit?
• Would I be able to run these tests myself and play around with the parameters?  If so, any pointers to how to get it up and running would be a huge help.  

Thanks again!

--

Eric

[Traun Leyden]

If you wanted to run these tests yourself, you could look at this repo:

https://github.com/couchbaselabs/perfrunner

and you can see what parameters / config files are used by looking here.

• Will the number of pushers and pullers directly impact the lag time? 

I am guessing yes, but I don't know the relationship here.

@Andy -- this is a good one to add to that performance guidance document you are brewing up!

 

• If so, could we achieve lag times in the neighborhood of 2 seconds if the number of pushers and pullers are kept to a limit?

I would think so, yes.  I will try to run some tests and get back to you.

 

• Would I be able to run these tests myself and play around with the parameters?  If so, any pointers to how to get it up and running would be a huge help.  

See above.

Thanks,

Traun

 

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/fe05366e-4748-4350-937c-ccc6dc458711%40googlegroups.com.

[Lukasz Dutka]

Eric,

any updates on the problem? I'm experiencing the similar issues on my test environment. The delay in delivering a document change via Walrus is <100ms but deviling via couchbase server is 2s. 

Regards

Lukasz

[Eric Levine]

There hasn't been any resolution on this.  I think there are some related bugs open regarding this on the Github repo.  As we are doing more with the sync gateway now, I'm seeing times with Couchbase that range from 200ms up to 1500ms.  My suspicion is that there is some sort of delay in the TAP or DCP feed that the Sync Gateway is listening to, which results in the clients being notified.

--

Eric