Why Couchbase can't use RAM from disk cache?

[Andrey Nikishaev]

Couchbase v1.8, CentOs 5.6

When disk cache fill all available memory, Couchbase starts to save data on disk instead of using memory that allocated to disk cache. And this is cause performance problems on my application.

Also i don't know why disk cache is used when all data saved in memory, and there are no disk read? 

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 3:56 AM, Andrey Nikishaev <cre...@gmail.com> wrote:

Couchbase v1.8, CentOs 5.6

When disk cache fill all available memory, Couchbase starts to save data on disk instead of using memory that allocated to disk cache. And this is cause performance problems on my application.

By default 1.8 picks it's ram quota trying to leave only 1 gig to OS, it's own executable code and etc. So there's typically not much page cache used in 1.8 installations.

BTW 2.0 is very different. It's evolving technology and especially when you use views, it actually relies quite heavily on reasonably large page cache. Even in pure key-value use case, my understanding is that you shouldn't allocate as much ram to couchbase bucket as in 1.8. And that's why 2.0's default is 60%, but it'll allow you to allocate more.

 

Also i don't know why disk cache is used when all data saved in memory, and there are no disk read? 

From your explanation I couldn't understand your problem. Can you elaborate ? You can send me direct email in Russian if that helps.

[Andrey Nikishaev]

Few days ago a got problem with Couchbase server. When i look at the stats i saw that server had no free RAM. 30% was filled with Couchbase, and else 70% with disk cache. Normally applications can use memory that allocated for disk cache but Couchbase seems to me not, because it started to save data directly to the disk. Because of this greatly increased disk reads and performance goes down.

For now i setup cron to flush disk cache every day. But by the day disk cache growing very vest. Server only used for Couchbase and don't have any disk heavy operation. So problem that Couchbase don't reuse memory allocated for disk cache as it should. 

[Andrey Nikishaev]

Here is screenshots.

Other data is a disk cache by the day.

http://gyazo.com/622501aa4e1bbc4fefe713046a0938ed

Server stats 
http://gyazo.com/d49fe573f5ebe6adf23462b50e9b2051

[Andrey Nikishaev]

Also after i clear cache Couchbase didn't relocate data from disk to memory. Resident stats still showing 80%
http://gyazo.com/82032ba14de2a72aef53dce64f37786b

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 10:56 AM, Andrey Nikishaev <cre...@gmail.com> wrote:

Few days ago a got problem with Couchbase server. When i look at the stats i saw that server had no free RAM. 30% was filled with Couchbase, and else 70% with disk cache. Normally applications can use memory that allocated for disk cache but Couchbase seems to me not, because it started to save data directly to the disk. Because of this greatly increased disk reads and performance goes down.

Couchbase makes eviction decisions not based on amount of ram free but on bucket quota. In case you've added more ram to this box you'll need to raise that quota to use that additional ram.

 

For now i setup cron to flush disk cache every day. But by the day disk cache growing very vest. Server only used for Couchbase and don't have any disk heavy operation. So problem that Couchbase don't reuse memory allocated for disk cache as it should. 

Doing that will not help couchbase or any other application in using this memory but will instead degrade performance. You should _never_ drop page cache in production. That API is intended for benchmarks, testing and development.

[Andrey Nikishaev]

Clearing cache really helps, in other way i got to many errors on get requests.

My server have 24GB Ram. Couchbase using 10GB and i didn't add any additional memory. 

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 11:17 AM, Andrey Nikishaev <cre...@gmail.com> wrote:

Clearing cache really helps, in other way i got to many errors on get requests.

That's weird. Maybe we're speaking about different things. What exactly you're doing ?

 

My server have 24GB Ram. Couchbase using 10GB and i didn't add any additional memory. 

And what is bucket quota ?

[Andrey Nikishaev]

I'm clearing disk cache like this:

echo 3 > /proc/sys/vm/drop_caches

Bucket quota is 20GB, 2 Couchbase server 1.8 with 10GB. I have only one, default bucket.

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 11:46 AM, Andrey Nikishaev <cre...@gmail.com> wrote:

I'm clearing disk cache like this:

echo 3 > /proc/sys/vm/drop_caches

Something is weird. Can you elaborate exactly how it helps? Is that something measurable ?

 

Bucket quota is 20GB, 2 Couchbase server 1.8 with 10GB. I have only one, default bucket.

Hm. Could not understand that. How those 20GB and 10GB are related ?

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 12:04 PM, Aliaksey Kandratsenka <alkond...@gmail.com> wrote:

On Mon, Dec 17, 2012 at 11:46 AM, Andrey Nikishaev <cre...@gmail.com> wrote:

I'm clearing disk cache like this:

echo 3 > /proc/sys/vm/drop_caches

Something is weird. Can you elaborate exactly how it helps? Is that something measurable ?

Also please tell us if there's any virtualization software involved.

[Andrey Nikishaev]

I have cluster with two servers by 10GB each, with only one bucket. So bucket limit on cluster is 20GB. Node quota is 10GB.

Clearing disk cache free memory that Couchbase can use, it must use it even if it used by disk cache, but for some reason not do this.
For example memory filled on 100% and Couchbase started to write data to disk, after i clear disk cache i got 50% free memory and Couchbase started to work normally writing data to memory not to disk.

[Andrey Nikishaev]

No we don't use any virtualization software and hardware.

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 12:10 PM, Andrey Nikishaev <cre...@gmail.com> wrote:

I have cluster with two servers by 10GB each, with only one bucket. So bucket limit on cluster is 20GB. Node quota is 10GB.

You mentioned above that your servers both have 24 gigs of RAM. Is that correct ? If so why per-node bucket quota is just 10 gigs ?

Clearing disk cache free memory that Couchbase can use, it must use it even if it used by disk cache, but for some reason not do this.
For example memory filled on 100% and Couchbase started to write data to disk, after i clear disk cache i got 50% free memory and Couchbase started to work normally writing data to memory not to disk.

That's not how it works. Clean pages of page cache are just like free pages and any reasonably modern OS can reuse them with ease. If explicitly getting rid of page cache helps in your case then there must be something else that's noteable.

Can you quantify that speedup?

[Andrey Nikishaev]

That's not how it works. Clean pages of page cache are just like free pages and any reasonably modern OS can reuse them with ease. If explicitly getting rid of page cache helps in your case then there must be something else that's noteable.

 

I know, but for some reason Couchbase cant use memory that allocated to disk cache.

Yes each server have 24GB, but for Couchbase used only 10GB.

There is no anything else on the server except Couchbase.  

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 12:24 PM, Andrey Nikishaev <cre...@gmail.com> wrote:

That's not how it works. Clean pages of page cache are just like free pages and any reasonably modern OS can reuse them with ease. If explicitly getting rid of page cache helps in your case then there must be something else that's noteable.

 

I know, but for some reason Couchbase cant use memory that allocated to disk cache.

Yes, it's not built for doing that. It'll try hard to never use more then bucket's quotas.

Yes each server have 24GB, but for Couchbase used only 10GB.

Then just bump per-node bucket quota to about 20 gigs and you'll be fine.

I'm pretty sure it'll solve your problem.

Meanwhile, may I have more results on drop_caches thing? Somehow people continue reporting that they use this in production. But there should be no reason at all for doing that. So I'm _really_ curious seeing if you can show us numbers or any other evidence that indeed dropping page cache and other caches (inode, dentry, etc) helps.

[Andrey Nikishaev]

Yes, it's not built for doing that. It'll try hard to never use more then bucket's quotas.

Yep but the quota is 10GB, but it started to write data to disk not when it hit 10GB limit, but when it use only ~5GB and other memory filled with disk cache.
You can check this. Just fill all memory with disk cache and then make load test with Couchbase, you will see that memory resident stats will drop bellow 100%.
That's mean that Couchbase thinks that there are no free memory(but in real server have memory allocated for disk cache and it can be used by Couchbase, but not used).

People clear disk cache for this reason, because they dont want that couchabse started to work from disk, not from the memory.

[Andrey Nikishaev]

Then just bump per-node bucket quota to about 20 gigs and you'll be fine.

I think this will make worse, because disk cache will fill up memory faster, and couchbase will use mostly disk. 

[Aliaksey Kandratsenka]

On Mon, Dec 17, 2012 at 1:03 PM, Andrey Nikishaev <cre...@gmail.com> wrote:

Then just bump per-node bucket quota to about 20 gigs and you'll be fine.

I think this will make worse, because disk cache will fill up memory faster, and couchbase will use mostly disk. 

I can't promise my explanation will be good enough but let me try.

Looks like you misunderstand one key point.

Bucket quota is amount RAM that couchbase is using as it's _own_ cache. There's no need to cache same data in page cache. 2.0 is a bit of different story, but in 1.8.1 you generally just give most of your ram to couchbase and it'll work best in this mode.

[Andrey Nikishaev]

Bucket quota is amount RAM that couchbase is using as it's _own_ cache. There's no need to cache same data in page cache.

Yes. It have two layers of handling data. First is RAM, and then disk. But under disk cache i mean system disk cache, that not handled by Couchbase.  
And when memory of server filled with system disk cache(Couchbase bucket quota can be filed even under 1%) Couchbase will use not RAM, but disk for saving data(this data will be only at disk).

[Aliaksey Kandratsenka]

So you're trying to say, that server will start evicting documents even when couchbase memory usage is well below bucket quota? And you can easily reproduce that?

If that's the case, then it's better to file jira bug and attach os-level and couchbase-level stats and any other evidence.

[Andrey Nikishaev]

So you're trying to say, that server will start evicting documents even when couchbase memory usage is well below bucket quota?

Yes.

Ok. I will send info, after i got fix for client library. 

[Mike Wiederhold]

Andrey,

I've added a comment on the bug you filed. Please review my comments as I would like to get this issue resolved for our 2.0.2 release.

http://www.couchbase.com/issues/browse/MB-7438#comment-51871