Huge "Memory Used" but very small meta data, user data and items?!
222 views
Skip to first unread message
Josh Deltener
Oct 10, 2012, 2:36:00 PM10/10/12
to couc...@googlegroups.com
This one has me stumped. We are using couch for our session data and inactive items are set to expire after 2 hours. We have about 11k items, each being about 2k or less.. We see items being cleaned every hour, yet Memory Used keeps going up no matter what?!
Ops per second: 6-13
Memory used: 550M !!
High Water: 471M
Low Water: 337M
Items: 11.2k
Avg Image Size: 2k
User data in RAM: 1.51M
Meta data in RAM: 25.6M
Matt Ingenthron
Oct 10, 2012, 2:57:59 PM10/10/12
to couc...@googlegroups.com
What's the quota and where are these stats coming from? Which platform?
Do be aware that generally the way memory allocation works is that processes don't shrink their memory usage later. I believe couchbase does actually try to return memory, but this will be based on how the actual memory is laid out. We leverage tcmalloc on Linux, for example.
Matt
Chad Kouse
Oct 10, 2012, 3:06:55 PM10/10/12
to couc...@googlegroups.com
On Linux I was able to free the buffer cache using this command:
sync && echo 3 > /proc/sys/vm/drop_caches
Then you can check your ram again and get a better idea of what memory is actually "in use"
Chad Kouse
Aliaksey Kandratsenka
Oct 10, 2012, 3:07:54 PM10/10/12
to couc...@googlegroups.com
On Wed, Oct 10, 2012 at 12:06 PM, Chad Kouse <ch...@tunewiki.com> wrote:
> On Linux I was able to free the buffer cache using this command:
>
> sync && echo 3 > /proc/sys/vm/drop_caches
Which is actually bad idea to do. Top will tell you which memory is
> On Linux I was able to free the buffer cache using this command:
>
> sync && echo 3 > /proc/sys/vm/drop_caches
used by programs and which by page cache and other kernel caches.
Chad Kouse
Oct 10, 2012, 3:12:38 PM10/10/12
to couc...@googlegroups.com
Is it a bad idea for performance reasons or is there potential for data loss/corruption? I used to do it all the time.
Chad Kouse
Aliaksey Kandratsenka
Oct 10, 2012, 3:15:41 PM10/10/12
to couc...@googlegroups.com
On Wed, Oct 10, 2012 at 12:12 PM, Chad Kouse <chad....@gmail.com> wrote:
> Is it a bad idea for performance reasons or is there potential for data
> loss/corruption? I used to do it all the time.
Performance reasons.
> Is it a bad idea for performance reasons or is there potential for data
> loss/corruption? I used to do it all the time.
It's purpose is to get you 'cold' system i.e. for benchmarking. AFAIK
it was never intended for production use.
Chad Kouse
Oct 10, 2012, 3:17:36 PM10/10/12
to couc...@googlegroups.com
Cool
Chad Kouse
Josh Deltener
Oct 10, 2012, 3:17:45 PM10/10/12
to couc...@googlegroups.com
600M quota. Linux, Server 1.8, PHP, this is a couchbase bucket. I'm getting these numbers from the web interface.
We are now getting some OOM errors, even though the max ram we SHOULD be using is around 45M. It's like it's not flushing any items that have expired.
Josh Deltener
Oct 10, 2012, 3:26:07 PM10/10/12
to couc...@googlegroups.com
Unless I'm totally missing something, this tells me items are being removed (2 hour lifetime), but memory is ever increasing.
Josh Deltener
Oct 10, 2012, 3:35:44 PM10/10/12
to couc...@googlegroups.com
Or, are the number if items NOT the total number of items in the bucket? Are there INACTIVE items using RAM?
Josh Deltener
Oct 10, 2012, 3:46:37 PM10/10/12
to couc...@googlegroups.com
Sorry, apparently my screenshots are hidden in the quoted text.
Chad Kouse
Oct 10, 2012, 3:50:01 PM10/10/12
to couc...@googlegroups.com
Something definitely looks wrong there...
--
Chad Kouse
Josh Deltener
Oct 10, 2012, 4:28:46 PM10/10/12
to couc...@googlegroups.com
Maybe this helps? This is a cbstats on the bucket. Still shows very low # of times.
accepting_conns: 1
auth_cmds: 1310318
auth_errors: 0
bucket_active_conns: 1
bucket_conns: 15
bytes_read: 2527807753
bytes_written: 21551421792
cas_badval: 0
cas_hits: 0
cas_misses: 0
cmd_flush: 0
cmd_get: 1310336
cmd_set: 732836
conn_yields: 9724
connection_structures: 492
curr_connections: 52
curr_conns_on_port_11209: 37
curr_conns_on_port_11210: 13
curr_items: 3437
curr_items_tot: 7056
daemon_connections: 4
decr_hits: 0
decr_misses: 0
delete_hits: 0
delete_misses: 0
ep_bg_fetched: 7610
ep_bg_load: 11381218
ep_bg_load_avg: 1495
ep_bg_max_load: 869726
ep_bg_max_wait: 366059
ep_bg_min_load: 12
ep_bg_min_wait: 29
ep_bg_num_samples: 7610
ep_bg_wait: 2249686
ep_bg_wait_avg: 295
ep_commit_num: 561118
ep_commit_time: 0
ep_commit_time_total: 29479
ep_data_age: 4
ep_data_age_highwat: 9
ep_db_cleaner_status: complete
ep_db_strategy: multiMTVBDB
ep_dbinit: 1
ep_dbname: /opt/couchbase/var/lib/couchbase/data/SessionCache-data/SessionCache
ep_dbshards: 4
ep_diskqueue_drain: 1987475
ep_diskqueue_fill: 1987340
ep_diskqueue_items: 0
ep_diskqueue_memory: 0
ep_diskqueue_pending: 0
ep_exp_pager_stime: 3600
ep_expired: 343513
ep_flush_all: false
ep_flush_duration: 0
ep_flush_duration_highwat: 5
ep_flush_duration_total: 29799
ep_flush_preempts: 0
ep_flusher_deduplication: 244338
ep_flusher_state: running
ep_flusher_todo: 0
ep_inconsistent_slave_chk: 0
ep_io_num_read: 7610
ep_io_num_write: 1370414
ep_io_read_bytes: 25491967
ep_io_write_bytes: 2141110647
ep_item_begin_failed: 0
ep_item_commit_failed: 0
ep_item_flush_expired: 514723
ep_item_flush_failed: 0
ep_items_rm_from_checkpoints: 2500569
ep_keep_closed_checkpoints: 0
ep_kv_size: 1013729
ep_latency_arith_cmd: 0
ep_latency_get_cmd: 1317946
ep_latency_store_cmd: 732836
ep_max_data_size: 209715200
ep_max_txn_size: 1000
ep_mem_high_wat: 157286400
ep_mem_low_wat: 125829120
ep_mem_tracker_enabled: true
ep_min_data_age: 0
ep_num_active_non_resident: 3332
ep_num_checkpoint_remover_runs: 153921
ep_num_eject_failures: 0
ep_num_eject_replicas: 134124
ep_num_expiry_pager_runs: 213
ep_num_non_resident: 6822
ep_num_not_my_vbuckets: 0
ep_num_pager_runs: 18740
ep_num_value_ejects: 241253
ep_onlineupdate: false
ep_onlineupdate_revert_add: 0
ep_onlineupdate_revert_delete: 0
ep_onlineupdate_revert_update: 0
ep_oom_errors: 0
ep_overhead: 18055081
ep_pending_ops: 0
ep_pending_ops_max: 0
ep_pending_ops_max_duration: 0
ep_pending_ops_total: 0
ep_queue_age_cap: 900
ep_queue_size: 0
ep_storage_age: 144
ep_storage_age_highwat: 1110
ep_storage_type: featured
ep_store_max_concurrency: 10
ep_store_max_readers: 9
ep_store_max_readwrite: 1
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_keepalive: 300
ep_tmp_oom_errors: 4150
ep_too_old: 1
ep_too_young: 0
ep_total_cache_size: 486054480
ep_total_del_items: 574244
ep_total_enqueued: 1987340
ep_total_new_items: 581300
ep_total_persisted: 1944658
ep_uncommitted_items: 0
ep_value_size: 371667
ep_vb_total: 684
ep_vbucket_del: 0
ep_vbucket_del_fail: 0
ep_version: 1.8.1_937_rel
ep_warmed_up: 0
ep_warmup: true
ep_warmup_dups: 0
ep_warmup_oom: 0
ep_warmup_thread: complete
ep_warmup_time: 8999
get_hits: 890222
get_misses: 420114
incr_hits: 0
incr_misses: 0
libevent: 2.0.11-stable
limit_maxbytes: 67108864
listen_disabled_num: 0
max_conns_on_port_11209: 1000
max_conns_on_port_11210: 9000
mem_used: 188734504
pid: 9028
pointer_size: 64
rejected_conns: 0
rusage_system: 40750.454744
rusage_user: 121639.982023
tap_checkpoint_end_received: 16980302
tap_checkpoint_end_sent: 4096340
tap_checkpoint_start_received: 16984577
tap_checkpoint_start_sent: 4098563
tap_connect_received: 693
tap_delete_received: 356011
tap_delete_sent: 352317
tap_mutation_received: 142558888
tap_mutation_sent: 146054335
tap_opaque_received: 10039
tap_opaque_sent: 1386
tap_vbucket_set_received: 1368
threads: 4
time: 1349898276
total_connections: 27521915
uptime: 4411260
vb_active_curr_items: 3437
vb_active_eject: 107129
vb_active_ht_memory: 8574624
vb_active_itm_memory: 427555
vb_active_num: 342
vb_active_num_non_resident: 3332
vb_active_ops_create: 290116
vb_active_ops_delete: 286679
vb_active_ops_reject: 0
vb_active_ops_update: 414908
vb_active_perc_mem_resident: 3
vb_active_queue_age: 0
vb_active_queue_drain: 991875
vb_active_queue_fill: 991875
vb_active_queue_memory: 0
vb_active_queue_pending: 0
vb_active_queue_size: 0
vb_dead_num: 0
vb_pending_curr_items: 0
vb_pending_eject: 0
vb_pending_ht_memory: 0
vb_pending_itm_memory: 0
vb_pending_num: 0
vb_pending_num_non_resident: 0
vb_pending_ops_create: 0
vb_pending_ops_delete: 0
vb_pending_ops_reject: 0
vb_pending_ops_update: 0
vb_pending_perc_mem_resident: 0
vb_pending_queue_age: 0
vb_pending_queue_drain: 0
vb_pending_queue_fill: 0
vb_pending_queue_memory: 0
vb_pending_queue_pending: 0
vb_pending_queue_size: 0
vb_replica_curr_items: 3619
vb_replica_eject: 134124
vb_replica_ht_memory: 8574624
vb_replica_itm_memory: 500325
vb_replica_num: 342
vb_replica_num_non_resident: 3490
vb_replica_ops_create: 291184
vb_replica_ops_delete: 287565
vb_replica_ops_reject: 0
vb_replica_ops_update: 374206
vb_replica_perc_mem_resident: 3
vb_replica_queue_age: 0
vb_replica_queue_drain: 995600
vb_replica_queue_fill: 995465
vb_replica_queue_memory: 0
vb_replica_queue_pending: 0
vb_replica_queue_size: 0
version: 1.4.4_494_ge4eafbd
Josh Deltener
Oct 10, 2012, 4:30:02 PM10/10/12
to couc...@googlegroups.com
These stats are on 1 node in a 3 node cluster.
Josh Deltener
Oct 11, 2012, 8:56:51 AM10/11/12
to couc...@googlegroups.com
Yesterday, we nuked this Couchbase bucket and moved it over to a Memcache bucket. It's confirmed, we never go above 10M of data and never above 12k items.. So why on earth would a Couchbase bucket suck up 550M! I should mention we are still having similar issues with our other Couchbase buckets.
Mike Wiederhold
Oct 11, 2012, 7:38:50 PM10/11/12
to couc...@googlegroups.com
I looked through your post and it appears that your bucket is not writing items to disk. Have you disabled persistence? If not can you upload you logs for me to look at? Instructions for how to do this can be found here:
Josh Deltener
Oct 11, 2012, 8:55:38 PM10/11/12
to couc...@googlegroups.com
It's writing to disk. We have a bunch of data on disk and items in/out of the write queue. We used Couchbase out of the box and haven't configured anything funky. I'll try to gather some logs for you.
Reply all
Reply to author
Forward
0 new messages