cbbackup does not complete

[Michael Liu]

Is there any way to see what is causing cbbackup to hang?

2014-12-15 22:43:06,803: mt cbbackup...

2014-12-15 22:43:06,803: mt  source : http://localhost:8091

2014-12-15 22:43:06,803: mt  sink   : backups/

2014-12-15 22:43:06,803: mt  opts   : {'username': '<xxx>', 'verbose': 1, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'seqno': 0.0, 'batch_max_size': 1000.0, 'report': 5.0, 'design_doc_only': 0.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'vbucket_list': None, 'threads': 4, 'mode': 'full', 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': None}

2014-12-15 22:43:06,819: mt skipping bucket that is not a couchbase-bucket: loop_session

2014-12-15 22:43:06,819: mt bucket: loop

  [################    ] 79.4% (5144/estimated 6479 msgs)

[Michael Liu]

With the -vvv option, execution stops here, which doesn't say much:

...

2014-12-15 22:48:22,022: Thread-1 Read 1024 bytes off the wire

2014-12-15 22:48:22,022: Thread-1 Read 1024 bytes off the wire

2014-12-15 22:48:22,022: Thread-1 Read 1024 bytes off the wire

2014-12-15 22:48:22,022: Thread-1 Read 1024 bytes off the wire

2014-12-15 22:48:22,022: Thread-1 Read 638 bytes off the wire

2014-12-15 22:48:22,130: w0 Send buffer size: 665821msgs)

2014-12-15 22:48:22,365: w0 Send buffer size: 698353

2014-12-15 22:48:22,518: w0 Send buffer size: 666519

2014-12-15 22:48:22,662: w0 Send buffer size: 677979

2014-12-15 22:48:22,809: w0 Send buffer size: 686251

2014-12-15 22:48:22,962: w0 Send buffer size: 720175msgs)

[Michael Liu]

Is there a recommended solution to move data if cbbackup fails to complete? I guess since I am down to a single node, i could theoretically just move the data files to a new node. What else would need to be copied over?

Although I'm not quite sure moving the data files will solve my problem.

[Aliaksey Kandratsenka]

On Tue, Dec 16, 2014 at 12:00 PM, Michael Liu <mike...@gmail.com> wrote:

Is there a recommended solution to move data if cbbackup fails to complete? I guess since I am down to a single node, i could theoretically just move the data files to a new node. What else would need to be copied over?

Although I'm not quite sure moving the data files will solve my problem.

Just copying the files should work given that we're append only.

[Michael Liu]

I was able to move the data to a new installation of couchbase, everything from web console looks ok to me. However -- I am still unable to complete a cbbackup. Given my data size is so small could there be a possibility that something is wrong with the vbucket structures? Is there any utility that can help sanitize or check the data files for corruption?

This problem is directly related to rebalance hanging as well, as I am unable to add a new server to the brand new cluster either because rebalance never completes.

Thanks!

[Aliaksey Kandratsenka]

On Wed, Dec 17, 2014 at 4:02 PM, Michael Liu <mike...@gmail.com> wrote:

I was able to move the data to a new installation of couchbase, everything from web console looks ok to me. However -- I am still unable to complete a cbbackup. Given my data size is so small could there be a possibility that something is wrong with the vbucket structures? Is there any utility that can help sanitize or check the data files for corruption?

I'm not aware of any specific "fsck" tool for couchstore files. But you can use couch_dbdump (which is shipped with couchbase server) to verify that at least basic structure is there and that (or if) byseq and byid btrees are ok and have matching doc bodies.

[Bruno Santos]

Michael,

Could you find the solution for this error?

I'm having exactly the same problem...

Thanks,

Bruno

[Michiel Bijland]

Hi,

Having the same problem:

2015-03-04 10:40:00,587: mt cbbackup...
2015-03-04 10:40:00,587: mt  source : couchbase://Username:Pass...@127.0.0.1:8091
2015-03-04 10:40:00,587: mt  sink   : /var/backups/couchbase/
2015-03-04 10:40:00,587: mt  opts   : {'username': None, 'verbose': 1, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 0.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'vbucket_list': None, 'threads': 4, 'mode': 'full', 'key': None, 'password': None, 'id': None, 'bucket_source': None}
2015-03-04 10:40:00,605: mt bucket: default
  [############        ] 60.4% (2719/estimated 4500 msgs)

-vvv

2015-03-04 10:36:58,353: Thread-1 Read 592 bytes off the wire
2015-03-04 10:36:58,353: Thread-1 Read 44 bytes off the wire
2015-03-04 10:36:58,353: Thread-1 Read 961 bytes off the wire
2015-03-04 10:36:58,353: Thread-1 Read 85 bytes off the wire
2015-03-04 10:36:58,354: Thread-1 Read 28 bytes off the wire
2015-03-04 10:36:58,392: w0 Send buffer size: 516601
2015-03-04 10:36:58,519: w0 Send buffer size: 507899
2015-03-04 10:36:58,663: w0 Send buffer size: 565071
2015-03-04 10:36:58,812: w0 Send buffer size: 564452msgs)

Anybody found a solution yet?
Michiel

[Michiel Bijland]

Issue is related to https://issues.couchbase.com/browse/MB-13386

This problem will probably be fixed in release 3.0.3

[Nick Wood]

I had the same issue with two of my buckets.  Not sure if it's feasible for you, or if it will work for you as well, but what fixed it for me was using XDCR to replicate to a new bucket, then delete/recreate the original bucket and replicate the documents back.  After that, backups worked great.  Hopefully it will stay that way.

  Nick