DB missing after power failure

[Vinay Saini]

Hello All,

I found some issue regarding DB corrupt after power failure in the mailing list :

https://groups.google.com/forum/#!searchin/2600hz-dev/power/2600hz-dev/_c_e5a226rE/altbmZLEAQAJ

https://groups.google.com/forum/#!searchin/2600hz-users/power/2600hz-users/n1feOmQ0GqI/vmUEJKbFlT8J 

But, did not able to find any proper solution for this problem. I noticed, usually it happens when we use SSD hard disk on the physical machine. Also  when I check database at the time of issue, I found only accounts related dbs were missing and all the other dbs like offnet,system_config were available.

So I am not sure if this issue is related to bigcouch or kazoo. Is there any way we can avoid such type of problem so that the system we build will be a production ready.

Thanks

Vinay   

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Hi Vinay,

If a particular database is corrupted and the local BigCouch/CouchDB
instance isn't able to read it properly, feel free to remove the file
from disk and let replication do its magic to repopulate the file. As
long as you have your two other nodes with copies of that database, your
cluster should continue to operate as expected.

Sometimes you need to restart the BigCouch/CouchDB process for it to
update its bookkeeping and initiate the replication job.

Of course, you can also reach out to the CouchDB community directly to
ask about this issue and let us know if there are best practices they
recommend that this community may not be aware of!

James

- --
James Aimonetti

Lead Systems Architect
"If Dialyzer don't care, I don't care"
2600HzPDX | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXj6CVAAoJENTKa+JPXCVgcI0IAItNc+v72ZKXPaY0xJGplSjb
nQ3hNyTvZ/B43LgeczyYcl+nhZzUAi4yS5sAH+vybmMhCdWcxsns+C0t+SBjwqko
gbSZ+3APsgpV8GFZAzMGBAlO+2k7dRd7a2OPiMUKjndKElT3cZI32ddrXMhFkISA
0t/TnD0ZU52C20nFA6pRCpxk9ULzUFPYUFPU2ceKHjJZ5v7KIQ5vUCR2p9iWmtbQ
l6b7v7Wpj7OrYbKw9IvecZZ4r6rXIRpUk3M9fi9DrVNXB8KkkBVfPIlylbNRMfee
Y2soudDXVXNSXDD+4d63bTu4AqVsbXkFWoBjTdaTS1XGiAwOvKrYWmjc+GOj0Y8=
=WqK+
-----END PGP SIGNATURE-----

[Vinay Saini]

Thanks James,

Actually I have single server installation, so I do not have database clustering and can not replicate from other instance. 

Yes sure, I will ask this question in the other community and will share the same here, If I get any solution.

Thanks

Vinay  

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ah, well, if this is single-server, then it is just for testing, right?

The clustering capabilities of CouchDB/BigCouch are what protect you
from individual servers having mishaps. Without that redundancy, you are
at the mercy of so many things that could go wrong.

If you can, try a 7 server installation for testing failure scenarios:

3 BigCouch nodes clustered
2 FreeSWITCH servers
2 Kamailio/Kazoo/RabbitMQ servers

Once all the services are connected up, you can play with simulating
power outages, high latency network connections, packet loss, etc.

- --
James Aimonetti

Lead Systems Architect
"If Dialyzer don't care, I don't care"
2600HzPDX | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXkM1KAAoJENTKa+JPXCVgUxwIAKWSA/v9J82v5hgxttLUStDi
kf79NC8pAq8WFva2Zh6zRhslsLU4F0+GU3LuhFmY/NL1N/d7HxEUvQy6kJ5PeFm8
BF+GKe/I8Tm+X9yCJPJfJkEJAc4+YPTknV3whnf1qWfG1hKwe+RkmFfeRV5NNGBa
SuYUJWno9IFIGy4OhoIfLSw34RL/QndbQ9T3PVTErJHYQW2gBa77KrJU7zMhdOVt
3tztD0ujf/lCGrasV6AsUhTNvHGMqlZsQOJZSIGY4Mo9ubEXdisBQ/rMiKmCFzg2
MLlJlpEN3LmmWKIdaHjSQEcieQQBytr4Uipke2IFqQYMlneQwNnjdtjf/wU5UgU=
=CBD5
-----END PGP SIGNATURE-----

[Joe]

There is a couchdb package for python

https://github.com/djc/couchdb-python

it includes some nice command line utilities for doing replication, dumping, and restoring to flat text files.  It probably won't save you at this point, but it makes for a quick and easy python script to automate backing up the db's that are most important to you on a regular basis

[James Aimonetti]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

There is also the db_archive SUP command:

https://github.com/2600hz/kazoo/blob/3.22/core/whistle_couch-1.0.0/src/whistle_couch_maintenance.erl#L99

sup whistle_couch_maintenance db_archive {DB_NAME} [{FILEPATH}]

It will put DB_NAME.json in /tmp (or in {FILEPATH} if you supply it,
with the most current documents. It does not archive attachments like
voicemails, faxes, etc.

In Kazoo 4, there is archive and import functionality for restoring
those JSON dumps.

- --
James Aimonetti

Lead Systems Architect
"If Dialyzer don't care, I don't care"
2600HzPDX | http://2600hz.com
sip:ja...@2600hz.com
tel:415.886.7905
irc:mc_ @ freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXoeflAAoJENTKa+JPXCVgK3EH/3SGto1KI8WOSuakJe1fTW8v
W3Cm968gKuYtnAlkpiYVE0IDOh9gKrQR5YUew2anjVxUQ1hXwGSkxeS/B7K8tgId
f2aeUElIm6dy+8r7rL9F2OXtLwjJeAEHEyJfVALxDppspRkZYGLogHIzAU6uEeFt
lFz1ral5fn5DXMvjXD2qH0rwNqj1bg836jncO+O3lDfYnO/h6UOvslVp6oMixeaH
qtbz/hKop+bCcf7+0R0y9M8G71e7sRoly6ATa8KcrzFOs4nNMnra7nKnHF4QNf63
1A2rIr0tPjhyV0g9aSk0+klPBrsJnCP16+xvCC4CgjFTE/lOITnQpW9/WHWtbL4=
=cOwd
-----END PGP SIGNATURE-----

[HaiBui]

I have same problem, how can I restore the Bigcouch DB ?

[Sergey Safarov]

Are you have secondary bigcouch server joined to cluster with first server?

Are you have backup of couchdb database folder?

Sergey

среда, 7 сентября 2016 г., 13:41:13 UTC+3 пользователь HaiBui написал:

[Hai Bui Duc Ha]

Hi Sergey Safarov,

Unfortunately, I run server all in one and don't have couchdb backup. Because, this is the first time I see this problem.

I see on admin port, data still have, but it don't have document which is pointed to the data on shards folder.

Then I try to "mapping" the data manually, and it work. But I need to restore all documents. 

You have any idea ?

Regards,

Hai Bui

--
You received this message because you are subscribed to a topic in the Google Groups "2600hz-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/2600hz-dev/TTcvio8-0wg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 2600hz-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hai Bui

VoIP engineer, Cvoice team, HTK-HCM Office

Mobile: +84-165-618-9876

[Gaurav Saini]

Hello Team,

We are facing same issue frequently on client end in case of power failure. I have asked couchdb community also and they mentioned about changing config setting delayed_commit Off. But when I checked it was already off and didn't helped me much. 

Can anyone please guide ? Also, we can't use 7 server cluster as we are developed a Hardware IP PBX which is single server and Kazoo based. So, single machine is installed on client's offices. 

Thanks

Gaurav

sip:...@2600hz.com

[Sergey Safarov]

If you not have backup, then docs can't be restored.

You cant recreate core database (like system_config and etc) and accounts docs (users, devices and etc).

Before any important changes backup must be.

Sergey

четверг, 8 сентября 2016 г., 13:19:11 UTC+3 пользователь HaiBui написал:

To unsubscribe from this group and all its topics, send an email to 2600hz-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sergey Safarov]

To protect Bigcouch (not CouchDB 1.x and 2.x) is enough second server on virtual machine.

Required crate bigcouch cluster from two machines and restore database via replication from backup server.

Sergey 

четверг, 29 декабря 2016 г., 21:28:34 UTC+3 пользователь Gaurav Saini написал:

[fred]

Is there a straight forward way to backup/restore bigcouch/couchdb?  I know there are some scripts around but not sure how well they work.

[Darren Schreiber]

If you just backup /srv/ and /etc/kazoo/bigcouch/ that’s literally all you do. Copy ALL (including hidden files) files or zip/tar them up.

--
You received this message because you are subscribed to the Google Groups "2600hz-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-dev+...@googlegroups.com.

[fred]

Thanks for the info Darren. Never occurred to me that it could be that easy.

[Darren Schreiber]

The biggest thing that seems to cause people to “lose” their data is misunderstanding how BigCouch sharding works and, thus, effectively failing to restore properly.

 

Basically, when restoring:

* Make sure BigCouch is OFF

* Copy the files back in place in /srv/ and /etc/kazoo/bigcouch/

* MAKE SURE THE FQDN HOSTNAME AND /ETC/HOSTS FILES ARE THE SAME AS BEFORE. Do not change the hostname! Do not use a new hostname!

* Make sure the permissions on the restored files in /srv/ are bigcouch:daemon

 

If you’ve got all the above right, start BigCouch back up. Should be good to go.

[Gaurav]

Hello Team,

As, I mentioned we are experimenting with Kazoo v3 with Centos 6 on single server hardware machine for IP PBX. We are currently facing similar issues with the single hardware machine that if in case of Power failure some databases are missing. I thought about the solutions mentioned in the thread that but we can't do clustering so replication is not possible. What best we did is to take backup of entire database and in case of any such failure we restore the last day backup.

Also, I discussed the same with bigcouch community here. https://github.com/cloudant/bigcouch/issues/141 

Please suggest how to proceed as on cloud/virtual server we are enjoying working with Kazoo but for on-premise installation in India it's becoming difficult. (as here we can have power failure sometimes.)

Thanks

Gaurav