info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
I have a quick question about exporting or dumping a database.
6 views
Skip to first unread message
jeff
Aug 9, 2012, 10:32:32 PM8/9/12
to
I have a server running Citadel which uses Berkeley DB on the back end
and I am trying to transfer the VM that is running it to a different
server and when I just copy the virtual hard drive and load it the
database is corrupt and cannot be fixed. When I try anything else like
db_hotbackup I get an input/output error when it tries to backup the
logs. The host system is running Fedora 16 and the VM is running Centos
6.3 Using qemu and KVM for the virtualization. I really have 2 questions.
1. How long should it take to dump a database? Mine is about 7.4 GB with
about 200 MB of log files right now.
2. What is the best way to transfer the database to a different system?
I would prefer a method that does not involve downtime as this is a
production email system.
and I am trying to transfer the VM that is running it to a different
server and when I just copy the virtual hard drive and load it the
database is corrupt and cannot be fixed. When I try anything else like
db_hotbackup I get an input/output error when it tries to backup the
logs. The host system is running Fedora 16 and the VM is running Centos
6.3 Using qemu and KVM for the virtualization. I really have 2 questions.
1. How long should it take to dump a database? Mine is about 7.4 GB with
about 200 MB of log files right now.
2. What is the best way to transfer the database to a different system?
I would prefer a method that does not involve downtime as this is a
production email system.
Florian Weimer
Aug 10, 2012, 2:34:06 PM8/10/12
to
* jeff:
> I have a server running Citadel which uses Berkeley DB on the back end
> and I am trying to transfer the VM that is running it to a different
> server and when I just copy the virtual hard drive and load it the
> database is corrupt and cannot be fixed.
This is odd. Are you copying a volume which is being written to? If
you copy a read-only snapshot, it should work.
> 1. How long should it take to dump a database? Mine is about 7.4 GB
> with about 200 MB of log files right now.
That mostly depends on fragmentation. About an hour would not be
entirely unheard of.
> 2. What is the best way to transfer the database to a different
> system? I would prefer a method that does not involve downtime as this
> is a production email system.
You need to make the database read-only at one point, otherwise some
writes will be lost. And if you change versions of Berkeley DB, you
have to perform environment recovery, which also requires downtime.
> I have a server running Citadel which uses Berkeley DB on the back end
> and I am trying to transfer the VM that is running it to a different
> server and when I just copy the virtual hard drive and load it the
> database is corrupt and cannot be fixed.
you copy a read-only snapshot, it should work.
> 1. How long should it take to dump a database? Mine is about 7.4 GB
> with about 200 MB of log files right now.
entirely unheard of.
> 2. What is the best way to transfer the database to a different
> system? I would prefer a method that does not involve downtime as this
> is a production email system.
writes will be lost. And if you change versions of Berkeley DB, you
have to perform environment recovery, which also requires downtime.
jeff
Aug 10, 2012, 4:20:53 PM8/10/12
to
On 08/10/2012 11:34 AM, Florian Weimer wrote:
> * jeff:
>
>> I have a server running Citadel which uses Berkeley DB on the back end
>> and I am trying to transfer the VM that is running it to a different
>> server and when I just copy the virtual hard drive and load it the
>> database is corrupt and cannot be fixed.
>
> This is odd. Are you copying a volume which is being written to? If
> you copy a read-only snapshot, it should work.
>
Yes it is being written to right now because it is production. I will
> * jeff:
>
>> I have a server running Citadel which uses Berkeley DB on the back end
>> and I am trying to transfer the VM that is running it to a different
>> server and when I just copy the virtual hard drive and load it the
>> database is corrupt and cannot be fixed.
>
> This is odd. Are you copying a volume which is being written to? If
> you copy a read-only snapshot, it should work.
>
see about scheduling some downtime for this.
>> 1. How long should it take to dump a database? Mine is about 7.4 GB
>> with about 200 MB of log files right now.
>
> That mostly depends on fragmentation. About an hour would not be
> entirely unheard of.
have taken 8-10 hours and either never finished or came up with an error.
>
>> 2. What is the best way to transfer the database to a different
>> system? I would prefer a method that does not involve downtime as this
>> is a production email system.
>
> You need to make the database read-only at one point, otherwise some
> writes will be lost. And if you change versions of Berkeley DB, you
> have to perform environment recovery, which also requires downtime.
IO even though the drives I am using are not exactly slow, but the
results of trying to copy does not seem to be effected by other virtual
machines running on that system. From other tests that I have done I
have verified that memory, and CPU are not being used much at all, but
the IO on the hard drive, mostly waiting for random seeks might be
causing problems. I have also noticed what appears to be some possible
corruption with other VMs which from what I can tell is affected by the
drive IO since moving to the new system with a larger RAID array seems
to have fixed the problems.
Florian Weimer
Aug 10, 2012, 4:42:37 PM8/10/12
to
* jeff:
> I have reason to believe that I am pushing the limits of the hard
> drive IO even though the drives I am using are not exactly slow, but
> the results of trying to copy does not seem to be effected by other
> virtual machines running on that system. From other tests that I have
> done I have verified that memory, and CPU are not being used much at
> all, but the IO on the hard drive, mostly waiting for random seeks
> might be causing problems.
Yes, Berkeley DB has a known bug which causes horrendous file system
fragmentation in its database files. Reading them sequentially is
very, very slow, even on beefy disk arrays.
> I have reason to believe that I am pushing the limits of the hard
> drive IO even though the drives I am using are not exactly slow, but
> the results of trying to copy does not seem to be effected by other
> virtual machines running on that system. From other tests that I have
> done I have verified that memory, and CPU are not being used much at
> all, but the IO on the hard drive, mostly waiting for random seeks
> might be causing problems.
fragmentation in its database files. Reading them sequentially is
very, very slow, even on beefy disk arrays.
0 new messages