[mongodb-user] Migrating mongodb database

[Joshua Partogi]

Dear all,

What would be the best approach to migrate mongodb database to another
machine? Do we just copy the directory that is pointed by dbpath? Is
this approach safe enough? Or is there more recommendable approach for
this?


Thank you very much in advance for the insights.

Kind regards,
Joshua

--
http://twitter.com/scrum8

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Scott Hernandez]

I have done this many times. Shutdown mongod, copy the dbpath
directory to a new machine, start the new mongod instance.

[Suno Ano]

Joshua> Dear all, What would be the best approach to migrate mongodb
Joshua> database to another machine? Do we just copy the directory that
Joshua> is pointed by dbpath? Is this approach safe enough? Or is there
Joshua> more recommendable approach for this?

As Scott already said, just copying works fine. If you have to copy a
lot of data, maybe trough some insecure and/or slow network (read:
Internet) and your connection might get canceled at some point, you can
resume it:

http://sunoano.name/ws/public_xhtml/debian_notes_cheat_sheets.html#resume_an_scp_transfer

[Joshua Partogi]

Thanks Suno,

That was really helpful. :-)

Kind regards,
Joshua

--
http://twitter.com/scrum8

On May 2, 1:12 am, Suno Ano <suno....@sunoano.org> wrote:
>  Joshua> Dear all, What would be the best approach to migrate mongodb
>  Joshua> database to another machine? Do we just copy the directory that
>  Joshua> is pointed by dbpath? Is this approach safe enough? Or is there
>  Joshua> more recommendable approach for this?
>
> As Scott already said, just copying works fine. If you have to copy a
> lot of data, maybe trough some insecure and/or slow network (read:
> Internet) and your connection might get canceled at some point, you can
> resume it:
>

> http://sunoano.name/ws/public_xhtml/debian_notes_cheat_sheets.html#re...

[Suno Ano]

Joshua> Thanks Suno, That was really helpful. :-)

I am glad it was; I figured that is a question that is coming up quite
often so I decided to add it to my MongoDB FAQs so hopefully it might
help others too.

http://sunoano.name/ws/public_xhtml/mongodb.html#how_do_i_physically_migrate_a_database

[GVP]

Hey guys;

I think that we're missing an important answer here.

Method #1: migrate a mongodb, quick migration, potentially lots of
downtime.
- Shutdown mongod
- Copy the dbpath directory to a new machine
- Start the new mongod instance with the same path variables

Method #2: migrate a mongodb, slower migration, very little downtime.
- Stop and re-start existing mongod as master (if it's not already
master)
- Install mongod on new machine and configure as slave
- Wait while the slave copies the DB, re-indexes and then "catches
up"
- Once the slave is caught up, do the following steps in a way that
minimizes down-time.
-* Turn off DB write access from whatever systems rely on Mongo (they
can still read during the switch)
-* Once the writes have all committed, restart the slave (new
computer) as a master.
-* Point all traffic at the new master.
-* Turn off the old server.
-* Turn on system writes.

On May 2, 8:45 am, Suno Ano <suno....@sunoano.org> wrote:
>  Joshua> Thanks Suno, That was really helpful. :-)
>
> I am glad it was; I figured that is a question that is coming up quite
> often so I decided to add it to my MongoDB FAQs so hopefully it might
> help others too.
>

> http://sunoano.name/ws/public_xhtml/mongodb.html#how_do_i_physically_...

[Suno Ano]

You are right sir, good catch. I updated my FAQs item accordingly:
http://sunoano.name/ws/mongodb.html#how_do_i_physically_migrate_a_database

[Damon P. Cortesi]

Suno, I noticed in that faq you mentioned using OpenVZ.

I'm not sure how common this is, but at least myself and a couple
other people have not been able to run MongoDB on an OpenVZ VPS. What
happens is that Mongo eventually consumes all available memory and
crashes. Other system services (ssh) also become unresponsive with
your typical "can not fork" errors. The best I can figure is that due
to how OpenVZ manages memory, Mongo's approach is not compatible and
the VM does not know when to release system memory.

I don't know if you're already using Mongo on OpenVZ (if you are, I'd
love to know if you had to configure anything specially), but wanted
to mention this.

Damon

On May 8, 6:38 am, Suno Ano <suno....@sunoano.org> wrote:
> You are right sir, good catch. I updated my FAQs item accordingly:http://sunoano.name/ws/mongodb.html#how_do_i_physically_migrate_a_dat...

[Suno Ano]

Damon> Suno, I noticed in that faq you mentioned using OpenVZ.

indeed


Damon> I'm not sure how common this is, but at least myself and a
Damon> couple other people have not been able to run MongoDB on an
Damon> OpenVZ VPS. What happens is that Mongo eventually consumes all
Damon> available memory and crashes.

Yes, I read those postings too. Those OpenVZ (or whatever other
virtualization technology came up) containers/environments (called VE
(Virtual Environment) with OpenVZ) simply do not have enough memory
assigned and/or might need a little OOM (out of memory) tweaking i.e.
basically allow some over-committing. In case of OpenVZ, elevating the
oomguarpages limit/barrier is key and makes things work smoothly.

By not enough memory I mean something around 512 MiB RSS (see below) or
even less. I can tell from experience, that if you assign a reasonable
amount of RSS (RAM basically) to your VE, all works fine.

Of course, you need to have control over the HN (Hardware Node) (read
host system) in order to make those settings which of course is a
problem if you use hosting where you rent a VE for some monthly fee.



Bottom Line:

- for most use cases 512 MiB RSS is simply not enough; here I am
talking about in physical memory pages (RAM) also known as i.e.
http://en.wikipedia.org/wiki/Resident_Set_Size, set with physpages in
OpenVZ. Simply treat this setting the same as you would for a
non-virtualized setup and you are fine ...

- if you allow your VE to over-commit, things just work; here I am
talking about virtual memory, oomguarpages in OpenVZ

So, MongoDB inside an OpenVZ VE is not a problem as long as you have a
reasonable amount of RSS assigned (one that fits your use case) to your
VE running MongoDB. For precautions, one can have settings to let the VE
over-commit its virtual memory requirements in order to avoid the OOM
killer caused problems.

[Pixy Misa]

In testing with MongoDB 1.2.2 I was able to repeatably crash and
corrupt my database on a 2GB OpenVZ VE.

Yes, the database was larger than 2GB, but that's the point. It's not
always feasible to allocate enough RAM to hold the entire database,
and having your database crash isn't an acceptable alternative.

This isn't inherently a fault with either OpenVZ or MongoDB, but I
would strongly advise against anyone trying to use the two together.
Which is a shame, because individually they are extremely useful.

[Eliot Horowitz]

We'll try and figure out what's going on with OpenVZ.
We haven't seen issues with anything else (vmware, parallels, zen) so
definitely something odd.

[Suno Ano]

Pixy> In testing with MongoDB 1.2.2 I was able to repeatably crash and
Pixy> corrupt my database on a 2GB OpenVZ VE.

I would be interested in your setup (kernel version etc.) and how you
configured your OpenVZ HN and VE. After I set the afore mentioned memory
settings appropriately, there were no issues whatsoever, everything runs
smoothly with Debian's linux-image-2.6.32-5-openvz-amd64 kernel.

Maybe you need to make some changes to sysctl[0] or simply touch the
afore mentioned memory settings.

- [0] http://sunoano.name/ws/openvz.html#sysctl_openvz

[Jesse Proudman]

We'd love to help resolve this issue Mongo / OpenVZ issue. We have
customers using Mongo and we can easily replicate the OOM issues
discussed here. I'm happy to provide any information / setup testing
environments for whomever has ideas on how to resolve this.

Thanks,

Jesse Proudman
Blue Box Group, LLC
p. 800-613-4305 x 801
www.blueboxgrp.com

[Eliot Horowitz]

Access to a couple of testing environments would be great.
Can you follow up on the ticket?
http://jira.mongodb.org/browse/SERVER-1121

[Pascal Jürgens]

OpenVZ is mainly an issue when you have a VPS run by some hosting provider and can't tune the settings. After several OOM situations where the system was unreachable, I pretty much concluded that Xen is the only way to go.

It would also be interesting to know whether/how OpenVZ's combined ram+swap messes with efficient memory mapping by mongo

(http://wiki.openvz.org/UBC_systemwide_configuration#Memory_and_swap_space)

Pascal

[Jesse Proudman]

I've gone ahead and updated that ticket.

Looking forward to helping create a solution!

Thanks

- Jesse

[Pablo Godel]

are there any updates / progress in relation to this topic?

Thank you

Pablo

[Eliot Horowitz]

Nothing good.
openvz does a number of strange things and we've had a very hard time
making progress.
We've gone so far as to issue a warning when you start on openvz since
once you cross the ram boundary, weird things start happening.
To make any further progress we'll probably need to talk to the openvz
developer team

[Pablo Godel]

Yes, that's how I started looking and found this thread.

what's the easiest way to recreate this problem? I would like to do some tests and configuration tweaks.

I also read that there was some feature planned in regards to memory management for 1.7 that would help?

[Eliot Horowitz]

Anything with more data than ram was causing odd things to happen.

The durability changes we're making in 1.7 won't help this unfortunately.

[Markus Gattol]

Pablo> What's the easiest way to recreate this problem? I would like to
Pablo> do some tests and configuration tweaks.

the sysctl settings are where I suggest you start looking
http://www.markus-gattol.name/ws/openvz.html#sysctl_openvz

[Pablo Godel]

Hi Markus,

great info you have there. do you think those tweaks suggested for Postgres would help with MongoDB? Would they help to deal with the problem permanently or just push it ahead for when more memory is needed?

Pablo

--

[Markus Gattol]

Pablo> Hi Markus, Great info you have there.

thanks :-)

Pablo> Do you think those tweaks suggested for Postgres would help with
Pablo> MongoDB? Would they help to deal with the problem permanently or
Pablo> just push it ahead for when more memory is needed?

I have not tried it yet but it is certainly worth a try. Because I have
not tried yet I can also not make a statement about mid to long-term
behavior of the whole shebang. If you have an OpenVZ box around and the
time to test, please report back to us. I will maybe give it a try some
time next week.

[Markus Gattol]

Without having done any testing here, sounds like it is the same cause
as reported in the "Migrating mongodb database" thread (the one where
the issue is about running MongoDB inside an OpenVZ VE (Virtual
Environment)).

Even if it is not exactly the same, then at least it has related
semantics.

[Pablo Godel]

I have plenty of those :)

I will try to do some testing and find out. I really want to run mongodb on openvz, it would be a HUGE bummer if it can't. 

--

[Markus Gattol]

Pablo> I will try to do some testing and find out. I really want to run
Pablo> mongodb on openvz, it would be a HUGE bummer if it can't.

I am confident it will be a non-issue in the not so distant future; it
is certainly just some funky memory thingy going on someone needs to
track down.

Once it is solved LXC (Linux Containers) will also work just fine. LXC
is in mainline. In fact OpenVZ developers are contributing a lot to
mainline which basically is LXC i.e. LXC is a feature subset of OpenVZ.

[Shrey Malhotra]

Any updates to this? I am running OpenVZ, and Mongodb and it runs out of memory.

Anyone knows how to fix it?

[Shrey Malhotra]

Anyone has a fix for this? I am using OpenVz and mongodb, but it runs out of memory.

On Saturday, 1 May 2010 19:32:58 UTC+5:30, Joshua Partogi wrote:

Dear all,

What would be the best approach to migrate mongodb database to another
machine? Do we just copy the directory that is pointed by dbpath? Is
this approach safe enough? Or is there more recommendable approach for
this?


Thank you very much in advance for the insights.

Kind regards,
Joshua

--
http://twitter.com/scrum8

[Asya Kamsky]

Please don't reply to old threads - your message is likely to be missed this way.

Recent versions of OpenVZ did not have the same problems as the old ones and it's very possible your memory problems have a different cause.

Start a new thread with your question, describe your problem is as much details as you can and we'll see if we can help you diagnose it.

Asya