mongodb backup recovery questions

[Ted Chyn]

There are 2 things for backup and recovery -journal and oplog. For single instance there is no oplog.

1. In single instance, after restore the mongodump backup, how is journal used to roll forward for point in time recovery ?

2. Is there jouranling take place in replicate set secondary for application of oplog ? or jouranling only take place in primary ?

3. For replicate set, is journal participate in rolling forward after mongorestore ? or journal is only used for extraction of oplog ?

[Asya Kamsky]

Ted:

Journaling and oplog have completely different roles and they are
entirely orthogonal. One is used to recover from a crash to a
consistent state (journal), the other is used to enable logical
replication of operations from primary to other nodes in the replica
set.

You ask about backup recovery - these are sort of two different
topics. Backup and restore is one. Recovery from crash is another.

I'll try to answer your questions inline but keep in mind the above comments.

On Sun, Aug 3, 2014 at 2:59 PM, Ted Chyn <ted....@gmail.com> wrote:
> There are 2 things for backup and recovery -journal and oplog. For single
> instance there is no oplog.

Since standalone instance has no oplog, but you can have a replica set
with one member only,
in which case you will have a single mongod instance with an oplog.

I'm going to assume that you are referring to standalone (non-replica)
nodes below.

> In single instance, after restore the mongodump backup, how is journal used
> to roll forward for point in time recovery ?

The journal is not used at all. You cannot use the journal for PIT
recovery, nor is it used after mongorestore.
After running mongorestore, the contents of your database will be in a
"consistent" state (although they may not represent a single
"point-in-time" - depending on how you created mongodump, whether you
already have data in the instance, etc.

> Is there jouranling take place in replicate set secondary for application of
> oplog ? or jouranling only take place in primary ?

Journal is turned on by default in every mongod instance (and should
*not* be disabled). This is true for every single mongod regardless
of whether its role is primary, secondary, stand-alone, config server,
arbiter, etc.

> For replica set, is journal participate in rolling forward after

> mongorestore ? or journal is only used for extraction of oplog ?

The journal is not used for anything like this. I'm not sure what
extraction of oplog means - the oplog is a collection that's used to
record and read operations that happened - it's a collection like user
collections, and it's not treated in any special way by the journal.

Hope this helps and feel free to ask follow-up questions.

Asya

[Ted Chyn]

thanks for the replies. some clarifications

1. for STABDALONE instance last backup  monday night, today is tuesday 2pm and I need to have point in time recovery to 1pm tuesday.

after I restore monday's backup, how do I roll forward from last night to 1pm tuesday ?

2.  for Replicate sets

 we have both journal and oplog, all replSet has journal and oplog. my questions

a)   journal(a file) contain all changes and oplogs(a collection) also contain changes for replication purpose. are entries in journal store redundantly in oplogs(may be in different format) ? can u tell me this is correct ?

b)  if qq a) above is true. what is the need for journal for point in time recovery ? can we just extract oplog.rs and find the timestamp and rollforward to whatever time in the past(like the article u presented in one of the forum) ?

Thanks in advance and I appreciate your insight.

Ted L Chyn

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/z8C3VkDTSIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJD%2BSXJxOQWDNuVSe%3DnXpCY-EBF22F1%3D9kWu5zjpF7%2B2cg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Asya Kamsky]

The journal is for mongod internal use only.

So you cannot use the journal for anything yourself - it's
automatically used by mongod when it starts up. So I'm going to
ignore the journal in my responses since it does not serve the use
case it looks like you are looking at.

More inline:

> 1. for STABDALONE instance last backup monday night, today is tuesday 2pm
> and I need to have point in time recovery to 1pm tuesday.
> after I restore monday's backup, how do I roll forward from last night to
> 1pm tuesday ?

If you are running standalone mongod there is no way to roll forward a
backup to any point.
Your only options are to go back to Monday night backup in whole or
stay on current version in whole.

To have point in time recovery you *must* have the oplog, which means
running a replica set (even if it's a replica set of one node, you can
use oplog for PIT recovery.

> 2. for Replicate sets
> we have both journal and oplog, all replSet has journal and oplog. my
> questions
> a) journal(a file) contain all changes and oplogs(a collection) also
> contain changes for replication purpose. are entries in journal store
> redundantly in oplogs(may be in different format) ? can u tell me this is
> correct ?

The journal does not include changes in any way that's usable except
by mongod when it starts up after abnormal shutdown. It is used to
make sure there are no "half-finished" writes in the instance (i.e.
consistent view of the data for *some* point in time right before the
abnormal shutdown).

The oplog contains the all the changes that have happened for some
long time period (determined by the size of the oplog as eventually it
rolls over and overwrites the oldest changes (like a circular buffer).
It is the only place where such changes are kept.

> b) if qq a) above is true. what is the need for journal for point in time
> recovery ? can we just extract oplog.rs and find the timestamp and
> rollforward to whatever time in the past(like the article u presented in one
> of the forum) ?

There is no need and no use for journal when you need PIT recovery.
It has a completely different purpose. The only thing that can help
with PIT recovery is the oplog.

Here are some detailed descriptions of the journal mechanism and of
the oplog and how it can be used for PIT recovery:

http://blog.mongodb.org/post/33700094220/how-mongodbs-journaling-works
http://stackoverflow.com/questions/15444920/modify-and-replay-mongodb-oplog/15451297#15451297

Asya

> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mongodb-user/CAKLPb0qEHv0W%2B6cOYnTrDGzCLSAYe3u6x6MMDsmgSco5mStOTg%40mail.gmail.com.

[Ted Chyn]

Many of us working with mySQL oracle and sqlserver and other dbs thought mongodb use journal as redo log and now I understand redo log = oplog.rs in mongodb. tyvm for the insight. 

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCHqZANFXWRvu3sAAAZD%2Bh9Jbf7dX864j8Smxp%2Buq8Zzw%40mail.gmail.com.

[Sunil Ghanta]

InnoDB redo logs in MySQL = Journal in MongoDB

Binary logs in MySQL = oplog.rs in MongoDB

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAKLPb0qno2XwAQFrsZ7WiURa%2BQPx%2Bh0P1TNX9AtAKc8mCUpoiw%40mail.gmail.com.

[Chirag Mewada]

Hello 

         Myself Chirag and i am doing research in NoSQL databases. I have a query regarding oplog in mongodb and redolog in oracle.

I am searching for the difference between them. As per the informations i found till now, i came on the decision that there are somany similarities between them. But i am interested to know whether they are exactly same? Are their purposes same ? Do they store exactly same information?

[Wan Bachtiar]

i am interested to know whether they are exactly same? Are their purposes same ? Do they store exactly same information?

Hi Chirag,

Please open a new discussion thread instead of re-opening a 4-year old discussion thread.

For your research please review MongoDB Replica Set Oplog. If you have questions about ORACLE redo log, I’d recommend to post on ORACLE Community Forum.

Regards,
Wan.

​