Question regarding order of backup operations in 3.0

[Barry Hagan]

I noticed after my upgrade from 2.5 to 3.0 that the order of backup operations for the db and indexes flipped in 3.0.  For 2.5 the indexes were backed up first, and then the ESENT backup ran. Now in 3.0 it is ESENT/Voron first, then indexes.  Was there a specific reason for this?

The theoretical issue I see is getting inconsistent data after a restore on the very rare chance that a write was committed and indexed between the time the db backup is completed and the index snapshot was taken.

For 2.5 the restore was safe:  During restore of the indexes, they may be behind the last ETag committed in the restored database, but they can just index those missed documents and get to consistency.

Now in 3.0, It seems possible (however slim of a chance) that a restored index could include docs for ETags that never made it into the database backup.  On restore you now have data in the index that is not in the database, and that would not be easy to clear short of resetting the index.  Or is there another mechanism I'm missing to keep consistency between the two pieces of a 3.0 backup?

[Chris Marisic]

It's possible a transactional snapshot of the indexes is taken at the start of the backup process.

[Oren Eini (Ayende Rahien)]

What actually happens is that when we restore, we use the earliest etag recorded, in the index or in the storage. That result in the same behavior. 

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry Hagan]

But that only works if the index could roll back to a commit point prior to the last Etag in the restored database.  The reset logic assumes the docs being re-indexed after the Etag reset are the same ones the index already saw so at worst it overwrites the same data in the index, but this will not be true for a restore.

Take a look at the test I added here, which fails for me:

https://github.com/barryhagan/ravendb/commit/d23f8add487ea7a59a488ad926b82a7b76af7772

It is a bit of a contrived example because I'm explicitly calling FlushMapindexes in my write loop, but it proves the possibility of getting a restored index that has data that is not in the restored db.  Of course for that to happen there must be a write that is committed (and flushed) to the index  in the window between the db and index backup.  But unless there is a good reason to run the db backup first, switching the order back to how it was in 2.5 will ensure this cannot happen.

[Oren Eini (Ayende Rahien)]

That actually only affect you when doing very low level operations, it isn't exposed externally.

When you query the index itself, we filter documents in the index and not on the database.

I recall we had a good reason to switch it, but I don't recall what it was. I'm checking on that.

[Oren Eini (Ayende Rahien)]

See the discussion here: http://issues.hibernatingrhinos.com/issue/RavenDB-3177#comment=67-6830

[Paweł Pekról]

Hi,

we have looked into the problem and it appears that you are absolutely right, at the time when this change was made it was a valid move, because we did not check the state of index every time (only during the unclean shutdown) + later on we have added more enhancements to last commited etag in index.

Here is my commit reverting this change https://github.com/ayende/ravendb/commit/8b3e8d139144d14cb78648ce08f177b00d1bee4b

Thanks for having a keen eye!

P.