Raven fails to load - attempt to recover gives -1018 (JET_errReadVerifyFailure)
75 views
Skip to first unread message
Brian Schau
Aug 9, 2016, 3:15:27 AM8/9/16
to RavenDB - 2nd generation document database
Hello,
Raven: #30100.
So, for some reason during the night I got this in the eventlog:
Raven (3260) 2-5qa1yDiinIf/94XOSSuDGiz-C:\RavenDB\Databases\esignatur\Data: The database page read from the file "C:\RavenDB\Databases\esignatur\Data" at offset 8560640 (0x000000000082a000) (database page 2089 (0x829)) for 4096 (0x00001000) bytes failed verification because it contains no page data. The read operation will fail with error -1019 (0xfffffc05). If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
and
Raven (3260) 2-5qa1yDiinIf/94XOSSuDGiz-C:\RavenDB\Databases\esignatur\Data: A request to write to the file "C:\RavenDB\Databases\esignatur\Data" at offset 96957558784 (0x00000016931ef000) for 393216 (0x00060000) bytes has not completed for 36 second(s). This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
I've stopped the raven process and I try to recover using:
cd \RavenDB\Databases\my_database
... but I get the following error:
Operation terminated with error -1018 (JET_errReadVerifyFailure, Checksum error on a database page) after 0.438 seconds.
I even tried to run the above command supplying /i and /a flags.
What can I do now? We're current offline ... :-(
Best regards,
Brian
Raven: #30100.
So, for some reason during the night I got this in the eventlog:
Raven (3260) 2-5qa1yDiinIf/94XOSSuDGiz-C:\RavenDB\Databases\esignatur\Data: The database page read from the file "C:\RavenDB\Databases\esignatur\Data" at offset 8560640 (0x000000000082a000) (database page 2089 (0x829)) for 4096 (0x00001000) bytes failed verification because it contains no page data. The read operation will fail with error -1019 (0xfffffc05). If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
and
Raven (3260) 2-5qa1yDiinIf/94XOSSuDGiz-C:\RavenDB\Databases\esignatur\Data: A request to write to the file "C:\RavenDB\Databases\esignatur\Data" at offset 96957558784 (0x00000016931ef000) for 393216 (0x00060000) bytes has not completed for 36 second(s). This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
I've stopped the raven process and I try to recover using:
cd \RavenDB\Databases\my_database
esentutl /r RVN /l logs /s system
... but I get the following error:
Operation terminated with error -1018 (JET_errReadVerifyFailure, Checksum error on a database page) after 0.438 seconds.
I even tried to run the above command supplying /i and /a flags.
What can I do now? We're current offline ... :-(
Best regards,
Brian
Oren Eini (Ayende Rahien)
Aug 9, 2016, 3:17:49 AM8/9/16
to ravendb
You have disk corruption of some sort, and you need to restore from backup.
Hibernating Rhinos Ltd
Oren Eini l CEO l Mobile: + 972-52-548-6969
Office: +972-4-622-7811 l Fax: +972-153-4-622-7811
--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Brian Schau
Aug 9, 2016, 3:22:53 AM8/9/16
to RavenDB - 2nd generation document database
Is that really the only option?
The crash of course happened just one hour before our daily backup :-) So I'll lose 23 hours of production data ... :-(
A chkdsk tells me that the disk is fine.
The crash of course happened just one hour before our daily backup :-) So I'll lose 23 hours of production data ... :-(
A chkdsk tells me that the disk is fine.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
Oren Eini (Ayende Rahien)
Aug 9, 2016, 3:30:28 AM8/9/16
to ravendb
The disk itself is fine, but there is corrupted data on the disk.
There isn't much that we can do if the data on disk is corrupted.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
Brian Schau
Aug 9, 2016, 3:34:22 AM8/9/16
to RavenDB - 2nd generation document database
That is understandable.
Now - why would the data suddenly be corrupted? We haven't had any sudden shutdowns (well, according to the event logs at least :-)
Is there no way to force esentutl to recover as much as possible? At 1 o'clock in the night we don't have that much traffic so a forced recovery could probably recover a nearly full database.
I don't mind rebuilding indexes ...
Now - why would the data suddenly be corrupted? We haven't had any sudden shutdowns (well, according to the event logs at least :-)
Is there no way to force esentutl to recover as much as possible? At 1 o'clock in the night we don't have that much traffic so a forced recovery could probably recover a nearly full database.
I don't mind rebuilding indexes ...
Brian Schau
Aug 9, 2016, 4:50:20 AM8/9/16
to RavenDB - 2nd generation document database
Replying to myself. I'm biting the bullet and restoring the database.
What a bummer.
What a bummer.
Oren Eini (Ayende Rahien)
Aug 9, 2016, 4:53:02 AM8/9/16
to ravendb
Anti Virus can do that, or driver issues.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
Brian Schau
Aug 9, 2016, 5:33:21 AM8/9/16
to RavenDB - 2nd generation document database
Well, we don't have any anti virus on the server and I see no trace in the event log of any driver issues.
Crap. I hate when problems like these arise. Gotta invest in a secondary server.
Crap. I hate when problems like these arise. Gotta invest in a secondary server.
Chris Marisic
Aug 9, 2016, 9:53:10 AM8/9/16
to RavenDB - 2nd generation document database
I strongly recommend a twice daily backup schedule. The most common schedule i use is 1pm and 11pm. If i had concerns about server health, i would easily increase this to 6am, 1pm, 6pm, 11pm.
The idea being the first backup gets all of the morning traffic. The second back up gets all of the evening traffic. Over night, few things happen (if it's a generic business system).
For true availability you need replication to do HA with a minimum of 2 servers, if not a triplicate set which is generally optimal in HA for reliability.
The idea being the first backup gets all of the morning traffic. The second back up gets all of the evening traffic. Over night, few things happen (if it's a generic business system).
For true availability you need replication to do HA with a minimum of 2 servers, if not a triplicate set which is generally optimal in HA for reliability.
Brian Schau
Aug 9, 2016, 3:06:45 PM8/9/16
to RavenDB - 2nd generation document database
Hi Chris,
Thanks for your input. I think we will reconfigure our backup to do incremental backup every 4 hours (and one full backup every week) in the short term.
And once that is up and running I'll add a hot-standby node (w. replication).
/brian
Thanks for your input. I think we will reconfigure our backup to do incremental backup every 4 hours (and one full backup every week) in the short term.
And once that is up and running I'll add a hot-standby node (w. replication).
/brian
Reply all
Reply to author
Forward
0 new messages