How to run recovery

[Phillip Haydon]

My development PC randomly rebooted itself and now when starting up raven.server, it throws an exception:

D:\Development\packages\RavenDB.1.0.603-Unstable\server>Raven.Server.exe

A critical error occurred while starting the server. Please see the exception details bellow for mor

e details:

System.InvalidOperationException: Could not open transactional storage: D:\Development\itcompiler.co

m\packages\RavenDB.1.0.603-Unstable\server\D\Raven-Data\Data ---> Microsoft.Isam.Esent.Interop.Esent

DatabaseDirtyShutdownException: Database was not shutdown cleanly. Recovery must first be run to pro

perly complete database operations for the previous shutdown.

at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in C:\Work\ravendb\SharedLibs\Sources\manage

desent-61618\EsentInterop\Api.cs:line 2736

at Raven.Storage.Esent.TransactionalStorage.EnsureDatabaseIsCreatedAndAttachToDatabase() in c:\Bu

ilds\raven-unstable\Raven.Storage.Esent\TransactionalStorage.cs:line 292

at Raven.Storage.Esent.TransactionalStorage.Initialize(IUuidGenerator uuidGenerator) in c:\Builds

\raven-unstable\Raven.Storage.Esent\TransactionalStorage.cs:line 209

--- End of inner exception stack trace ---

at Raven.Storage.Esent.TransactionalStorage.Initialize(IUuidGenerator uuidGenerator) in c:\Builds

\raven-unstable\Raven.Storage.Esent\TransactionalStorage.cs:line 222

at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration) in c:\Builds\r

aven-unstable\Raven.Database\DocumentDatabase.cs:line 181

at Raven.Server.RavenDbServer..ctor(RavenConfiguration settings)

at Raven.Server.Program.RunServerInDebugMode(RavenConfiguration ravenConfiguration, Boolean lauch

Browser)

at Raven.Server.Program.RunInDebugMode(Nullable`1 anonymousUserAccessMode, RavenConfiguration rav

enConfiguration, Boolean launchBrowser)

at Raven.Server.Program.<>c__DisplayClass1a.<InteractiveRun>b__10()

at Raven.Server.Program.InteractiveRun(String[] args)

at Raven.Server.Program.Main(String[] args)

Press any key to continue...

How can i run recovery?

[Matt Warren]

You need to use the command line tool esentutl, see http://support.microsoft.com/kb/930832
for more info.

You can also type this at a command line prompt for more info:
esentutl /?

I think you'll need to run the command prompt as Administrator.

Also I recommend making a backup of the entire folder first, it's a
low-level tool.

On Jan 28, 3:29 am, Phillip Haydon <phillip.hay...@gmail.com> wrote:
> My development PC randomly rebooted itself and now when starting up
> raven.server, it throws an exception:
>
> D:\Development\packages\RavenDB.1.0.603-Unstable\server>Raven.Server.exe
> A critical error occurred while starting the server. Please see the
> exception details bellow for mor
> e details:
> System.InvalidOperationException: Could not open transactional storage:
> D:\Development\itcompiler.co
> m\packages\RavenDB.1.0.603-Unstable\server\D\Raven-Data\Data --->
> Microsoft.Isam.Esent.Interop.Esent

> DatabaseDirtyShutdownException: *Database was not shutdown cleanly.
> Recovery must first be run to pro*
> *perly complete database operations for the previous shutdown.*

[Oren Eini (Ayende Rahien)]

esentutl /r is how you do that.

[Fitzchak Yitzchaki]

I create a feature request to investigate auto detecting this situation on startup:

http://issues.hibernatingrhinos.com/issue/RavenDB-4 

[Phillip Haydon]

Since it was on my dev pc I ended up just blowing away the data folder and starting again, re-importing all the data. 

But good to know there's a solution so if it occurs in the future it can be fixed.

Cheers.

[Chris Marisic]

I went to post a comment on this issue but got "Can't process openId return URL Constrains validation exception. Causes: 1: Can't create user due to license restrictions: users limit exceeded. " when i tried to login using openid.

Something that would be really awesome with this feature is if you are able to cause data to be dumped out that is showing the transaction that are about to be lost. From the way I understand it, the /r option with esentutl will remove atleast 1 uncompleted transaction permanently correct? Anything that could be dumped out unless it's raw binary would likely provide alot of value in figuring out what transaction(s) were skipped and to figure out which documents need to be corrected.

[Oren Eini (Ayende Rahien)]

That is actually handled at a lower level than ravendb, so I don't think that we can do that.

[Chris Marisic]

Could you physically ready the transaction log? Like File.Read, or is it all compiled and would be pointless?

[Oren Eini (Ayende Rahien)]

The transaction log is binary data, mostly full with things like file pointers, offsets, and partial dumps.

It wouldn't make sense to a human without a LOT of work.

[mare]

This is such a stupid error. Running RavenDB.Embedded and shutting down the IIS Express during testing the site locally results in this.

 

I mean, seriously?

 

And this site was running in read only mode, no writes were performed to it? Then I tried to copy the entire Raven DB data directory (App_Data in my case) and it said files were in use/locked. That's when I shutdown IIS Express, figuring nothing could be wrong with that, because I am used to do it all the time with SQL Server and other JSON&XML file based storage. Only to found out it leaves Raven DB  in dirty state.

 

Why would you even want to implement it to work with Microsoft.Isam.Esent.Interop.EsentDatabaseDirtyShutdownException?

 

so many strange design decision in this RavenDB...should be enough reasons for me to try out Mongo or Couch..

[codin...@googlemail.com]

In England we use the word mare as shortform to mean nightmare...

Just saying...

Sent from my iPhone

[mare]

hahaha how insightful and funny;)

yet completely irrelevant to the topic and a valid problem at hand..

[Oren Eini (Ayende Rahien)]

Mare,

Shutting down IIS may mean that you have effectively killed the process while it was doing something.

And while you may not think that you were doing writes, RavenDB needs to manage a lot of things, and it would do some writes even if all you are doing are queries.

In general, shutting down in this manner works, and we will auto recover without an issue.

[mare]

Hey Oren,

 

Clearly it didn't in my case and when I run into such errors even in my Development scenarios I start to doubt the robustness of a solution. Whatever it was doing, it shouldn't be doing it anymore (it was few minutes after the last time I refreshed the site locally that I shutdown the IIS Express, so all under the hood processes should be already finished by then).

 

Would you mind providing a little bit more detail on how and where to run esentutl rather than just saying run esentutl /r?

 

I ran it within "logs" folder and this is what it did:

 

> esentutl /r rvn

 

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 6.1
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode...
    Logfile base name: rvn
            Log files: <current directory>
         System files: <current directory>

Performing soft recovery...

Operation completed successfully in 0.140 seconds.

 

However, database is still in dirty shutdown mode...

[Oren Eini (Ayende Rahien)]

From the ravendb directory NOT the logs directory.

esentutl /r RVN /l logs /s system

[mare]

still no change, still in recovery mode.

 

I guess I'll have to scrap this DB, it was only for testing purposes but still..I'm kinda skeptical what to do when this happens in production..I'm sure it will happen, you can't predict all the scenarios..

 

What is the correct way to shutdown RavenDB? For instance, when running embedded within IIS Express (Ctrl+F5 from VS2012)?

[Oren Eini (Ayende Rahien)]

Call EmbeddedDocumentStore.Dispose

[Justin A]

Mare - u said you're using RavenDb with IIS ---EXPRESS---

I'm taking a wild stab at whatever/whoever gets close to me .. but i'm going to guess this means you're doing some -development- work on your dev/localhost machine?

If so ... have a peeky-poo at this doc: http://ravendb.net/kb/22/using-ravendb-in-day-to-day-development

At the bottom of that doc, there is a link to a YouTube video explaining in detail, the content of that document.

Maybe that might make things waaaay better for you?

[mare]

Thanks Justion on this. Actually this might come in handy but not all the time. Mostly I am dealing with projects that do require some persistant storage at least for a couple of days.

 

On the topic of recovery - I was unable to make recovery work no matter which switches I used BUT I was able to make the REPAIR work as Matt also said it in his thread https://groups.google.com/forum/?fromgroups#!topic/ravendb/R6mOsG4Io5M

 

Oren, maybe check this out because it seems that recovery doesn't work at all, while the repair (esentutl /p switch) warns twice about doing it before the database is in clean shutdown mode but still manages to bring the database back online.

[Justin A]

if you want to have some persistent storage, then follow those instructions .. but don't do the -RAM thing. just double-click ravendb.server.exe instead.

Personally, the only reason I would use Embedded is

1) unit/integration TESTS

2) some application that would never have access to a) IIS or b) install the raven service thingy.

but each to their own. GL! 

(at least you're using RavenDB :) that's the first major win!)