NullReferenceException for all database operations

[Andrej Krivulčík]

All operations (studio and C# code) with my production database fail with the following:

System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object.

   at Raven.Database.Indexing.WorkContext.<>c.<RecoverIndexingErrors>b__91_1(ListItem x) in C:\Builds\RavenDB-3.5-RC\Raven.Database\Indexing\WorkContext.cs:line 151

   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)

   at System.Linq.OrderedEnumerable`1.<GetEnumerator>d__0.MoveNext()

   at Raven.Database.Indexing.WorkContext.RecoverIndexingErrors() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Indexing\WorkContext.cs:line 153

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, DocumentDatabase systemDatabase, TransportState recievedTransportState) in C:\Builds\RavenDB-3.5-RC\Raven.Database\DocumentDatabase.cs:line 261

   at Raven.Database.Server.Tenancy.DatabasesLandlord.<>c__DisplayClass27_0.<TryGetOrCreateResourceStore>b__1() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Server\Tenancy\DatabaseLandlord.cs:line 200

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Database.Server.Tenancy.DatabasesLandlord.<GetResourceInternal>d__26.MoveNext() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Server\Tenancy\DatabaseLandlord.cs:line 148

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)

   at Raven.Database.Common.ResourceApiController`2.get_Resource() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Common\ResourceApiController.cs:line 156

   at Raven.Database.Common.ResourceApiController`2.get_MaxSecondsForTaskToWaitForResourceToLoad() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Common\ResourceApiController.cs:line 46

   at Raven.Database.Common.ResourceApiController`2.<TrySetupRequestToProperResource>d__33.MoveNext() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Common\ResourceApiController.cs:line 258

---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.

   at Raven.Database.Indexing.WorkContext.<>c.<RecoverIndexingErrors>b__91_1(ListItem x) in C:\Builds\RavenDB-3.5-RC\Raven.Database\Indexing\WorkContext.cs:line 151

   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)

   at System.Linq.OrderedEnumerable`1.<GetEnumerator>d__0.MoveNext()

   at Raven.Database.Indexing.WorkContext.RecoverIndexingErrors() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Indexing\WorkContext.cs:line 153

   at Raven.Database.DocumentDatabase..ctor(InMemoryRavenConfiguration configuration, DocumentDatabase systemDatabase, TransportState recievedTransportState) in C:\Builds\RavenDB-3.5-RC\Raven.Database\DocumentDatabase.cs:line 261

   at Raven.Database.Server.Tenancy.DatabasesLandlord.<>c__DisplayClass27_0.<TryGetOrCreateResourceStore>b__1() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Server\Tenancy\DatabaseLandlord.cs:line 200

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.Tasks.Task.Execute()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Database.Server.Tenancy.DatabasesLandlord.<GetResourceInternal>d__26.MoveNext() in C:\Builds\RavenDB-3.5-RC\Raven.Database\Server\Tenancy\DatabaseLandlord.cs:line 148<---

This is #35169.

This was the sequence of events before this issue:

1. Ran a program with new MyIndex().SideBySideExecute(documentStore), twice. Both attempts failed with a "Task was cancelled" error.
2. Restarted the RavenDB database. This took a long time but the database stopped on its own. Actually crashed, see below.
3. Started the database.
4. The database was very active after start, for a long time. I let it finish all activity.
5. The database is unusable, as any operation ends with the exception above.

I'm attaching the event export from the crash/start. The first event (09:12:37 UTC) is the previous successful start, no events are logged between that and the crash (12:20:23 UTC). The files mentioned in the "Windows Error Reporting" events are not present on the server.

Any suggestions about how to recover from this situation?

[Andrej Krivulčík]

Attaching the events.

[Oren Eini (Ayende Rahien)]

http://issues.hibernatingrhinos.com/issue/RavenDB-5349

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Tue, Sep 27, 2016 at 4:11 PM, Andrej Krivulčík <kriv...@gmail.com> wrote:

Attaching the events.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Grisha Kotler]

Hi,

How old is that database? (When was it created?) Did you upgrade from 3.0?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

--

[Andrej Krivulčík]

The database is new, I restored it from export after it failed (here: https://groups.google.com/forum/#!topic/ravendb/zbp0xrvZQIw , there was subsequent private communication with Tal Weiss).

Database was created on 2016-09-16, on the same build - #35169 - no upgrades.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Grisha Kotler]

Can you send the database?

Hibernating Rhinos Ltd  

Grisha Kotler l RavenDB Core Team Developer l Mobile: +972-54-586-8647

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

RavenDB paving the way to "Data Made Simple" http://ravendb.net/

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Andrej Krivulčík]

I can, but it's 200+ GB. I'll prepare it and send a download link. It's much smaller compressed.

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/dgoj9it7Xtw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.

[Andrej Krivulčík]

I sent the download link to sup...@ravendb.net.

[Andrej Krivulčík]

Any progress on this? I've recovered the database from previous export since then. The smuggler import failed 2 times before I could recover it successfully, but I can't easily reproduce it. Here's the console output if anyone is interested: https://gist.github.com/krivulcik/1d3331be9d5f1b401e4153fddaa23219 . The database was running just fine before, during and after the import, I didn't notice anything unusual.

This was the second failure, I didn't save the console output for the first one.

[Fitzchak Yitzchaki]

Hello,

Sorry for the late response, we had the holidays in the beginning of this week.

I'm downloading the file and will investigate this.

Best Regards,

Hibernating Rhinos Ltd  

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

[Maxim Buryak]

Andrej, 

do you still have the RavenDB server side logs from the time of the import failures?

[Andrej Krivulčík]

If they are database-specific then no, I deleted the database (and all the files) before the successful import. If the logs are server-wide, they might be available but I'll need a hint where to find them. The Admin Logs (/studio/index.html#admin/settings/adminLogs) don't contain historic logs. I can't find anything relevant in Windows Event Viewer.

[Fitzchak Yitzchaki]

Hello,

We was able to reproduce the issue with your data. 

How did you got to this stage?

How long ago this database was created?

Did you restored this database from a backup or import?

Best Regards,

Hibernating Rhinos Ltd  

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

[Andrej Krivulčík]

Inline below:

On Thursday, October 6, 2016 at 2:59:10 PM UTC+2, Fitzchak Yitzchaki wrote:

Hello,

We was able to reproduce the issue with your data. 

How did you got to this stage?

Detailed steps are in the original post. This is the maximum detail I could remember:

1. Ran a program with new MyIndex().SideBySideExecute(documentStore), twice. Both attempts failed with a "Task was cancelled" error.
2. Restarted the RavenDB database. This took a long time but the database stopped on its own. Actually crashed, see below.
3. Started the database.
4. The database was very active after start, for a long time. I let it finish all activity.
5. The database is unusable, as any operation ends with the exception above.

 

How long ago this database was created?

Database was created on 2016-09-16, on the same build - #35169 - no upgrades.

 

Did you restored this database from a backup or import?

Import, using smuggler: 

PS .....\RavenDB-3.5.0-rc-35166.Tools> .\Raven.Smuggler.exe in http://localhost:80 G:\data-prod-backup\2016-09-25-20-51.ravendb-full-dump --database=data-prod --negative-metadata-filter:@id=Raven/Replication/Destinations --negative-metadata-filter:@id=Raven/Backup/Periodic/Setup --negative-metadata-filter:@id=Raven/Backup/Periodic/Status --negative-metadata-filter:@id=Raven/Replication/VersionHilo

 

Best Regards,

Hibernating Rhinos Ltd  

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

On Thu, Oct 6, 2016 at 2:40 PM, Andrej Krivulčík <kriv...@gmail.com> wrote:

If they are database-specific then no, I deleted the database (and all the files) before the successful import. If the logs are server-wide, they might be available but I'll need a hint where to find them. The Admin Logs (/studio/index.html#admin/settings/adminLogs) don't contain historic logs. I can't find anything relevant in Windows Event Viewer.

On Thursday, October 6, 2016 at 12:56:50 PM UTC+2, Maxim Buryak wrote:

Andrej, 

do you still have the RavenDB server side logs from the time of the import failures?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Andrej Krivulčík]

There might be confusion about the database creation date and the smuggler command used to restore it. The command was actually:

PS .....\RavenDB-3.5.0-rc-35166.Tools> .\Raven.Smuggler.exe in http://localhost:80 G:/data-prod-backup\2016-09-12-14-24.ravendb-full-dump --database=data-prod --negative-metadata-filter:@id=Raven/Replication/Destinations --negative-metadata-filter:@id=Raven/Backup/Periodic/Setup --negative-metadata-filter:@id=Raven/Backup/Periodic/Status --negative-metadata-filter:@id=Raven/Replication/VersionHilo

The command I pasted before was copied from restoring the database after this failure. The command in this message was used to restore the data before this failure (after the other failure: https://groups.google.com/forum/#!topic/ravendb/zbp0xrvZQIw).

[Fitzchak Yitzchaki]

There is some sort of data corruption that I found, which prevented the db to come up.

I fixed this issue, and now the database come up:

https://github.com/ayende/ravendb/pull/4539

Please try with the next build and let us know.

It is a good idea to export the data and import it again to a new database.

Best Regards,

Hibernating Rhinos Ltd  

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Andrej Krivulčík]

Thanks, I'll try this when the new build will be available. I still have the corrupted database available.

I already recovered the database from an export, as I need to work with it.

Thanks for looking into this.

[Andrej Krivulčík]

Opening the database with RC3 (build #35173) works okay, the database starts correctly.

Thanks for taking care of this.