Problem going from 4.0 RC to final

[Sander Rijken]

I'm running into this problem after upgrading the server to 4.0 final.

Any idea how to fix this?

Raven.Client.Exceptions.Database.DatabaseLoadFailureException: Failed to start database *****

At C:\Raven\Databases\***** ---> Voron.Exceptions.VoronUnrecoverableErrorException: Specified argument was out of the range of valid values.

Parameter name: index ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.

Parameter name: index

   at Voron.Data.Tables.TableValueReader.ThrowIndexOutOfRange() in C:\Builds\RavenDB-Stable-4.0\src\Voron\Data\Tables\TableValueReader.cs:line 91

   at Voron.Data.Tables.TableValueReader.Read(Int32 index, Int32& size) in C:\Builds\RavenDB-Stable-4.0\src\Voron\Data\Tables\TableValueReader.cs:line 75

   at Voron.Data.Tables.Table.DeleteValueFromIndex(Int64 id, TableValueReader& value) in C:\Builds\RavenDB-Stable-4.0\src\Voron\Data\Tables\Table.cs:line 401

   at Voron.Data.Tables.Table.Delete(Int64 id) in C:\Builds\RavenDB-Stable-4.0\src\Voron\Data\Tables\Table.cs:line 315

   at Voron.Data.Tables.Table.Update(Int64 id, TableValueBuilder builder, Boolean forceUpdate) in C:\Builds\RavenDB-Stable-4.0\src\Voron\Data\Tables\Table.cs:line 256

   at Voron.Data.Tables.Table.Set(TableValueBuilder builder, Boolean forceUpdate) in C:\Builds\RavenDB-Stable-4.0\src\Voron\Data\Tables\Table.cs:line 1297

   at Raven.Server.Storage.Schema.Updates.Documents.From14.Update(UpdateStep step) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Storage\Schema\Updates\Documents\From14.cs:line 57

   at Raven.Server.Storage.Schema.SchemaUpgrader.InternalUpgrader.Upgrade(Transaction readTx, Transaction writeTx, Int32 currentVersion, Int32& versionAfterUpgrade) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Storage\Schema\SchemaUpgrader.cs:line 92

   at Voron.StorageEnvironment.UpgradeSchema(Int32 schemaVersionVal, UpgraderDelegate upgrader) in C:\Builds\RavenDB-Stable-4.0\src\Voron\StorageEnvironment.cs:line 374

   at Voron.StorageEnvironment.UpgradeSchemaIfRequired() in C:\Builds\RavenDB-Stable-4.0\src\Voron\StorageEnvironment.cs:line 352

   --- End of inner exception stack trace ---

   at Voron.Exceptions.VoronUnrecoverableErrorException.Raise(StorageEnvironment env, String message, Exception inner) in C:\Builds\RavenDB-Stable-4.0\src\Voron\Exceptions\VoronUnrecoverableErrorException.cs:line 49

   at Voron.StorageEnvironment.UpgradeSchemaIfRequired() in C:\Builds\RavenDB-Stable-4.0\src\Voron\StorageEnvironment.cs:line 358

   at Voron.StorageEnvironment.LoadExistingDatabase() in C:\Builds\RavenDB-Stable-4.0\src\Voron\StorageEnvironment.cs:line 321

   at Voron.StorageEnvironment..ctor(StorageEnvironmentOptions options) in C:\Builds\RavenDB-Stable-4.0\src\Voron\StorageEnvironment.cs:line 145

   at Raven.Server.Documents.DocumentsStorage.Initialize(StorageEnvironmentOptions options) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DocumentsStorage.cs:line 315

   at Raven.Server.Documents.DocumentsStorage.Initialize(Boolean generateNewDatabaseId) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DocumentsStorage.cs:line 252

   at Raven.Server.Documents.DocumentDatabase.Initialize(InitializeOptions options) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DocumentDatabase.cs:line 291

   at Raven.Server.Documents.DatabasesLandlord.CreateDocumentsStorage(StringSegment databaseName, RavenConfiguration config) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DatabasesLandlord.cs:line 541

   --- End of inner exception stack trace ---

   at Raven.Server.Documents.DatabasesLandlord.CreateDocumentsStorage(StringSegment databaseName, RavenConfiguration config) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DatabasesLandlord.cs:line 558

   at Raven.Server.Documents.DatabasesLandlord.ActuallyCreateDatabase(StringSegment databaseName, RavenConfiguration config) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DatabasesLandlord.cs:line 494

   at Raven.Server.Documents.DatabasesLandlord.<>c__DisplayClass29_0.<CreateDatabaseUnderResourceSemaphore>b__0() in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Documents\DatabasesLandlord.cs:line 446

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Server.Routing.RouteInformation.<UnlikelyWaitForDatabaseToLoad>d__14.MoveNext() in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Routing\RouteInformation.cs:line 120

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Server.Routing.RouteInformation.<WaitForDb>d__19.MoveNext() in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Routing\RouteInformation.cs:line 158

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Server.Routing.RequestRouter.<HandlePath>d__6.MoveNext() in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Routing\RequestRouter.cs:line 63

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()

   at Raven.Server.RavenServerStartup.<RequestHandler>d__11.MoveNext() in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\RavenServerStartup.cs:line 159

[Oren Eini (Ayende Rahien)]

Hi,

Can you send the database to us so we can look at it?

Alternatively, you can use the Voron.Recovery tool to generate an export and then import the db.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sander Rijken]

This was a 2 node cluster, I managed to fix it by exporting from the other (not yet updated) node, and importing that. I can see if I still have the DB if you want?

Sander

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

If you can send it, that would be great, yes

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Sander Rijken]

I'm still seeing problems by the way, I think it's the same error still:

2018-02-19T19:37:32.3840747Z, 9, Information, B, Raven.Server.Rachis.RachisConsensus, Failed to talk to remote follower: A, EXCEPTION: System.IO.InvalidDataException: Expected to get type of 'InstallSnapshotResponse' message, but got unknown message: {"Type":"Error","ExceptionType":"VoronErrorException","Message":"Invalid index EtagIndexName on Items, attempted to delete value but the value from 8858531 wasn't in the index","Exception":"Voron.Exceptions.VoronErrorException: Invalid index EtagIndexName on Items, attempted to delete value but the value from 8858531 wasn't in the index\r\n   at Voron.Data.Tables.Table.ThrowInvalidAttemptToRemoveValueFromIndexAndNotFindingIt(Int64 id, Slice indexDefName) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Voron\\Data\\Tables\\Table.cs:line 380\r\n   at Voron.Data.Tables.Table.DeleteValueFromIndex(Int64 id, TableValueReader& value) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Voron\\Data\\Tables\\Table.cs:line 388\r\n   at Voron.Data.Tables.Table.Delete(Int64 id) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Voron\\Data\\Tables\\Table.cs:line 315\r\n   at Raven.Server.Rachis.Follower.InstallSnapshot(TransactionOperationContext context) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\Rachis\\Follower.cs:line 555\r\n   at Raven.Server.Rachis.Follower.NegotiateWithLeader(TransactionOperationContext context, LogLengthNegotiation negotiation) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\Rachis\\Follower.cs:line 410\r\n   at Raven.Server.Rachis.Follower.Run(Object obj) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\Rachis\\Follower.cs:line 786"}

   at Raven.Server.Rachis.RemoteConnection.ThrowUnexpectedMessage(String type, String expectedType, BlittableJsonReaderObject json) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Rachis\RemoteConnection.cs:line 500

   at Raven.Server.Rachis.RemoteConnection.ValidateMessage(String expectedType, BlittableJsonReaderObject json) in C:\Builds\RavenDB-Stable-4.0\src\Raven.Server\Rachis\Rem

Also the cluster keeps voting, term number is at 5862 now, is that normal behavior?

Sander

[Oren Eini (Ayende Rahien)]

No, that is a migration problem. 

Please export / import the database, I guess we missed something when doing the upgrade.

Details: we change an internal structure, and it _should_ be auto updated. In your case, it looked like it failed, and that cause the issue

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Note that this is actually from the _server_, not the database, so might be easier to drop the system store and recreate the cluster

[Sander Rijken]

OK, there's just one DB in there, so I'll just export that, reconfigure everything, and start over; that's probably easiest.. Also fixes the server cert for now :)

-- 

Sander Rijken

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/-Jx9bnGysvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+unsubscribe@googlegroups.com.