Failed to de-serialize metadata of document

[Igor K]

Hello,

I am unable to load a document or delete it due to this error.

Has this been encountered before?

System.IO.InvalidDataException: Failed to de-serialize metadata of document apikeys/3c93ae96-aa62-42b0-afa3-4bb693cbcdd4/xxxxxxxxxxxxxxxxxxxxxx ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.

   at Raven.Abstractions.Extensions.StreamExtensions.ReadEtag(Stream stream) in c:\Builds\RavenDB-Stable-3.0\Raven.Abstractions\Extensions\StreamExtensions.cs:line 162

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.ReadDocumentMetadata(String normalizedKey, Slice sliceKey, Int32& size)

   --- End of inner exception stack trace ---

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.ReadDocumentMetadata(String normalizedKey, Slice sliceKey, Int32& size)

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.DocumentByKey(String key)

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.<GetDocumentsWithIdStartingWith>d__15.MoveNext()

   at Raven.Database.Actions.DocumentActions.<>c__DisplayClassf.<GetDocumentsWithIdStartingWith>b__d(IStorageActionsAccessor actions)

   at Raven.Storage.Voron.TransactionalStorage.ExecuteBatch(Action`1 action)

   at Raven.Storage.Voron.TransactionalStorage.Batch(Action`1 action)

   at Raven.Database.Actions.DocumentActions.GetDocumentsWithIdStartingWith(String idPrefix, String matches, String exclude, Int32 start, Int32 pageSize, CancellationToken token, Int32& nextStart, Action`1 addDoc, String transformer, Dictionary`2 transformerParameters, String skipAfter)

   at Raven.Database.Actions.DocumentActions.GetDocumentsWithIdStartingWith(String idPrefix, String matches, String exclude, Int32 start, Int32 pageSize, CancellationToken token, Int32& nextStart, String transformer, Dictionary`2 transformerParameters, String skipAfter)

   at Raven.Database.Server.Controllers.DocumentsController.DocsGet()

   at lambda_method(Closure , Object , Object[] )

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()

   at Raven.Client.Connection.Implementation.HttpJsonRequest.<CheckForErrorsAndReturnCachedResultIfAnyAsync>d__20.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Implementation\HttpJsonRequest.cs:line 451

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Implementation.HttpJsonRequest.<>c__DisplayClasse.<<SendRequestInternal>b__d>d__10.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Implementation\HttpJsonRequest.cs:line 231

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at Raven.Client.Connection.Implementation.HttpJsonRequest.<RunWithAuthRetry>d__15`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Implementation\HttpJsonRequest.cs:line 295

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Implementation.HttpJsonRequest.<ReadResponseJsonAsync>d__9.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Implementation\HttpJsonRequest.cs:line 204

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Async.AsyncServerClient.<>c__DisplayClass1a7.<<StartsWithAsync>b__1a2>d__1ad.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Async\AsyncServerClient.cs:line 1271

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.ReplicationInformerBase`1.<TryOperationAsync>d__29`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\ReplicationInformerBase.cs:line 454

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.ReplicationInformerBase`1.<ExecuteWithReplicationAsync>d__19`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\ReplicationInformerBase.cs:line 343

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Client.Connection.Async.AsyncServerClient.<ExecuteWithReplication>d__2c9`1.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Connection\Async\AsyncServerClient.cs:line 0

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Raven.Abstractions.Util.AsyncHelpers.<>c__DisplayClassb`1.<<RunSync>b__8>d__d.MoveNext() in c:\Builds\RavenDB-Stable-3.0\Raven.Abstractions\Util\AsyncHelpers.cs:line 82

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Raven.Abstractions.Util.AsyncHelpers.RunSync[T](Func`1 task) in c:\Builds\RavenDB-Stable-3.0\Raven.Abstractions\Util\AsyncHelpers.cs:line 90

   at Raven.Client.Document.DocumentSession.LoadStartingWith[T](String keyPrefix, String matches, Int32 start, Int32 pageSize, String exclude, RavenPagingInformation pagingInformation, String skipAfter) in c:\Builds\RavenDB-Stable-3.0\Raven.Client.Lightweight\Document\DocumentSession.cs:line 1001

   at Transperfect.Portal.Web.Api.Api.ProjectAController.GenerateKey(String username) in C:\GitSource\TransPort\Transperfect.Portal\Transperfect.Portal.Web.Api\Api\ProjectAController.cs:line 730; Line number: 730; User: ; Error Path: ; Raw Url: ; Exception Type: Raven.Abstractions.Connection.ErrorResponseException; Client IP: 

Date: 2016-12-02 09:30:10; Level: Info; Log-Msg: FlexMembershipStore: Getting user by username. Username: ; Error-Msg: ; Source:  in -> | line:  file: ; Line number: ; User: ; Error Path: ; Raw Url: ; Exception Type: ; Client IP: 

Date: 2016-12-02 09:30:10; Level: Debug; Log-Msg: ProjectAController: Updating api key for yyyyyyyyyyy...@transperfect.com by user Users/9c332fce-5bdf-44d2-9511-708e845bc944; Error-Msg: ; Source: ; Line number: ; User: ; Error Path: ; Raw Url: ; Exception Type: ; Client IP: 

Date: 2016-12-02 09:30:10; Level: Error; Log-Msg: ; Error-Msg: Url: "/databases/TransPort/docs?startsWith=ApiKeys%2F&matches=*/yyyyyyyyyyy...@transperfect.com&exclude=&start=0&pageSize=25"

System.IO.InvalidDataException: Failed to de-serialize metadata of document apikeys/3c93ae96-aa62-42b0-afa3-4bb693cbcdd4/xxxxxxxxxxxxxxxxxxxxxx ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.

   at Raven.Abstractions.Extensions.StreamExtensions.ReadEtag(Stream stream) in c:\Builds\RavenDB-Stable-3.0\Raven.Abstractions\Extensions\StreamExtensions.cs:line 162

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.ReadDocumentMetadata(String normalizedKey, Slice sliceKey, Int32& size)

   --- End of inner exception stack trace ---

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.ReadDocumentMetadata(String normalizedKey, Slice sliceKey, Int32& size)

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.DocumentByKey(String key)

   at Raven.Database.Storage.Voron.StorageActions.DocumentsStorageActions.<GetDocumentsWithIdStartingWith>d__15.MoveNext()

   at Raven.Database.Actions.DocumentActions.<>c__DisplayClassf.<GetDocumentsWithIdStartingWith>b__d(IStorageActionsAccessor actions)

   at Raven.Storage.Voron.TransactionalStorage.ExecuteBatch(Action`1 action)

   at Raven.Storage.Voron.TransactionalStorage.Batch(Action`1 action)

   at Raven.Database.Actions.DocumentActions.GetDocumentsWithIdStartingWith(String idPrefix, String matches, String exclude, Int32 start, Int32 pageSize, CancellationToken token, Int32& nextStart, Action`1 addDoc, String transformer, Dictionary`2 transformerParameters, String skipAfter)

   at Raven.Database.Actions.DocumentActions.GetDocumentsWithIdStartingWith(String idPrefix, String matches, String exclude, Int32 start, Int32 pageSize, CancellationToken token, Int32& nextStart, String transformer, Dictionary`2 transformerParameters, String skipAfter)

   at Raven.Database.Server.Controllers.DocumentsController.DocsGet()

   at lambda_method(Closure , Object , Object[] )

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)

   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext(); Source: Raven.Abstractions.Connection.ErrorResponseException: Url: "/databases/TransPort/docs?startsWith=ApiKeys%2F&matches=*/yyyyyyyyyyy...@transperfect.com&exclude=&start=0&pageSize=25"

Replication also stopped due to this error.

Please help.

[Oren Eini (Ayende Rahien)]

What build are you using? What storage engine?

Did you had hardware issues? This looks like a problem in the  storage itself.

Also, it looks like you have api keys in there that you might want to change.

[Igor K]

I am using 3.0.30153, Voron.

We noticed 4 related logs this week where an index threw a Disk Full exception, despite the disk having enough space.

I just want to delete the corrupted records.

These API keys are not RavenDB api keys, they are our own objects.

Our IT department recently (2 weeks ago) added a backup tool to our VMs called VEEAM.

[Oren Eini (Ayende Rahien)]

You can just issue a delete to those documents, IIRC, and that would work.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Igor K]

We tried deleting it, but no matter how we tried the doc metadata needs to be loaded apparently and that step fails.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Try overwriting it and then deleting it?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Igor K]

This produces the same error.

Running Raven in a separate process produces the attached error which is visible on the server.

We tried loading the file via index query, session load, http call, no matter how we try to load or delete it give the same error.

This corrupt record also gets included in all backups we tried.

[Oren Eini (Ayende Rahien)]

Can you export the database? Otherwise, you can use the data recovery tool:

https://github.com/ravendb/ravendb/tree/v3.5/Raven.StorageExporter

It is available here:

http://hibernatingrhinos.com/downloads/RavenDB%20Tools/35180

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Igor K]

We are trying to do this now.

We will let you know what the result is.

[Igor K]

Hi Oren,

Based on the source code of Raven.StorageExporter I wrote a small console application that creates a database configuration object, starts a transactional storage. It gives me the ability to manipulate documents at a much lower level.

In DocumentStorageActions class I found a private method WriteDocumentMetadata, I wrote a public method to expose it. This way I was able to overwrite the metadata of this erroneous object and make it accessible through the API, we were also able to load and delete it from the database in Raven Studio.

Here is the text of my console app:

static void Main(string[] args)
       {
           var ravenConfiguration = new RavenConfiguration
           {
               DataDirectory = ConfigurationManager.AppSettings["DataDirectory"],
               CacheDocumentsInMemory = false,
               RunInMemory = false,
               Storage =
               {
                   PreventSchemaUpdate = true,
                   SkipConsistencyCheck = true
               }
           };
           Console.WriteLine("Configuration initialized.");

           CreateTransactionalStorage(ravenConfiguration);
           Console.WriteLine("Transactional storage created.");

           //Stream sDoc = null;
           //JsonDocument jDoc = null;
           //storage.Batch(a => jDoc = a.Documents.DocumentByKey("Lookup/Currencies"));

           try
           {
               /**
                * Try deleting the document
                */
               //RavenJObject outMetadata;
               //Etag outEtag;

               //string documentKey = ConfigurationManager.AppSettings["DocumentKey"];
               //Etag etag = ConfigurationManager.AppSettings["InvalidEtag"] == "true" ? Etag.InvalidEtag : null;

               //storage.Batch(a => a.Documents.DeleteDocument(documentKey, etag, out outMetadata, out outEtag));


               /**
                * Trying to overwrite the metadata
                */
               JsonDocumentMetadata metadata = new JsonDocumentMetadata()
               {
                   Etag = Etag.Empty,
                   Key = ConfigurationManager.AppSettings["DocumentKey"],
                   Metadata = new RavenJObject()
               };
               storage.Batch(a => a.Documents.WriteDocumentMetadataPub(metadata, ConfigurationManager.AppSettings["DocumentKey"], true));


               Console.WriteLine("Operation succeeded.");
           }
           catch (Exception ex)
           {
               throw ex;
           }
           finally
           {
               storage.Dispose();
           }

           Console.Read();
       }


Please let us know if this may have any repercussions.

Regards,
Iskandar

[Iskandar Rafiev]

Hi Oren,

I used the source code of the Raven.StorageExporter which creates a database configuration file and loads the database file, then added a public function to DocumentsStorageActions that gives me access to private WriteDocumentMetadata method:

public bool WriteDocumentMetadataPub(JsonDocumentMetadata metadata, Slice key, bool shouldIgnoreConcurrencyExceptions = false)
        {
            return WriteDocumentMetadata(metadata, key, true);
        }

then I called this method with the following parameters:

JsonDocumentMetadata metadata = new JsonDocumentMetadata()
                {
                    Etag = Etag.Empty,

                    Key = ConfigurationManager.AppSettings["DocumentKey"], // broken document key
                    Metadata = new RavenJObject()
                };


Slice key = ConfigurationManager.AppSettings["DocumentKey"]; // broken document key


shouldignoreConcurrencyExceptions = true;

This overwrote the object's metadata and we were able to load it in Raven Studio and delete it from the database.

The full text of the console application:

Please let us know if this is going to cause any repercussions.

Regards,

Iskandar

[Iskandar Rafiev]

Hi Oren,

Based on the source code of Raven.StorageExporter I wrote a small console application that creates a database configuration object, starts a transactional storage. It gives me the ability to manipulate documents at a much lower level.

In DocumentStorageActions class I found a private method WriteDocumentMetadata, I wrote a public method to expose it. This way I was able to overwrite the metadata of this erroneous object and make it accessible through the API, we were also able to load and delete it from the database in Raven Studio.

Here is the text of my console app:

Please let us know if this may have any repercussions.

Regards,

Iskandar

On Friday, December 2, 2016 at 9:58:56 AM UTC-5, Igor K wrote:

[Oren Eini (Ayende Rahien)]

For safety's sake, please export the db, create a new one and import it.

It is possible that there is additional issues that you are not currently seeing.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.