3.0.3498 - More detail of Voron scratch buffer error

67 views
Skip to first unread message

James Tan

unread,
Oct 18, 2014, 9:58:29 PM10/18/14
to rav...@googlegroups.com
Testing environment, isolated master to slave (3.0.3496) replication, No compression 

1. The scratch buffer error started after about 970K doc replicated.
2. This error is unable recover itself, had to restart slave


2014-10-18 21:41:09.4091,Raven.Bundles.Replication.Tasks.ReplicationTask,Warn,ltss,Replication to V3Voron had failed,"System.Net.WebException: Error: System.AggregateException: Error when executing write ---> Voron.Exceptions.ScratchBufferSizeLimitException: Cannot allocate more space for the scratch buffer.
Current size is: 393,280 KB.
Limit: 524,288 KB.
Requested Size: 393,284 KB.
Already flushed and waited for 5,015 ms for read transactions to complete.
Do you have a long running read transaction executing?
Debug info:
Requested number of pages: 1 (NearestPowerOfTwo: 1)
Oldest active transaction: 357 (snapshot: 357)
Oldest active transaction when flush was forced: 357
Next write transaction id: 358
Active transactions:
Id: 357 - Read
Scratch files usage:
scratch.0000000004.buffers - size: 393,280 KB, in active use: 12,736 KB
scratch.0000000003.buffers - size: 524,352 KB, in active use: 240,396 KB
Most available free pages:
scratch.0000000004.buffers
Size:1, ValidAfterTransactionId: 358
Size:32, ValidAfterTransactionId: 358
Size:64, ValidAfterTransactionId: 358
Size:4, ValidAfterTransactionId: 358
Size:16, ValidAfterTransactionId: 358
Size:8, ValidAfterTransactionId: 358
Size:2, ValidAfterTransactionId: 358
scratch.0000000003.buffers
Size:1, ValidAfterTransactionId: 352
Size:2, ValidAfterTransactionId: 352
Size:4, ValidAfterTransactionId: 352
Size:32, ValidAfterTransactionId: 352
Size:16, ValidAfterTransactionId: 344
Size:8, ValidAfterTransactionId: 338
Size:64, ValidAfterTransactionId: 344

   at Voron.Impl.Scratch.ScratchBufferPool.Allocate(Transaction tx, Int32 numberOfPages)
   at Voron.Impl.Transaction.InitTransactionHeader()
   at Voron.Impl.Transaction..ctor(StorageEnvironment env, Int64 id, TransactionFlags flags, IFreeSpaceHandling freeSpaceHandling)
   at Voron.StorageEnvironment.NewTransaction(TransactionFlags flags, Nullable`1 timeout)
   at Voron.Impl.TransactionMergingWriter.HandleActualWrites(OutstandingWrite mine, CancellationToken token)
   --- End of inner exception stack trace ---
   at Voron.Impl.TransactionMergingWriter.Write(WriteBatch batch)
   at Raven.Database.Storage.Voron.Impl.TableStorage.Write(WriteBatch writeBatch) in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Storage\Voron\Impl\TableStorage.cs:line 139
   at Raven.Storage.Voron.TransactionalStorage.ExecuteBatch(Action`1 action) in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Storage\Voron\TransactionalStorage.cs:line 195
   at Raven.Storage.Voron.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Storage\Voron\TransactionalStorage.cs:line 173
   at Raven.Database.Bundles.Replication.Controllers.ReplicationController.<DocReplicatePost>d__c.MoveNext() in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Bundles\Replication\Controllers\ReplicationController.cs:line 140
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
---> (Inner Exception #0) Voron.Exceptions.ScratchBufferSizeLimitException: Cannot allocate more space for the scratch buffer.
Current size is: 393,280 KB.
Limit: 524,288 KB.
Requested Size: 393,284 KB.
Already flushed and waited for 5,015 ms for read transactions to complete.
Do you have a long running read transaction executing?
Debug info:
Requested number of pages: 1 (NearestPowerOfTwo: 1)
Oldest active transaction: 357 (snapshot: 357)
Oldest active transaction when flush was forced: 357
Next write transaction id: 358
Active transactions:
Id: 357 - Read
Scratch files usage:
scratch.0000000004.buffers - size: 393,280 KB, in active use: 12,736 KB
scratch.0000000003.buffers - size: 524,352 KB, in active use: 240,396 KB
Most available free pages:
scratch.0000000004.buffers
Size:1, ValidAfterTransactionId: 358
Size:32, ValidAfterTransactionId: 358
Size:64, ValidAfterTransactionId: 358
Size:4, ValidAfterTransactionId: 358
Size:16, ValidAfterTransactionId: 358
Size:8, ValidAfterTransactionId: 358
Size:2, ValidAfterTransactionId: 358
scratch.0000000003.buffers
Size:1, ValidAfterTransactionId: 352
Size:2, ValidAfterTransactionId: 352
Size:4, ValidAfterTransactionId: 352
Size:32, ValidAfterTransactionId: 352
Size:16, ValidAfterTransactionId: 344
Size:8, ValidAfterTransactionId: 338
Size:64, ValidAfterTransactionId: 344

   at Voron.Impl.Scratch.ScratchBufferPool.Allocate(Transaction tx, Int32 numberOfPages)
   at Voron.Impl.Transaction.InitTransactionHeader()
   at Voron.Impl.Transaction..ctor(StorageEnvironment env, Int64 id, TransactionFlags flags, IFreeSpaceHandling freeSpaceHandling)
   at Voron.StorageEnvironment.NewTransaction(TransactionFlags flags, Nullable`1 timeout)
   at Voron.Impl.TransactionMergingWriter.HandleActualWrites(OutstandingWrite mine, CancellationToken token)<---
 ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.GetResponse()
   at Raven.Abstractions.Connection.HttpRavenRequest.SendRequestToServer(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 195
   --- End of inner exception stack trace ---
   at Raven.Abstractions.Connection.HttpRavenRequest.SendRequestToServer(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 225
   at Raven.Abstractions.Connection.HttpRavenRequest.ExecuteRequest(CancellationToken cancellationToken) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 180
   at Raven.Bundles.Replication.Tasks.ReplicationTask.TryReplicationDocuments(ReplicationStrategy destination, RavenJArray jsonDocuments, String& lastError) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Replication\Tasks\ReplicationTask.cs:line 710"






Oren Eini (Ayende Rahien)

unread,
Oct 19, 2014, 2:14:26 AM10/19/14
to ravendb
Thanks, we have a repro, and we'll be fixing this this week, we hope

Hibernating Rhinos Ltd  

Oren Eini l CEO Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages