Testing environment, isolated master to slave (3.0.3496) replication, No compression
1. The scratch buffer error started after about 970K doc replicated.
2. This error is unable recover itself, had to restart slave
2014-10-18 21:41:09.4091,Raven.Bundles.Replication.Tasks.ReplicationTask,Warn,ltss,Replication to V3Voron had failed,"System.Net.WebException: Error: System.AggregateException: Error when executing write ---> Voron.Exceptions.ScratchBufferSizeLimitException: Cannot allocate more space for the scratch buffer.
Current size is: 393,280 KB.
Limit: 524,288 KB.
Requested Size: 393,284 KB.
Already flushed and waited for 5,015 ms for read transactions to complete.
Do you have a long running read transaction executing?
Debug info:
Requested number of pages: 1 (NearestPowerOfTwo: 1)
Oldest active transaction: 357 (snapshot: 357)
Oldest active transaction when flush was forced: 357
Next write transaction id: 358
Active transactions:
Id: 357 - Read
Scratch files usage:
scratch.0000000004.buffers - size: 393,280 KB, in active use: 12,736 KB
scratch.0000000003.buffers - size: 524,352 KB, in active use: 240,396 KB
Most available free pages:
scratch.0000000004.buffers
Size:1, ValidAfterTransactionId: 358
Size:32, ValidAfterTransactionId: 358
Size:64, ValidAfterTransactionId: 358
Size:4, ValidAfterTransactionId: 358
Size:16, ValidAfterTransactionId: 358
Size:8, ValidAfterTransactionId: 358
Size:2, ValidAfterTransactionId: 358
scratch.0000000003.buffers
Size:1, ValidAfterTransactionId: 352
Size:2, ValidAfterTransactionId: 352
Size:4, ValidAfterTransactionId: 352
Size:32, ValidAfterTransactionId: 352
Size:16, ValidAfterTransactionId: 344
Size:8, ValidAfterTransactionId: 338
Size:64, ValidAfterTransactionId: 344
at Voron.Impl.Scratch.ScratchBufferPool.Allocate(Transaction tx, Int32 numberOfPages)
at Voron.Impl.Transaction.InitTransactionHeader()
at Voron.Impl.Transaction..ctor(StorageEnvironment env, Int64 id, TransactionFlags flags, IFreeSpaceHandling freeSpaceHandling)
at Voron.StorageEnvironment.NewTransaction(TransactionFlags flags, Nullable`1 timeout)
at Voron.Impl.TransactionMergingWriter.HandleActualWrites(OutstandingWrite mine, CancellationToken token)
--- End of inner exception stack trace ---
at Voron.Impl.TransactionMergingWriter.Write(WriteBatch batch)
at Raven.Database.Storage.Voron.Impl.TableStorage.Write(WriteBatch writeBatch) in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Storage\Voron\Impl\TableStorage.cs:line 139
at Raven.Storage.Voron.TransactionalStorage.ExecuteBatch(Action`1 action) in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Storage\Voron\TransactionalStorage.cs:line 195
at Raven.Storage.Voron.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Storage\Voron\TransactionalStorage.cs:line 173
at Raven.Database.Bundles.Replication.Controllers.ReplicationController.<DocReplicatePost>d__c.MoveNext() in c:\Builds\RavenDB-3.0-PreRelease\Raven.Database\Bundles\Replication\Controllers\ReplicationController.cs:line 140
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
---> (Inner Exception #0) Voron.Exceptions.ScratchBufferSizeLimitException: Cannot allocate more space for the scratch buffer.
Current size is: 393,280 KB.
Limit: 524,288 KB.
Requested Size: 393,284 KB.
Already flushed and waited for 5,015 ms for read transactions to complete.
Do you have a long running read transaction executing?
Debug info:
Requested number of pages: 1 (NearestPowerOfTwo: 1)
Oldest active transaction: 357 (snapshot: 357)
Oldest active transaction when flush was forced: 357
Next write transaction id: 358
Active transactions:
Id: 357 - Read
Scratch files usage:
scratch.0000000004.buffers - size: 393,280 KB, in active use: 12,736 KB
scratch.0000000003.buffers - size: 524,352 KB, in active use: 240,396 KB
Most available free pages:
scratch.0000000004.buffers
Size:1, ValidAfterTransactionId: 358
Size:32, ValidAfterTransactionId: 358
Size:64, ValidAfterTransactionId: 358
Size:4, ValidAfterTransactionId: 358
Size:16, ValidAfterTransactionId: 358
Size:8, ValidAfterTransactionId: 358
Size:2, ValidAfterTransactionId: 358
scratch.0000000003.buffers
Size:1, ValidAfterTransactionId: 352
Size:2, ValidAfterTransactionId: 352
Size:4, ValidAfterTransactionId: 352
Size:32, ValidAfterTransactionId: 352
Size:16, ValidAfterTransactionId: 344
Size:8, ValidAfterTransactionId: 338
Size:64, ValidAfterTransactionId: 344
at Voron.Impl.Scratch.ScratchBufferPool.Allocate(Transaction tx, Int32 numberOfPages)
at Voron.Impl.Transaction.InitTransactionHeader()
at Voron.Impl.Transaction..ctor(StorageEnvironment env, Int64 id, TransactionFlags flags, IFreeSpaceHandling freeSpaceHandling)
at Voron.StorageEnvironment.NewTransaction(TransactionFlags flags, Nullable`1 timeout)
at Voron.Impl.TransactionMergingWriter.HandleActualWrites(OutstandingWrite mine, CancellationToken token)<---
---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Raven.Abstractions.Connection.HttpRavenRequest.SendRequestToServer(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 195
--- End of inner exception stack trace ---
at Raven.Abstractions.Connection.HttpRavenRequest.SendRequestToServer(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 225
at Raven.Abstractions.Connection.HttpRavenRequest.ExecuteRequest(CancellationToken cancellationToken) in c:\Builds\RavenDB-Stable\Raven.Abstractions\Connection\HttpRavenRequest.cs:line 180
at Raven.Bundles.Replication.Tasks.ReplicationTask.TryReplicationDocuments(ReplicationStrategy destination, RavenJArray jsonDocuments, String& lastError) in c:\Builds\RavenDB-Stable\Raven.Database\Bundles\Replication\Tasks\ReplicationTask.cs:line 710"