RavenDB 4 - Memory usage during index streaming grows indefinitely
216 views
Skip to first unread message
Andrej Krivulčík
May 25, 2018, 9:11:03 AM5/25/18
to RavenDB - 2nd generation document database
When streaming from index, server memory usage grows indefinitely. On a large enough data, the usage grows until the client fails with the exception below [1] (some formatting and cleanup by me). Also, note the typo strem instead of stream.
Synthetic reproduction code is provided below ([2]).
Output of /admin/debug/memory/stats from before the processing is in [3], after processing ("breakpoint here" in the code) in [4].
Reproduced on 4.0.4-nightly-20180525-0400.
[1] Exception when processing large production database
Unhandled Exception: System.AggregateException: One or more errors occurred.
---> System.InvalidOperationException: Invalid JSON. Last 4KB of read strem : 'nContext.ReadObjectInternal(Object builder, String documentId, UsageMode mode, IBlittableDocumentModifier modifier) in C:\Builds\RavenDB-4.0-Nightly\src\Sparrow\Json\JsonOperationContext.cs:line 568
at Raven.Server.Documents.Queries.Results.QueryResultRetrieverBase.ReturnProjection(DynamicJsonValue result, Document doc, Single score, JsonOperationContext context) in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Queries\Results\QueryResultRetrieverBase.cs:line 222
at Raven.Server.Documents.Queries.Results.QueryResultRetrieverBase.GetProjection(Document input, Single score, String lowerId, IState state) in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Queries\Results\QueryResultRetrieverBase.cs:line 146
at Raven.Server.Documents.Queries.Results.MapReduceQueryResultRetriever.Get(Document input, Single score, IState state) in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Queries\Results\MapReduceQueryResultRetriever.cs:line 45
at Raven.Server.Documents.Indexes.Persistence.Lucene.IndexReadOperation.<Query>d__11.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Indexes\Persistence\Lucene\IndexReadOperation.cs:line 117
at Raven.Server.Documents.Indexes.Index.<QueryInternal>d__152`1.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Indexes\Index.cs:line 2027"{"Url":"/databases/data-prod/streams/queries","Type":"Sparrow.LowMemory.EarlyOutOfMemoryException","Message":"The amount of available memory to commit on the system is low. Commit charge: 36.1 GBytes / 38 GBytes. Memory: 31.703 GBytes / 32 GBytes","Error":"Sparrow.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commiton the system is low. Commit charge: 36.1 GBytes / 38 GBytes. Memory: 31.703 GBytes / 32 GBytes
at Raven.Server.Documents.Queries.StreamDocumentQueryResult.HandleException(Exception e) in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Queries\StreamDocumentQueryResult.cs:line 41
at Raven.Server.Documents.Indexes.Index.<QueryInternal>d__152`1.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Indexes\Index.cs:line 2040
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Raven.Server.Documents.Indexes.Index.<StreamQuery>d__150.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Indexes\Index.cs:line 1904
--- End of stack trace from previous location whereexception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Raven.Server.Documents.Handlers.StreamingHandler.<StreamQueryPost>d__3.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Documents\Handlers\StreamingHandler.cs:line 131
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Raven.Server.Routing.RequestRouter.<HandlePath>d__7.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\Routing\RequestRouter.cs:line 105
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo. Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Raven.Server.RavenServerStartup.<RequestHandler>d__11.MoveNext() in C:\Builds\RavenDB-4.0-Nightly\src\Raven.Server\RavenServerStartup.cs:line 160"}'
at Sparrow.Json.Parsing.UnmanagedJsonParserHelper.ThrowInvalidJson(PeepingTomStream peepingTomStream)
at Raven.Client.Documents.Session.Operations.StreamOperation.YieldStreamResults.<MoveNextAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Raven.Client.Documents.Session.AsyncDocumentSession.YieldStream`1.<MoveNextAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at [my code]
[2] Reproduction code
using Raven.Client.Documents;
using Raven.Client.Documents.Indexes;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
namespace RavenDB4Tests.Tryouts
{
public static class StreamingMemory
{
public static async Task StreamingMemoryTest()
{
var rng = new Random();
using (var store = new DocumentStore
{
Urls = new[] { "http://localhost:8080" },
Database = "StreamingMemoryTest"
}.Initialize())
{
new DocsIndex().Execute(store);
if (await ShouldInitData(store))
{
await InitializeData(store);
}
using (var session = store.OpenAsyncSession())
{
Console.WriteLine("Creating stream.");
var query = session.Query<Doc, DocsIndex>();
var stream = await session.Advanced.StreamAsync(query);
while (await stream.MoveNextAsync())
{
var doc = stream.Current.Document;
if (doc.Id.EndsWith("000"))
{
Console.WriteLine($"Processing {doc.Id}");
}
if (doc.Id == "doc/500000")
{
// breakpoint here
}
session.Advanced.Evict(doc);
}
Console.WriteLine("Finished processing.");
}
}
}
private static async Task<bool> ShouldInitData(IDocumentStore store)
{
using (var session = store.OpenAsyncSession())
{
var doc = await session.LoadAsync<Doc>("doc/1");
return doc == null;
}
}
private static async Task InitializeData(IDocumentStore store)
{
var start = 0;
var batches = 50;
Console.WriteLine("Generating data.");
var rng = new Random();
for (int batchNo = start; batchNo < start + batches; batchNo++)
{
Console.WriteLine($"{DateTime.Now.ToLongTimeString()}: Generating batch {batchNo + 1}/{batches}");
using (var session = store.OpenAsyncSession())
{
for (int i = 1; i <= 10000; i++)
{
await session.StoreAsync(new Doc { Id = "doc/" + (batchNo * 10000 + i), IntVal = i, IntVals = Enumerable.Range(1, 50).ToDictionary(x => x.ToString(), _ => rng.Next()) });
}
await session.SaveChangesAsync();
}
}
Console.WriteLine("Data generated.");
}
public class Doc
{
public string Id { get; set; }
public int IntVal { get; set; }
public Dictionary<string, int> IntVals { get; set; }
}
public class DocsIndex : AbstractIndexCreationTask<Doc>
{
public DocsIndex()
{
Map = docs =>
from doc in docs
select new
{
doc.Id,
doc.IntVal,
doc.IntVals,
};
Reduce = results =>
from result in results
group result by result.Id
into g
let doc = g.First()
select new
{
Id = g.Key,
doc.IntVal,
doc.IntVals,
};
}
}
}
}[3] /admin/debug/memory/stats output before processing
{
"WorkingSet": 1569935360,
"TotalUnmanagedAllocations": 2687752,
"ManagedAllocations": 21806736,
"TotalMemoryMapped": 3246981120,
"PhysicalMem": "15,91 GBytes",
"FreeMem": "7,31 GBytes",
"HighMemLastOneMinute": "8,76 GBytes",
"LowMemLastOneMinute": "6,28 GBytes",
"HighMemLastFiveMinute": "8,76 GBytes",
"LowMemLastFiveMinute": "6,28 GBytes",
"HighMemSinceStartup": "8,76 GBytes",
"LowMemSinceStartup": "6,28 GBytes",
"Humane": {
"WorkingSet": "1,46 GBytes",
"TotalUnmanagedAllocations": "2,56 MBytes",
"ManagedAllocations": "20,8 MBytes",
"TotalMemoryMapped": "3,02 GBytes"
},
"Threads": [
{
"Name": "Unknown",
"Allocations": 2687752,
"HumaneAllocations": "2,56 MBytes",
"Ids": [
{
"Id": 6164,
"ManagedThreadId": 1,
"Allocations": 983234,
"HumaneAllocations": "960,19 KBytes"
},
{
"Id": 6344,
"ManagedThreadId": 13,
"Allocations": 885124,
"HumaneAllocations": "864,38 KBytes"
},
{
"Id": 9288,
"ManagedThreadId": 10,
"Allocations": 311490,
"HumaneAllocations": "304,19 KBytes"
},
{
"Id": 19028,
"ManagedThreadId": 17,
"Allocations": 229376,
"HumaneAllocations": "224 KBytes"
},
{
"Id": 8660,
"ManagedThreadId": 9,
"Allocations": 180224,
"HumaneAllocations": "176 KBytes"
},
{
"Id": 8944,
"ManagedThreadId": 4,
"Allocations": 98304,
"HumaneAllocations": "96 KBytes"
},
{
"Id": 6448,
"ManagedThreadId": 18,
"Allocations": 0,
"HumaneAllocations": "0 Bytes"
}
]
},
{
"Name": "'StreamingMemoryTest' Transaction Merging Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 1332,
"ManagedThreadId": 20
},
{
"Name": "Consensus Leader - A in term 3",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 3096,
"ManagedThreadId": 14
},
{
"Name": "RavenDB Tasks Executer",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 18212,
"ManagedThreadId": 12
},
{
"Name": "Voron Global Flushing Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 10600,
"ManagedThreadId": 11
},
{
"Name": "Low memory notification thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 18876,
"ManagedThreadId": 7
},
{
"Name": "Logging Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 14604,
"ManagedThreadId": 5
}
],
"Mappings": [
{
"Directory": "\\Databases\\StreamingMemoryTest\\Indexes\\DocsIndex",
"TotalDirectorySize": 2147483648,
"HumaneTotalDirectorySize": "2 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 2147483648,
"HumaneFileSize": "2 GBytes",
"TotalMapped": 2147483648,
"HumaneTotalMapped": "2 GBytes",
"Mappings": [
{
"Size": 2147483648,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest",
"TotalDirectorySize": 1073741824,
"HumaneTotalDirectorySize": "1 024 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 1073741824,
"HumaneFileSize": "1 024 MBytes",
"TotalMapped": 1073741824,
"HumaneTotalMapped": "1 024 MBytes",
"Mappings": [
{
"Size": 1073741824,
"Count": 1
}
]
}
}
},
{
"Directory": "\\System",
"TotalDirectorySize": 16777216,
"HumaneTotalDirectorySize": "16 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 16777216,
"HumaneFileSize": "16 MBytes",
"TotalMapped": 16777216,
"HumaneTotalMapped": "16 MBytes",
"Mappings": [
{
"Size": 16777216,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Configuration",
"TotalDirectorySize": 8388608,
"HumaneTotalDirectorySize": "8 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 8388608,
"HumaneFileSize": "8 MBytes",
"TotalMapped": 8388608,
"HumaneTotalMapped": "8 MBytes",
"Mappings": [
{
"Size": 8388608,
"Count": 1
}
]
}
}
},
{
"Directory": "\\System\\Temp",
"TotalDirectorySize": 196608,
"HumaneTotalDirectorySize": "192 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 131072,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Indexes\\DocsIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Configuration\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
}
]
}
[4] /admin/debug/memory/stats output before processing finish
{
"WorkingSet": 2709032960,
"TotalUnmanagedAllocations": 5128968,
"ManagedAllocations": 552524464,
"TotalMemoryMapped": 3247112192,
"PhysicalMem": "15,91 GBytes",
"FreeMem": "5,95 GBytes",
"HighMemLastOneMinute": "6,56 GBytes",
"LowMemLastOneMinute": "5,93 GBytes",
"HighMemLastFiveMinute": "8,76 GBytes",
"LowMemLastFiveMinute": "5,93 GBytes",
"HighMemSinceStartup": "8,76 GBytes",
"LowMemSinceStartup": "5,93 GBytes",
"Humane": {
"WorkingSet": "2,52 GBytes",
"TotalUnmanagedAllocations": "4,89 MBytes",
"ManagedAllocations": "526,93 MBytes",
"TotalMemoryMapped": "3,02 GBytes"
},
"Threads": [
{
"Name": "Unknown",
"Allocations": 5128968,
"HumaneAllocations": "4,89 MBytes",
"Ids": [
{
"Id": 6344,
"ManagedThreadId": 13,
"Allocations": 1311108,
"HumaneAllocations": "1,25 MBytes"
},
{
"Id": 6164,
"ManagedThreadId": 1,
"Allocations": 835778,
"HumaneAllocations": "816,19 KBytes"
},
{
"Id": 10632,
"ManagedThreadId": 25,
"Allocations": 557056,
"HumaneAllocations": "544 KBytes"
},
{
"Id": 9288,
"ManagedThreadId": 10,
"Allocations": 426178,
"HumaneAllocations": "416,19 KBytes"
},
{
"Id": 19028,
"ManagedThreadId": 17,
"Allocations": 360448,
"HumaneAllocations": "352 KBytes"
},
{
"Id": 16460,
"ManagedThreadId": 23,
"Allocations": 344064,
"HumaneAllocations": "336 KBytes"
},
{
"Id": 8660,
"ManagedThreadId": 9,
"Allocations": 311296,
"HumaneAllocations": "304 KBytes"
},
{
"Id": 6448,
"ManagedThreadId": 18,
"Allocations": 278528,
"HumaneAllocations": "272 KBytes"
},
{
"Id": 17076,
"ManagedThreadId": 22,
"Allocations": 180224,
"HumaneAllocations": "176 KBytes"
},
{
"Id": 20088,
"ManagedThreadId": 27,
"Allocations": 131072,
"HumaneAllocations": "128 KBytes"
},
{
"Id": 17624,
"ManagedThreadId": 24,
"Allocations": 131072,
"HumaneAllocations": "128 KBytes"
},
{
"Id": 18552,
"ManagedThreadId": 28,
"Allocations": 131072,
"HumaneAllocations": "128 KBytes"
},
{
"Id": 20416,
"ManagedThreadId": 26,
"Allocations": 131072,
"HumaneAllocations": "128 KBytes"
}
]
},
{
"Name": "Indexing of DocsIndex of StreamingMemoryTest",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 9644,
"ManagedThreadId": 34
},
{
"Name": "'StreamingMemoryTest' Transaction Merging Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 1332,
"ManagedThreadId": 20
},
{
"Name": "Consensus Leader - A in term 3",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 3096,
"ManagedThreadId": 14
},
{
"Name": "RavenDB Tasks Executer",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 18212,
"ManagedThreadId": 12
},
{
"Name": "Voron Global Flushing Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 10600,
"ManagedThreadId": 11
},
{
"Name": "Low memory notification thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 18876,
"ManagedThreadId": 7
}
],
"Mappings": [
{
"Directory": "\\Databases\\StreamingMemoryTest\\Indexes\\DocsIndex",
"TotalDirectorySize": 2147483648,
"HumaneTotalDirectorySize": "2 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 2147483648,
"HumaneFileSize": "2 GBytes",
"TotalMapped": 2147483648,
"HumaneTotalMapped": "2 GBytes",
"Mappings": [
{
"Size": 2147483648,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest",
"TotalDirectorySize": 1073741824,
"HumaneTotalDirectorySize": "1 024 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 1073741824,
"HumaneFileSize": "1 024 MBytes",
"TotalMapped": 1073741824,
"HumaneTotalMapped": "1 024 MBytes",
"Mappings": [
{
"Size": 1073741824,
"Count": 1
}
]
}
}
},
{
"Directory": "\\System",
"TotalDirectorySize": 16777216,
"HumaneTotalDirectorySize": "16 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 16777216,
"HumaneFileSize": "16 MBytes",
"TotalMapped": 16777216,
"HumaneTotalMapped": "16 MBytes",
"Mappings": [
{
"Size": 16777216,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Configuration",
"TotalDirectorySize": 8388608,
"HumaneTotalDirectorySize": "8 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 8388608,
"HumaneFileSize": "8 MBytes",
"TotalMapped": 8388608,
"HumaneTotalMapped": "8 MBytes",
"Mappings": [
{
"Size": 8388608,
"Count": 1
}
]
}
}
},
{
"Directory": "\\System\\Temp",
"TotalDirectorySize": 327680,
"HumaneTotalDirectorySize": "320 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 262144,
"HumaneFileSize": "256 KBytes",
"TotalMapped": 262144,
"HumaneTotalMapped": "256 KBytes",
"Mappings": [
{
"Size": 262144,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Indexes\\DocsIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Configuration\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "\\Databases\\StreamingMemoryTest\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
}
]
}
Oren Eini (Ayende Rahien)
May 25, 2018, 11:27:02 AM5/25/18
to ravendb
What machine are you running this on? On my end, running your code, I get:
working set | native mem | managed mem | mmap size | reqs/sec | docs (all dbs)
* 3.12 GBytes | 34.15 MBytes | 846.57 MBytes | 4.12 GBytes | 0 | 500,000
The managed memory goes up to about 1GB and then down to 500MB on a pretty consistent basis.
That said, this is still too much allocations, we are looking further into this
Hibernating Rhinos Ltd
Oren Eini l CEO l Mobile: + 972-52-548-6969
Office: +972-4-622-7811 l Fax: +972-153-4-622-7811
--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Oren Eini (Ayende Rahien)
May 25, 2018, 11:56:05 AM5/25/18
to ravendb
I'm pretty sure that this related to the fact that we need to basically read a lot of data back from Lucene stored fields.
What the profiler shows is that we allocate like crazy inside the Field Reader. Depending on the size of your stored fields, that is very likely
Andrej Krivulčík
May 25, 2018, 11:59:15 AM5/25/18
to RavenDB - 2nd generation document database
Synthetic test was done on my laptop - 16 GB RAM. Production problem occurred on an Azure L4s instance (32 GB RAM). Having more (and/or more complex) records in the synthetic test would probably make the issue more visible but in my case, it was sufficient to demonstrate it.
Managed memory oscillates but grows in my case, something like 280 -> 90 -> 300 -> 100 -> 320 -> 110 -> 350 -> 120 -> 380 -> 140 etc...
Oren Eini (Ayende Rahien)
May 25, 2018, 11:59:31 AM5/25/18
to ravendb
And that is _not_ the answer, actually. Looks like we need to pin the data from Lucene to read it natively, but we unpin it only at the dispose of the context.
Thanks, we'll be fixing that shortly. By Monday you should have a build with this fix
Oren Eini (Ayende Rahien)
May 25, 2018, 12:02:35 PM5/25/18
to ravendb
Oren Eini (Ayende Rahien)
May 25, 2018, 12:05:43 PM5/25/18
to ravendb
Yes, this isn't _quite_ a leak. What is going on is that we have pinned buffered that will only be removed when the system is idle or there is a low memory event.
But even so, the low memory event need to happen when a particular memory isn't in used, etc.
At any rate, this is fairly easy to fix now that we know where it is at
Hibernating Rhinos Ltd
Oren Eini l CEO l Mobile: + 972-52-548-6969
Office: +972-4-622-7811 l Fax: +972-153-4-622-7811
Andrej Krivulčík
May 25, 2018, 12:16:25 PM5/25/18
to RavenDB - 2nd generation document database
Great, thanks for the investigation, confirmation and fix. However - about the removal when there is a low memory event - is this consistent with my server running out of memory? I.e. the memory wasn't unpinned because some particular memory was used?
Oren Eini (Ayende Rahien)
May 25, 2018, 12:28:15 PM5/25/18
to ravendb
What is likely to happen is that there will be some period of high memory, and the server will recover from it after a while
--
Oren Eini (Ayende Rahien)
May 28, 2018, 8:44:40 AM5/28/18
to ravendb
This has been fixed now
Andrej Krivulčík
May 29, 2018, 4:22:20 AM5/29/18
to RavenDB - 2nd generation document database
Thanks Oren, the fix seems to have helped somewhat, I can now process the production data without running out of memory.
However, memory usage during the processing still seems to grow.
Stats in console during processing:
working set | native mem | managed mem | mmap size | reqs/sec | docs (all dbs)
* 24.79 GBytes | 3.98 GBytes | 931 MBytes | 385.61 GBytes | 1 | 28,081,224
Studio resource usage a while after processing finished (drop in CPU usage indicates when the processing finished):
Task Manager and Resource Monitor screenshot during processing:
/admin/debug/memory/stats during processing:
{
"WorkingSet": 26991132672,
"TotalUnmanagedAllocations": 4705390182,
"ManagedAllocations": 865302232,
"TotalMemoryMapped": 414047993856,
"PhysicalMem": "32 GBytes",
"FreeMem": "4.5 GBytes",
"HighMemLastOneMinute": "10.53 GBytes",
"LowMemLastOneMinute": "4.5 GBytes",
"HighMemLastFiveMinute": "26.31 GBytes",
"LowMemLastFiveMinute": "4.5 GBytes",
"HighMemSinceStartup": "29.82 GBytes",
"LowMemSinceStartup": "4.5 GBytes",
"Humane": {
"WorkingSet": "25.14 GBytes",
"TotalUnmanagedAllocations": "4.38 GBytes",
"ManagedAllocations": "825.22 MBytes",
"TotalMemoryMapped": "385.61 GBytes"
},
"Threads": [
{
"Name": "Unknown",
"Allocations": 4704996966,
"HumaneAllocations": "4.38 GBytes",
"Ids": [
{
"Id": 4924,
"ManagedThreadId": 27,
"Allocations": 604176384,
"HumaneAllocations": "576.19 MBytes"
},
{
"Id": 6088,
"ManagedThreadId": 18,
"Allocations": 558138702,
"HumaneAllocations": "532.28 MBytes"
},
{
"Id": 6072,
"ManagedThreadId": 32,
"Allocations": 520192000,
"HumaneAllocations": "496.09 MBytes"
},
{
"Id": 15220,
"ManagedThreadId": 17,
"Allocations": 503546212,
"HumaneAllocations": "480.22 MBytes"
},
{
"Id": 9016,
"ManagedThreadId": 9,
"Allocations": 453821806,
"HumaneAllocations": "432.8 MBytes"
},
{
"Id": 10212,
"ManagedThreadId": 35,
"Allocations": 452984832,
"HumaneAllocations": "432 MBytes"
},
{
"Id": 9804,
"ManagedThreadId": 29,
"Allocations": 419610624,
"HumaneAllocations": "400.17 MBytes"
},
{
"Id": 10836,
"ManagedThreadId": 37,
"Allocations": 402833408,
"HumaneAllocations": "384.17 MBytes"
},
{
"Id": 7016,
"ManagedThreadId": 36,
"Allocations": 402751488,
"HumaneAllocations": "384.09 MBytes"
},
{
"Id": 9672,
"ManagedThreadId": 53,
"Allocations": 385875968,
"HumaneAllocations": "368 MBytes"
},
{
"Id": 1528,
"ManagedThreadId": 1,
"Allocations": 557250,
"HumaneAllocations": "544.19 KBytes"
},
{
"Id": 8588,
"ManagedThreadId": 33,
"Allocations": 294912,
"HumaneAllocations": "288 KBytes"
},
{
"Id": 9480,
"ManagedThreadId": 14,
"Allocations": 213380,
"HumaneAllocations": "208.38 KBytes"
},
{
"Id": 14876,
"ManagedThreadId": 30,
"Allocations": 0,
"HumaneAllocations": "0 Bytes"
},
{
"Id": 10016,
"ManagedThreadId": 31,
"Allocations": 0,
"HumaneAllocations": "0 Bytes"
}
]
},
{
"Name": "Indexing of CompanyFullIndex of data-prod",
"Allocations": 212992,
"HumaneAllocations": "208 KBytes",
"Id": 10496,
"ManagedThreadId": 48
},
{
"Name": "Consensus Leader - A in term 20",
"Allocations": 180224,
"HumaneAllocations": "176 KBytes",
"Id": 5408,
"ManagedThreadId": 13
},
{
"Name": "Indexing of ReportIndex of data-prod",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 7804,
"ManagedThreadId": 52
},
{
"Name": "Indexing of LocationSuggestionIndex of data-prod",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 11896,
"ManagedThreadId": 51
},
{
"Name": "Indexing of LocationIndex of data-prod",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 9800,
"ManagedThreadId": 50
},
{
"Name": "Indexing of PersonSearchIndex of data-prod",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 2404,
"ManagedThreadId": 49
},
{
"Name": "Indexing of CompanyFullIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 15140,
"ManagedThreadId": 47
},
{
"Name": "Indexing of AccountsIndex of data-prod",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 9524,
"ManagedThreadId": 46
},
{
"Name": "Indexing of UsersByNameSortIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 10976,
"ManagedThreadId": 45
},
{
"Name": "Indexing of ReportIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 13764,
"ManagedThreadId": 44
},
{
"Name": "Indexing of PersonSearchIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 5356,
"ManagedThreadId": 43
},
{
"Name": "Indexing of LocationSuggestionIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 10280,
"ManagedThreadId": 42
},
{
"Name": "Indexing of LocationIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 8968,
"ManagedThreadId": 41
},
{
"Name": "Indexing of AccountsIndex of data-subset",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 8448,
"ManagedThreadId": 40
},
{
"Name": "Indexing of UsersByNameSortIndex of data-prod",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 6892,
"ManagedThreadId": 39
},
{
"Name": "'data-prod' Transaction Merging Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 928,
"ManagedThreadId": 26
},
{
"Name": "'data-subset' Transaction Merging Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 5960,
"ManagedThreadId": 20
},
{
"Name": "RavenDB Tasks Executer",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 2568,
"ManagedThreadId": 12
},
{
"Name": "Voron Global Flushing Thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 15244,
"ManagedThreadId": 11
},
{
"Name": "Low memory notification thread",
"Allocations": 0,
"HumaneAllocations": "0 Bytes",
"Id": 6360,
"ManagedThreadId": 7
}
],
"Mappings": [
{
"Directory": "E:\\Raven\\Databases\\data-prod",
"TotalDirectorySize": 274877906944,
"HumaneTotalDirectorySize": "256 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 274877906944,
"HumaneFileSize": "256 GBytes",
"TotalMapped": 274877906944,
"HumaneTotalMapped": "256 GBytes",
"Mappings": [
{
"Size": 274877906944,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\CompanyFullIndex",
"TotalDirectorySize": 105226698752,
"HumaneTotalDirectorySize": "98 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 105226698752,
"HumaneFileSize": "98 GBytes",
"TotalMapped": 105226698752,
"HumaneTotalMapped": "98 GBytes",
"Mappings": [
{
"Size": 105226698752,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\PersonSearchIndex",
"TotalDirectorySize": 17179869184,
"HumaneTotalDirectorySize": "16 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 17179869184,
"HumaneFileSize": "16 GBytes",
"TotalMapped": 17179869184,
"HumaneTotalMapped": "16 GBytes",
"Mappings": [
{
"Size": 17179869184,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\LocationSuggestionIndex",
"TotalDirectorySize": 4294967296,
"HumaneTotalDirectorySize": "4 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 4294967296,
"HumaneFileSize": "4 GBytes",
"TotalMapped": 4294967296,
"HumaneTotalMapped": "4 GBytes",
"Mappings": [
{
"Size": 4294967296,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\AccountsIndex",
"TotalDirectorySize": 4294967296,
"HumaneTotalDirectorySize": "4 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 4294967296,
"HumaneFileSize": "4 GBytes",
"TotalMapped": 4294967296,
"HumaneTotalMapped": "4 GBytes",
"Mappings": [
{
"Size": 4294967296,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\LocationIndex",
"TotalDirectorySize": 3221225472,
"HumaneTotalDirectorySize": "3 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 3221225472,
"HumaneFileSize": "3 GBytes",
"TotalMapped": 3221225472,
"HumaneTotalMapped": "3 GBytes",
"Mappings": [
{
"Size": 3221225472,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset",
"TotalDirectorySize": 3221225472,
"HumaneTotalDirectorySize": "3 GBytes",
"Details": {
"Raven.voron": {
"FileSize": 3221225472,
"HumaneFileSize": "3 GBytes",
"TotalMapped": 3221225472,
"HumaneTotalMapped": "3 GBytes",
"Mappings": [
{
"Size": 3221225472,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\CompanyFullIndex",
"TotalDirectorySize": 1073741824,
"HumaneTotalDirectorySize": "1,024 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 1073741824,
"HumaneFileSize": "1,024 MBytes",
"TotalMapped": 1073741824,
"HumaneTotalMapped": "1,024 MBytes",
"Mappings": [
{
"Size": 1073741824,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\PersonSearchIndex",
"TotalDirectorySize": 268435456,
"HumaneTotalDirectorySize": "256 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 268435456,
"HumaneFileSize": "256 MBytes",
"TotalMapped": 268435456,
"HumaneTotalMapped": "256 MBytes",
"Mappings": [
{
"Size": 268435456,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\LocationSuggestionIndex",
"TotalDirectorySize": 134217728,
"HumaneTotalDirectorySize": "128 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 134217728,
"HumaneFileSize": "128 MBytes",
"TotalMapped": 134217728,
"HumaneTotalMapped": "128 MBytes",
"Mappings": [
{
"Size": 134217728,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\AccountsIndex",
"TotalDirectorySize": 67108864,
"HumaneTotalDirectorySize": "64 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 67108864,
"HumaneFileSize": "64 MBytes",
"TotalMapped": 67108864,
"HumaneTotalMapped": "64 MBytes",
"Mappings": [
{
"Size": 67108864,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\LocationIndex",
"TotalDirectorySize": 67108864,
"HumaneTotalDirectorySize": "64 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 67108864,
"HumaneFileSize": "64 MBytes",
"TotalMapped": 67108864,
"HumaneTotalMapped": "64 MBytes",
"Mappings": [
{
"Size": 67108864,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\ReportIndex",
"TotalDirectorySize": 33554432,
"HumaneTotalDirectorySize": "32 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 33554432,
"HumaneFileSize": "32 MBytes",
"TotalMapped": 33554432,
"HumaneTotalMapped": "32 MBytes",
"Mappings": [
{
"Size": 33554432,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\ReportIndex",
"TotalDirectorySize": 33554432,
"HumaneTotalDirectorySize": "32 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 33554432,
"HumaneFileSize": "32 MBytes",
"TotalMapped": 33554432,
"HumaneTotalMapped": "32 MBytes",
"Mappings": [
{
"Size": 33554432,
"Count": 1
}
]
}
}
},
{
"Directory": "D:\\Raven\\System",
"TotalDirectorySize": 16777216,
"HumaneTotalDirectorySize": "16 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 16777216,
"HumaneFileSize": "16 MBytes",
"TotalMapped": 16777216,
"HumaneTotalMapped": "16 MBytes",
"Mappings": [
{
"Size": 16777216,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Configuration",
"TotalDirectorySize": 8388608,
"HumaneTotalDirectorySize": "8 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 8388608,
"HumaneFileSize": "8 MBytes",
"TotalMapped": 8388608,
"HumaneTotalMapped": "8 MBytes",
"Mappings": [
{
"Size": 8388608,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Configuration",
"TotalDirectorySize": 8388608,
"HumaneTotalDirectorySize": "8 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 8388608,
"HumaneFileSize": "8 MBytes",
"TotalMapped": 8388608,
"HumaneTotalMapped": "8 MBytes",
"Mappings": [
{
"Size": 8388608,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\UsersByNameSortIndex",
"TotalDirectorySize": 8388608,
"HumaneTotalDirectorySize": "8 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 8388608,
"HumaneFileSize": "8 MBytes",
"TotalMapped": 8388608,
"HumaneTotalMapped": "8 MBytes",
"Mappings": [
{
"Size": 8388608,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\UsersByNameSortIndex",
"TotalDirectorySize": 8388608,
"HumaneTotalDirectorySize": "8 MBytes",
"Details": {
"Raven.voron": {
"FileSize": 8388608,
"HumaneFileSize": "8 MBytes",
"TotalMapped": 8388608,
"HumaneTotalMapped": "8 MBytes",
"Mappings": [
{
"Size": 8388608,
"Count": 1
}
]
}
}
},
{
"Directory": "D:\\Raven\\System\\Temp",
"TotalDirectorySize": 327680,
"HumaneTotalDirectorySize": "320 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 262144,
"HumaneFileSize": "256 KBytes",
"TotalMapped": 262144,
"HumaneTotalMapped": "256 KBytes",
"Mappings": [
{
"Size": 131072,
"Count": 2
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\CompanyFullIndex\\Temp",
"TotalDirectorySize": 262144,
"HumaneTotalDirectorySize": "256 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 2
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 2
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\CompanyFullIndex\\Temp",
"TotalDirectorySize": 262144,
"HumaneTotalDirectorySize": "256 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 131072,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 131072,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\LocationSuggestionIndex\\Temp",
"TotalDirectorySize": 196608,
"HumaneTotalDirectorySize": "192 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 131072,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\LocationSuggestionIndex\\Temp",
"TotalDirectorySize": 196608,
"HumaneTotalDirectorySize": "192 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 131072,
"HumaneFileSize": "128 KBytes",
"TotalMapped": 131072,
"HumaneTotalMapped": "128 KBytes",
"Mappings": [
{
"Size": 131072,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Configuration\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\AccountsIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\UsersByNameSortIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Configuration\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\AccountsIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\ReportIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\ReportIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\PersonSearchIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\LocationIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-subset\\Indexes\\PersonSearchIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\UsersByNameSortIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
},
{
"Directory": "E:\\Raven\\Databases\\data-prod\\Indexes\\LocationIndex\\Temp",
"TotalDirectorySize": 131072,
"HumaneTotalDirectorySize": "128 KBytes",
"Details": {
"compression.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
},
"scratch.0000000000.buffers": {
"FileSize": 65536,
"HumaneFileSize": "64 KBytes",
"TotalMapped": 65536,
"HumaneTotalMapped": "64 KBytes",
"Mappings": [
{
"Size": 65536,
"Count": 1
}
]
}
}
}
]
}
This is not an issue for me anymore as I can process all the data I need, but it seems that the memory usage still grows with no apparent bound.
If you can find any useful information in this, that would be great. If needed, I'll be happy to help with investigating this issue, if it's difficult to reproduce with synthetic data.
Oren Eini (Ayende Rahien)
May 29, 2018, 5:15:00 AM5/29/18
to ravendb
Let's break this down:
working set | native mem | managed mem | mmap size | reqs/sec | docs (all dbs)
* 24.79 GBytes | 3.98 GBytes | 931 MBytes | 385.61 GBytes | 1 | 28,081,224
RavenDB is holding 25GB in memory.
However, out of that memory, we have about 4 GB in native memory and 1 GB in managed memory.
All the rest are just mmap memory that is currently resident in RAM because you just touched a large part of the DB.
You can confirm it using VM Map: https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
In short, this isn't a problem as we consider it. As the system need more memory, the OS will evict the mmap pages out because it can restore them cheaply.
Andrej Krivulčík
May 29, 2018, 5:43:16 AM5/29/18
to RavenDB - 2nd generation document database
Great, thanks for the explanation. I expected something like this but wasn't sure.
Reply all
Reply to author
Forward
0 new messages