Raven 4.0 Bulk Insert Performance

[Shubha Lakshmi]

I have installed Raven 4.0, and acquired a free license for it from raven's site. I am currently using a single node, with 16 GB RAM and 2 cores.

I tried testing the speed of bulk insert operation using the following code :

List<Employee> employees = new List<Employee>();
            for (int i = 0; i < 1000 * 1000; i++)
            {
                employees.Add(new Employee
                {
                    FirstName = "FirstName #" + i,
                    LastName = "LastName #" + i
                });
            }
            swch.Start();
            using (BulkInsertOperation bulkInsert = store.BulkInsert())
            {
                foreach(var emp in employees)
                bulkInsert.Store(emp);
               
            }


            Debug.WriteLine(swch.Elapsed.ToString(@"m\:ss\.ff"));

The operation is taking approx 4 minutes to complete. Would the operation be faster in a cluster setup, with each machine having more than 2 cores. Any suggestions to make bulk insert faster would be helpful.

TIA

[Federico Lois]

Probably could not be the case on your actual code, but if it follows this kind of pattern, you can actually parallelize it and use multiple cores. The DB server is hardly doing anything at that point, the amount you are inserting is negligible; most of the cost there is in actually converting the objects to send them over the wire. That is also being done on a single thread. You can create multiple BulkInsertOperations, one on each thread and it will be faster. Also, you are probably going to be bound by CPU at the client code which in turn will slow down the server if you are running all on the same machine; if you are not, you have to take care of the latency between client and server which can be improved doing the insertion in parallel. 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shubha Lakshmi]

Hi, Thanks for the suggestion. I tried the following piece of code, but it is going into infinite loop somewhere :

List<Employee> employees = new List<Employee>();


            for (int i = 0; i < 1000 * 1000; i++)
            {
                employees.Add(new Employee
                {
                    FirstName = "FirstName #" + i,
                    LastName = "LastName #" + i
                });
            }

            //Perform batching
            var batches = employees.Batch(200);
            List<Task> taskList = new List<Task>();
            Task[] taskArray = new Task[200];

            foreach (var batch in batches)
            {
                List<Employee> tempList = batch.ToList();
                Task insertBatchTask = Task.Run(() =>
                {                  
                    using (BulkInsertOperation bulkInsert = store.BulkInsert())
                    {
                        foreach (var emp in tempList)
                            bulkInsert.Store(emp);


                    }

                });
                taskList.Add(insertBatchTask);

            }
            swch.Start();
            Task.WaitAll(taskList.ToArray());

            Debug.WriteLine(swch.Elapsed.ToString(@"m\:ss\.ff"));

Where "Batch" is standard IEnumerable Batching code.

Could it be due to the fact that DocumentStoreHolder is implemented as a singleton and we need new store instances ... TIA

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Shubha Lakshmi]

I was able to debug the issue .. It was "Access to modified closure" ..I rectified it and now insertion is taking 2 minutes 45 seconds .... It is still exceeding 1 minute .. I will try to optimize my code further ...

[Shubha Lakshmi]

Just detailing out the system configuration here : 4 cores(Logical Processors), 16 GB RAM out of which 6 GB is being used by raven server due to limitation of free license, I assume bulk writes will be much faster in a higher configuration machine which comes with trial/professional license utilizing upto 40 cores per node.

[Shubha Lakshmi]

I compared both Parallel and non parallel versions of BulkInsert operations, and found parallel version to be faster .

I could not find async version of "BulkInsert" , so I am just using " await Task.WhenAll" after batching the collection to be inserted.

Detailing the statistics :

Insertion on Employee Collection

Number of records : 1 M

Clocked Time format : mm:ss:.ff

Parallel Insertion using await task.WhenAll

first time on empty collection : 3:46.65

second time with 1 M records in collection : 2:31.42

Third time with 2M records in collection : 2:29.56

Fourth Time with 3M records in collection : 2:26.54

Non parallel Insertion :

first time on empty collection : 4:07.14

second time with 1 M records in collection : 3:43.01

Third time with 2M records in collection : 3:40.39

Fourth Time with 3M records in collection : 3:39.98

I have a couple of questions based on above observation :

1. Can we conclude that Parallel Bulk insert operations are faster or is it due to utilization of only 3 cores by free license version of raven.

2. Why is async version of "BulkInsert" not available.

3. What would be the best configuration and strategy (parallel versus non parallel)  when we are dealing with 6GB Bulk insert record size.

[Oren Eini (Ayende Rahien)]

1) Parallel operations are faster, but try batch size of 10,000 or 50,000.

2) BulkInsert has StoreAsync and DisposeAsync.

3) See 1. Note that the 3 cores limit is likely affecting you here, because there is a limit on parsing. 

Also try to run it on a separate machine. to avoid client and server fighting over the same resource.

There is a branch out where we are trying to see how we can improve the performance there, with some promising results, but it isn't there yet.

Note that you should also check your I/O, but for such numbers, I don't think you are saturating it.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Shubha Lakshmi]

Thanks Oren, I will try batch size of 50000 and a separate client system,  is there any way to store entire collection, as in bulkInsert.Store(employees) where employees is IEnumerable<Employee> instead of storing individual employee entity at a time .. Would it make any difference...

[Justin A]

Out of interest, I tried to whip up a test program to try some tests on this. I couldn't get BulkInsert to work locally. Kept getting an exception thrown:

Raven.Client.Exceptions.Documents.BulkInsert.BulkInsertAbortedException: Failed to execute bulk insert ---> System.AggregateException: One or more errors occurred. (System.InvalidOperationException: Failed to process bulk insert Inserted 0 documents in 0 batches. ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.

   at Raven.Server.Documents.Handlers.BatchRequestParser.ThrowUnexpectedEndOfStream() in C:\Builds\RavenDB-4.0-RC\src\Raven.Server\Documents\Handlers\BatchRequestParser.cs:line 605

   at Raven.Server.Documents.Handlers.BatchRequestParser.<RefillParserBuffer>d__23.MoveNext() in C:\Builds\RavenDB-4.0-RC\src\Raven.Server\Documents\Handlers\BatchRequestParser.cs:line 597

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

........

This was the code I was trying to do, to compare my results to yours Shubha (yes, different hardware spec, etc...)

            var batches = employees.Batch(10);

            var stopwatch = Stopwatch.StartNew();

            foreach (var batch in batches)

                {

                    await BulkInsertEmployees(batch, documentStore);

                }

            stopwatch.Stop();

            Console.WriteLine($"Inserted {employees.Count:N0} employee's in {stopwatch.Elapsed.TotalSeconds:N2} seconds.");

-----

        private static async Task BulkInsertEmployees(IEnumerable<Employee> batch, IDocumentStore documentStore)

        {

            using (var operation = documentStore.BulkInsert())

            {

                var tasks = batch.Select(employee => operation.StoreAsync(employee)).ToArray();

                await Task.WhenAll(tasks);

            }

        }

I thought i was doing what you were doing, but just in a c# 7.1 core 2 console app with an async Main method.

:(

-me-

[Shubha Lakshmi]

I am suspecting "Access to modified closure" issue .. "https://stackoverflow.com/questions/235455/access-to-modified-closure"

[Justin A]

Initially, my code did that and assploded.

So i refactored it (to the code above) and i'm still getting an exception (not sure if it' the same one) and resharper it not showing 'access to modified closure' warning (with that code above).

when i have it in a LINQ statement, then I get that.

:(

[Oren Eini (Ayende Rahien)]

Can you send us a repro of this?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Justin A]

https://github.com/PureKrome/RavenDbBulkInsert

On Sunday, 29 October 2017 19:35:30 UTC+11, Oren Eini wrote:

Can you send us a repro of this?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Sat, Oct 28, 2017 at 1:58 PM, Justin A <jus...@adler.com.au> wrote:

Initially, my code did that and assploded.

So i refactored it (to the code above) and i'm still getting an exception (not sure if it' the same one) and resharper it not showing 'access to modified closure' warning (with that code above).

when i have it in a LINQ statement, then I get that.

:(

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

You are doing 10 x 10 bulk inserts. That is pretty bad thing to do.

There is an initial cost for bulk insert, and the idea is that it is well worth the time because you are going to sends thousands or millions of documents through.

For 10, just use a session.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Justin A]

:) my initial numbers we 1000x1000 as in the example above.

I just dropped it to 10 x 10 to see if it was a SIZE issue, so try pumping the number up to something silly and then batch em.

eg. batches of 10,000 or 50,000 like you said.

[Oren Eini (Ayende Rahien)]

The underlying issue is here:

https://github.com/PureKrome/RavenDbBulkInsert/blob/master/src/RavenDbBulkInsert/Program.cs#L63

var tasks = batch.Select(employee => operation.StoreAsync(employee)).ToArray();

You aren't waiting for the StoreAsync to complete before starting the new one, which can cause issues.

Fixing this (to be):

 using (var operation = documentStore.BulkInsert())

 {

     foreach (var employee in batch)

     {

         await operation.StoreAsync(employee);

     }

 }

And using 10,000 batches.

Inserted 1,000,000 employee's in 36.09 seconds.

Note that doing this with a single batch gives

Inserted 1,000,000 employee's in 34.64 seconds.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Justin A]

Inline

On Monday, 30 October 2017 21:47:18 UTC+11, Oren Eini wrote:

The underlying issue is here:

https://github.com/PureKrome/RavenDbBulkInsert/blob/master/src/RavenDbBulkInsert/Program.cs#L63

var tasks = batch.Select(employee => operation.StoreAsync(employee)).ToArray();


You aren't waiting for the StoreAsync to complete before starting the new one, which can cause issues.

AH! So i was trying to be too tricky. I had 'x' number of async operations occuring simultaneously which was causing fighting to occur.

kewl!

 

Inserted 1,000,000 employee's in 36.09 seconds.

For me on my crappy macbook pro - 

Starting bulk insert testing app.

Created all fake employees.

Inserted 1,000,000 employee's in 133.25 seconds.

-- Press any key to quit.

 

Oooo! where is this info-graphic stat found?

Also - thanks for spending a bit of time looking at my issue. It's testing/playing around with scenario's like this that help us newbies get a better understanding and hopefully 'silly' things we newbies make, end up in documentation samples for others to learn from :)

-me- 

[Oren Eini (Ayende Rahien)]

The graphics are from the latest nightly, already available here:

http://4.live-test.ravendb.net/studio/index.html

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

[Johannes Rudolph]

Sorry for reviving an old thread, but we've been trying to optimize bulk insert performance on our end and with 4.1.5 and 4.2 and we do get nowhere near the performance numbers posted here. 

I'm seeing about 13.3k docs/second inserted. That's about half the numbers posted by Oren on 10/30/17 and a far cry from the claimed "150k writes/s on commodity hardware" on the website.

I'm using https://github.com/PureKrome/RavenDbBulkInsert at rev e36cbcfe3e35027021ea722305c29a23ce2030cb with the 4.2 client libs and on .NETCore.App/2.1.9 on OS X. RavenDB 4.2. is running on docker using the ubuntu image using a docker volume (i.e. not a host mount which would have known performance problems).

```

Inserted 1,000,000 employee's in 75.08 seconds.

-- Press any key to quit.

```

Machine is a 2018 MacBook Pro 13" 2,7 GHz Intel Core i7 which has 4.x GHz Turbo (on PSU) and should have insanely fast NVMe SSD. I get similar results on my i7-8700 desktop machine. 

The performance numbers match what we're seeing on our real application that we're in the process of moving to RavenDB (about 8-10k docs/s with ~10 bulk inserts running in parallel) .

I've also tried running the native binary on OS X and get the same performance, so it's not something with docker. The stats for 4.1.5 and 4.2. are also very similar.

The number from the test here come in at about 2 MiB/s according to the RavenDB dashboard. I'd have expected to see more like 100 MiB/s on that sort of hardware. After all bulk insert is just pretty much writing sequentially to the journal, is it? 

I've attached a screenshot of the studio dashboard and an i/o stats export. Has there been some kind of performance regression between 4.0 and 4.1.5/4.2? 

To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

A single client is typically not able to really push enough data to RavenDB.

Try running multiple instances of the bulk insert at the same time. 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To post to this group, send email to rav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/3a0fd49c-0499-4998-8914-a80df31a4b49%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Oren Eini CEO   /   Hibernating Rhinos LTD Mobile:  +972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

[vlko]

Hi Oren

It would be great to have sample app, we can reuse. For some scenarios we need some "super fast" importer. For example initial import or batch import from external sources during night where there is no load on database.

To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/3a0fd49c-0499-4998-8914-a80df31a4b49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johannes Rudolph]

That what I thought too - but our real app is already using multiple threads for pushing the data and on top scales horizontally to multiple workers - nonetheless our RavenDB instance tops out at approx. 10k docs/s total across all parallel bulk inserts.

That's by the way also the speed we see when importing a ravendump with a couple of million docs in it, which makes me believe it's somewhat of a global limit that we're running into.

I've tried modifying the test program by https://github.com/PureKrome/RavenDbBulkInsert to run 10 bulk inserts in parallel. Brings the total runtime down to 50s instead of 75s but still not great. With 4 bulk inserts at a batch size of 250k I was able to get 35s, which is an improvement x2 but not the order of magnitude I was hoping for to get into the range of 150k writes/s? This appears to be the best batch size from the ones I've tried.

We'd love to better understand what kind of limit we're running into (i.e. is it CPU/io/client/server) and whether there's something we can do to get the most out of our ravens? I.e. how can we expect bulk insert performance to scale with vCPUs, clients / parallelism etc.? 

 

Here's the test program I'm using:

var batches = employees.Batch(100000);

var stopwatch = Stopwatch.StartNew();

try

{

var tasks = batches

.Select(x => Task.Run(() => BulkInsertEmployees(x, documentStore).Wait()));

await Task.WhenAll(tasks);

}

On Monday, June 10, 2019 at 3:04:37 PM UTC+2, Oren Eini wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/3a0fd49c-0499-4998-8914-a80df31a4b49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

The fastest way to go about it is to generate the dump files and then point RavenDB to the directory in question.

https://github.com/ravendb/ravendb/blob/v4.2/src/Raven.Server/Smuggler/Documents/Handlers/SmugglerHandler.cs#L276  

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/be1d3953-af95-47fb-853b-aa245728207d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

You are probably still processing most of this in sequence. Try creating multiple bulk insert tasks that compete on pulling from a concurrent queue?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/3028b212-7862-4f0f-99c9-2ed645167816%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrej Krivulčík]

I just tried to achieve high performance with bulk insert. I used code that can be found at https://github.com/krivulcik/RavenDBBulkInsertTest

On my local dev laptop (i7 with 4 cores/8 threads, 32 GB RAM, NVMe SSD disk with lots of iops), I could achieve around 50k writes/second.

I spun up some azure VMs to test it:

L8s_v2 (8 vcpus, 64 GiB memory) machine with 1.9 TB host-local NVMe SSD (it doesn't get much closer to metal in cloud) for database. I formatted the disk as NTFS with 64 kB blocks (is this the best block size?).

2 x Standard H8 (8 vcpus, 56 GiB memory) machines with irrelevant storage as "work" nodes which generate data and push it to the database.

When running in a single thread:

dotnet RavenDBBulkInsertTest/bin/Debug/netcoreapp2.2/publish/RavenDBBulkInsertTest.dll http://IP:8080 BulkInsertTestSingle single W1 1000000

I get the following results:

[10:46:17] [Info] [5]: Inserted 1000000 employees in 44.84 seconds.
[10:46:17] [Info] [5]: 22301.76 records/s

When running in 16 threads:

dotnet RavenDBBulkInsertTest/bin/Debug/netcoreapp2.2/publish/RavenDBBulkInsertTest.dll http://IP:8080 BulkInsertTestParallel parallel W1 16 1000000

[10:44:34] [Info] [1]: Inserted 16000000 employees in 168.49 seconds.
[10:44:34] [Info] [1]: 94962.09 records/s

When running in 16 threads from two machines at the same time (32 writers):

dotnet RavenDBBulkInsertTest/bin/Debug/netcoreapp2.2/publish/RavenDBBulkInsertTest.dll http://IP:8080 BulkInsertTestParallelMulti parallel W1 16 1000000
dotnet RavenDBBulkInsertTest/bin/Debug/netcoreapp2.2/publish/RavenDBBulkInsertTest.dll http://IP:8080 BulkInsertTestParallelMulti parallel W2 16 1000000

[10:36:55] [Info] [1]: Inserted 16000000 employees in 331.82 seconds.
[10:36:55] [Info] [1]: 48218.33 records/s

[10:36:55] [Info] [1]: Inserted 16000000 employees in 331.51 seconds.
[10:36:55] [Info] [1]: 48264.06 records/s

It seems that this particular server with this particular data is limited at around 100k writes/second.

I believe that it doesn't get much easier than this:

• Very small documents.
• Bulk insert is probably optimized to sustain high write speeds.
  
• Inserting into empty database.
• NVMe storage with extreme IOPS capacity.
• Pushing data from unloaded writer nodes to unloaded database server.
• No replication, no encryption, no crypto (unsecured mode).
• VMs are in the same azure vnet, I didn't measure network latency/throughput but I think that it doesn't get much better than this.
  

Is this correct usage of parallelized BulkInsert?

Is the performance expected?

Is there anything else that could be done to achieve higher speeds in this scenario? https://ravendb.net/features/high-performance states: RavenDB can perform over 150,000 writes per second and 1 million reads on simple commodity hardware. This is not exactly simple commodity hardware and in the easiest scenario I can come up with, and still it doesn't reach this speed.

Are there any reference benchmarks we could run to check HW speed etc.?

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/3028b212-7862-4f0f-99c9-2ed645167816%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

The benchmark we usually use is wrk, script found here: 

https://github.com/ravendb/ravendb/tree/v4.2/bench/Scripts 

Try using multiple processes, not just threads, there are some per-process limits that might be hitting you.

I'll try to setup a standalone benchmark you can use this week 

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/cbfcba57-11a4-4e71-86f7-820a3bfd2b48%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrej Krivulčík]

I just tried running 4 processes with 4 threads each, on two servers each (so 8 processes in total, 32 threads in total). I get very similar results of roughly 100k writes/s as with single process with 32 threads.

Having a standalone benchmark would be great, thanks.

However, is there anything I can tweak to get as much performance as possible? Filesystem block size optimization etc? There are some system configuration recommendations here: (https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/system-configuration-recommendations ) but these are for Linux only. Are there any documented best practices for deploying on Windows? https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/deployment-considerations has some recommendations like not running from HDDs, but a comprehensive list of best practices would be very useful (like the issues that pop up in notifications - running with swap on HDD even though SSD is available etc.).

On Tuesday, June 11, 2019 at 2:41:02 PM UTC+2, Oren Eini wrote:

The benchmark we usually use is wrk, script found here: 

https://github.com/ravendb/ravendb/tree/v4.2/bench/Scripts 

Try using multiple processes, not just threads, there are some per-process limits that might be hitting you.

I'll try to setup a standalone benchmark you can use this week 

[Oren Eini (Ayende Rahien)]

Here is a blog post (will be up tomorrow) with the full details.

https://ayende.com/blog/187618-A/benchmarking-ravendb-bulk-insert-on-commodity-hardware?key=a6dd0a0bdc3149f99c993be686b0653c 

In short, I spend a few hours and managed to get to 100K writes/sec on AWS using  ie3n.xlarge machine.

The two issues are:

* Latency of I/O ( vs. throughput)

* The client being able to generate the data fast enough. 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To post to this group, send email to rav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/2238fccd-9f81-4222-8325-bc75dd1be425%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Oren Eini CEO   /   Hibernating Rhinos LTD

[Justin A]

Hi Oren - loved the blog post!!! There's heaps of gems in there that really really are awesome. Also loved the journey through the post -> that's just as valuable as the summary, imo. maybe even _more_ valuable, actually.

Just a few suggestions for some potential edits, to the post:

SIDE NOTE: I've just re-read my post below and there's a lot of stuff I'm saying, so just work with me through my journey :) It's mainly about clarification of disk setups and configurations.

> Here you can see a single write, for 124KB, that took 200ms.

Ok .. so if that's slow, what is an _expected_ target write speed?  10ms? 50ms? 2ms?

> setup the journals’ directory for this database to point to the new drive

Is the suggestion here to split this 'temp' data to a separate drive with respect to the 'users-data' data to avoid disk-io-thrashing/fighting? Similar in concept to how in other old school DB's like MSSql you can move TempDB to a different filesystem to avoid fighting.

Also, if you're going to move the Journal's data to a different disk, what size would you be thinking? In the examples you added a new 8GB disk for Journals .. why 8GB?

a) no real reason - journals don't use much size and it was $$-cheap but had some ok IOPS

b) you knew the final size of the data and it was a math formula (10% of final db size or the size of writes-per-second .. or something)

Why did you only go for 400 IOPS for the Journal disk? You know (through experience) that faster IOPS for Journal data (e.g. size of journal data + the amount of writes) start to add less value with respect to the $$-cost?

>  switch instances again .. to i3en.xlarge instance ... mostly interested in the NVMe drive

Wait what? an NVMe drive? but .. the previous pictures sorta suggested this was already happening?? (NOTE: I come from a Windows + Azure background, so with the heaps of AWS info in this blog post, I've quickly made fast-assumptions to what things are/mean with respect to AWS terminology. I do know what an NVMe HD is though...).

Here is a screenie from what I was seeing ...

So i kept thinking ... these disks were all just folders on an NVMe disk.

Lets go back and see what I misunderstood ...

> The machine has an 8GB drive that I’m using to host RavenDB

Ok - my guess this will host the OS and RavenDb and doesn't get used in the read/writes. (EDIT: more on this assumption, later)

> separate volume for the data itself. I create a 512GB gp2 volume (with 1536 IOPS) to start with

Ohhh... I read that as this was a disk that was created with the VM .. not some attached network disk. Also .. now I looked up what a gp2 is ... which == a 'typical' SSD. 

so this means ...

OS + RavenDb + Journal -> 8GB .. SSD?

Data -> 512 GB "normal" SSD

and later

OS + RavenDb -> 8GB .. SSD?

Data -> 512 GB "normal" SSD

Journal -> 8GB "fast" SSD.

Phew. ok.

continuing ..

> This time to i3en.xlarge instance (4 cores, 30GB, 2 TB NVMe drive)

So now it's like this...

OS + RavenDb + Data + Journal -> 2TB NVMe

and then ..

> On the same i3en.xlarge system, I attached the two volumes (512GB gp2 and 8GB io2) with the same setup (journals on the io2 volume)

Which is then...

OS + RavenDb -> 2TB NVMe

Data -> 512GB gp2 "normal" SSD

Journal -> 8GB io2 "fast" SSD

which has _really_ similar results for the last machine (99k and 93K) ??? so how can the results be so much higher than the previous machine when

- data and journals are on the same 'attached' network disks [gp2 and io2]

- 4 cores, both machines? [i3en.xlarge  and t3a.xlarge]

How is the NVMe being used in the last image when data + journal's are on different disks? Is RavenDb actually using the NVMe for something else so the idea of the Journal being tempData is not 100% accurate? There's other tempdata that writes to somewhere, also?

SIDE NOTE: I just jumped into the "Playground" and noticed this...

so maybe some explanation about what/how TempFiles is and how that played a part in the bulk insert stuff?

---

So yeah ... A few things I sorta got confused with so hopefully some clarifications could be added to the post?

I'm pretty damn certain I'll be hitting that blog post _heaps_ of times in the future to help test some hardware setups we plan to do with future projects. Sure _we_ don't need 100K writes/sec but it's nice to see what we can do/get with smaller setups and then have _some expectations_.

Sorry for sorta rambling .. but I thought it was important and I found the post very exciting and helpful.

regards,

- little ol' me - 

[Johannes Rudolph]

Thanks, this post provides a lot of useful info and reference numbers. We'll be continuing our quest for more performance with that input.

On Thursday, June 13, 2019 at 3:17:15 AM UTC+2, Oren Eini wrote:

Here is a blog post (will be up tomorrow) with the full details.

https://ayende.com/blog/187618-A/benchmarking-ravendb-bulk-insert-on-commodity-hardware?key=a6dd0a0bdc3149f99c993be686b0653c 

In short, I spend a few hours and managed to get to 100K writes/sec on AWS using  ie3n.xlarge machine.

The two issues are:

* Latency of I/O ( vs. throughput)

* The client being able to generate the data fast enough. 

On Tue, Jun 11, 2019 at 4:25 PM Andrej Krivulčík <kriv...@gmail.com> wrote:

I just tried running 4 processes with 4 threads each, on two servers each (so 8 processes in total, 32 threads in total). I get very similar results of roughly 100k writes/s as with single process with 32 threads.

Having a standalone benchmark would be great, thanks.

However, is there anything I can tweak to get as much performance as possible? Filesystem block size optimization etc? There are some system configuration recommendations here: (https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/system-configuration-recommendations ) but these are for Linux only. Are there any documented best practices for deploying on Windows? https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/deployment-considerations has some recommendations like not running from HDDs, but a comprehensive list of best practices would be very useful (like the issues that pop up in notifications - running with swap on HDD even though SSD is available etc.).

On Tuesday, June 11, 2019 at 2:41:02 PM UTC+2, Oren Eini wrote:

The benchmark we usually use is wrk, script found here: 

https://github.com/ravendb/ravendb/tree/v4.2/bench/Scripts 

Try using multiple processes, not just threads, there are some per-process limits that might be hitting you.

I'll try to setup a standalone benchmark you can use this week 

Mobile:  +972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/2238fccd-9f81-4222-8325-bc75dd1be425%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johannes Rudolph]

Thanks, this post provides a lot of useful info and reference numbers. We'll be continuing our quest for more performance with that input.

On Thursday, June 13, 2019 at 3:17:15 AM UTC+2, Oren Eini wrote:

Here is a blog post (will be up tomorrow) with the full details.

https://ayende.com/blog/187618-A/benchmarking-ravendb-bulk-insert-on-commodity-hardware?key=a6dd0a0bdc3149f99c993be686b0653c 

In short, I spend a few hours and managed to get to 100K writes/sec on AWS using  ie3n.xlarge machine.

The two issues are:

* Latency of I/O ( vs. throughput)

* The client being able to generate the data fast enough. 

On Tue, Jun 11, 2019 at 4:25 PM Andrej Krivulčík <kriv...@gmail.com> wrote:

I just tried running 4 processes with 4 threads each, on two servers each (so 8 processes in total, 32 threads in total). I get very similar results of roughly 100k writes/s as with single process with 32 threads.

Having a standalone benchmark would be great, thanks.

However, is there anything I can tweak to get as much performance as possible? Filesystem block size optimization etc? There are some system configuration recommendations here: (https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/system-configuration-recommendations ) but these are for Linux only. Are there any documented best practices for deploying on Windows? https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/deployment-considerations has some recommendations like not running from HDDs, but a comprehensive list of best practices would be very useful (like the issues that pop up in notifications - running with swap on HDD even though SSD is available etc.).

On Tuesday, June 11, 2019 at 2:41:02 PM UTC+2, Oren Eini wrote:

The benchmark we usually use is wrk, script found here: 

https://github.com/ravendb/ravendb/tree/v4.2/bench/Scripts 

Try using multiple processes, not just threads, there are some per-process limits that might be hitting you.

I'll try to setup a standalone benchmark you can use this week 

Mobile:  +972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/2238fccd-9f81-4222-8325-bc75dd1be425%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

Another thing that I forgot to mention (but updated in the post), look at:

DocumentStore.Conventions.BulkInsert.TrySerializeEntityToJsonStream

[Andrej Krivulčík]

Oren, thanks for this blog post. Very informative and useful!

I went ahead and tried to break this 100k writes/s barrier I've been hitting in my benchmarks.

Good news is that I could break it and hit 200k writes/s. However, the journey was not very straightforward.

The best result so far was ~100k writes/s on Azure servers - standalone dedicated DB server with NVMe storage (200k write iops) and separate compute server which fed data to this DB server. When I added a second server, the write speed stayed the same and the clients were not fully utilized, so the client throughput was not the bottleneck anymore.

Okay, what could be the bottleneck here?

You mentioned in the post that on physical hardware, the limit is higher. Let's try this on bare metal.

My laptop has NVMe SSD with around 160k iops random write capacity. We connected by coworker's PC (6 cores/12 threads CPU) over a gigabit network (100 Mbps was a limiting factor here), and ran the benchmark again. 100k writes/s again.

Okay, what could be the bottleneck here?

Screw persistent storage and let's go to RAM. I set up a ramdisk which I used as database storage on the Azure VM and ran the benchmarks again. (I needed to format a file on the ramdisk as RavenDB refused to create journal files directly on the ramdisk, if anyone would like to try this.) 100k writes/s again.

Okay, what could be the bottleneck here?

Taking a look at IO Stats chart, we noticed an intriguing pattern: The journal writes were extremely fast and short, and there were gaps between all of them, both when the storage was NVMe and RAM - it was more pronounced when storing to ramdisk. Seems like the disk storage is no longer the bottleneck. In this particular case, the database stores the data in a single thread, where the iops capacity is much lower. 30k iops on my laptop, 25k iops on the Azure VM.

I ran the benchmark on two databases in parallel. One server pushed the data to one database, the other server to another database. The performance was a little bit better - 100k writes/s and a change. Not pinned to 100k anymore but not much better. Also, the servers generating the data started to slack off.

Okay, what could be the bottleneck here?

Turns out that the network between the Azure VMs was saturated now, at around 40 MB/s (400 Mbps). Back to bare metal and direct gigabit connection!

I set up two databases on my laptop and the other PC started to shove the data in parallel. Much to our delight, the write rate was around 200k writes/s! The storage capacity was finally being used. When we tried to add a third database, the speed didn't really increase and the laptop started throttling (thermal design is not the best on this one) so we didn't really care to continue.

Anyway, in this particular scenario - bulk inserting to one database - the 100k writes/s limit probably won't get any better. However, it's pretty good speed, especially as I started at around 15k with my attempts :-). In other scenarios - multiple imports, ordinary operation etc. - the write capacity is higher, which is good to know.

Again, thanks for the story.

On Thursday, June 13, 2019 at 3:17:15 AM UTC+2, Oren Eini wrote:

Here is a blog post (will be up tomorrow) with the full details.

https://ayende.com/blog/187618-A/benchmarking-ravendb-bulk-insert-on-commodity-hardware?key=a6dd0a0bdc3149f99c993be686b0653c 

In short, I spend a few hours and managed to get to 100K writes/sec on AWS using  ie3n.xlarge machine.

The two issues are:

* Latency of I/O ( vs. throughput)

* The client being able to generate the data fast enough. 

On Tue, Jun 11, 2019 at 4:25 PM Andrej Krivulčík <kriv...@gmail.com> wrote:

I just tried running 4 processes with 4 threads each, on two servers each (so 8 processes in total, 32 threads in total). I get very similar results of roughly 100k writes/s as with single process with 32 threads.

Having a standalone benchmark would be great, thanks.

However, is there anything I can tweak to get as much performance as possible? Filesystem block size optimization etc? There are some system configuration recommendations here: (https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/system-configuration-recommendations ) but these are for Linux only. Are there any documented best practices for deploying on Windows? https://ravendb.net/docs/article-page/4.2/Csharp/start/installation/deployment-considerations has some recommendations like not running from HDDs, but a comprehensive list of best practices would be very useful (like the issues that pop up in notifications - running with swap on HDD even though SSD is available etc.).

On Tuesday, June 11, 2019 at 2:41:02 PM UTC+2, Oren Eini wrote:

The benchmark we usually use is wrk, script found here: 

https://github.com/ravendb/ravendb/tree/v4.2/bench/Scripts 

Try using multiple processes, not just threads, there are some per-process limits that might be hitting you.

I'll try to setup a standalone benchmark you can use this week 

Mobile:  +972-52-548-6969 Sales:  sa...@ravendb.net Skype:  ayenderahien Support:  sup...@ravendb.net

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rav...@googlegroups.com.

To post to this group, send email to rav...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/2238fccd-9f81-4222-8325-bc75dd1be425%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oren Eini (Ayende Rahien)]

inline

On Thu, Jun 13, 2019 at 7:12 AM Justin A <jus...@adler.com.au> wrote:

Hi Oren - loved the blog post!!! There's heaps of gems in there that really really are awesome. Also loved the journey through the post -> that's just as valuable as the summary, imo. maybe even _more_ valuable, actually.

Just a few suggestions for some potential edits, to the post:

SIDE NOTE: I've just re-read my post below and there's a lot of stuff I'm saying, so just work with me through my journey :) It's mainly about clarification of disk setups and configurations.

> Here you can see a single write, for 124KB, that took 200ms.

Ok .. so if that's slow, what is an _expected_ target write speed?  10ms? 50ms? 2ms?

Basically, translate that to MB/s. This comes out to ~600KB/s, which is really bad.

I would want to see stuff in the range of < 15 ms for all writes under MB, usually.

 

> setup the journals’ directory for this database to point to the new drive

Is the suggestion here to split this 'temp' data to a separate drive with respect to the 'users-data' data to avoid disk-io-thrashing/fighting? Similar in concept to how in other old school DB's like MSSql you can move TempDB to a different filesystem to avoid fighting.

No, temp data is different. We have this option, but I didn't bother with it because on AWS, it typically doesn't matter. On Azure, you usually have the D drive and can use it for this.

Here we are talking about the journals (which are usually what slow us down in the critical commit path). You can now parallelize the writes to the journals and the data file at the same time.

 

Also, if you're going to move the Journal's data to a different disk, what size would you be thinking? In the examples you added a new 8GB disk for Journals .. why 8GB?

Journals are usually only kept around until the data is flushed. In this case, the load on the server was ~0.75 GB of journals on a consistent basis, so there was no point in having much more.

If you had a HDD in there, where a lot more data may reside in memory, you might need more. See also: https://ayende.com/blog/174916/the-guts-n-glory-of-database-internals-what-goes-inside-the-transaction-journal

 

a) no real reason - journals don't use much size and it was $$-cheap but had some ok IOPS

b) you knew the final size of the data and it was a math formula (10% of final db size or the size of writes-per-second .. or something)

Basically, the idea is to compute how much journals are taking under load, and have a nice padding factor. In most cases, unless you have very large transactions, it is going to be pretty small.

 

Why did you only go for 400 IOPS for the Journal disk? You know (through experience) that faster IOPS for Journal data (e.g. size of journal data + the amount of writes) start to add less value with respect to the $$-cost?

Max amount of IOPS you can get from 8gb io2 disk

 

>  switch instances again .. to i3en.xlarge instance ... mostly interested in the NVMe drive

Wait what? an NVMe drive? but .. the previous pictures sorta suggested this was already happening?? (NOTE: I come from a Windows + Azure background, so with the heaps of AWS info in this blog post, I've quickly made fast-assumptions to what things are/mean with respect to AWS terminology. I do know what an NVMe HD is though...).

Different NVMe, Linux may show that this is a NVMe drive, but it is an EBS volume.  Basically, I suspect that the OS is lying to me about how that.

The i3en has a dedicated NVMe that is in the machine, not over the network.

 

Here is a screenie from what I was seeing ...

So i kept thinking ... these disks were all just folders on an NVMe disk.

Lets go back and see what I misunderstood ...

> The machine has an 8GB drive that I’m using to host RavenDB

Ok - my guess this will host the OS and RavenDb and doesn't get used in the read/writes. (EDIT: more on this assumption, later)

> separate volume for the data itself. I create a 512GB gp2 volume (with 1536 IOPS) to start with

Ohhh... I read that as this was a disk that was created with the VM .. not some attached network disk. Also .. now I looked up what a gp2 is ... which == a 'typical' SSD. 

so this means ...

OS + RavenDb + Journal -> 8GB .. SSD?

Data -> 512 GB "normal" SSD

and later

OS + RavenDb -> 8GB .. SSD?

Data -> 512 GB "normal" SSD

Journal -> 8GB "fast" SSD.

Phew. ok.

Yes, correct.

 

continuing ..

> This time to i3en.xlarge instance (4 cores, 30GB, 2 TB NVMe drive)

So now it's like this...

OS + RavenDb + Data + Journal -> 2TB NVMe

Actually, I think it also had a 8GB for the OS itself, and the 2TB was just for data / journal

 

and then ..

> On the same i3en.xlarge system, I attached the two volumes (512GB gp2 and 8GB io2) with the same setup (journals on the io2 volume)

Which is then...

OS + RavenDb -> 2TB NVMe

Data -> 512GB gp2 "normal" SSD

Journal -> 8GB io2 "fast" SSD

which has _really_ similar results for the last machine (99k and 93K) ??? so how can the results be so much higher than the previous machine when

- data and journals are on the same 'attached' network disks [gp2 and io2]

- 4 cores, both machines? [i3en.xlarge  and t3a.xlarge]

See Andrej comment, we have a single writer thread per database, so once you get past the costs of writing to the IO, we are bottleneck by how fast we can process this in a single thread.

 

How is the NVMe being used in the last image when data + journal's are on different disks? Is RavenDb actually using the NVMe for something else so the idea of the Journal being tempData is not 100% accurate? There's other tempdata that writes to somewhere, also?

Journal is _the opposite_ of temp data. Even though it is temporary :-)

Journal is a _durable_ record of changes to the database. This is how Voron gets is durability / ACID from.

Once we apply changes to the data file (and sync), we no longer need the journal and can be discarded / reuse.

Temp data are files that we write / read (temporary tx buffers) but aren't important for the db and re-created on each startup.

 

SIDE NOTE: I just jumped into the "Playground" and noticed this...

so maybe some explanation about what/how TempFiles is and how that played a part in the bulk insert stuff?

Temp files in this case is a memap buffer to which we write data while the transaction is in play.

 

---

So yeah ... A few things I sorta got confused with so hopefully some clarifications could be added to the post?

The Guts N Glory series might be a good place to start.

This is non trivial topic, so hard to explain simply.

 

I'm pretty damn certain I'll be hitting that blog post _heaps_ of times in the future to help test some hardware setups we plan to do with future projects. Sure _we_ don't need 100K writes/sec but it's nice to see what we can do/get with smaller setups and then have _some expectations_.

Sorry for sorta rambling .. but I thought it was important and I found the post very exciting and helpful.

regards,

- little ol' me - 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To post to this group, send email to rav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/b011becc-bacc-4efc-bcad-db5ea3dcea02%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Justin A]

@Oren - thanks heaps for the reply! really helped :)

@andrej - wow, that's another awesome story, also :) Kewl!!! 

I know only part of all of these stories has stuck into my head, so I'm pretty sure i'll be coming back to this thread and these 2x blog posts (Batch inserts + Guts n Glory) a number of times, in the future. So thanks heaps for this information :)

-me-