3.5 Performance

[Jahmai Lay]

I am performance tuning our applications and observed that adding a single database read to a use-case (like reading a user record) has quite a significant and seemingly disproportionate impact on the end-to-end latency of that use-case, so I went ahead and did some micro benchmarks on raven to figure out why.

The test machine is:

i7-5390K (6 phsyical / 12 logical core - 3.5ghz)

32GB RAM

Samsung 950 Pro m2 SSD

The test methodology is:

Single Read/Write - One thread in a loop performing the read/write operation

Concurrent Read/Write - One thread per core in a loop performing the read/write operation concurrently

Batch Write - One thread in a loop writing 1000 records at a time

DatabaseCommands - The test was performed using an instance of IDatabaseCommands from a DocumentStore, using the Get / Put methods.

DocumentActions - The test was performed using an instance of DocumentActions from a DocumentDatabase, using the Get / Put methods.

The benchmark is hosted inside an xunit unit test and run via the Resharper unit test runner, in Release / x64 configuration.

Each test is performed 5 times, and each iteration runs for 5 seconds and measures the number of operations / second.

The tests are executed in serial.

The test source is attached, note that is uses some library code in our product so can't be compiled without modification.

The raw test results are also attached for reference.

I acknowledge that these kinds of micro benchmarks can defy real world performance, but some of the results support other observations I have in the performance penalty of accessing Raven in our products use-cases so they are valuable in my case.

Observations:

- I suspect the immense performance increase of Voron over Esent for reads is due to reading the same 1000 documents over and over and Voron having those pages already loaded in memory. I am impressed none the less.

- I expected more from Voron writes. I am confused as to whether there is a misconfiguration here, or it's just how Voron behaves when writing large numbers of new records to a new database.

- Finally, and most critically, I am baffled by the enormous penalty of reading records via IDatabaseCommands vs DocumentActions when using Esent. For writes, the penalty appears to be something in the vicinity of 10-20%, which considering there is serialisation, HTTP protocol overhead and network access, seems reasonable. But then for reads, there is a 5x drop in performance for concurrent reads, and a 30x (!) drop in performance for single threaded reads. This appears to be what is dramatically affecting our application performance, and I am looking for some insights as to why this might be the case.

[Oren Eini (Ayende Rahien)]

Out of interest, could you run the same on 4.0?

What are the results when you use the async API?

Note that the level of things that have to happen for DocumentsActions vs DatabaseCommands are huge. 

We need to serialize it, send it over the network, deserialize it, etc.

That creates a lot of memory pressure that can impact this.

What actually happens in the DocumentsActions scenario is that we give you instances that were fetch directly from the internal cache, that means that what we have to do is basically:

- B+Tree lookup.

- Cache lookup

- Done

On the DatabaseCommands side of things:

- All of the above

- Serializing the data to the network

- Deserializing the data from the network

- Deal with allocations, json parsing costs, GC, etc.

With single thread operations, that is our worst scenario, because you have to wait for the whole thing, and you aren't gaining anything from the async work methods.

With writes, you don't really see it because the cost of going to disk is so high that it alleviate the difference.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jahmai Lay]

I'll run it on 4.0 and using async API and get back to you.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Jahmai Lay]

There is no Raven.Database nuget project for 4.0 server. Do I still only need Raven.Database.dll and Raven.Abstractions.dll or is it more complex now?

[Oren Eini (Ayende Rahien)]

Just Raven.Server, although it would be easier to do it against a remote db

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Jahmai Lay]

I haven't done 4.0 yet, but here are some interesting results in a couple of new 3.5 tests:

DatabaseCommands_Single_Read|esent: 7745 reads in 5.0002523 seconds (1548 per second).

DatabaseCommands_Single_Read|esent: 7826 reads in 5.0000266 seconds (1565 per second).

DatabaseCommands_Single_Read|esent: 7779 reads in 5.0001305 seconds (1555 per second).

DatabaseCommands_Single_Read|esent: 7841 reads in 5.0006551 seconds (1567 per second).

DatabaseCommands_Single_Read|esent: 7794 reads in 5.0003173 seconds (1558 per second).

AsyncDatabaseCommands_Single_Read|esent: 7987 reads in 5.000086 seconds (1597 per second).

AsyncDatabaseCommands_Single_Read|esent: 8001 reads in 5.0001393 seconds (1600 per second).

AsyncDatabaseCommands_Single_Read|esent: 8052 reads in 5.0004048 seconds (1610 per second).

AsyncDatabaseCommands_Single_Read|esent: 8055 reads in 5.0000439 seconds (1610 per second).

AsyncDatabaseCommands_Single_Read|esent: 7987 reads in 5.0002991 seconds (1597 per second).

DatabaseCommands_Concurrent_Read|esent: 12362 reads in 5.0011508 seconds (2471 per second).

DatabaseCommands_Concurrent_Read|esent: 38874 reads in 5.0010916 seconds (7773 per second).

DatabaseCommands_Concurrent_Read|esent: 38697 reads in 5.0005325 seconds (7738 per second).

DatabaseCommands_Concurrent_Read|esent: 38837 reads in 5.0006701 seconds (7766 per second).

AsyncDatabaseCommands_Concurrent_Read|esent: 42172 reads in 5.0009754 seconds (8432 per second).

AsyncDatabaseCommands_Concurrent_Read|esent: 42403 reads in 5.0009181 seconds (8479 per second).

AsyncDatabaseCommands_Concurrent_Read|esent: 41499 reads in 5.0009974 seconds (8298 per second).

AsyncDatabaseCommands_Concurrent_Read|esent: 42411 reads in 5.0009181 seconds (8480 per second).

AsyncDatabaseCommands_Concurrent_Read|esent: 42053 reads in 5.0009025 seconds (8409 per second).

As you can see, async doesn't improve much, however:

DatabaseCommands_Single_MultiGet|sequential|100|esent: 31800 reads in 5.0137222 seconds (6342 per second).

DatabaseCommands_Single_MultiGet|sequential|100|esent: 31500 reads in 5.0020688 seconds (6297 per second).

DatabaseCommands_Single_MultiGet|sequential|100|esent: 31800 reads in 5.0026138 seconds (6356 per second).

DatabaseCommands_Single_MultiGet|sequential|100|esent: 31900 reads in 5.0126452 seconds (6363 per second).

DatabaseCommands_Single_MultiGet|sequential|100|esent: 31400 reads in 5.0037426 seconds (6275 per second).

DatabaseCommands_Single_MultiGet|parallel|100|esent: 83100 reads in 5.002608 seconds (16611 per second).

DatabaseCommands_Single_MultiGet|parallel|100|esent: 83700 reads in 5.0047585 seconds (16724 per second).

DatabaseCommands_Single_MultiGet|parallel|100|esent: 90700 reads in 5.0039441 seconds (18125 per second).

DatabaseCommands_Single_MultiGet|parallel|100|esent: 93200 reads in 5.0030808 seconds (18628 per second).

DatabaseCommands_Single_MultiGet|parallel|100|esent: 92000 reads in 5.0012391 seconds (18395 per second).

DatabaseCommands_Concurrent_MultiGet|sequential|100|esent: 86500 reads in 5.0075925 seconds (17273 per second).

DatabaseCommands_Concurrent_MultiGet|sequential|100|esent: 146400 reads in 5.0197814 seconds (29164 per second).

DatabaseCommands_Concurrent_MultiGet|sequential|100|esent: 151500 reads in 5.0452868 seconds (30028 per second).

DatabaseCommands_Concurrent_MultiGet|sequential|100|esent: 176100 reads in 5.0341453 seconds (34981 per second).

DatabaseCommands_Concurrent_MultiGet|sequential|100|esent: 186100 reads in 5.0254123 seconds (37031 per second).

DatabaseCommands_Concurrent_MultiGet|parallel|100|esent: 56800 reads in 5.1071263 seconds (11121 per second).

DatabaseCommands_Concurrent_MultiGet|parallel|100|esent: 121100 reads in 5.0275505 seconds (24087 per second).

DatabaseCommands_Concurrent_MultiGet|parallel|100|esent: 162400 reads in 5.0243965 seconds (32322 per second).

DatabaseCommands_Concurrent_MultiGet|parallel|100|esent: 182100 reads in 5.0201605 seconds (36273 per second).

DatabaseCommands_Concurrent_MultiGet|parallel|100|esent: 195100 reads in 5.0337229 seconds (38758 per second).

Using multiget brings performance back into the realm of acceptable, "parallel" means &parallel=yes is used in the request and the 100 is how many GET's per multi-get.

This suggests most of the performance loss is in the http request, rather than serialization. It is localhost so network latency should be minimal, but is nagle algorithm enabled?

I will get round to trying 4.0 soon, but for now I think I can work some kind of client batching and multi-get requests for our high throughput use-cases.

[Tal Weiss]

Jahmai have you read https://ayende.com/blog/176257/10x-speedup-utilizing-nagle-algorithm-in-business-application

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hibernating Rhinos Ltd  

Tal Weiss l Core Team Developer l Mobile:+972-54-802-4849

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811l Skype: talweiss1982

RavenDB paving the way to "Data Made Simple" http://ravendb.net/ 

[Jahmai Lay]

Hi Tal,

After 5 years of developing a realtime network application for mobile phones I absolutely agree that application level Nagle is far better than the socket level Nagle. Being able to batch your compression / encryption and reduce protocol header overhead is a very good thing. So I assume from your response that socket level Nagle is disabled in Raven 3.5? If so, then I consider that a good thing.

Hibernating Rhinos Ltd  

[Oren Eini (Ayende Rahien)]

Yes, that is disabled.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Gleb Chermennov]

Have you continued the research yet?

пятница, 6 января 2017 г., 3:00:03 UTC+3 пользователь Jahmai Lay написал:

[Jahmai Lay]

Not yet, there is some work I need to do to make the tests work with 4.0, since they are in-process using DocumentDatabase directly rather than to an externally running server.