RavenFS performance problems

[Michael Davis]

We've started to see some major performance issues when downloading files from a RavenFS filesystem in our production RavenDB server.  Specifically, we're commonly seeing requests to download even small (<20KB) files with a duration of anywhere from 5 to 60 seconds (as recorded by the Traffic Monitor; if I make the request from a browser, it generally seems to be even longer than that before the file actually finishes downloading).  As far as I can tell, the duration doesn't have anything to do with the actual file size, and it seems to be fairly random to me.  The RavenDB side of things seems to be fine by comparison; it's rare to see a DB request that even takes as long as 1 second, and the average looks to be around 50 ms.

I'm not sure if this might help, but one thing I've noticed while watching it is that I've repeatedly seen multiple FS requests with large (but not necessarily equal) durations appear all at the same time in the Traffic Monitor, as though they were all waiting on some shared resource that was finally released.

We're currently running build 30153.  The filesystem having the problem is using the Encryption bundle and the ESENT storage engine.  Any recommendations for how we might be able to help diagnose or fix this would be appreciated.

As an aside, is there any way to have the Traffic Monitor listen for requests for just a filesystem if you have both a filesystem and a database with the same name (i.e., db/Foo and fs/Foo)?  It doesn't seem to let you use the prefix to pick one or the other, and when I just enter the shared name, I only get results for the database, not the filesystem.  To get results for the filesystem, I'm having to use the admin view to get everything.

Thanks for any help that anyone can provide.

Michael Davis

[Fitzchak Yitzchaki]

How much files do you have?

What is the biggest file that you have there?

And how much concurrent requests do you have?

Regards the filtering I opened an issue:

http://issues.hibernatingrhinos.com/issue/RavenDB-5315

Best Regards,

Hibernating Rhinos Ltd  

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ar...@ayende.com]

Hi Michael,

in addition to the above questions, can you provide us with the following stuff:

1) debug logs (https://ravendb.net/docs/article-page/3.0/csharp/studio/management/admin-logs) configured for `Raven.Database.FileSystem` category and `Debug` level taken at the time of experiencing slow file download
2) Fiddler capture taken at the same time (https://ravendb.net/docs/article-page/3.0/Csharp/server/troubleshooting/sending-support-ticket)

Do you have files synchronization enabled (master/slave or master/master)?

Regards,
Arek

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Michael Davis]

The filesystem contains 171,253 files.  The largest of them appears to be 13.17 MB.  I don't believe we have very many concurrent requests.  The highest I've seen the requests per second counter go up to in the last few minutes is about 2.5, and most of the time it's staying below 1.  I don't have any synchronization set up.

The requested logs and a Fiddler capture are available here: https://alabama.box.com/s/6a604pt6tv4ch2g5ie4ogoyz1uy7codt

[Arkadiusz Palinski]

According to the logs and Fiddler output the request took 51 seconds. About 25 seconds of that time was processing by RavenDB server (Temp-Request-Time header in the response), including the writing a file's content to the response.

Is the host 'http://optimus:8080' a remote machine or localhost? Do you have any antivirus software installed there? Has any machine configuration changed meanwhile, since you didn't experience that issue before?

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/G3fqBmCbRVU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Michael Davis]

It's the local machine.  I'm not aware of any antivirus software on that server, nor am I aware of any configuration changes on that server.

I think the problems may have been getting gradually worse for a while now, though I'm not certain.  We made several changes to our application to try and cache some of the data so we didn't need to go to the database as often, and that may have helped for a while, but we're about at the limit of what we can do there.  Some of the performance issues we encountered in the past may have been due to problems with another third-party application we depend on, so that problem might have hidden this one for a while until we found a workaround.

[Arkadiusz Palinski]

Can you send us the backup of that file system so we would be able to track it on our side (sup...@ravendb.net)?

[Michael Davis]

Sorry, but I don't think that will be possible.  For one thing, it's over 100 GB.  More importantly, it contains sensitive information, and that information belongs to our client (who also owns the server), not us, so I don't have the authority to give you access to it, either.

Do you have any other suggestions for things we can look at on our end?

[Oren Eini (Ayende Rahien)]

Can we schedule a call with you? And can you setup an environment where we can test it without affecting prod?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Michael Davis]

We can probably set up something tomorrow between 1-5PM CDT (UTC-5).  If that works for you, send me a direct email and we can work out the details.

I don't think we've seen anything like this in any test environment we have, though none of them have anywhere near the amount of data that the production DB contains.  Is there anything in particular you're looking for in a test environment?

[Oren Eini (Ayende Rahien)]

We are working on this with another customer, hopefully we'll have something by next week to report

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[ar...@ayende.com]

http://issues.hibernatingrhinos.com/issue/RavenDB-5340

Fixed in 3.0.30156-hotfix