Diagnosing Erratic CPU Performance

[Jeremiah Barndt]

For a little over a week now, our RavenDB setup has been experiencing issues in relation to CPU usage that has in some cases caused unresponsiveness in the silverlight interface and as well as the server responding to requests for data.

Essentially, for most of the past few months we've been using Raven, CPU usage has been very low (0-5%) at all times of the day. The only time it would spike up high (as much as 100%) would be when we'd be doing massive operations like bulk inserts and such. To give some additional background on our setup, the majority of all requests (read-only) going to Raven is through our front-end Web API setup.

A little over a week ago we noticed that Raven was unresponsive and was failing to return data for requests. The CPU would be locked between 97-100% usage, all attributed to the Raven.Server.exe process and we wouldn't be able to recover other than to force close the service and start it up again. It happened multiple times over a few day period, each requiring a restart, after which, CPU would go back down to normal for a while.

We were running build 2330 at the time and decided we'd try our luck upgrading to 2375 to see if that would make a difference. Since then, we've still had high CPU spikes, but they have not locked up the server in any way and the high CPU usage seems to go down after 5-10 minutes when it does happen. It does not adversely affect anything, but is still something I have concern for as to why it would be occurring. I've loaded up perfmon and looked into the metrics as well as turned on NLog debugging to see if I was getting an influx of requests around the high CPU times. This does not seem to be the case. Request load is steady (2-10 req/second) and does not appear to usually fluctuate much during high CPU load. In addition, I've noticed that the high CPU load seems to happen at certain times throughout each hour. Right now, generally the last 10 minutes of each hour the CPU hovers between 50% and 95% CPU usage, then goes back down, usually until around the half hour mark where for a couple minutes it goes up to 50% - 95% again, then back down, rinse and repeat. It seems to be consistent from hour to hour overall.

I did try turning off IIS once (Raven is running as a service) so that Web API was not passing requests to Raven and the one time I tried that I did not see the CPU usage go up like it typically would at the certain points throughout the hour, which would make me think it was more requests causing high CPU, but both the IIS logs and perfmon metrics beg to differ with no significant difference in requests counts during high CPU.

So I guess I'm trying to figure out something else I should be looking at to try and figure out what is happening. Appreciate the help!

[Oren Eini (Ayende Rahien)]

Are you using periodic backup, by any chance?

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jeremiah Barndt]

No, we haven't enabled that feature, actually.

[Oren Eini (Ayende Rahien)]

Can you take a dump while it is using high cpu?

[Karl Cassar]

To add to this, this has been happening as well to me on two different applications - exactly the same thing. Raven.Server.exe utilising CPU between 50 - 100% for no apparent reason, and it does not even close when I try to stop it, but I have to forcefully end the process.

Also, stopping the application who is the only client will still leave Raven.Server running at 50 - 100% CPU usage, and this happened both on the latest stable build, and latest unstable build.  However, with the latest unstable build, it never became unresponsive - Only slows down the server due to massive CPU usage.  

Regards,

Karl

[Jeremiah Barndt]

It would figure that I haven't been having the CPU spikes occur in the last day or so. Nothing has changed, so I will keep a watchful eye and if I can get a dump for you I will.

[Kijana Woodard]

Weekends don't make a difference?

[Jeremiah Barndt]

Not at all. The best I could tell this had nothing to do with the amount of requests coming in at a given time judging by the logs. Weekends/day/night time was not immune to these spikes. Request/sec rarely goes above 10 or so. Average requests/sec is closer to be 1-4.

I should also clarify my previous comment about periodic backup. The bundle is "activated" on the database but not configured/enabled. I'm not sure if this makes a difference, but essentially, when I created the database I think periodic backup was a default option and I just let it go. Looking at the configuration in the interface though it's not actually enabled, so my assumption is that this isn't effecting anything.

As I write this my colleague has informed me that over last night we started getting 40% CPU spikes for a minute or two at the same time every hour. The rest of the time it's 1-5% CPU usage (fairly normal). Looking at the details myself, it looks like for about the last 9-10 hours, right around the 53-54 minute mark there's a 40% or so CPU spike. This is different from my original post for a couple reasons. I was seeing spikes generally twice per hour and they were much higher and for longer than 2 minutes. Unfortunately I can't definitively say these 40% spikes were the Raven process (even though, besides IIS, is the only service running) since I've been manually looking at the task manager when the CPU was spiking, so at least for the time being I would disregard that since I have no additional information.

[Oren Eini (Ayende Rahien)]

Do you have logs?