RavenDB process suddenly starts to consume 100% CPU and server becomes unresponsive - build 35215

[Ian Cross]

Hi guys,

We have hit an issue twice in the last week on a server that has been running for the last 2 years. Our last upgrade was to RavenDB 3.54 (35215) back in August.

Last week at 04:30 in the morning and today at 16:10 this afternoon. RavenDB was doing fine at roughly 0-30% CPU and then all of a sudden.... boom (see charts)...

The only recovery from this is to kill the RavenDB service and it can take a long time to even get onto the server to even launch Task Manager. We managed to disable the indexes from another server via the RavenDB console but that didn't help. It is very hard to operate the console or server when this happens but we might be able to if we catch it earlier next time.

We have looked through event logs, alerts, console logs but nothing to give us anything obvious. There is a single application for this database and the load is not unusual and we can't see anything funny in the logs. The client application is able to function to start with but gradually slows down until it becomes unresponsive. 

Are there any known issues that have been fixed since July that exhibit similar behaviour. We have both RavenFS and Replication but only active / passive (if that's relevant).

Cheers,

Ian

[Oren Eini (Ayende Rahien)]

Nothing that pops to mind, no. 

Can you setup procdump to watch this and then send us the minidump?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Cross]

Many thanks Oren....

I have set up procdump to watch for > 98% for 60 seconds and to take 3 mini dumps.

Hopefully we will catch it using this.

Cheers,

Ian

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Ian Cross]

Ok, the 100% CPU hit yesterday and again today. I have managed to get 2 mini-dumps, a few minutes apart.

I assume I would send a link to support at ravendb.net? They are roughly 30MB each.

Kind regards,

Ian

[Ian Cross]

Ok, I have sent links to support.

Cheers,

Ian

On Tuesday, November 14, 2017 at 1:28:51 AM UTC, Oren Eini wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.