Major issues since #35202 upgrade

[ZNS]

Hello,

We have ravendb setup running multiple databases  and replication on two separete servers and we recently made two changes:

1. Upgraded from #35183 to #35202
2. All our databases we're running master->slave machine 01 replicated to 02 but 02 did not replicate anything. However one of our databases had some special needs so we also set only this database to master<->master so 02 replicates to 01 for this database only.

Since this time we have had two incidents where our second server, the one that runs as slave for all databases (except one), has used 100% CPU for several hours. This causes the master server to throw ALOT of "Failed to replicate.." errors both for documents, indexes and transfomers, due to timeout. Which after several hours seems to degrade the master server so that it also stops responding to requests, they time out. The strange thing is that the slave server which runs on 100% CPU does not log anything in the ravendb logs.

Is it safe to downgrade back to #35183?

[Oren Eini (Ayende Rahien)]

Can you take a debug info package and maybe a minidump and send it to us?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ZNS]

At which point in time can I take those, so that they are of any use to you? Only during the 100% cpu-load, or right after? During the server is pretty much unreachable so that can be hard. Also would it be safe to downgrade to #35183 we might need to do that If this isn't resolved pretty soon.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

During the high CPU, yes.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Oren Eini (Ayende Rahien)]

And I would rather not downgrade, we can look into this pretty easily.

[ZNS]

Happened again but this time on the master server. Was unable to get any debug data due to that the server was completely unresponsive running on 100% CPU. Most annoyed that the failover doesn't seem to work, we use "ReadFromAllServers" for our API (which gets data from ravendb) but the application failed when master went down, failed when only the slave went down also. We don't have any choice but to downgrade at this point.

[Oren Eini (Ayende Rahien)]

Can you take a minidump at this stage?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[ZNS]

I'm unable to reach the server with remote desktop. I'm looking into using procdump to see if that could work. https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

[Oren Eini (Ayende Rahien)]

That would work, yes

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[ZNS]

Took a while longer this time, but it happened again. Got at least two dumps this time. I'm sending them to support.

[Oren Eini (Ayende Rahien)]

All the highly used threads seems to be request processing / thread pool stuff.

Do you have a high number of requests while this is going on?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[ZNS]

I haven't checked but it seems unlikely. Almost all requests to ravendb comes from an API we host, and there are not an unusal amount of requests to the API at the time of the cpu load. Also The cpu load goes from normal (avg. ~20%) to 100% almost instantly and then stays at 100% until the machine is rebooted.

http://prntscr.com/fn7pji

[Michael Yarichuk]

Hi,

Just to be sure, can you check the server's Event Viewer for errors?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best regards,

 

Hibernating Rhinos Ltd  

Michael Yarichuk l RavenDB Core Team 

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

RavenDB paving the way to "Data Made Simple"   http://ravendb.net/  

[ZNS]

There's unfortunately nothing of interest in the windows event logs...

[ZNS]

Without anything to back it up it feels like it's some specific request that triggers the cpu load, perhaps causing an infinite loop? Things that support that theory are that it happens at what seems random times. It's most often several days, but not a specific number of days, between occurances but in one case just a few hours. It happens on both the master and the slave server, never at once but randomly. This could be explained by that we use the failover "ReadFromAllServers".

[Oren Eini (Ayende Rahien)]

Assuming that this is the case, this should show up in the Debug Package Info, which capture stack traces, or with a minidump, for the same reason

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Ian Cross]

This seems to be very similar to the behaviour I am seeing. Was there anything we can learn from the experience here please?

We have also setup procdump this morning to try and track it down. Did anything come from this thread?

Cheers,

Ian

[Oren Eini (Ayende Rahien)]

Note that I recall, no

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--