website not responding, Raven.Server.exe running at almost 100% CPU usage constantly, management studio not loading

[Karl Cassar]

Hi,

For some reason, the Raven.Server.exe all of a sudden started taking up to 100% cpu usage constantly (for over 5 minutes).  The website accessing the server resulted in not loading, and even the management studio is not loading.

Any insights on how I can know what is causing this?  I switched off the website, and the CPU usage still remained at 100% for Raven.Server, with no apparent clients connected as the website is the sole client to RavenDB.

Regards,

Karl

[Oren Eini (Ayende Rahien)]

What build?
And can you get to the stats?
What does the perf counter says?

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kijana Woodard]

Replication turned on?

--

[Karl Cassar]

Oren: 

Build is #2261.  

What do you mean by "can you get to the stats?" and "what does the perf counter say?"

Kijana: 

No, replication is not turned on.

Regards,

Karl

[Oren Eini (Ayende Rahien)]

Can you try using a later version?

2261 is quite old, and 2360 should work better.

--

[Karl Cassar]

Ok will do, and see if it happens again.  Any thing I should know when upgrading from 2261 to 2360?

Regards,

Karl

[Oren Eini (Ayende Rahien)]

Not that pops to mind

--

[Karl Cassar]

Hi,

I had upgraded to the latest build last Friday, and today it happened again - website not responding (probably due to it blocking on database queries), and Raven.Server.exe running at 70-100% CPU usage indefinitely, and raven Studio unaccessible.  

Co-incidentally, this happened also on another website I have using RavenDB, which has a totally different code base.  Also, when I stop the website for both web=applications (who is the only client for RavenDB), the server still shows 70 - 100% CPU usage.  The ravenDB studio is unaccessible, and even if I go to services.msc and try to stop the service, it never stops - It displays the 'didn't respond in a timely manner' error message and the service is marked as 'Stopping...' indefinitely.   I then have to manually end the process, and restart it.  Up till now, whenever I restarted it this way, no corruption in the indexes or data was detected later at least.

Any insights on how I can start looking at the problem, and if it could be code-related or RavenDB? I don't see any case how a query could be allowed to block RavenDB indefinitely, even with the application stopped.  

Regards,  

Karl

[Oren Eini (Ayende Rahien)]

Can you create a dump of the ravendb server when this happens? That would allow us to analyze what is going on.

Better would also having the debug logs, so we can tell what it is doing.

--

[Karl Cassar]

Which is the best way to take a dump? I had already created one dump via the task manager, but it was of build 2261. Is that good for you to debug? Id have to wait until it happens again on the latest build, for a dump of build 2330.

Is it an issue regarding performance to switch on debug logging? I guess i would have to leave it on until it happens, right?

Regards,
Karl

You received this message because you are subscribed to a topic in the Google Groups "ravendb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/dR36TwZ5TLo/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Please send us the dump, yes.

And there is some perf hit, but not a major one.

[Karl Cassar]

Unfortunately somehow the dump I created some days ago had been deleted, so I'll have to wait until it happens again.  I have also switched on the debug logs, and will send them as well when this happens again.

Just to be sure - what is the best way to create the dump?  I normally do it from Task Manager > Right click on 'Raven.Server.exe' > Create Dump File.  Would this suffice?

Regards,

Karl

[Oren Eini (Ayende Rahien)]

Yes, that would work

--

[Karl Cassar]

Hi,

The dump file is quite big (210MB compressed), so I've attached it via WeTransfer - You can download it using just the browser by following this link: http://we.tl/ily0yomKP4

Debug log files can be downloaded as well via this link: http://we.tl/QKBVkmj1Ti.

The issue happened at approximately 21/05/2013 16:16 server time.  

I was looking at the log file, and I found a lot of entries like:

2013-05-21 17:24:39.1389,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'sectiondata/75' was found,

2013-05-21 17:24:39.4197,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'sectiondata/85' was found,

2013-05-21 17:24:39.4197,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'sectiondata/88' was found,

2013-05-21 17:24:39.4197,Raven.Storage.Esent.StorageActions.DocumentStorageActions,Debug,Document with key 'sectiondata/63' was found,

Does that mean 4 seperate queries, or just 1 which loaded them all?  Is there any way I can know the activity on the server?  Are the logs a good indication?

Regards,

Karl

[Oren Eini (Ayende Rahien)]

Probably a single query that does that.

Anything in the http side of thing that might be expensive?

[Karl Cassar]

In what sense 'http side of things'?  RavenDB server is hosted on same server as website, (localhost) - so HTTP calls shouldn't be expensive I guess?

Is there any way I can know the activity going on on the server?  I'd guess number of actual queries is what counts?

Regards,

Karl

[Kijana Woodard]

You should see something in the logs before those entries that looks like this:

2013-05-20 16:32:18.6634,Raven.Database.Indexing.Index.Querying,Debug,xyz,Issuing query on index ProductIndex for all documents,

[Kijana Woodard]

Not to mention lines like these:

2013-05-20 16:32:15.6947,Raven.Database.Server.HttpServer,Debug,,"Request #12,990: GET     - 1,507 ms - xyz    - 200 - /indexes/ProductIndex?query=&pageSize=60&sort=__random%3BgU3dX17P_kSS7CgNc4Rvww",

2013-05-20 16:32:15.6947,Raven.Database.Server.HttpServer,Debug,," Query: 

Time: 1,477 ms

Index: ProductIndex

Results: 60 returned out of 730,880 total.

",

[Karl Cassar]

Hi Kijana,

Yes, managed to find those before so it is a normal query I assume. How would one explain why the Raven.Server.exe process still continues running at 70 - 100% cpu usage, when the sole website application accessing RavenDB is switched off?  Also, currently it is using 1.2GB of Memory (Private Working Set) in task manager.  Trying to stop the service in the services.msc window never completes (Error 1053: The service did not respond to the start or control request in a timely fashion), and I have to forcefully stop it via 'End Process'. I've left the website switched off for 20 minutes, as at this time I can afford to do it.  I assumed maybe it had a 'backlog' of work to do, maybe by time CPU usage would go down.  However, it is still running at full CPU usage, and server performance is disastrous.

Is there any way I can actually know what the server is doing?  I have debug logs switched on, and no queries / updates to the log file are done.  I have also taken another crash dump after I have switched off the website, and on client is connected.  Would it provide any further insights?

Just in case, I've uploaded the dump file of Raven.Server.exe and log files - link to download:  http://bit.ly/10SILP0

Dump was taken at 22/05/2013 02:02, server-time. 

Regards,

[Karl Cassar]

Any updates on this, because it is happening very frequently lately, almost most of the time?

Any help would be greatly appreciated !

Regards,

Karl

[Oren Eini (Ayende Rahien)]

any way for us to check what is going on there live? Via skype or something like that?

--

[Karl Cassar]

Apologies for the late reply, but I was abroad for the past days.

The problem is that when this happens, the server slows down to a crawl and we cannot leave it like that for a lot of time as it has got other production websites running on it.  Will any log files help? Did you manage to take a look at the crash dump?  It has just happened again, and I've taken another crash dump just in case, if it is of any help.

Regards,

Karl Cassar, B. Sc. I.T. (Hons) (Melit.)
Technical Director

CasaSoft Ltd., 6, Spencer Flats, Flat 4, Triq Dun Gorg Preca, Hamrun, HMR 1605, Malta (Europe) |  View Map

Tel: (+356) 2166 2204 | Mob: (+356) 7979 2211 | Email: ka...@casasoft.com.mt | Web: www.casasoft.com.mt

Skype:  karl.casasoft |  |  | 

---

Be Earth-friendly, please do not print this e-mail unless there is truly a need. 
This email message (including attachments) contains information which may be confidential and/or legally privileged. Unless you are the intended recipient, you may not use, copy or disclose to anyone the message or any information contained in the message or from any attachments that were sent with this email, and If you have received this email message in error, please advise the sender by email, and delete the message. Unauthorised disclosure and/or use of information contained in this email may result in civil and criminal liability. 

--
You received this message because you are subscribed to a topic in the Google Groups "ravendb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/dR36TwZ5TLo/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Yes, the log file will help, as well as a crash dump

[Karl Cassar]

Hi,

How 'unstable' is the unstable 2.5 build?  I would like to give that a try before, to see if maybe it fixes the problem.  Or would you say I should better wait until the official stable release is finalised?  If you suggest I can try the unstable build, can I easily upgrade from v2 to v2.5, without any issues?

Regards,

Karl

[Kijana Woodard]

I can confirm that the upgrade is just copying the new binaries over the old. If it's complex to recreate your data  you should do this on a copy since you can't downgrade the data.

--

[Oren Eini (Ayende Rahien)]

We are going to go to code freeze next week

--