Rhodecode server stopped responding

[gbr]

Up until very recently, Rhodecode was running ine. However, without any obvious changes, it stopped responding to requests and just hangs. I have to ssh in to the server and restart the service to get it back up and running again. I checked the logs, but nothing obvious was present. These are the last items written when I encountered this behaviour:

2013-09-02 14:16:14.460 INFO  [rhodecode.lib.base] IP: 111.111.111.111 User: <AuthUser('id:None[None] ip:111.111.111.111 auth:False')> accessed /favicon.ico

2013-09-02 14:16:14.539 INFO  [rhodecode.RequestWrapper] IP: 111.111.111.111 Request to /favicon.ico time: 0.221s

2013-09-02 14:16:14.729 INFO  [rhodecode.RequestWrapper] IP: 111.111.111.111 Request to /favicon.ico time: 0.002s

2013-09-02 14:16:14.779 INFO  [rhodecode.lib.base] IP: 111.111.111.111 User: <AuthUser('id:None[None] ip:111.111.111.111 auth:False')> accessed /favicon.ico

2013-09-02 14:16:14.832 INFO  [rhodecode.RequestWrapper] IP: 111.111.111.111 Request to /favicon.ico time: 0.103s

2013-09-02 14:16:29.988 INFO  [rhodecode.lib.base] IP: 111.111.111.111 User: <AuthUser('id:None[None] ip:111.111.111.111 auth:False')> accessed /

2013-09-02 14:16:29.998 WARNI [rhodecode.lib.auth] user <AuthUser('id:None[None] ip:111.111.111.111 auth:False')> NOT authenticated on func: HomeController:index

2013-09-02 14:16:30.000 INFO  [rhodecode.RequestWrapper] IP: 111.111.111.111 Request to / time: 0.061s

This was pretty much the last thing that the server did...

The server is running as www-data user and the repos is owned by this user (I don't think it's a permission issue as the same happened as root).

Any idea what could be wrong?

[Marcin Kuzminski]

Hi,

Two things that come to my mind, server runned out of memory, or some git subprocesses failed. Next time it happens search the process list for some git processes running in the background.

Marcin Kuzminski

--
You received this message because you are subscribed to the Google Groups "rhodecode" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhodecode+...@googlegroups.com.
To post to this group, send email to rhod...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[gbr]

Thanks for the swift response.

Git can't be the problem, as the server hasn't gotten git installed (it's using mercurial). Memory should also be sufficient. Just to make sure, I disabled a few processes that use up a lot of memory to see if this problem occurs again.

Any other idea or could this also be a problem with mercurial?

[Marcin Kuzminski]

Ok so if we can remove git from that equation it's easier.

Only thing that comes to my mind is someone trying to do large remote clone/or there was a connection issues on that clone.

Marcin Kuzminski

[gbr]

I hadn't done any clones, just a few push/pulls and browsing the site (mainly journal and changeset section). A Jenkins server is also running on the machine, pulling upon code changes.

It's been running ok since I freed up some more memory (about 12 hours now). If this turns out to be the problem, can I allocate a defined amount of memory to the process (there was enough memory when I checked (swap+ram) after rhodecode had crashed, but maybe the memory requirements peaked at some stage)?

Any other suggestions are more than welcome...

[gbr]

Another thing I noticed when the server becomes inresponsive is that the CPU usage goes through the roof (only for 1 core though).