Site performance issues

[Travis Huinker]

Greetings,

We've recently been running into performance issues where we keep getting spikes resulting in extremely long load times or the Oops error pages. When investigating through New Relic it points to the database element of "MySQL entity select" being the cause of the long loads. Here are the queries it says are the slow ones:

New Relic reports the same occurring error messages of:

• Out of resources when opening file './reason/entity.MYD' (Errcode: 24 - Too many open files)
  
• Out of resources when opening file './reason/meta.MYD' (Errcode: 24 - Too many open files)
  
• Out of resources when opening file './reason/relationship.MYD' (Errcode: 24 - Too many open files)
  
• Out of resources when opening file './reason/page_node.MYD' (Errcode: 24 - Too many open files)
  

We haven't done any major changes to the code recently so we're all a bit perplexed as to the recent and ongoing performance issues. Server wise, nothing is being overloaded in terms of hardware. However, our server admins report "apachectl, showing MANY waiting connections. Not sure on the exact nature of the wait, but it seems to have implications. Also, take note of the KeepAlive stuff too.."

 Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process
'www-browser -dump http://localhost:80/server-status' failed.
Maybe you need to install a package providing www-browser or you
need to adjust the APACHE_LYNX variable in /etc/apache2/envvars

Throwing this out as maybe someone experienced similar issues or might possibly have some directions to point us to in solving these ongoing issues. Here is an example of the spikes we're experiencing:

Any input would be most appreciated, thanks!

[Tom Brice]

It sounds like you are hitting the limit of open files in mysql. I found this that matches your error in New Relic:

https://stackoverflow.com/a/31645955/8179

If that is it, the question is why. Is someone running a bulk export of some nature? Is there another database hosted in the same mysql that is experiencing unusual load or traffic?

Hope that helps,

Tom

Tom Brice
Senior Web Application Developer
Carleton College | (507)222-7175

--
You received this message because you are subscribed to the Google Groups "Reason Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.
Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

[Brian Jones]

Bingbot and msnbot have been hitting the Luther site with start dates far enough in the past to cause a "PHP Fatal error:  Allowed memory size of 67108864 bytes exhausted" error. (By the way https://apps.carleton.edu/calendar/?start_date=2010-01-27&textonly=1 returns a 500 error, so the problem is perhaps an events core issue). These queries can take a while before they error out. Any speculation that this could be part of the problem?

Thanks.

[Tom Brice]

Brian,

We see that error come through sometimes but not enough to take down the server. Seems like bots throttle themselves enough so that they are not pounding a server.

What is the setting for open_files_limit  in you mysql configuration? The Error 24 you are seeing is due to exceeding that limit.

As to the source of the other errors, I don't know. Maybe someone else has an idea.

Tom

Tom Brice
Senior Web Application Developer
Carleton College | (507)222-7175

--