Status Alert: Investigating Slowness, 2014-09-02

[Justin Goff]

Hello!

(You may have already seen this via our Status Alert mailing list, but just in case I wanted to post an update here.)

We are currently investigating a database issue this morning that has been causing slowness and 502 Errors throughout Haiku Learning. The slowness would have been noticeable starting around 7:50 a.m. PDT.

We are working to resolve the issue as soon as possible. Further updates will be posted to Twitter and to status.haikulearning.com.

Thank you for your patience.

Best regards,

Justin

[Justin Goff]

Hello, again!

Looks like the situation is resolved! Here are the details from our Status Alert mailing list:

The Haiku Learning team would like to report that we’ve resolved this morning’s incident and that Haiku Learning is functioning normally at this time.

At 7:52 a.m. PDT, a database replication delay issue with one of our database servers caused 6 minutes of downtime. Around 8:00 a.m. we removed the server suffering from delay out of production. Around 8:32 a.m., as part of an attempt to deal with the underlying cause of the replication delay, we executed a statement that overwhelmed our database server. This resulted in 9 more minutes of downtime. Haiku Learning has recovered and performance is now normal.

We are continuing to monitor the recovery and are taking optimization steps to remedy this issue long-term.

Thank you again for your patience and your understanding.

Best regards,

Justin