Songkick API downtime 06:22-06:44

[Graham Bleach]

Hi all,

Between about 06:22 and 06:44  UTC this morning, that is 2013-03-29T06:22:00Z and 2013-03-29T06:44:00Z there was an outage to the Songkick API.

This was caused by a couple of factors coinciding:

1) A recent change to the syslog configuration on our API servers, which filled the local filesystems on these machines.

2) A failure in our monitoring, due to an error in the way we were detecting filesystems that were about to fill. This didn't tell us that the filesystems were filling until it was too late to avoid an outage.

The problems described in 1) have already been addressed and problem 2) will be resolved in the next few days, by returning to a previous, better understood filesystem monitoring method.

We'd like to say sorry for any related disruption your applications suffered as a result.

Regards,

Graham

--
Head of TechOps, Songkick.com