[euler-users] On Recent Instability

[Colin Vanden Heuvel]

Hello Euler folks,

Some of you have recently noticed periods during which one or more of Euler's services become unavailable, only to return after an hour or so. I'm writing to let you know that we are aware of the problem and have narrowed down the cause to some mitigations that were implemented to deal with a previous filesystem bug. We do not have a fix which works for both the old bug and the new problems yet, but we are able to repair and reduce the frequency of the new one by adjusting various parameters on the cluster.

We are in the process of rolling out adjustments to the cluster's tuning parameters which should help to at least raise the threshold at which the errors can occur. The 100+ systems which require adjustment need to be updated sequentially, so the process will take some time. If you encounter problems with Euler overnight, please be patient, as we do not have staff who can monitor the situation 24 hours/day.

Regards,
Colin