Error: No space left on device

[Quinton Heney]

Could you help with this error (causes model training to fail after a few seconds for an image model):

Error: wbuffer_write: write failed unix error: No space left on device vips2png: unable to write to target target

Only some of the students are seeing this error when training their models. Is it an issue with one of the image files in the training "buckets"?

[Dale Lane]

Sorry about that. 

That was an error on my servers - the error referred to a failure on my side when my code tried to resize their training image(s) to be the right size for training a model.

I can see from the logs on my side that two of the three instances of the ML4K servers had this problem - which is why some of your students will have been okay. They will have used the third server which was okay. (Requests are randomly distributed across all three instances)).

I do have some automation that looks out for this sort of problem, and from the logs on my side I can see that it was triggered 18 minutes ago - that restarted the two servers that had the problem. This will have resolved the problem at that time.

But I'm sorry for the inconvenience it caused your students in the meantime. 

I'll look into what I can do to prevent that happening again in future.

Kind regards

D