can't connect to the server.... again

[Meg M]

would like this fixed, or at least an explanation as to why it happens so often.  I do appreciate this service, when it works...

[Francis Y. Yan]

My apologies for the inconvenience. The most recent three "server" issues were caused by the continuous retraining of our own research algorithm Fugu -- Fugu is a learning-based ABR algorithm and it retrains five neural network models every day. For some reason (probably memory overutilization?), it failed to save one of the five models very occasionally, which prevented the retraining on the next day and brought down the entire system. I don't currently have a good way to reliably reproduce the symptom, so unfortunately it might happen again in the future.

In general, Puffer runs many research algorithms behind the scenes without thoroughly profiling them; this greatly reduces the turnaround time compared with the best industrial practice, at the cost of reliability and availability.

By the way, prior to this, the downtime issues were often caused by something else -- the file receiver process that we run on the Puffer server to receive encoded media chunks from other servers could become CPU-bound and fail to catch up with the three senders sometimes. I fixed that issue by introducing two additional file receiver processes, which seems to be working a lot better.

Best,
Francis

On Mon, Nov 1, 2021 at 7:08 AM Meg M <megm...@gmail.com> wrote:

would like this fixed, or at least an explanation as to why it happens so often.  I do appreciate this service, when it works...

--
You received this message because you are subscribed to the Google Groups "Puffer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puffer-stanfo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puffer-stanford/950ae9ca-703b-4469-af04-eefdbbb69f74n%40googlegroups.com.