Current downtime

[Stephen Chan]

Hi all,

I was running some large maintenance operations on the CoralNet server this weekend, but now as I am attempting to get the server back up, it seems to keep getting overloaded by something. I already tried a few things, so now I'm just leaving it alone for a while in hopes that the load will just clear up, hopefully by the time the weekend is over. I will keep looking into it when I have the chance as well. Sorry for the inconvenience.

[Stephen Chan]

Seems like it needed another 4 hours to resolve the situation, but has since been stable.

[Stephen Chan]

Sorry, it's happening again. I already told the database to roll back the operation I was attempting, but I guess it just takes a while to get back on track. To prevent confusion and further overloading, I won't restart the server until the performance monitor shows the database has stabilized. I will study the behavior more closely so that it goes more smoothly next weekend.

[Stephen Chan]

And now we're back. At least I learned a way to get the database back on track faster.

[Zach Ferris]

Thank you, Stephen, for the maintenance updates. Please note that the site appears to still be down (error: "504 Gateway Time-out; nginx").

[Stephen Chan]

Indeed it is still down, I should have kept a closer eye on it. I'll see what I can do this time.

[Stephen Chan]

Okay, so there was a long database operation I started running last weekend; I had decided to abort it, clean up, and try again next weekend. But I think something about one of the database indexes (used for boosting performance) was not entirely cleaned up, because now any site operation which would use that index (mainly annotation exports) takes way too long and brings the infrastructure to its knees. So now I'm trying to re-run the aforementioned operation to completion, hopefully fixing the problem. My estimate for that operation is 5-6 hours, and then I'll have to check if that was actually the fix.

Apologies for having this happen during the weekdays, this was the result of a few missteps I made during some maintenance that turned out to be rather complex.

[Stephen Chan]

The operation only took around 3 hours, but I then had to make an additional fix. After that, things have been looking more stable. Hopefully that keeps up.

[Zach Ferris]

Thank you for the additional updates and fixes. I was able to log in briefly this morning and view most pages, but I could not access the annotation tool. Now, the website appears to be down again, as the login page is no longer appearing.

[Stephen Chan]

Looks like it's been calmer the past 3 to 3.5 hours, but indeed the performance was very bad for a long stretch before that. I'll keep an eye on it and see what else I can do to help.

[Nikita Jukenti]

Hello Stephen, 

Thank you for fixing the issues with the server, appreciate it. However, it looks like classifiers are not annotating images for my sources. 

Coral Regards, 

Nikita Jukenti,

Marine Biologist

--
You received this message because you are subscribed to the Google Groups "CoralNet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coralnet-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/coralnet-users/a5470ad8-2fed-4726-a9ae-597b6f5f3a2an%40googlegroups.com.

[Stephen Chan]

Thanks for the notice, Nikita - there was indeed a process that was still stuck. It should be un-stuck and catching up on the backlog now.