CoralNet site is impossibly slow: does not work

[Virginia Sofia Campanella]

Hi, I'm experiencing extremely slow performances. Since two days ago (April 14th 2025), from the home page to anything else, nothing is loading so it's impossible to work with the website (like to open the home page it takes 2h of loading, I'm not joking). 

Is someone experiencing the same? I have deadlines and I really need this to work. 

Thank you! 

[Antonie Kooymans]

Hi Virginia,

I am experiencing the same issue! Hopefully it will be resolved soon.

Antonie

Op woensdag 16 april 2025 om 18:04:44 UTC+2 schreef Virginia Sofia Campanella:

[Stephen Chan]

Hi Virginia, Antonie,

I need some more time to fix this, but the slowness seems to be limited to certain pages, such as the homepage. If you can, try to access your source directly by typing the URL into your address bar to avoid the homepage. In the meantime, please let me know if there are other pages that cannot seem to load.

[Stephen Chan]

I should note that other pages are also somewhat slower due to the extra site load from folks accessing the homepage, so the slowness is not "limited" to the homepage, but from my observation so far it seems like it's centered on the homepage.

[Anika Kowald]

Hi all, I'm experiencing the same issue. It's impossibly slow or simply not letting me in. I can go anywhere except the home page and my projects... which is kind of vital. Anyone have anyway around this (its been this way for about 3 days now)

[Stephen Chan]

It appears that the site's been back to its usual levels of responsiveness in the past 18 or so hours.

After investigating the site logs, I believe I was wrong about the slowness being limited to certain pages. I just hadn't tested enough before I made that assessment.

Also, it looks like the slowness was caused by several extremely long page loads happening at once, mainly browse patches and browse images in some of the site's larger sources. This situation used to mostly happen with annotation exports, until I improved performance of those exports sometime last year. So now the page-loads bottleneck has moved, and with a bit of bad luck that bottleneck was exposed in the past few days.

I still want to analyze the site logs from another angle to double check that this new assessment is correct, but assuming it is, I'll get to work on optimizing browse patches and browse images. I've had a few ideas for this for a while now, but never got around to them because they didn't seem like as big of a priority before.

I'll also double check if there are any sources that are of real outlier size and see if anything needs to be done about that, such as splitting up the source. Note that I've never actually proceeded with such an action before on CoralNet, and I'd certainly contact any source admins for a discussion before proceeding to do so, but just giving a heads up that it might be necessary in some cases.

[Jordan Pierce - NOAA Affiliate]

Hey Stephen, just curious, do you know of many people actually using the patches? I ask because there are potentially a lot of patches for a project, but does anyone need them?

[Stephen Chan]

Hi Jordan,

Patch images are only generated when someone heads over to Browse Patches (or the label pages) to actually view them. Only the 20 patches on the page you're viewing actually get generated, and the patch images are generated in their own processes rather than as part of the Browse page load. So, there shouldn't be an issue with generated patches going unused. I suspect that the page-load slowness is more about determining which annotations will have their patches shown on the page (this would be a database query, or maybe multiple queries).

Also, I know that both Browse Images and Browse Patches can get heavy when a particular metadata field has many possible values, leading to a very long dropdown box in the search form. I'm thinking of addressing that by changing it to a free text field when there are enough possible values (e.g. more than 30).

[Ewout Knoester]

Hi Stephen,

The last few weeks the site has also been slow for me, is this specifically for our project (REEFolution Kenya) or still site wide? The loading for any page type typically takes 15-20 minutes, and is successful half of the time, the other times we get the error '502 Bad Gateway - nginx'.

Best,

Ewout

[Stephen Chan]

Hi Ewout,

Looking at the server logs, I'm seeing that most of the site requests were very slow for about an hour, close to the time of your message here. Indeed there were about ten 502 Bad Gateway errors from that period; I think that happens when there are too many site requests queued up without being able to be served, so it's basically another symptom of the site being overloaded.

In the past six days, there were 3-4 other slowness periods of 1-2 hours each. And, one particularly bad stretch of about 8 hours between May 15th-16th. The week before that seemed to be relatively smooth.

I don't have a more automatic way to record slowness periods yet, and these server logs only go back about 2 weeks, so I'm not really sure how it was going from late April to May 8th. If you have more specific information about when it was slow for you, please let me know.

It still looks like the main causes of the slowness are the same, so I will keep working towards the fixes I've mentioned earlier in this thread. I unfortunately got somewhat sidetracked in my recent work, but am hopefully re-focusing better now.

[Stephen Chan]

Browse-induced slowness seems to be in full force at the moment. I've recently pushed some changes, but there is more work to be done.