Sliders not working on griddap graph/data pages

[Chris Woelkers - NOAA Federal]

The ERDDAP server I admin has had an odd issue since the upgrade to 2.25.1, and continues to be an issues in 2.30. The griddap graph and data pages usually have sliders to make it easy to dial in the needed data. However since the upgrade the sliders do not work. The slider is visually there but it is not on the "track", appears as a static image, and cannot be moved. The issue has been seen on Firefox and Chrome.

The server is running Tomcat 10.1.54 on top of Temurin Java 25. I would provide configs but I'm not certain which ones may be relevant here so will do so when requested. 

Here is an one page among many showing the issue: https://apps.glerl.noaa.gov/erddap/griddap/LS_fvcom_h_zeta.graph

[Roy Mendelssohn - NOAA Federal]

HI Chris:

Ouch! Anything to ruin my weekend, yes? Usually but not always this has to do with how things are being proxied from apache2 (or nginx) to the tomcat. I would need to know the relevant portions how that has been done, suitably sanitized, Also, I would think somewhere in the logs an error is being thrown, either in the apache logs, the tomcat logs, or the ERDDAP™ logs. Also can you double check that you have installed whatever fonts you declare are being used in your setup.xml? Java used to contain the basic font in the download but it no longer does and must be installed separately, I have to go back in my notes if that is the problem.

So let me know, sorry you are having problems.

-Roy

PS - Are you running a Docker image or just a plain ERDDAP™. If a Docker image have you updated to the latest image?

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/35bf46bb-9ebd-45fd-a0fe-3999ecbf04efn%40googlegroups.com.

[Roy Mendelssohn - NOAA Federal]

Hi Chris:

A quick follow-up to show my brain is at least partially working this morning - by knowing to ask AI to give me the answer since I had asked it previously.  If you are using the Deja Vu fonts,  they are not automatically installed,   Here is an answer thanks to Claude just in case this is your problem:

The problem: Java used to bundle the DejaVu fonts (via fontconfig integration), but modern headless JDK builds (especially on minimal cloud VMs like GCP's Ubuntu images) no longer pull in font packages as dependencies. ERDDAP's graphics generation hits missing fonts and either throws errors or produces ugly output.

The fix — two packages on Ubuntu/Debian:

bash

sudo apt update
sudo apt install fontconfig fonts-dejavu

fontconfig alone often works because it pulls in some fonts as a dependency, but being explicit about fonts-dejavu ensures the DejaVu family (DejaVu Sans, DejaVu Serif, DejaVu Sans Mono) is definitely present — those are what ERDDAP's graph rendering expects.

After installing, you can verify the fonts are visible to the Java font subsystem:

bash

fc-list | grep -i dejavu

You should see entries like /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf.

On RHEL/CentOS/Rocky:

bash

sudo dnf install fontconfig dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts

One extra step sometimes needed on headless servers — after font install, rebuild the font cache:

bash

sudo fc-cache -f -v

And if Tomcat was already running when you installed the fonts, restart Tomcat — the JVM caches available fonts at startup.

I apologize for the extra graphics,  it was easier to just cut and paste,  Well actually I should have edited all of this to make it look like I remembered all of this off the top of my head,  but nah!

Try the font fix first,  if that doesn’t do it then send me th proxy info and any errors you are getting.

HTH and thank Claude,

-Roy

[Chris Woelkers - NOAA Federal]

Hi Roy,

So first ERDDAP is being run on its own. Docker is not, nor ever will be if I have anything to say about it, involved. It even has its own instance of Tomcat to keep things clean.
Second I checked the fonts installed, via fc-list | grep -i dejavu, and it was listed along with multiple variants.

That leaves the proxy and it got more complex yesterday, though the issue has existed for much longer, so I'll share the older setup. As Apache is serving sites on this server I also have it performing the proxy role. Here are the lines from the relevant Apache config that have anything to do with rewrites and proxies.

<VirtualHost *:80>

    RewriteEngine On
    RewriteCond %{SERVER_PORT} 80
    RewriteRule ^(.*)$ https://apps.glerl.noaa.gov/$1 [R,L]

</VirtualHost>

<VirtualHost *:443>

    SSLProxyEngine On
    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass /erddap http://localhost:8081/erddap
    ProxyPassReverse /erddap http://localhost:8081/erddap
</VirtualHost>

This all worked before the issue popped up and nothing was changed until yesterday when I introduced Anubis to stop the AI bots from killing the site. That only changed the proxy port numbers to point to Anubis which then points directly to http://localhost:8081.

[Roy Mendelssohn - NOAA Federal]

What does the port configuration look like in server.xml. I don’t know anything about Anubis and who this might affect things, I will have to look. I would also suggest again going through the logs, my guess is something is throwing an error.

Let me know.

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/6dccdc2a-c9f3-4f74-955c-b7a51d2a14cdn%40googlegroups.com.

[Roy Mendelssohn - NOAA Federal]

Hi Chris:

Well you have gotten beyond where I know much,  but again I asked Claude about Anubis and whether it could be causing what you are seeing,  Below is what is says,  Give  these things a try and let me know.

-Roy

Yes, Anubis could absolutely be causing this, and there are a few very plausible mechanisms.

What Anubis does: It sits between users and the origin service as a reverse proxy, forwarding requests once challenge conditions are met, using a JavaScript-based proof-of-work challenge. That JS challenge is the key issue here. Help Net Security

Why ERDDAP sliders are likely breaking:

ERDDAP's "Data Access Form" pages (the ones with the sliders) are heavily JavaScript-dependent. When a user first hits the site, Anubis intercepts and serves its own challenge page. Once the challenge is solved it sets a cookie and proxies through to ERDDAP. The problem is likely one or more of these:

1. Partial page loading / response interception. Anubis sits in the middle of every HTTP transaction. It has to process all whole HTTP packets transmitted between the reverse proxy and the final app, which is less efficient than a reverse proxy and increases internal bandwidth usage and CPU usage. If ERDDAP's HTML page and its associated JavaScript/CSS resources are being served through Anubis in a way that disrupts the load order or timing, sliders that depend on JS initialization can fail silently. GitHub
2. Missing or mangled HTTP headers. ERDDAP generates absolute URLs in its HTML based on the Host and X-Forwarded-* headers it sees. If Anubis isn't passing through X-Forwarded-Host, X-Forwarded-Proto, or X-Forwarded-Portcorrectly, ERDDAP may generate broken resource URLs (pointing to the wrong port/scheme), causing JS and CSS files to 404 silently — and sliders that depend on those files break.
3. Cookie interference. Anubis sets its own challenge cookie. If there's any conflict with ERDDAP's own session cookies, or if Anubis is stripping/modifying cookie headers in transit, ERDDAP's stateful form elements can misbehave.
4. Content-Security-Policy headers. Anubis may be injecting its own security headers (CSP, etc.) that then block ERDDAP's inline JavaScript from executing — which ERDDAP uses extensively for slider initialization.

Things to check/try:

• Look at the browser's Developer Tools → Network tab when the broken page loads. Are the JS/CSS resources returning 200? Are any being blocked by CSP errors in the Console?
• Check what headers Anubis is forwarding. The user should confirm Anubis is passing X-Forwarded-For, X-Forwarded-Host, and X-Forwarded-Proto to ERDDAP.
• In ERDDAP's setup.xml, verify baseHttpsUrl and baseUrl match what users actually see in the browser (not the internal localhost:8081).
• Try accessing ERDDAP directly on localhost:8081 (bypassing Anubis entirely) to confirm it still works — that isolates whether Anubis is the culprit.
• Check Anubis's bot policy configuration to see if it has any rules that might be intercepting or modifying responses for certain content types.

The most likely culprit is missing X-Forwarded-* headers or Anubis injecting CSP headers that block ERDDAP's JavaScript. The network tab in the browser will tell the story quickly.

[Roy Mendelssohn - NOAA Federal]

It does look like a proxying problem, and I just thought of something else, given the timing of events. In your setup.xml what do you have for:

<useHeadersForUrl>

If it is not set, then now by default it is true. So one thing you can try is to explicitly set it to false, make certain how your baseURLS are defined in setup.xml, and see if the problem goes away. If you want keep it as true, you may have to set headers to make certain things are passed correctly, for example you might need to add something along the lines of:

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
RequestHeader set X-Forwarded-Host "coastwatch.pfeg.noaa.gov"

Of course replacing coastwatch.pfeg.noaa.gov <http://coastwatch.pfeg.noaa.gov/> with your info. Also, I have included in my :8080 port definition:

scheme="https"
secure="true"
proxyPort="443"
proxyName="coastwatch.pfeg.noaa.gov"
relaxedQueryChars='[]|'
redirectPort="8443" />

So try setting <useHeadersForUrl> to false and see if that solves the problem, and if not try modifying you proxying as above and see, also try making an offering to the god, goddesses, spirits or whatever you believe in to appease them, and let me know what you find. But I think the key is to look for errors in the logs and/or as in my last email track what ERDDAP™ sees and what is replies, it sure looks like ERDDAP™ is not sending the Javascript to where it should be going.

HTH,

-Roy

[Chris John - NOAA Affiliate]

At a quick look in the console on the page, I believe the root problem in the browser is the mixed content errors:

Mixed Content: The page at 'https://apps.glerl.noaa.gov/erddap/griddap/LS_fvcom_h_zeta.graph' was loaded over HTTPS, but requested an insecure stylesheet 'http://apps.glerl.noaa.gov/erddap/images/erddap2.css'. This request has been blocked; the content must be served over HTTPS.
Mixed Content: The page at 'https://apps.glerl.noaa.gov/erddap/griddap/LS_fvcom_h_zeta.graph' was loaded over HTTPS, but requested an insecure script 'http://apps.glerl.noaa.gov/erddap/images/wz_tooltip.js'. This request has been blocked; the content must be served over HTTPS.
Mixed Content: The page at 'https://apps.glerl.noaa.gov/erddap/griddap/LS_fvcom_h_zeta.graph' was loaded over HTTPS, but requested an insecure script 'http://apps.glerl.noaa.gov/erddap/images/wz_dragdrop.js'. This request has been blocked; the content must be served over HTTPS.

Essentially the css and javascript used to handle the drag/drop and tooltips is not being loaded.

This is likely due to proxy configuration, though it's possible it could be as simple as making sure the baseUrl and baseHttpsUrl are set properly in setup.xml (you might want to set them both to https://apps.glerl.noaa.gov).

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi

To view this discussion, visit https://groups.google.com/d/msgid/erddap/E9181994-CC89-4783-AAB7-56973BE242D9%40noaa.gov.

[Roy Mendelssohn - NOAA Federal]

Great thanks,, This would suggest adding:

> RequestHeader set X-Forwarded-Proto "https"

and to be doubly certain add what I suggested to the port definition in server.xml.

-Roy

[Chris Woelkers - NOAA Federal]

Here are the connectors from server.xml.

<Connector port="8081" protocol="HTTP/1.1"
           connectionTimeout="300000"
           relaxedQueryChars="[]|"
           redirectPort="8444" />
    <Connector port="8444" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true" connectionTimeout="300000" relaxedQueryChars="[]|" >
        <SSLHostConfig protocols="TLSv1.2+TLSv1.3">
                <Certificate certificateKeyFile="/etc/letsencrypt/live/apps.glerl.noaa.gov/privkey.pem"
                        certificateFile="/etc/letsencrypt/live/apps.glerl.noaa.gov/fullchain.pem"
                    type="RSA" />
        </SSLHostConfig>
    </Connector>

I know that the letsencrypt certs are not working as port 8444 is not open. I'm thinking of removing the redirectPort from the 8081 and disabling the SSL connector to let Apache handle the SSL portion.

The <useHeadersForUrl> setting has not been set. I'm going to set it to False before playing with the connectors to see how that works.

In setup.xml both baseUrl and baseHttpsUrl are set to https://apps.glerl.noaa.gov.

As for the logs there are no error like entries in any related log. I checked the tomcat journald log, the tomcat localhost_access log, and the erddap log. The only entries were in the localhost_access log.
As for Anubis that was added yesterday and the error was there before then. It may be clouding the issue but should not be the cause. Especially as my IP is whitelisted by it and I still see the issue. That said here are responses to the various things to try brought up by Claude.

[Chris Woelkers - NOAA Federal]

And after seeing Roy's latest message I've modified the RequestHeader to be https instead of http as it was initially.

[Roy Mendelssohn - NOAA Federal]

In similar situations adding:

> RequestHeader set X-Forwarded-Proto "https"

has fixed the problem, and what I suggested adding to the port definition sort of makes certain that tomcat views it as https, But we have at least isolated exactly where the problem lies.

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/7a55cf8a-48bc-4f1a-a753-468ab1ad289an%40googlegroups.com.

[Chris Woelkers - NOAA Federal]

And now it all works.

It looks like adding <useHeadersForUrl>false</useHeadersForUrl> to server.xml and setting RequestHeader set X-Forwarded-Proto "https" in the Apache config did it. Though it's possible that just the first would have worked the second hasn't broken anything so I'll leave it be for now.

Thanks all for the help.

[Roy Mendelssohn - NOAA Federal]

And we have Apache handle the certs, it appears to keep things cleaner in the proxying. If you want to keep the certs in tomcat, you can proxy to :8444 from Apache using http, there are some things you put in the port definition to make that work, I will have to look that up if you want to try that.

Let us know what you find out.

-Roy

[Roy Mendelssohn - NOAA Federal]

Yay great. This is helpful for us also, it is how we learn to fix issues as they arise, I learn a lot through these things, though it often strains my brain. Donations to my coffee fund or retirement always accepted (actually i should be careful saying these things as I am sure NOAA has Gemini scanning all of our emails for whatever and this is all in jest). I assume you also made the requisite offering which I am certain helped.

One thing is if you don’t need <useHeadersForUrl> don’t worry about it. It is useful for certain kinds of proxies, it can also be helpful in testing so that the “final” setup can be used in the test environment.

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/88ec7163-49df-46e9-b419-fdea2302ede5n%40googlegroups.com.