Datasets failed to Load -- default datasets.xml

[David Hiltz - NOAA Federal]

Hello --

   I'm trying to get an initial setup of ERRDAP working with the default datasets.xml file.

  ERRDAP: 2.02

  OS: CentOS 7.8

  Tomcat: 8.5.49

The status page shows:

   Here is errdap part of the catalina.out file:

////**** ERD Low Level Startup
localTime=2020-09-25T14:05:24-04:00
erddapVersion=2.02
Java 1.8.0_262 (64 bit, Oracle Corporation) on Linux (3.10.0-1127.10.1.el7.x86_64).
MemoryInUse=    15 MB (highWaterMark=    15 MB) (Xmx ~= 910 MB)
logLevel=info: verbose=true reallyVerbose=false
ERROR in File2.deleteIfOld: dir=/erddap/dataset/_FileVisitor/ isn't a directory.
bigParentDirectory=/erddap/
contextDirectory=/opt/tomcat/webapps/erddap/
java.awt.HeadlessException
 at gov.noaa.pfel.coastwatch.sgt.SgtUtil.isBufferedImageAccelerated(SgtUtil.java:368)
 at gov.noaa.pfel.erddap.util.EDStatic.<clinit>(EDStatic.java:1574)
 at gov.noaa.pfel.erddap.Erddap.<init>(Erddap.java:214)

Any ideas or things I can check would be appreciated.   thanks, David Hiltz

[Bob Simons]

If there are problems with ERDDAP starting up (e.g., it won't even return a web page), then the error messages are usually in the Tomcat/catalina log files.

If ERDDAP is running (showing web pages), then the error messages almost always in the /bigParentDirectory/logs/log.txt file.

As I said in a separate email:

My ERDDAP is getting a huge number of requests from your ERDDAP.

It is clear (because your ERDDAP shows 0 datasets) that there is a configuration error causing this.

So I blacklisted your IP address temporarily.

Please temporarily change your datasets.xml so that it only has 1 or 2 active EDDGridFromErddap or EDDTableFromErddap datasets until the problem is resolved (set active="false").

Let me know when you've done that and I'll unblacklist you.

After that is done, to resolve the problem with loading these EDDGridFromErddap or EDDTableFromErddap datasets:

go to your ERDDAP's   https://yourDomain/erddap/status.html (which flushes all pending messages to the log.txt file)

then look in the log.txt file to see the error message related to loading one of the datasets.

If you need help understanding the error message, please send me (or this group) a complete error message (including stack trace) and the relevant diagnostic information right above it, from the log.txt file.

Or, you can use the DasDds tool to diagnose almost all dataset configuration problems faster and easier because everything is in one place and the attempts take much less time.

https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#DasDds

I hope that helps.

[Roy Mendelssohn - NOAA Federal]

Hi David:

Adding on to what Bob said, just a guess. I assume that you have checked that the directory shown exists (/erddap). I also assume you are running tomcat as a user tomcat? If so, you have to make certain that the tomcat user has write access to the directory /erddap. Also, depending on your OS, and if you have SELinux running, you may need to do a little bit more on permissions to allow the writes, you can look at the SELinux settings with "ls -alZ". SELinux can block writes across different parts of the system.

HTH,

-Roy

> On Sep 25, 2020, at 11:37 AM, 'Bob Simons' via ERDDAP <erd...@googlegroups.com> wrote:
>
> If there are problems with ERDDAP starting up (e.g., it won't even return a web page), then the error messages are usually in the Tomcat/catalina log files.
> If ERDDAP is running (showing web pages), then the error messages almost always in the /bigParentDirectory/logs/log.txt file.
>
> As I said in a separate email:
> My ERDDAP is getting a huge number of requests from your ERDDAP.
> It is clear (because your ERDDAP shows 0 datasets) that there is a configuration error causing this.
> So I blacklisted your IP address temporarily.
> Please temporarily change your datasets.xml so that it only has 1 or 2 active EDDGridFromErddap or EDDTableFromErddap datasets until the problem is resolved (set active="false").
> Let me know when you've done that and I'll unblacklist you.
>
> After that is done, to resolve the problem with loading these EDDGridFromErddap or EDDTableFromErddap datasets:
> go to your ERDDAP's https://yourDomain/erddap/status.html (which flushes all pending messages to the log.txt file)
> then look in the log.txt file to see the error message related to loading one of the datasets.
>
> If you need help understanding the error message, please send me (or this group) a complete error message (including stack trace) and the relevant diagnostic information right above it, from the log.txt file.
>
> Or, you can use the DasDds tool to diagnose almost all dataset configuration problems faster and easier because everything is in one place and the attempts take much less time.
> https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#DasDds
>
> I hope that helps.
>
> On Friday, September 25, 2020 at 11:21:08 AM UTC-7 David Hiltz - NOAA Federal wrote:
> Hello --
> I'm trying to get an initial setup of ERRDAP working with the default datasets.xml file.
>
> ERRDAP: 2.02
> OS: CentOS 7.8
> Tomcat: 8.5.49
>
> The status page shows:
>
>
>

> Here is errdap part of the catalina.out file:
>
> ////**** ERD Low Level Startup
> localTime=2020-09-25T14:05:24-04:00
> erddapVersion=2.02
> Java 1.8.0_262 (64 bit, Oracle Corporation) on Linux (3.10.0-1127.10.1.el7.x86_64).
> MemoryInUse= 15 MB (highWaterMark= 15 MB) (Xmx ~= 910 MB)
> logLevel=info: verbose=true reallyVerbose=false
> ERROR in File2.deleteIfOld: dir=/erddap/dataset/_FileVisitor/ isn't a directory.
> bigParentDirectory=/erddap/
> contextDirectory=/opt/tomcat/webapps/erddap/
> java.awt.HeadlessException
> at gov.noaa.pfel.coastwatch.sgt.SgtUtil.isBufferedImageAccelerated(SgtUtil.java:368)
> at gov.noaa.pfel.erddap.util.EDStatic.<clinit>(EDStatic.java:1574)
> at gov.noaa.pfel.erddap.Erddap.<init>(Erddap.java:214)
>
>
> Any ideas or things I can check would be appreciated. thanks, David Hiltz
>
>

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/793a1e09-1830-46ff-b4e0-9738bc6b4f26n%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[David Hiltz]

    Directory /opt/tomcat/content/erddap exists.

    Yes, tomcat process is running as tomcat.

    tomcat owns the "/opt/tomcat/content/erddap" directory and the
files inside of it.

    SeLinux is disabled.

    I didn't have a /opt/tomcat/content/erddap/logs directory or the
log.txt file below so I created both and made tomcat the owner of both.

    Bob:   I remove many of the datasetID's in datasets.xml and
restarted tomcat.   Sorry, I figured the default datasets.xml file would
be ok to use.

###&@&###
David Hiltz
Server Administrator
NOAA/NMFS/NEFSC
166 Water St
Woods Hole, MA 02543
508-495-2218

[Roy Mendelssohn - NOAA Federal]

Hi David:

That is not the directory. In your setup.xml you defined a "BigDirectory". based on the Java error message, it appears as if it is set at the directory /erddap, but I could be wrong. So you need to do the following:

Go into the file setup.xml and find the line:

<bigParentDirectory>/</bigParentDirectory>

Make sure that that directory exists, and that that directory can be written to be the user tomcat. And if you will be having a pretty big ERDDAP, the term "bigParentDirectory", it may require a lot of space. The error message in catalina.out is that it is trying to write to a file in /erddap/. Also, in your $TOMCAT_HOME have you defined a setenv.sh file and made it executable?

Thanks,

-roy

[Bob Simons]

As Roy points out, ERDDAP will create the logs directory and other directories in the bigParentDirectory (perhaps /erddap/, perhaps /home/erddap/, whatever, its your choice, but not "/") as specified by <bigParentDirectory> in setup.xml. That directory should not be inside the tomcat directory.

It is essential that the user running tomcat have write privileges for bigParentDirectory and all subdirectories and files. You can fix problems or ensure this is true with (as su)

  chown -R tomcatUser bigParentDirectory

Let ERDDAP create the logs directory and log.txt file. That is proof that it can do that.

Yes, normally, you can use the datasets.xml that comes with ERDDAP. However, you're having problems. So I'm only asking you to make changes until the problems are fixed.

I have removed your IP address from my ERDDAP's blacklist. 

So, now, once all the things above are done and understood, try running ERDDAP.

Visit status.html to check ERDDAP's progress on loading the datasets.

When it finishes the initial attempt to load the datasets (see the messages at the top of status.html saying it finished),

if there are errors, then shut down tomcat (for now, while we're diagnosing this startup problem)

and look in bigParentDirectory/logs/log.txt for error messages.

If the logs directory or log.txt file doesn't exist, look in the catalina logs -- there will be an error message there.

[David Hiltz]

  I figured out my problem.    Thanks to you Bob and Roy for the ideas.   They helped point me in the right direction.

  my <bigParentDirectory> was set to "/erddap" because as the directions suggested, pick a directory outside of the tomcat directory tree.   My problem was the permissions on some of the sub-directories under /erddap.     Our datasets that query our Oracle database are also working - I just had to get the right "ojdbc8.jar' file.

   I plan to write a script to check several of the setting for erddap so I can quickly see if something is not right (checks like: does the <bigParentDirectory> exist and are all the files/directories inside writable by tomcat).    I also need a check for the version of java defined by JAVA_HOME in the "setenv.sh" file.  It many times doesn't get updated when Java on the server is updated.

   Thanks again for your help.

###&@&###
David Hiltz
Server Administrator
NOAA/NMFS/NEFSC
166 Water St
Woods Hole, MA 02543
508-495-2218

To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/bdb04437-7565-4137-b599-40d36faa6766n%40googlegroups.com.