ERDDAP AWS

[Kim Hyde - NOAA Federal]

Hello again,

I am struggling to figure out exactly how to serve files that are stored on AWS S3.  The files are here.  The first part of the xml file and the dasdds output are below.  I'm guessing there is an issue with the prefix and key, but it isn't clear to me exactly how to input the key so that it finds multiple files.  I've tried several iterations for the cacheFromUrl variable and every time I get the error that key does not exist and no files were found. I've been doing my best to follow the ERDDAP AWS S3 Buckets documentation, but I must be missing something.  Any help or guidance would be greatly appreciated.  

Thanks!

Kim

  <dataset type="EDDGridFromNcFiles" datasetID="duPontavice_bottom_temp_aws" active="true">
    <reloadEveryNMinutes>10080</reloadEveryNMinutes>
    <updateEveryNMillis>10000</updateEveryNMillis>
    <fileDir>/home/erddap/data/bottomtemp_cache/</fileDir>
    <cacheFromUrl>https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/testdata/</cacheFromUrl>
    <cacheSizeGB>10</cacheSizeGB>
    <fileNameRegex>.*\.nc</fileNameRegex>
    <recursive>true</recursive>
    <pathRegex>.*</pathRegex>
    <metadataFrom>last</metadataFrom>
    <matchAxisNDigits>20</matchAxisNDigits>

    <fileTableInMemory>false</fileTableInMemory> 

*** DasDds duPontavice_bottom_temp_aws

*** deleting cached dataset info for datasetID=duPontavice_bottom_temp_aws

EDD.oneFromDatasetsXml(duPontavice_bottom_temp_aws)...

SimpleXmlReader encoding=

*** constructing EDDGridFromFiles(xmlReader)...

*** constructing EDDGridFromFiles duPontavice_bottom_temp_aws

dir/file table doesn't exist: /erddapData/dataset/ws/duPontavice_bottom_temp_aws/dirTable.nc

dir/file table doesn't exist: /erddapData/dataset/ws/duPontavice_bottom_temp_aws/fileTable.nc

creating new dirTable and fileTable (dirTable=null?true fileTable=null?true badFileMap=null?false)

WARNING: FileVisitorSubdir.visitFileFailed: /home/erddap/data/bottomtemp_cache

doQuickRestart=false

getUrlConnInputStream https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/tesdata/ requestCompression=true

  Warning: HTTP status code=404

ERROR from url=https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/tesdata/ :

java.io.IOException: HTTP status code=404 java.io.FileNotFoundException: https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/tesdata/

(<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>erddap/tesdata/</Key><RequestId>KA2CH0NBKBPBWN20</RequestId><HostId>imdmku2nWaClKSzmo+296IOVMKWJi+4LOKAhZfkNDoqeRf5hGWrwFsCmfnwa9o0mNGKwZz1ZgeYcba+F2jPc6U+53sLR34+7</HostId></Error>)

 at gov.noaa.pfel.coastwatch.util.SSR.getBufferedInputStream(SSR.java:1962)

 at gov.noaa.pfel.coastwatch.util.SSR.getUrlConnBufferedInputStream(SSR.java:1874)

 at gov.noaa.pfel.coastwatch.util.SSR.getUrlConnBufferedInputStream(SSR.java:1755)

 at gov.noaa.pfel.coastwatch.util.SSR.getBufferedUrlReader(SSR.java:2031)

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.addToWAFUrlList(FileVisitorDNLS.java:1282)

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStep(FileVisitorDNLS.java:621)

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStepCache(FileVisitorDNLS.java:671)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.getFileInfo(EDDGridFromFiles.java:1820)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:824)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:96)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFiles.<init>(EDDGridFromNcFiles.java:103)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:302)

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:445)

 at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:566)

 at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11187)

 at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:131)

 at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:157)

*** An error occurred while trying to load duPontavice_bottom_temp_aws:

java.lang.RuntimeException: datasets.xml error on or before line #1047: No matching files at https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/tesdata/

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:486)

 at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:566)

 at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11187)

 at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:131)

 at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:157)

Caused by: java.lang.Exception: No matching files at https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/tesdata/

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.getFileInfo(EDDGridFromFiles.java:1823)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:824)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:96)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFiles.<init>(EDDGridFromNcFiles.java:103)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:302)

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:445)

 ... 4 more

[Roy Mendelssohn - NOAA Federal]

Hi Kim:

I usually try to answer these questions, but I don't have much experience in setting up for AWS S3 buckets. I know there are people on this list who are using AWS and I hope they will respond, I will do some reading to see if I can figure out the problem (I do not have a test AWS environment I suppose I should set one up). , or hopefully Bob will see this and respond. But give me some time, we will get this worked out. Mainly responding to you now so you know that we are working on it.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/5608df48-afd6-493a-a689-144481f1e273n%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[Kim Hyde - NOAA Federal]

Thank you Roy,

We are slowly adding datasets to our ERDDAP, but right now we don't have a solution for serving larger gridded datasets.  Our ERDDAP server does not have the capacity to hold large datasets (it is old, but maybe someday we will get the funds for a new larger server) and we don't have a means for moving data from inside our firewall to the public server so our IT department is suggesting that we put the data on AWS and serve it from there.  I'm open to other suggestions if you have them.

Thanks,

Kim 

--

Kimberly J. W. Hyde (she/her)

Operations Research Analyst

Ecosystem Dynamics and Assessment Branch

Northeast Fisheries Science Center

NOAA Fisheries | U.S. Department of Commerce

Office: (401) 782-3226

www.fisheries.noaa.gov

[Roy Mendelssohn - NOAA Federal]

Serving on AWS is fine.  As Bob notes in the instructions:

Disclaimer: Working with AWS S3 buckets is a lot of extra work. AWS is a huge ecosystem of services and features. There's a lot to learn. It takes time and effort, but it is do-able. Be patient and you'll get things working

So let’s be patient and we can get this worked out,  which is the correct solution,  I just don’t have a lot of experience with this,  so I can’t quickly see what a problem might be.  As I said,  I know of other installations that are using AWS and hopefully they can help also.

-Roy

[Roy Mendelssohn - NOAA Federal]

Hi Kim:

Let’s take a step back,  and follow Bob’s instructions step by step.  So can you please try to do:

https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#AwsS3MakeAnEDDTableFromFileNamesDataset

and let’s see if we can get that to work  (and note most importantly about where to put your AWS credentials).  Once we get that,  we can go on to the next steps.

-Roy

[bobsimons2.00]

Thanks for providing lots of information, especially the error message.

I think the problem here is that you are not specifying the AWS URL in the format that ERDDAP requires. 

In between s3. and .amazonaws.com, you need to specify the AWS region where the data is stored (in the format that AWS likes).

See https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#AwsS3Buckets

So 

   <cacheFromUrl>https://noaa-nmfs-nefsc-pds.s3.amazonaws.com/erddap/testdata/</cacheFromUrl>
should be something like

   <cacheFromUrl>https://noaa-nmfs-nefsc-pds.s3.us-east-1.amazonaws.com/erddap/testdata/</cacheFromUrl>

If you don't use the format that ERDDAP requires, ERDDAP doesn't notice that this is an AWS URL (which it treats specially via the AWS Java SDK) and tries to open the URL as if it were an ordinary web accessible folder. Since it isn't, AWS responds with an error.

As for the bigger picture: is your server a local server (not on an AWS compute instance)? If so, it may technically work to access the data on AWS S3 but the data access will be painfully slow and thus not useful. If the server is in an AWS instance, access to S3 files is decently fast. But my experience with using ERDDAP on my PC and accessing S3 files was super slow. AWS promised NOAA free data egress, but they didn't specify the speed for those downloads. Sometimes, e.g. initially, it is reasonably fast, but usually it is (for me) painfully slow.  The much better approaches are:

1) run ERDDAP on an AWS compute instance to connect to the data on S3

2) run ERDDAP on your external server and attach additional storage to hold the data.  Hard drives are cheap now. Roy can give you some options for inexpensive enclosures to hold multiple drives. Even if the drive enclosure is connected via USB 3 (or better SATA or similar), access will be fast.

Best wishes.

[Roy Mendelssohn - NOAA Federal]

Hi Kim:

I just looked at your files, and they are all netcdf-3 file. One thing you can do for the nonce is either convert them to netcdf4 or gzip the files. For example I did the following:

nccopy -d4 bottom_temp_1959.nc test1.nc
gzip bottom_temp_1959.nc

Here is a comparison in size:

18M Oct 19 12:50 bottom_temp_1959.nc
4.5M Oct 19 12:44 bottom_temp_1959.nc.gz
4.5M Oct 19 12:45 test1.nc

ERDDAP can handle either of these options, and voila you all of a sudden have a lot more space. You can write a script to do all of the files, or if that is not something you do not know, I can help with that. Be warned that gzip as I show above will delete the original file. There is an option for it to retain the original file. If you do the nccopy option you likely would not need to change your datasets.xml, if you use the gzip option you would need to make a small modification to the regex for the files names.

HTH,

-Roy

[bobsimons2.00]

Roy is correct. Those are two good ways to save lots of disk space.

Note that gzip is better if you want to keep the original files (although in compressed form).

But .nc4 will perform better if you are using a local server and local files. 

gzip and nc4 will both perform well on an aws server and s3 storage.

[Kim Hyde - NOAA Federal]

Hello again,

The AWS credentials have been added, and I think they have been done correctly because I am getting a new error now.  Could you please help me figure out what I need to fix next?

Thanks,

Kim

EDD.oneFromDatasetsXml(duPontavice_bottom_temp_aws)...

SimpleXmlReader encoding=

*** constructing EDDGridFromFiles(xmlReader)...

*** constructing EDDGridFromFiles duPontavice_bottom_temp_aws

dir/file table doesn't exist: /erddapData/dataset/ws/duPontavice_bottom_temp_aws/dirTable.nc

dir/file table doesn't exist: /erddapData/dataset/ws/duPontavice_bottom_temp_aws/fileTable.nc

creating new dirTable and fileTable (dirTable=null?true fileTable=null?true badFileMap=null?false)

WARNING: FileVisitorSubdir.visitFileFailed: /home/erddap/data/bottomtemp_cache

doQuickRestart=false

*** An error occurred while trying to load duPontavice_bottom_temp_aws:

java.lang.RuntimeException: datasets.xml error on or before line #1047: IOException: java.lang.IllegalArgumentException: Expected a profile definition on line 1

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:486)

 at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:566)

 at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11187)

 at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:131)

 at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:157)

Caused by: java.io.IOException: java.lang.IllegalArgumentException: Expected a profile definition on line 1

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStep(FileVisitorDNLS.java:538)

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStepCache(FileVisitorDNLS.java:671)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.getFileInfo(EDDGridFromFiles.java:1820)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:824)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:96)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFiles.<init>(EDDGridFromNcFiles.java:103)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:302)

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:445)

 ... 4 more

[Roy Mendelssohn - NOAA Federal]

Hi Kim:

Can you please send the xml snippet as you presently have it, as well as point out which line in the snippet would correspond to line #1047 in the full datasets.xml. We are making progress. Step by step we will get this to work.

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/dd61c95b-d0bc-4a37-85a3-68789e5566a8n%40googlegroups.com.

[bobsimons2.00]

In addition to the things Roy asked for: if the error message had more stack trace information below what you provided, please include the full error message.

[Roy Mendelssohn - NOAA Federal]

Hi Kim:

Bob's email had me go back and look at your error message and then at your files, because I didn't remember seeing a file with that name. From what I can see the only file that has as part of its name "duPontavice_bottom_temp" is a pdf file. The suggests that your filename regex is not working properly, so we would need to see the xml snippet. If you want to do a quick test, remove all the non-netcdf files from the bucket. That fact that the error was thrown on the last file suggests that the others were read correctly (not a guarantee unless I saw all of your log).

Let me know.

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/1609f8a2-f54c-446c-bc68-7f18868065ddn%40googlegroups.com.

[Kim Hyde - NOAA Federal]

Hello Roy and Bob,

Thank you for continued support. Attached is the full text from datasets.xml for the bottom temp product and the Dasdds log.  I thought I set up the regex to just look for .nc files, but I'm still learning that syntax as well so that could easily be my error.  Line 1047 of the full datasets.xml is </dataset> and the end of the file is line 1048 </erddapDatasets>.

At this time I don't have access to add or remove files from the AWS bucket.  I would have to put in an IT request for them to do it.  

Kim

[Roy Mendelssohn - NOAA Federal]

Hi KIm:

I see in the xml file:

<cacheFromUrl>https://noaa-nmfs-nefsc-pds.s3.us-east-1.amazonaws.com/erddap/tesdata/</cacheFromUrl>

I believe that should be:

<cacheFromUrl>https://noaa-nmfs-nefsc-pds.s3.us-east-1.amazonaws.com/erddap/testdata/</cacheFromUrl>

At least that is what is in the original link you sent that sends me to the file list n AWS.

HTH,

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/924d56a7-4134-471a-b207-a2105aa05215n%40googlegroups.com.
> <aws_bt_test_errorlog.log><aws_bt_test.xml>

[Kim Hyde - NOAA Federal]

Yes, it appears that I was missing a "t" in the url (great catch!), however I am still getting an error.

Kim

*** DasDds duPontavice_bottom_temp_aws

*** deleting cached dataset info for datasetID=duPontavice_bottom_temp_aws

EDD.oneFromDatasetsXml(duPontavice_bottom_temp_aws)...

SimpleXmlReader encoding=

*** constructing EDDGridFromFiles(xmlReader)...

*** constructing EDDGridFromFiles duPontavice_bottom_temp_aws

dir/file table doesn't exist: /erddapData/dataset/ws/duPontavice_bottom_temp_aws/dirTable.nc

dir/file table doesn't exist: /erddapData/dataset/ws/duPontavice_bottom_temp_aws/fileTable.nc

creating new dirTable and fileTable (dirTable=null?true fileTable=null?true badFileMap=null?false)

WARNING: FileVisitorSubdir.visitFileFailed: /home/erddap/data/bottomtemp_cache

doQuickRestart=false

*** An error occurred while trying to load duPontavice_bottom_temp_aws:

java.lang.RuntimeException: datasets.xml error on or before line #1047: IOException: java.lang.IllegalArgumentException: Expected a profile definition on line 1

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:486)

 at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:566)

 at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11187)

 at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:131)

 at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:157)

Caused by: java.io.IOException: java.lang.IllegalArgumentException: Expected a profile definition on line 1

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStep(FileVisitorDNLS.java:538)

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStepCache(FileVisitorDNLS.java:671)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.getFileInfo(EDDGridFromFiles.java:1820)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:824)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:96)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFiles.<init>(EDDGridFromNcFiles.java:103)

 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:302)

 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:445)

 ... 4 more

Caused by: java.lang.IllegalArgumentException: Expected a profile definition on line 1

 at software.amazon.awssdk.utils.Validate.isTrue(Validate.java:76)

 at software.amazon.awssdk.profiles.internal.ProfileFileReader.readPropertyDefinitionLine(ProfileFileReader.java:117)

 at software.amazon.awssdk.profiles.internal.ProfileFileReader.parseLine(ProfileFileReader.java:83)

 at software.amazon.awssdk.profiles.internal.ProfileFileReader.lambda$parseFile$0(ProfileFileReader.java:59)

 at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)

 at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)

 at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)

 at software.amazon.awssdk.profiles.internal.ProfileFileReader.parseFile(ProfileFileReader.java:59)

 at software.amazon.awssdk.profiles.ProfileFile$BuilderImpl.build(ProfileFile.java:309)

 at software.amazon.awssdk.profiles.ProfileFile.lambda$addCredentialsFile$0(ProfileFile.java:152)

 at java.base/java.util.Optional.ifPresent(Optional.java:178)

 at software.amazon.awssdk.profiles.ProfileFile.addCredentialsFile(ProfileFile.java:149)

 at software.amazon.awssdk.utils.builder.SdkBuilder.applyMutation(SdkBuilder.java:61)

 at software.amazon.awssdk.profiles.ProfileFile.defaultProfileFile(ProfileFile.java:98)

 at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$mergeGlobalDefaults$3(SdkDefaultClientBuilder.java:273)

 at software.amazon.awssdk.utils.builder.SdkBuilder.applyMutation(SdkBuilder.java:61)

 at software.amazon.awssdk.core.client.config.SdkClientConfiguration.merge(SdkClientConfiguration.java:66)

 at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.mergeGlobalDefaults(SdkDefaultClientBuilder.java:271)

 at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:180)

 at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:36)

 at software.amazon.awssdk.services.s3.DefaultS3ClientBuilder.buildClient(DefaultS3ClientBuilder.java:25)

 at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:150)

 at com.cohort.util.File2.getS3Client(File2.java:1019)

 at gov.noaa.pfel.coastwatch.util.FileVisitorDNLS.oneStep(FileVisitorDNLS.java:418)

 ... 11 more

[bobsimons2.00]

Thank you for including more/all of the error message. It is clear from the additional information (stack traces are wonderful things) that the troublesome "Profile" has to do with AWS's processing of your AWS credentials file. Please re-read the AWS instructions for the profile file and try to figure out the problem with the credentials file.

[Kim Hyde - NOAA Federal]

I think we are getting close!  We were able to update the credentials and get the dataset to show up on ERDDAP.  However, they just tried re-imaging the ERDDAP container with the same credentials and now I'm getting mixed results.  Attached is the dasdds log, which looks good for the dataset itself, but then there are EmailThread session errors that continue to repeat.  Is this email error associated with the AWS credentials or something else.  I don't get this email error when I run the dasdds on the local files, which makes me think it is somehow associated with the AWS files, but I don't know what is different.  Nothing changed in the datasets.xml file.

Thanks again for your help,

Kim

[Roy Mendelssohn - NOAA Federal]

HI Kim:

I believe this is your problem (remember I know little about AWS):

#48 bad file: removing fileTable row for /home/erddap/data/bottomtemp_cache/bottom_temp_2007.nc
java.lang.RuntimeException: AxisVariable=time
isn't ascending sorted (DoubleArray isn't sorted in ascending order: [290]=21184.0 > [291]=20820.0.)
or descending sorted (DoubleArray isn't sorted in descending order: [0]=20894.0 < [1]=20895.0.).

Check the times in your files.

HTH,

-Roy

> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/f125b3ae-2d32-4e36-8ea5-30a5e53eb4dfn%40googlegroups.com.
> <DasDds_Btemp_20231030.log>