Issue with data ingesting.

70 views
Skip to first unread message

Chuan-Yuan Hsu

unread,
Aug 4, 2021, 6:05:08 PM8/4/21
to ERDDAP
Hi 

I was trying to ingest some of my colleagues' generated netCDF files to the ERDDAP. However, the ERDDAP always gave me the error message saying that 

*** An error occurred while trying to load R4_x260_204_0034_1f7f_3f0e_f233:
java.lang.RuntimeException: datasets.xml error on or before line #541: cdm_data_type=TimeSeries, but the dataset doesn't have a time variable.
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:443)
 at gov.noaa.pfel.erddap.dataset.EDD.oneFromDatasetsXml(EDD.java:524)
 at gov.noaa.pfel.erddap.dataset.EDD.testDasDds(EDD.java:11123)
 at gov.noaa.pfel.erddap.DasDds.doIt(DasDds.java:129)
 at gov.noaa.pfel.erddap.DasDds.main(DasDds.java:155)
Caused by: com.cohort.util.SimpleException: cdm_data_type=TimeSeries, but the dataset doesn't have a time variable.
 at gov.noaa.pfel.erddap.dataset.EDDTable.accessibleViaNcCF(EDDTable.java:13440)
 at gov.noaa.pfel.erddap.dataset.EDD.ensureValid(EDD.java:783)
 at gov.noaa.pfel.erddap.dataset.EDDTable.ensureValid(EDDTable.java:666)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.<init>(EDDTableFromFiles.java:1882)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromNcCFFiles.<init>(EDDTableFromNcCFFiles.java:97)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.fromXml(EDDTableFromFiles.java:519)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:430)
 ... 4 more

Well,  one interesting thing is this error only prints out on the screen, but I cannot see it saved in the DasDds.out, but DasDds.log and log.txt. 

The netCDF has the time variable, and I can also see the time variable in the XML by using "grep 'sourceName' XML" or "grep 'destinationName' XML". Could anyone give me a hint on how to debug this?  Spend over 2 whole days but still cannot figure this out. 

ps: all of the files can be access at https://erddap2.gcoos.org/message/erddap_group/ 
ps2: version of ERDDAP: v2.11


Thanks,
Chuan-Yuan Hsu

Roy Mendelssohn - NOAA Federal

unread,
Aug 4, 2021, 6:15:10 PM8/4/21
to Chuan-Yuan Hsu, ERDDAP
I will take a look it when I get a chance, but I am only so-so at figuring out these types of problems. Bob is likely out of contact till next week, so if you don't hear back from him until he catches up on things next week, don't take it as a sign your problem isn't considered important.

-Roy
> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/54404c41-a4d3-4ad0-bcf3-0a4bcbbedcdbn%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

Roy Mendelssohn - NOAA Federal

unread,
Aug 4, 2021, 6:46:57 PM8/4/21
to Chuan-Yuan Hsu, ERDDAP
Just a guess here but could this be causing the problem:

> <dataVariable>
> <sourceName>time</sourceName>
> <destinationName>time2</destinationName>
>

Any reason you are re-naming "time" to "time2". If I understand the inner workings of ERDDAP correctly (and often I don't), ERDDAP will then see that variable as 'time2' which is not a CF compatible name for 'time', and thus throw an error.

-Roy


> On Aug 4, 2021, at 3:05 PM, Chuan-Yuan Hsu <ch...@tamu.edu> wrote:
>

Kevin O'Brien - NOAA Affiliate

unread,
Aug 4, 2021, 7:36:50 PM8/4/21
to Roy Mendelssohn - NOAA Federal, Chuan-Yuan Hsu, ERDDAP
I think that Roy is on the right track, and I would look further at the time variable in the NetCDF file.  The configuration looks like:
<dataVariable>
<sourceName>time</sourceName>
<destinationName>time2</destinationName>
<dataType>double</dataType>
<!-- sourceAttributes>
<att name="_ChunkSizes" type="int">115053</att>
<att name="_FillValue" type="double">-99999.0</att>
<att name="axis">T</att>
<att name="calendar">gregorian</att>
<att name="comment">unknown</att>
<att name="coverage_content_type">physicalMeasurement</att>
<att name="ioos_category">time</att>
<att name="long_name">time</att>
<att name="missing_value" type="double">-99999.0</att>
<att name="standard_name">time</att>
<att name="standard_name_url">http://mmisw.org/ont/ioos/parameter/time</att>
<att name="units">1970-01-01 00:00:00</att>
<att name="valid_max" type="double">1.444987919E9</att>
<att name="valid_min" type="double">1.4380848E9</att>
</sourceAttributes -->
<addAttributes>
<att name="_ChunkSizes">null</att>
<att name="colorBarMaximum" type="double">1.446E9</att>
<att name="colorBarMinimum" type="double">1.436E9</att>
<att name="comment">null</att>
<att name="ioos_category">Time</att>
</addAttributes>
</dataVariable>

It seems that the units for the "time" variable ERDDAP saw in the NetCDF file were "1970-01-01 00:00:00".
I would expect something more like "days since 1970-01-01 00:00:00"  and this is probably why ERDDAP gave it the time2 destination name.   Fix the destinationName and correct the units and it may (should) work.

best
kevin






To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/78ADE82C-5837-45B4-8AD7-C129DD8592C1%40noaa.gov.


--
Kevin O'Brien
Senior Research Scientist
UW/CICOES
NOAA/PMEL/SDIG
206-526-6751
ORCiD: 0000-0003-4167-3028
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the  National Oceanic and Atmospheric Administration."

Roy Mendelssohn - NOAA Federal

unread,
Aug 4, 2021, 7:43:08 PM8/4/21
to Chuan-Yuan Hsu, ERDDAP, Kevin O'Brien - NOAA Affiliate


Good catch by Kevin. It is actually incorrect in the netcdf file (I just checked). ERDDAP xml allows you to redefine the units. But if this is representative of a series of netcdf files, I would also go back and talk to the people creating the files, as the units are off.

-Roy

Chuan-Yuan Hsu

unread,
Aug 4, 2021, 10:48:30 PM8/4/21
to Roy Mendelssohn - NOAA Federal, ERDDAP, Kevin O'Brien - NOAA Affiliate
Wonderful… Good catch!! I do see the 1970 things in XML, but thinking it is normal so that I didn’t check the unit is presented in a wrong way.  After I modified the units, it works. 

Really appreciated you two, Kevin and Roy. 

Thanks,
Chuan-Yuan Hsu, Ph.D.
+—————————————————————+
Postdoctoral Research Scientist,
Product Developer, Data Scientist,
Gulf of Mexico Coastal Ocean Observing System (GCOOS)
Texas A&M University — Department of Oceanography

Office: 979-845-3956
Mobile’: 734-926-5394
Email: ch...@tamu.edu
Web: http://gcoos.org

On Aug 4, 2021, at 6:43 PM, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:



Good catch by Kevin.  It is actually incorrect in the netcdf file  (I just checked).    ERDDAP xml allows you to redefine the units.  But if this is representative of a series of netcdf files, I would also go back and talk to the people creating the files,  as the units are off.

-Roy


On Aug 4, 2021, at 4:36 PM, Kevin O'Brien - NOAA Affiliate <kevin.m.o'br...@noaa.gov> wrote:

I think that Roy is on the right track, and I would look further at the time variable in the NetCDF file.  The configuration looks like:
<dataVariable>
<sourceName>time</sourceName>
<destinationName>time2</destinationName>
<dataType>double</dataType>
<!--  sourceAttributes>
           <att name="_ChunkSizes" type="int">115053</att>
           <att name="_FillValue" type="double">-99999.0</att>
           <att name="axis">T</att>
           <att name="calendar">gregorian</att>
           <att name="comment">unknown</att>
           <att name="coverage_content_type">physicalMeasurement</att>
           <att name="ioos_category">time</att>
           <att name="long_name">time</att>
           <att name="missing_value" type="double">-99999.0</att>
           <att name="standard_name">time</att>

ps2: version of ERDDAP: v2.11


Thanks,
Chuan-Yuan Hsu

-- 
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980


"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

-- 
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.



-- 
Kevin O'Brien
Senior Research Scientist
UW/CICOES
NOAA/PMEL/SDIG
206-526-6751
ORCiD: 0000-0003-4167-3028
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the  National Oceanic and Atmospheric Administration."


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980

Roy Mendelssohn - NOAA Federal

unread,
Aug 4, 2021, 10:49:49 PM8/4/21
to Chuan-Yuan Hsu, ERDDAP, Kevin O'Brien - NOAA Affiliate
Glad you got it to work.

-Roy
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/
Reply all
Reply to author
Forward
0 new messages