row.getFileName

[Tuomo Saari]

Hi,

I created an EDDTableFromFileNames dataset on ERDDAP v2.30.0, and tried to script new variables using =row.getFileName() and =row.getFullFileName(). The result of both are empty. What coul dbe wrong? I didn't find anything useful in ERDDAP log files. 

I even tested with a simple version like below, with no luck. What could be wrong?

<dataVariable>
        <sourceName>=row.getFileName()</sourceName>
        <destinationName>row_FileName</destinationName>
        <dataType>String</dataType>
        <addAttributes>
            <att name="ioos_category">Identifier</att>
            <att name="long_name">ERDDAP source filename</att>
        </addAttributes>
    </dataVariable>

Thank you! 

/Tuomo

[Roy Mendelssohn - NOAA Federal]

Hi Tuomo:

This is might be one for Chris as I had to do a lot of searching to find anything even searching the code for “getFullFileName” only turned up a jexlscript script at:

https://erddap.github.io/docs/dokka/-e-r-d-d-a-p/com.cohort.util/-script-row

However, even were this can be used (more on this below) I believe what would be used is:

<sourceName>=row.fileName</sourceName>

or:

<sourceName>=row.fullFileName</sourceName>

No parentheses, no "get" prefix — treat it as a property, not a function call. however, when i look at the EDDTableFromFileNames skeleton at:

https://erddap.github.io/docs/server-admin/datasets?_highlight=eddtablefromfilenames#eddtablefromfilenames-skeleton-xml

I don’t see that as an option. Not being certain of this myself, I asked Claude which said:

> EDDTableFromFileNames doesn't read inside files at all
> For this dataset type, ERDDAP isn't opening each file and extracting a row of data from its contents — it's just cataloging directory entries. The "rows" ARE the files. The native source columns for this dataset type are exactly: url, name, lastModified, and size — nothing else, because there's no file content being parsed.
> That's almost certainly why row.getFileName() (or the property-corrected row.fileName) came up blank: fileName/fullFileName in ScriptRow are meant for dataset types that actually open and read a source data file per row (EDDTableFromNcFiles, EDDTableFromMultidimNcFiles, EDDTableFromAsciiFiles, etc.) so the script can tag each output row with which underlying file it came from. In EDDTableFromFileNames, there is no "file being read to produce this row" in that sense — the file's name is already the data — so that ScriptRow property likely has nothing populated to return.
>
> The actual fix
> No script needed. Just declare name as a native sourceName:
>
> <dataVariable>
> <sourceName>name</sourceName>

> <destinationName>row_FileName</destinationName>
> <dataType>String</dataType>
> <addAttributes>
> <att name="ioos_category">Identifier</att>
> <att name="long_name">ERDDAP source filename</att>
> </addAttributes>
> </dataVariable>
>

> And if the admin wants the full path rather than just the bare filename, use url instead:
>
> <sourceName>url</sourceName>
>

Of course, if this is correct I will take the credit, if wrong blame Claude. But seriously at:

https://erddap.github.io/docs/server-admin/dataset-examples/video-datasets

you can see an example of creating a EDDTableFromFileNames and you see:

<dataVariable>
<sourceName>url</sourceName>
<destinationName>url</destinationName>
<dataType>String</dataType>
<!-- sourceAttributes>

<att name="ioos_category">Identifier</att>

<att name="long_name">URL</att>
</sourceAttributes -->
<addAttributes>
</addAttributes>
</dataVariable>
<dataVariable>
<sourceName>name</sourceName>
<destinationName>name</destinationName>
<dataType>String</dataType>
<!-- sourceAttributes>

<att name="ioos_category">Identifier</att>

<att name="long_name">File Name</att>
</sourceAttributes -->
<addAttributes>
</addAttributes>
</dataVariable>

and that successfully created a dataset with the filenames:

https://coastwatch.pfeg.noaa.gov/erddap/tabledap/fed_HAGE_VisualSurvey.html


HTH, and let me know if this fixes things.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/b7339706-fe16-45e1-b08a-47b53ce03952n%40googlegroups.com.

[Tuomo Saari]

Hi Roy,

It was not clear to me from the documentation if these methods were supposed to work for this type of dataset. 

What you said would explain the behavior perfectly.

I was also interested whether I could get the parent directory name, but the "url" is available so I guess I will just parse it from there.

Thank you!

/Tumo

[Roy Mendelssohn - NOAA Federal]

Glad it worked,. It wasn’t clear to me either, I wasn’t kidding when I said I had to a lot of searching. One reason we are starting the sample datasets.xml snippets which I referred to, which I hope people will contribute to, is to give working examples of certain things. Also, queries like yours hope us to see where our docs need improving.

Let me know if it all works - again look at the example that is posted and the final result,

-Roy

> To view this discussion, visit https://groups.google.com/d/msgid/erddap/6063a911-5470-42e2-b40a-8722e482d65dn%40googlegroups.com.

[Mathew Biddle - NOAA Federal]

Could you use ***pathName? I use that to extract info from file names in some of my datasets.

https://erddap.github.io/docs/server-admin/datasets?_highlight=pathname#eddtablefromfiles 

> The format for the pseudo sourceName which gets the value from a file's absolute path name is ***pathName,dataType,extractRegex,captureGroupNumber

https://github.com/MathewBiddle/ioos_datasets_xml/blob/a701f1ad035c9f54215ab8795b7991620f596a74/xml/gts_regional_statistics.xml#L42-L50

For the Year variable in https://erddap.ioos.us/erddap/tabledap/gts_regional_statistics.html.

But that might have the same issue as before.

Matt

To view this discussion, visit https://groups.google.com/d/msgid/erddap/13B6CC32-2B32-40B5-A8F3-0CCB05096B14%40noaa.gov.

--

Mathew Biddle, Physical Scientist
NOAA/NOS
US Integrated Ocean Observing System Office
1315 East-West Highway
Silver Spring MD 20910

240-533-9470

ORCiD: 0000-0003-4897-1669

http://www.ioos.noaa.gov/

[Tuomo Saari]

That is an interesting suggestion, thanks Matt! 

Even if it doesn't work here, I feel like it can come in handy in some other cases.

/Tuomo

[Chris John - NOAA Affiliate]

EDDTableFromFileNames does not support row.getFileName() and row.getFullFileName(). You can get the file name using:

 <dataVariable>
  <sourceName>name</sourceName>
  <destinationName>name</destinationName>
  <dataType>String</dataType>
  <!-- sourceAttributes>
    <att name=\"ioos_category\">Identifier</att>

    <att name=\"long_name\">File Name</att>
  </sourceAttributes -->
  <addAttributes>
  </addAttributes>
</dataVariable>

You can also extract part of the file name as a variable. This is detailed here:
https://erddap.github.io/docs/server-admin/datasets?_highlight=eddtablefromfilenames#from-file-names-data-table-contents

You should also be able to use the name variable in scripts if you need to do something more than extract from it.

If you need the full file path, the I believe the previously mentioned ***pathName will provide that.

Christopher John

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, Contractor with TSPi in support of 

NOAA Fisheries Office of Southwest Fisheries Science Center (SWFSC) | U.S. Department of Commerce

www.fisheries.noaa.gov

To view this discussion, visit https://groups.google.com/d/msgid/erddap/99c822b2-5e26-4a37-8e52-67d3adde9979n%40googlegroups.com.