ingest.properties in ogpIngest

[Garey Mills]

Hi - I am having a problem with getting ogpIngest to work. It is not reading the WEB-INF/ingest.properties file.

I know there is a bean defined called ingestProperties.propFile that references the class org.OpenGeoPortal.Ingest.PropertyFileIngestProperties and 'WEB-INF/ingest.properties'

but I don't know whether that is supposed to make the property file directly available to that code or not. 

Not knowing my way around XML bean definition, I don't know how to debug this;

Garey Mills

--

Generate messages about directories that cannot  be

read,  files  that  cannot be opened ... rather than being silent ... 

(from `man du`)

[Chris Barnett]

Hi Garey,

You read that correctly. The file at ‘WEB-INF/ingest.properties’ is injected as a resource to the class org.OpenGeoPortal.Ingest.PropertyFileIngestProperties.

Are you getting an error that explicitly says that the properties file is not found (if it didn’t find it I think you would get a Spring error on startup…probably Bean Creation Error) or are you seeing some behavior that you don’t expect? Unfortunately the properties file for ogpIngest is quite arcane to try to accommodate a lot of different metadata practices.

thanks,

Chris

--
You received this message because you are subscribed to the Google Groups "OpenGeoportal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opengeoportal-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Garey Mills]

Chris - I am trying to find the exact error message, but the behaviour that I don't expect is that the Location tag is empty, despite the ingest.properties attached.

Here is the exact error from the ogpIngest ingest screen

UCB-28722/bk001542h01

Solr Ingest Warnings: location<location>:Online Location Error getting links from properties files:null-

[Chris Barnett]

Hi Garey,

Is this for FGDC or ISO metadata?

thanks,

Chris

<ingest.properties>

[Garey Mills]

ISO

[Chris Barnett]

Hi Garey,

With a random ISO file my Location field is being set properly if I use your ingest.properties. What doesn’t work for me is the layer ‘Name’.  Where in the metadata are you storing the geoserver layer name? Can you send me an example ISO record that you are using?

thanks,

Chris

[Garey Mills]

Chris -

Here's the ISO record we are trying to upload. As for the name, I am not sure what tag is being picked up. Perhaps the <fileIdentifier>?

Garey

[Chris Barnett]

Hi Garey,

The file worked for me. What do you see in your logs on ingest?

I see this (annotated):

2014-11-06 17:11:14 BasicMetadataUploadSubmitter [INFO] 21012425-06a8-4400-901f-f731dcfdc69e

2014-11-06 17:11:14 MetadataUploadController [INFO] JobId: 21012425-06a8-4400-901f-f731dcfdc69e

2014-11-06 17:11:14 BasicUploadMetadataJob [INFO] Trying to parse metadata...

2014-11-06 17:11:14 Iso19139LocationResolver [INFO] areLinks:false

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] number of links: 0

# no links were found in the metadata document

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] setting location

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] finished handleLocation

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] publisher

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] originator

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] bounds

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Alameda county (California)

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword type: place

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: 2012

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword type: temporal

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Local transit

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Bus rapid transit

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Urban transit

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Public transit

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Mass transit

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword type: theme

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keyword value: Downloadable Data

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] keywords

java.lang.NullPointerException

at org.OpenGeoPortal.Ingest.Metadata.Iso19139ParseMethod.getAttributeValue(Iso19139ParseMethod.java:544)

at org.OpenGeoPortal.Ingest.Metadata.Iso19139ParseMethod.handleAccess(Iso19139ParseMethod.java:720)

at org.OpenGeoPortal.Ingest.Metadata.AbstractXmlMetadataParseMethod.marshallMetadata(AbstractXmlMetadataParseMethod.java:74)

at org.OpenGeoPortal.Ingest.Metadata.FlexibleMetadataConverter.handleAsXml(FlexibleMetadataConverter.java:238)

at org.OpenGeoPortal.Ingest.Metadata.FlexibleMetadataConverter.handleMetadata(FlexibleMetadataConverter.java:169)

at org.OpenGeoPortal.Ingest.Metadata.FlexibleMetadataConverter.parse(FlexibleMetadataConverter.java:85)

at org.OpenGeoPortal.Ingest.BasicUploadMetadataJob.ingestXmlMetadata(BasicUploadMetadataJob.java:169)

at org.OpenGeoPortal.Ingest.BasicUploadMetadataJob.uploadMetadata(BasicUploadMetadataJob.java:85)

at org.OpenGeoPortal.Ingest.BasicUploadMetadataJob.run(BasicUploadMetadataJob.java:255)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

#Didn’t find an expected tag for access

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] access

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] resolveVectorToGeometryType

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] DataType:Polygon

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] datatype

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] fulltext

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] Date: 2014

2014-11-06 17:11:14 Iso19139ParseMethod [INFO] date

2014-11-06 17:11:14 FlexibleMetadataConverter [INFO] owsname is null: false

2014-11-06 17:11:14 FlexibleMetadataConverter [INFO] institution is null: false

2014-11-06 17:11:14 FlexibleMetadataConverter [INFO] ows name:28722/bk001542h01

2014-11-06 17:11:14 BasicUploadMetadataJob [INFO] Metadata Parsed...

2014-11-06 17:11:14 BasicUploadMetadataJob [INFO] Metadata parsed?: true

2014-11-06 17:11:14 BasicUploadMetadataJob [INFO] Trying Solr ingest...[28722/bk001542h01]

2014-11-06 17:11:14 OwsSolrIngest [INFO] OWS Name: 28722/bk001542h01

2014-11-06 17:11:14 OwsSolrIngest [INFO] 28722/bk001542h01

2014-11-06 17:11:14 OwsSolrIngest [INFO] Berkeley.28722/bk001542h01

2014-11-06 17:11:14 OwsSolrIngest [INFO] Links: 0

2014-11-06 17:11:14 LocationLink [INFO] LocationType: wms

2014-11-06 17:11:14 LocationLink [INFO] URL: http://gis-gs.lib.berkeley.edu:8080/geoserver/wms

2014-11-06 17:11:14 LocationLink [INFO] LocationType: wfs

2014-11-06 17:11:14 LocationLink [INFO] URL: http://gis-gs.lib.berkeley.edu:8080/geoserver/wfs

2014-11-06 17:11:14 OwsSolrIngest [INFO] Property Links: 2

#added 2 links from the properties file

2014-11-06 17:11:14 OwsSolrIngest [INFO] {"wfs": "http://gis-gs.lib.berkeley.edu:8080/geoserver/wfs","wms": ["http://gis-gs.lib.berkeley.edu:8080/geoserver/wms”]}

#Location field sent to solr

Looking at your metadata record, it looks like ingest grabbed the wrong date. I’ll take a look at what tag(s) the code is looking at.

The “Name” is populated with the <fileIdentifier>. Do you have a way to associate the GeoServer layer name with the metadata record? Otherwise OGP won’t know how to find your layer for preview and download. You could add a simple class that looks at a table that associates fileIdentifiers with layer names or the record from EZID and reference it in:

org.OpenGeoPortal.Ingest.AbstractSolrIngest or OwsSolrIngest with the method: processName

thanks,

Chris

<ACTransit_ISO19139.xml>

[Garey Mills]

Chris - We deviate, but why? Here is what I have in the logs

...

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] access

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] resolveVectorToGeometryType

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] DataType:Polygon

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] datatype

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] fulltext

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] Date: 2014

2014-11-07 14:01:52 Iso19139ParseMethod [INFO] date

2014-11-07 14:01:52 FlexibleMetadataConverter [INFO] owsname is null: false

2014-11-07 14:01:52 FlexibleMetadataConverter [INFO] institution is null: false

2014-11-07 14:01:52 FlexibleMetadataConverter [INFO] ows name:28722/bk001542h01

2014-11-07 14:01:52 BasicUploadMetadataJob [INFO] Metadata Parsed...

2014-11-07 14:01:52 BasicUploadMetadataJob [INFO] Metadata parsed?: true

2014-11-07 14:01:52 BasicUploadMetadataJob [INFO] Trying Solr ingest...[UCB-28722/bk001542h01]

2014-11-07 14:01:52 OwsSolrIngest [INFO] OWS Name: UCB-28722/bk001542h01

2014-11-07 14:01:52 OwsSolrIngest [INFO] Links: 0

2014-11-07 14:01:52 SolrJClient [INFO] committing add to Solr

2014-11-07 14:01:52 OwsSolrIngest [INFO] Successfully added layer to solr.

2014-11-07 14:01:52 BasicUploadMetadataJob [WARN] Solr Ingest Warnings:1

and that ends the log entries. 

On other topics, I will look at the layer name/fileIdentifier table.

Garey