Search index rebuild results in missing dcCreated fields (1.6)

[edmore moyo]

Hi,

We are gearing up for an upgrade from 1.4.x to 1.6 but we have come across an issue:

https://opencast.jira.com/browse/MH-10566

Has anyone else encountered such an issue or is also able to reproduce it? After the index rebuilds all dates in the media module appear as n.a.

Cheers,

Edmore

[Karen Dolan]

Edmore,

This is a long shot, but do you see anything in the logs like "does not conform to W3C-DTF encoding scheme"?

- Karen

To unsubscribe from this group and stop receiving emails from it, send an email to matterhorn-use...@opencast.org.

[edmore moyo]

Hi Karen,

Yes I checked for that, went through the code as well - nothing :-(

Are you able to reproduce the issue though?

Cheers,

Edmore

[Karen Dolan]

Edmore,

It looks like it might be possible for the Search Service to activate and start rebuilding it's index before the StaticMetadataService (StaticMetadataServiceDublinCoreImpl) has a chance to load and be available to parse the dublin core catalog metadata. The OSGi injection policy for the StaticMetadataService is dynamic, not static, so this service does not have to be available in order for the Search service to run.  The Search Service's SolrIndexManager does have a "not null" check on it's StaticMetadataService collection, but the collection is never null because it is initialized as an empty collection by the Search Service. 

Do the logs give any indication that the search index might have activated and started rebuilding its index before "StaticMetadataServiceDublinCoreImpl" was activated?

If it did, you could try stopping matterhorn, removing the search index files again, and restarting Matterhorn to see if the timing is better. If this is the issue then the Search service needs to be changed to wait for at least one Static Metadata Service to be available before trying to rebuild it's index.

Best of Luck!

Karen

[edmore moyo]

Hi Karen,

Thanks. I will check for the activation.

Cheers,

Edmore

[edmore moyo]

Hi Karen,

Your assumptions are correct:

devslsmth001:/data/matterhorn/local # tail -f  /opt/matterhorn/logs/opencast.log | grep StaticMetadataServiceDublinCoreImpl

2014-12-12 10:39:22 DEBUG [FelixStartLevel] (StaticMetadataServiceDublinCoreImpl:92) - activate()

The StaticMetadataServiceDublin ... is activated much later than the search service actually starts reindexing. So allowing the index to rebuild by restarting the search service impl via the system console is a workaround. There is definitely a timing issue.

Thank you.

Cheers,

Edmore

[Karen Dolan]

Edmore,

It's great that the issue is identified! Would you mind doing a test where you change policy from dynamic to static on the name="staticMetadata" in the modules/matterhorn-search-service-impl/src/main/resources/search-service.xml?

In theory, this should prevent the Search Service from loading until at least one StaticMetadataService is available and, theory, it should allow more than one to load dynamically. 

- Karen

FYI: Matterhorn loads 2 classes that implement the StaticMetadataService interface, one parses media package metadata, StaticMetadataServiceMediaPackageImpl , and the other parses dublin core catalog metadata, StaticMetadataServiceDublinCoreImpl.   

If both are needed, then the cardinality should be 2..n. For some reason, that I no longer recall, when I looked at the code yesterday, it seemed like the Episode Service relies on both, but Search Service only really needs the dublin core parser.

[edmore moyo]

Hi Karen,

Sure. I will test that and let you know the outcome.

Cheers,
Edmore

[edmore moyo]

Hi Karen,

Just tested your suggestion. Unfortunately setting the policy to "static" in the search-service.xml file still results in the dates in the media module appearing as n.a.

Cheers,
Edmore