Search index rebuild errors

43 views
Skip to first unread message

s.la...@googlemail.com

unread,
Aug 23, 2024, 10:52:35 AM8/23/24
to Opencast Users
Hi,

I updated our opencast test cluster to latest 16.4 (coming from 15).
After I figured out, how to configure elasticsearch correctly so that the presentation node can access it, I was able to start opencast without further start up errors.

I subsequently started to rebuild the indexes which yielded the following results:
  • Series index rebuild without error
  • Scheduler rebuild without error
  • AssetManager rebuild WITH errors caused by missing events but finished successfully
  • Comments index rebuild WITH errors caused by missing events but finished successfully
  • Workflow index rebuild without error
  • Search index rebuild WITH warnings and errors caused by missing events
The asset manager errors where already with older opencast versions. How can I clean up opencast to get rid of those orphaned events, which lead to errors? Is there any health check implemented, so opencast checks it's database, filesystem (and indexes) for consistency errors?
To me it seems, the search index rebuild stopped after about 10 % because of the error (see log file snippet below). 

2024-08-23T09:45:56,926 | INFO | (IndexRebuildService:214) - Finished rebuilding the Workflow index
2024-08-23T09:45:56,926 | INFO | (IndexRebuildService:206) - Starting to rebuild the Search index
2024-08-23T09:45:56,938 | INFO | (AbstractIndexProducer:44) - Starting Search index rebuild with 12285 search
2024-08-23T09:45:57,803 | WARN | (RequestLogger:65) - request [PUT http://ELASTICSEARCH.IP:9200/opencast_search/_doc/007c063c-5fc3-4557-a4f6-6684489e201c?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "Error parsing [null] as date in [null_value] on field [deleted]); [null_value] will be ignored"]
2024-08-23T09:45:58,254 | WARN | (RequestLogger:65) - request [PUT http://ELASTICSEARCH.IP:9200/opencast_search/_doc/00a6fcd9-2eea-443b-856a-8519487bccce?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "Error parsing [null] as date in [null_value] on field [deleted]); [null_value] will be ignored"]
2024-08-23T09:45:58,288 | WARN | (RequestLogger:65) - request [PUT http://ELASTICSEARCH.IP:9200/opencast_search/_doc/8c85f035-1b6d-4454-8073-4a1d35a67cef?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "Error parsing [null] as date in [null_value] on field [deleted]); [null_value] will be ignored"]
2024-08-23T09:46:01,285 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 0181a1d8-8d5e-467e-8c77-6526ae30fb9e, omitting the link from the indexed data
2024-08-23T09:46:02,099 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 01f43dd6-a54e-4f83-a7f8-aecbe45bd71b, omitting the link from the indexed data
2024-08-23T09:46:04,006 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 150/12285 finished, 1% complete.
2024-08-23T09:46:06,652 | WARN | (RequestLogger:65) - request [PUT http://ELASTICSEARCH.IP:9200/opencast_search/_doc/c4178142-36e1-4507-a818-7cd7bc948a6d?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "Error parsing [null] as date in [null_value] on field [deleted]); [null_value] will be ignored"]
2024-08-23T09:46:07,784 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 250/12285 finished, 2% complete.
2024-08-23T09:46:12,816 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 400/12285 finished, 3% complete.
2024-08-23T09:46:15,931 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 500/12285 finished, 4% complete.
2024-08-23T09:46:18,191 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 0b668572-e650-4899-a7ac-30380bb8858e, omitting the link from the indexed data
2024-08-23T09:46:20,661 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 650/12285 finished, 5% complete.
2024-08-23T09:46:21,447 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 0d87f731-6d6d-4e0e-bbe3-c0899b112333, omitting the link from the indexed data
2024-08-23T09:46:23,269 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 0ebd6a00-d104-41e4-aa8b-aff3ab8495d5, omitting the link from the indexed data
2024-08-23T09:46:23,883 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 750/12285 finished, 6% complete.
2024-08-23T09:46:27,590 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 119ff2ca-94b1-4724-977c-84eff2586a48, omitting the link from the indexed data
2024-08-23T09:46:28,964 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 900/12285 finished, 7% complete.
2024-08-23T09:46:32,891 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 1000/12285 finished, 8% complete.
2024-08-23T09:46:36,409 | INFO | (AbstractIndexProducer:133) - Search index rebuild: 1100/12285 finished, 8% complete.
2024-08-23T09:46:38,215 | WARN | (SearchServiceIndex:265) - Series 0 not found during index of event 17ca2741-998e-4758-836c-e79dad1b2f93, omitting the link from the indexed data
org.opencastproject.util.NotFoundException: null
at org.opencastproject.workspace.impl.WorkspaceImpl.downloadIfNecessary(WorkspaceImpl.java:535) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl$2.xapply(WorkspaceImpl.java:557) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl$2.xapply(WorkspaceImpl.java:554) ~[?:?]
at org.opencastproject.util.data.Function$X.apply(Function.java:104) ~[?:?]
at org.opencastproject.util.IoSupport.locked(IoSupport.java:478) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl.get(WorkspaceImpl.java:402) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl.read(WorkspaceImpl.java:437) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.loadDublinCore(DublinCoreUtil.java:76) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.lambda$loadEpisodeDublinCore$0(DublinCoreUtil.java:64) ~[?:?]
at java.util.Optional.map(Optional.java:265) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.loadEpisodeDublinCore(DublinCoreUtil.java:64) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.indexMediaPackage(SearchServiceIndex.java:255) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.lambda$repopulate$3(SearchServiceIndex.java:471) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.repopulate(SearchServiceIndex.java:457) ~[?:?]
at org.opencastproject.elasticsearch.index.rebuild.IndexRebuildService.rebuildIndex(IndexRebuildService.java:209) ~[?:?]
at org.opencastproject.elasticsearch.index.rebuild.IndexRebuildService.rebuildIndex(IndexRebuildService.java:137) ~[?:?]
at org.opencastproject.elasticsearch.index.endpoint.IndexEndpoint.lambda$rebuildIndex$3(IndexEndpoint.java:167) ~[?:?]
at org.opencastproject.security.util.SecurityContext.lambda$runInContext$0(SecurityContext.java:68) ~[?:?]
at org.opencastproject.security.util.SecurityContext.runInContext(SecurityContext.java:58) ~[?:?]
at org.opencastproject.security.util.SecurityContext.runInContext(SecurityContext.java:67) ~[?:?]
at org.opencastproject.elasticsearch.index.endpoint.IndexEndpoint.lambda$rebuildIndex$4(IndexEndpoint.java:164) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-08-23T09:46:38,581 | ERROR | (AbstractIndexProducer:201) - Error updating the Search index: 1161/12285 could be finished.
org.opencastproject.util.NotFoundException: null
at org.opencastproject.workspace.impl.WorkspaceImpl.downloadIfNecessary(WorkspaceImpl.java:535) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl$2.xapply(WorkspaceImpl.java:557) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl$2.xapply(WorkspaceImpl.java:554) ~[?:?]
at org.opencastproject.util.data.Function$X.apply(Function.java:104) ~[?:?]
at org.opencastproject.util.IoSupport.locked(IoSupport.java:478) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl.get(WorkspaceImpl.java:402) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl.read(WorkspaceImpl.java:437) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.loadDublinCore(DublinCoreUtil.java:76) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.lambda$loadEpisodeDublinCore$0(DublinCoreUtil.java:64) ~[?:?]
at java.util.Optional.map(Optional.java:265) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.loadEpisodeDublinCore(DublinCoreUtil.java:64) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.indexMediaPackage(SearchServiceIndex.java:255) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.lambda$repopulate$3(SearchServiceIndex.java:471) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.repopulate(SearchServiceIndex.java:457) ~[?:?]
at org.opencastproject.elasticsearch.index.rebuild.IndexRebuildService.rebuildIndex(IndexRebuildService.java:209) ~[?:?]
at org.opencastproject.elasticsearch.index.rebuild.IndexRebuildService.rebuildIndex(IndexRebuildService.java:137) ~[?:?]
at org.opencastproject.elasticsearch.index.endpoint.IndexEndpoint.lambda$rebuildIndex$3(IndexEndpoint.java:167) ~[?:?]
at org.opencastproject.security.util.SecurityContext.lambda$runInContext$0(SecurityContext.java:68) ~[?:?]
at org.opencastproject.security.util.SecurityContext.runInContext(SecurityContext.java:58) ~[?:?]
at org.opencastproject.security.util.SecurityContext.runInContext(SecurityContext.java:67) ~[?:?]
at org.opencastproject.elasticsearch.index.endpoint.IndexEndpoint.lambda$rebuildIndex$4(IndexEndpoint.java:164) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-08-23T09:46:38,581 | ERROR | (AbstractIndexProducer:185) - Error updating the Search index.
java.lang.RuntimeException: Internal Index Rebuild Failure
at org.opencastproject.search.impl.SearchServiceIndex.lambda$repopulate$3(SearchServiceIndex.java:475) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.repopulate(SearchServiceIndex.java:457) ~[?:?]
at org.opencastproject.elasticsearch.index.rebuild.IndexRebuildService.rebuildIndex(IndexRebuildService.java:209) ~[?:?]
at org.opencastproject.elasticsearch.index.rebuild.IndexRebuildService.rebuildIndex(IndexRebuildService.java:137) ~[?:?]
at org.opencastproject.elasticsearch.index.endpoint.IndexEndpoint.lambda$rebuildIndex$3(IndexEndpoint.java:167) ~[?:?]
at org.opencastproject.security.util.SecurityContext.lambda$runInContext$0(SecurityContext.java:68) ~[?:?]
at org.opencastproject.security.util.SecurityContext.runInContext(SecurityContext.java:58) ~[?:?]
at org.opencastproject.security.util.SecurityContext.runInContext(SecurityContext.java:67) ~[?:?]
at org.opencastproject.elasticsearch.index.endpoint.IndexEndpoint.lambda$rebuildIndex$4(IndexEndpoint.java:164) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.opencastproject.util.NotFoundException
at org.opencastproject.workspace.impl.WorkspaceImpl.downloadIfNecessary(WorkspaceImpl.java:535) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl$2.xapply(WorkspaceImpl.java:557) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl$2.xapply(WorkspaceImpl.java:554) ~[?:?]
at org.opencastproject.util.data.Function$X.apply(Function.java:104) ~[?:?]
at org.opencastproject.util.IoSupport.locked(IoSupport.java:478) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl.get(WorkspaceImpl.java:402) ~[?:?]
at org.opencastproject.workspace.impl.WorkspaceImpl.read(WorkspaceImpl.java:437) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.loadDublinCore(DublinCoreUtil.java:76) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.lambda$loadEpisodeDublinCore$0(DublinCoreUtil.java:64) ~[?:?]
at java.util.Optional.map(Optional.java:265) ~[?:?]
at org.opencastproject.metadata.dublincore.DublinCoreUtil.loadEpisodeDublinCore(DublinCoreUtil.java:64) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.indexMediaPackage(SearchServiceIndex.java:255) ~[?:?]
at org.opencastproject.search.impl.SearchServiceIndex.lambda$repopulate$3(SearchServiceIndex.java:471) ~[?:?]
... 12 more
2024-08-23T09:46:38,582 | INFO | (IndexRebuildService:214) - Finished rebuilding the Search index
2024-08-23T09:54:07,803 | INFO | (StaticResourceServlet:116) - Added static file authorization for [^/([^/]+)/oaipmh-default/.*$]

How can I or from where do I have to delete this failing event so the re-indexing runs successfully? 


Huber, Daniel

unread,
Aug 26, 2024, 11:17:55 AM8/26/24
to us...@opencast.org
Hi,

I had a similar issue while rebuilding the index after an update from 14.x to 16.1. The rebuild failed due to some events no longer existing on the file system. For some reason, in the oc_search database table the deletion_date field was set to NULL while I guess it should be set to some date for deleted events. As this problem only affected our test system, I took the risk and manually added some timestamps for the events I could not find on the file system. Afterwards, the index rebuild finished successfully.

Not sure if I would do the same on a productive system, thou. Especially as the events in question still show up in the admin UI for some reason. So something still seems to be odd but I had no time to take a closer look, yet.

Best regards,

Daniel


________________________________________
Von: 's.la...@googlemail.com' via Opencast Users <us...@opencast.org>
Gesendet: Freitag, 23. August 2024 12:52:34
An: Opencast Users
Betreff: [OC Users] Search index rebuild errors

Hi,

I updated our opencast test cluster to latest 16.4 (coming from 15).
After I figured out, how to configure elasticsearch correctly so that the presentation node can access it, I was able to start opencast without further start up errors.

I subsequently started to rebuild the indexes which yielded the following results:

* Series index rebuild without error
* Scheduler rebuild without error
* AssetManager rebuild WITH errors caused by missing events but finished successfully
* Comments index rebuild WITH errors caused by missing events but finished successfully
* Workflow index rebuild without error
* Search index rebuild WITH warnings and errors caused by missing events
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org<mailto:users+un...@opencast.org>.

s.la...@googlemail.com

unread,
Aug 29, 2024, 8:13:26 AM8/29/24
to Opencast Users
Hi Daniel,

thanks for sharing your solution, which was really helpful. I needed to fix a few events, which where not available anymore.
So I did an "iterative rebuild", running the rebuild until an error occured, set delete date of the event in the database and re-running the rebuild.

Regards
Sven

Reply all
Reply to author
Forward
0 new messages