Workflow hanging

[Hartmut Trüe]

Hello,

I keep finding hanging workflows, sometimes they block further processing. In the logfile I find exceptions like the one shown below.

To clean up I killed the jobs manually later.

It is Opencast 13.4.0

Any suggestions?

Regards,

Hartmut

2023-04-21T12:04:42,053 | WARN  | (WorkflowServiceImpl:1757) - [>a761ee49] Exception while accepting job Job {id:1435585, operation:START_OPERATION, status:RUNNING}
2023-04-21T12:04:42,053 | WARN  | (WorkflowServiceImpl:1765) - [>a761ee49] Unable to parse workflow instance
org.opencastproject.serviceregistry.api.ServiceRegistryException: This argument list for operation 'START_OPERATION' does not meet expectations
        at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1749) [!/:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2101) [!/:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2067) [!/:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
        at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
        at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
        at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
        at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
        at java.util.Collections$UnmodifiableList.get(Collections.java:1310) ~[?:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1730) ~[?:?]
        ... 6 more
2023-04-21T12:04:44,656 | INFO  | (AnalyzeTracksWorkflowOperationHandler:79) - Running analyze-tracks workflow operation on workflow 1435556
2023-04-21T12:04:44,657 | INFO  | (AnalyzeTracksWorkflowOperationHandler:120) - Finished analyze-tracks workflow operation adding the properties: {presenter_work_aspect=16/9, presenter_work_video=true, presenter_work_framerate=23.976025, presenter_work_media=true, presenter_work_resolution_x=1920, presenter_work_resolution_y=1080, presenter_work_audio=true}

[Greg Logan]

Hi Hartmut,

Did you upgrade Opencast between that workflow starting, and the error?  We would need more information about job 1435585 to really be able to tell what's going on.

G

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

[Hartmut Trüe]

Hi Greg,

no, there was no upgrade or restart. There where 20 videos in process, and one job was blocking. I deleted that job via API and the others continued. In the end 19 videos were successfully processed, one video stayed in status "processing" and some jobs were left over, but nothing happened anymore. So I killed that jobs later, the processing failed and I restarted the processing of the failed video again. After that it ran successfully.

Hartmut

[Hartmut Trüe]

... another two jobs hanging since yesterday 16:09. I killed the jobs, and processed one after the other with the publish workflow. Now both successfully processed. Looks like an issue with parallel processing.

2023-04-26T16:09:42,130 | WARN  | (WorkflowServiceImpl:1757) - [>1f9497e8] Exception while accepting job Job {id:1447613, operation:START_OPERATION, status:RUNNING}
2023-04-26T16:09:42,131 | WARN  | (WorkflowServiceImpl:1765) - [>1f9497e8] Unable to parse workflow instance

org.opencastproject.serviceregistry.api.ServiceRegistryException: This argument list for operation 'START_OPERATION' does not meet expectations
        at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1749) [!/:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2101) [!/:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2067) [!/:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
        at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
        at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
        at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
        at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
        at java.util.Collections$UnmodifiableList.get(Collections.java:1310) ~[?:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1730) ~[?:?]
        ... 6 more

2023-04-26T16:09:42,345 | WARN  | (WorkflowServiceImpl:1757) - [>1f9497e8] Exception while accepting job Job {id:1447614, operation:START_OPERATION, status:RUNNING}
2023-04-26T16:09:42,345 | WARN  | (WorkflowServiceImpl:1765) - [>1f9497e8] Unable to parse workflow instance

org.opencastproject.serviceregistry.api.ServiceRegistryException: This argument list for operation 'START_OPERATION' does not meet expectations
        at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1749) [!/:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2101) [!/:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2067) [!/:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
        at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
        at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
        at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
        at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
        at java.util.Collections$UnmodifiableList.get(Collections.java:1310) ~[?:?]
        at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1730) ~[?:?]
        ... 6 more