Processing new events fails at final 'cleanup' step

[m b]

Was just about pulling my hair out trying to figure out the problem yesterday. 

Nothing in my config has changed to my knowledge, and opencast has been working fine. One day events are processing fine, the next they hang at the final step. 

It looks like it hangs here:

2022-06-01T00:33:06,851 | INFO  | (CleanupWorkflowOperationHandler:287) - Successfully deleted external URI http://opencast:8080/files/mediapackage/324be986-a700-4b96-ae0e-805e76f26f4d/86b1a8b1-b690-41fe-ba9a-6f3dccd59446
2022-06-01T00:33:06,852 | WARN  | (TrustedHttpClientImpl:597) - URI http://opencast/files/mediapackage/324be986-a700-4b96-ae0e-805e76f26f4d/86b1a8b1-b690-41fe-ba9a-6f3dccd59446 does not support digest authentication
2022-06-01T00:33:06,853 | INFO  | (CleanupWorkflowOperationHandler:291) - Unable to delete external URI http://opencast/files/mediapackage/324be986-a700-4b96-ae0e-805e76f26f4d/86b1a8b1-b690-41fe-ba9a-6f3dccd59446, status code '301' returned
2022-06-01T00:37:27,959 | INFO  | (WorkingFileRepositoryImpl:226) - [>29579069] Unable to delete non existing media package element 86b1a8b1-b690-41fe-ba9a-6f3dccd59446@324be986-a700-4b96-ae0e-805e76f26f4d
2022-06-01T00:37:27,981 | INFO  | (CleanupWorkflowOperationHandler:287) - Successfully deleted external URI http://opencast:8080/files/mediapackage/324be986-a700-4b96-ae0e-805e76f26f4d/1f7902e8-7236-458c-9903-19876ada8216
2022-06-01T00:37:27,982 | WARN  | (TrustedHttpClientImpl:597) - URI http://opencast/files/mediapackage/324be986-a700-4b96-ae0e-805e76f26f4d/1f7902e8-7236-458c-9903-19876ada8216 does not support digest authentication
2022-06-01T00:37:27,983 | INFO  | (CleanupWorkflowOperationHandler:291) - Unable to delete external URI http://opencast/files/mediapackage/324be986-a700-4b96-ae0e-805e76f26f4d/1f7902e8-7236-458c-9903-19876ada8216, status code '301' returned

When I navigate to that URL I get "The resource you requested does not exist."

Full logs are attached. If anyone has any suggestions on what to do, that would be appreciated.

(The logs include a fresh start of the service, hitting some errors on a couple stuck jobs, deleting those jobs, and then starting a brand new job, which seems to run smoothly all the way until hanging at the end)

[Lars Kiesow]

Hi m,
what's failing according to the logs is the `image` operation. Does
cleanup also fail? Or does it just give you the warnings and is
potentially slow?

Do you use a shared file system for all your Opencast nodes? If so, you
can speed up the cleanup operation and get rid of the warnings easily
by setting `delete-external=false` on that operation [1]. But you
should only do that with a shared file system.

Best regards,
Lars

[1]
https://docs.opencast.org/r/11.x/admin/#workflowoperationhandlers/cleanup-woh/

[m b]

Thanks for the response. I don't use a shared file system. I wonder why the image operation started failing? Clean up doesn't fail outright, it just hangs indefinitely. After spending a few too many hours on this, I ended up just installing from scratch on a new box, and it is working out of the gate. Just need to get all the content uploaded. I am curious what caused this though if there is a simple answer.

[m b]

Finding myself stuck once again. I uploaded a bunch of events. Think I ran out of disk space. Added some more. All events have workflows that show succeeded in the workflow details page and via /workflows/ endpoint. However in the GUI and via the /events/ endpoint they have 

workflow_state: "RUNNING",

And I'm trying to figure out how I can get these to update, as I'd like to be able to republish metadata in the future.

Any thoughts?

[Greg Logan]

Hi m,

Running out of disk space can cause all kinds of weirdness.  You could try manually stopping the workflow(s) with the workflow endpoint[1].  Failing that, rebuilding the index[2] should take care of things, at the cost of a longer rebuild time depending on what volume of data you have in the system already.

G

1: https://stable.opencast.org/docs.html?path=/workflow#stop-5

2: https://stable.opencast.org/docs.html?path=/index

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.