URGENT - Publish workflow fails after update 3.6 > 4.4

[Sven Laudel]

Hi,

we recently upgraded our Opencast installation from 3.6 to 4.4.

Except for a minor database problem everything seemed to went well and it looks functional.

But unfortunately some recordings, which were already published, disappeared. I don't know where they have gone, because in the Admin interface they are shown as publish, but when one tries to watch them, they are not found. 

I'm trying to republish them without any luck. 

Also after ingesting a new recording the whole workflow comes to a halt at "publish-engage" and quits with an error.

Log from presentation:

2018-09-10 10:34:00,679 | INFO  | (WorkspaceImpl:392) - Downloading https://urzopencastadm2.url:8443/files/mediapackage/38977ce2-ac22-416e-9faa-78c1d9987e9f/7df5af75-fd23-43c5-b9b4-d01260bcf925/concat.mp4 to /opt/opencast/data/prod/workspace/https_urzopencastadm2.url_8443/files/mediapackage/38977ce2-ac22-416e-9faa-78c1d9987e9f/7df5af75-fd23-43c5-b9b4-d01260bcf925/concat.mp4

2018-09-10 10:35:00,176 | WARN  | (DownloadDistributionServiceImpl:359) - Error distributing https://urzopencastadm2.url:8443/files/mediapackage/ceca977c-2960-4b62-a37b-b720b7c31a34/0af871ad-c5d8-4f36-8348-da6036136785/concat.mp4

org.opencastproject.security.api.TrustedHttpClientException: java.net.SocketTimeoutException: Read timed out

        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:392)[69:matterhorn-kernel:4.4.0]

        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:345)[69:matterhorn-kernel:4.4.0]

Log from admin:

2018-09-10 10:33:45,788 | INFO  | (StreamingDistributionServiceRemoteImpl:94) - Distributing 1 elements to engage-player@streaming

2018-09-10 10:35:03,419 | ERROR | (WorkflowOperationWorker:182) - Workflow operation 'operation:'publish-engage', position:39, state:'FAILED'' failed

org.opencastproject.workflow.api.WorkflowOperationException: One of the distribution jobs did not complete successfully

        at org.opencastproject.workflow.handler.distribution.PublishEngageWorkflowOperationHandler.start(PublishEngageWorkflowOperationHandler.java:357)[70:matterhorn-distribution-workflowoperation:4.4.0]

I pasted the complete log from the presentation node to https://pastebin.com/Q1qYMVcs and from the admin node to https://pastebin.com/ma1kEHUn.

First opencast distributes the recordings to the streaming server successfully. After the it distributes the same recordings to the engage download where it fails.

For me it looks like Opencast tries to connect to some service which is not available or which does not answer?

i don't have a clue what is not working.

Does anyone have any idea what i could try to get it working again?

Thanks in advance an best regards

Sven

[kdo...@dce.harvard.edu]

Hi Sven,

Is there any chance that the downloading is happening from the same server that has the DNS https://urzopencastadm2.url?

I ask because the timeout might be port related. See accepted answer here: https://stackoverflow.com/questions/32428141/wget-for-https-website-from-dedicated-server

In other words, try get the "https://urzopencastadm2.url:8443/files/mediapackage/ceca977c-2960-4b62-a37b-b720b7c31a34/0af871ad-c5d8-4f36-8348-da6036136785/concat.mp4" file using the command line on the server that has the download service to make sure it’s not a network permission issue.

Best of luck,
Karen

> --
> You received this message because you are subscribed to the Google Groups "Opencast Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

[Sven Laudel]

Hi Karen,

just tried to curl:

[root@urzopencastengage2 user]# curl -sSLO https://urzopencastadm2.url:8443/files/mediapackage/ceca977c-2960-4b62-a37b-b720b7c31a34/0af871ad-c5d8-4f36-8348-da6036136785/concat.mp4

curl: (56) Received HTTP code 503 from proxy after CONNECT

Ok, we are using proxy to connect to the outside world, but this setting was active before the update to 4.4.

When using the following curl i get just a html document which says some thing like "not logged in".

[root@urzopencastengage2 user]# curl -k --noproxy urzopencastadm2.url -sSLO https://urzopencastadm2.url:8443/files/mediapackage/ceca977c-2960-4b62-a37b-b720b7c31a34/0af871ad-c5d8-4f36-8348-da6036136785/concat.mp4

But what i'm wondering about is why does the distribution to streaming still work? 

So does opencast honor proxy environment settings and if so, what should i do?

 

Best regards

Sven

[kdo...@dce.harvard.edu]

> just tried to curl:
> [root@urzopencastengage2 user]# curl -sSLO https://urzopencastadm2.url:8443/files/mediapackage/ceca977c-2960-4b62-a37b-b720b7c31a34/0af871ad-c5d8-4f36-8348-da6036136785/concat.mp4
> curl: (56) Received HTTP code 503 from proxy after CONNECT
>
> Ok, we are using proxy to connect to the outside world, but this setting was active before the update to 4.4.

If these are new servers, verify that their iptables are set up the same way as your 3.6 server’s iptables.
Reference: https://docs.opencast.org/develop/admin/configuration/https/opencast.only/#port-forwarding-required
(I don’t see that page in r/4.x, but it should be equally applicable)

- Karen

[Sven Laudel]

Thanks for your recommendations.

Unfortunately the servers are the same , i didn't change them.

I just uninstalled opencast3, made an update to the underlying centos7.

Afterwards I installed opencast4 and configured it accordingly.

Best regards

Sven

[Sebastian Schulze]

Hi Sven,

"made an update to the underlying centos7"

Maybe an SELinux-related problem?

Sebastian

[Greg Logan]

Hi Sven,

HTTP 503 is saying that the backend service is unavailable.  Is the *proxy* configuration correct as well?

G

[Sven Laudel]

Hi Sebastian,

i don't think so, because SELinux is set to permissive.

Best regards

Sven

[Sven Laudel]

Hi Greg,

thanks for your recommendation.

I just added a no_proxy rule to the environment, so the admin can be reached without the proxy.

This must be configured, but unfortunately it didn't solve the problem.

It is still the same error.

By the way, before the upgrade to 4.4 it worked flawlessly.

Best regards

Sven

[Dietmar Zenker]

Hi Sven,

just a vague presumption (as you stated that you had updated the OS): checking your logs I've noticed that your opencast nodes are running on https alt port 8443 - did you check if your config in org.ops4j.pax.web.cfg is correct and your keystore and associated passwords are valid? Can you access endpoints of one node from another node?

Cheers,

Dietmar

[Sven Laudel]

Hi,

i just checked it and i think it's configured correctly.

What makes me wonder is, every workflow task is running correctly even the distribution to the streaming server but not the engage download task. I don't understand, why this fails.

I tried the following in my workflow, i disabled the engage download

    <operation

     id="publish-engage"

     if="${publishToEngage}"

     max-attempts="2"

     exception-handler-workflow="ng-partial-error"

     description="Publishing to Opencast Media Module">

     <configurations>

<!--        <configuration key="download-source-flavors">dublincore/*,security/*</configuration>

       <configuration key="download-source-tags">engage-download,atom,rss,mobile</configuration>-->

       <configuration key="streaming-source-tags">engage-streaming</configuration>

       <configuration key="check-availability">true</configuration>

     </configurations>

   </operation>

Now the workflow runs through smoothly and the recording is in published state.

But it can not be watched, because "No compatible source was found for this media.".

How should theodul be configured to show the streaming version of the recording? This would be  work around, but not fix the problem.

I also tried to delete some recordings, but without any success. They remain visible in the admin ui, even retracting a recording did not work.

Best regards

Sven 

[Dietmar Zenker]

Hi Sven,

as a new dedicated Wowza streaming module was introduced with Opencast 4 that replaced the former matterhorn-distribution-service-streaming module, is it possible that this is incorrectly configured? (pls. refer to the corresponding docu)

Again, just a a vague presumption...

Cheers,

Dietmar

[Karen Dolan]

Sven,

I notice that the coverimage.png (small image file) appears to have downloaded Ok and not caused a "SocketTimeoutException: Read timed out READ",  just prior to the mp4 socket read timeout. If this that socket read timeout is only happening on the mp4, there may be a keep-alive or some other web/proxy timeout param that needs adjustment in your 4.4.

2018-09-10 10:34:00,098 | INFO  | (WorkspaceImpl:392) - Downloading https://urzopencastadm2.url:8443/files/mediapackage/38977ce2-ac22-416e-9faa-78c1d9987e9f/fba44b89-1a49-4cc9-927c-a436d0751237/coverimage.png to /opt/opencast/data/prod/workspace/https_urzopencastadm2.url_8443/files/mediapackage/38977ce2-ac22-416e-9faa-78c1d9987e9f/fba44b89-1a49-4cc9-927c-a436d0751237/coverimage.png   <-- downloaded OK? the WorkspaceImpl looped to downloading the mp4 right after this.

2018-09-10 10:34:00,679 | INFO  | (WorkspaceImpl:392) - Downloading https://urzopencastadm2.url:8443/files/mediapackage/38977ce2-ac22-416e-9faa-78c1d9987e9f/7df5af75-fd23-43c5-b9b4-d01260bcf925/concat.mp4 to /opt/opencast/data/prod/workspace/https_urzopencastadm2.url_8443/files/mediapackage/38977ce2-ac22-416e-9faa-78c1d9987e9f/7df5af75-fd23-43c5-b9b4-d01260bcf925/concat.mp4

2018-09-10 10:35:00,176 | WARN  | (DownloadDistributionServiceImpl:359) - Error distributing https://urzopencastadm2.url:8443/files/mediapackage/ceca977c-2960-4b62-a37b-b720b7c31a34/0af871ad-c5d8-4f36-8348-da6036136785/concat.mp4

org.opencastproject.security.api.TrustedHttpClientException: java.net.SocketTimeoutException: Read timed out

        at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:392)[69:matterhorn-kernel:4.4.0]

https://pastebin.com/Q1qYMVcs

- Karen

[Sven Laudel]

Hi Dietmar,

as far as is can see the configuration seems straight forward.

######### STREAMING AND DOWNLOAD #########

# The base URL of the streaming server (ususally "rtmp://<SERVER_URL>/matterhorn-engage").

# ${org.opencastproject.server.url} can not be used, because the streaming server does not use the HTTP protocol.

# Streaming is not included in the default workflow, since the Red5 streaming server is a 3rd party component that

# requires separate installation.

org.opencastproject.streaming.url=rtmp://urzopencaststream1.url/opencast

# The directory where Opencast stores the streams

org.opencastproject.streaming.directory=/opt/opencast/data/prod/streams

# The port to use for the streaming server, only used by a Wowza streaming server.

org.opencastproject.streaming.port=1935

# Server URL for Adaptive Streaming, usually the same server like RTMP streaming with a different protocol

org.opencastproject.adaptive-streaming.url=http://urzopencaststream1.url/opencast

# Port for Adaptive Streaming (for Wowza usually 1935)

org.opencastproject.adaptive-streaming.port=1935

# Some newer streaming server versions expect an "flv:" tag within the rtmp URL.

# Not every RTMP-streaming server is compatible with this (i.e. nginx), so this

# is the compatibility mode to the old syntax.

# true  = without "flv:" tag - old syntax

# false = with    "flv:" tag - new syntax

#org.opencastproject.streaming.flvcompatibility=true

Regards

Sven

[Dietmar Zenker]

Hi Sven,

you have reported in the german-speaking group that you had "played around with the NFS mount options". As Karen stated, small files are downloaded correctly, but large files fail. Thus, maybe there is a problem with your NFS share that causes problems?

Greets,

Dietmar