Strange problem uploading recordings from Moodle and pyCA

[s.la...@googlemail.com]

Hi,

we are experiencing strange problems with our opencast 13.10 installation.
Everything was fine last semester. While the semester break we upgraded Moodle to 4.1 including the Moodle Opencast tool chain.
After this we had a dozen uploads from moodle so I don't think this caused the problem.

What I changed last, was that I moved the whole nfs based filesystem from the old nfs to a new nfs server with a lot more space and updated the OS of all opencast machines.
I used the following command to rsync the files:
rsync -av -H --delete --numeric-ids /old/ /new/
After the sync I changed the switched the mount points.

Uploading recordings via opencast web frontend seems to work okay.

Problems arise when we try to upload recordings from moodle (block_/tool_opencast, wether it's ingest or api upload) or from our capture agents (pyCA running on windows machines with OBS). As long as the upload takes less then 30 or 60 seconds (didn't figure out why sometimes 30 and else 60) it works. But if the upload files is bigger so the upload takes more time, it will eventually run into an error. This happens wether we upload via ingest endpoint or via api.

Direct upload via opencast frontend from the CA machines works for bigger files and longer times.
So for me it seems like a problem with the api and ingest endpoint.

The error log looks like:
"2023-09-29T14:00:28,909 | INFO | (IngestServiceImpl:666) - Created mediapackage 2cc5a589-4eb8-4f8f-a2be-785c343068d7
2023-09-29T14:00:29,256 | INFO | (IngestServiceImpl:765) - Start adding track cb2196c0-4f6a-41d2-8bbc-b3a773833035 from input stream on mediapackage 2cc5a589-4eb8-4f8f-a2be-785c343068d7
2023-09-29T14:00:59,233 | ERROR | (EventsEndpoint:664) - Unable to create event
org.opencastproject.index.service.exception.IndexServiceException: Unable to parse new event.
at org.opencastproject.index.service.impl.util.EventHttpServletRequest.createFromHttpServletRequest(EventHttpServletRequest.java:187) ~[!/:?]
at org.opencastproject.external.endpoint.EventsEndpoint.createNewEvent(EventsEndpoint.java:641) [!/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
...
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [!/:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [!/:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [!/:9.4.43.v20210629]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [!/:9.4.43.v20210629]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.eclipse.jetty.io.EofException: Early EOF
at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1207) ~[!/:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1195) ~[!/:9.4.43.v20210629]
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:333) ~[!/:9.4.43.v20210629]
at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:1027) ~[?:?]
at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:931) ~[?:?]
at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:102) ~[!/:2.8.0]
at org.opencastproject.util.ProgressInputStream.read(ProgressInputStream.java:84) ~[!/:?]
at java.security.DigestInputStream.read(DigestInputStream.java:162) ~[?:?]
at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1158) ~[!/:2.8.0]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:878) ~[!/:2.8.0]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1135) ~[!/:2.8.0]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:854) ~[!/:2.8.0]
at org.opencastproject.workingfilerepository.impl.WorkingFileRepositoryImpl.put(WorkingFileRepositoryImpl.java:397) ~[?:?]
at org.opencastproject.ingest.impl.IngestServiceImpl.addContentToRepo(IngestServiceImpl.java:1711) ~[?:?]
at org.opencastproject.ingest.impl.IngestServiceImpl.addTrack(IngestServiceImpl.java:771) ~[?:?]
at org.opencastproject.ingest.impl.IngestServiceImpl.addTrack(IngestServiceImpl.java:747) ~[?:?]
at org.opencastproject.index.service.impl.util.EventHttpServletRequest.ingestFile(EventHttpServletRequest.java:215) ~[!/:?]
at org.opencastproject.index.service.impl.util.EventHttpServletRequest.createFromHttpServletRequest(EventHttpServletRequest.java:174) ~[!/:?]
... 133 more
2023-09-29T14:00:59,236 | WARN | (WebApplicationExceptionMapper:73) - javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
at org.opencastproject.external.endpoint.EventsEndpoint.createNewEvent(EventsEndpoint.java:665)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
...
2023-09-29T14:06:41,504 | INFO | (ServiceRegistryJpaImpl:2308) - State reset to NORMAL for service org.opencastproject.ingest on host https://ADMINURL through sanitize method"

Maybe someone experienced similar problems or has any knowledge, what we can do to fix this problem, as lectures will start in a bit more than a week.

Regards

Sven

[Katrin Ihler]

Looks like something timed out and the upload wasn't completed. But I'm not certain where that happened...

Katrin

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

-- 
ELAN e.V.
Heger-Tor-Wall 12
D-49074 Osnabrück
https://elan-ev.de

[Katrin S]

Hi Sven,

the EOF error hints at the video file not being fully uploaded. Have you changed anything in the proxy Server, especially regarding buffering? Does the new NFS Server have restrictions regarding filesize?

Katrin

[s.la...@googlemail.com]

Hi Katrin,

the proxy server for us is our institutional loadbalancing/ssl termination server. I spoke to the administrator of this load balancer and we tried to figure out what was happening together but didn't come to any conclusion.

What I've noticed is that uploads to Opencast are generally very slow, really slow. Manual uploads are slow, but they don't stop after 30 seconds, they can sometimes take 10 minutes, but are then finally successful.

I still need to figure out why the uploads are slow and whether it might be the load balancer. But I'm still surprised at the difference.
The NFS doesn't seem to be the problem, when I moved the file system I at least achieved a write speed of over 100 MB/s.

Sven

[Katrin S]

Hi Sven, 

it's a little bit of a shot in the dark, but what you describe sounds like there is a problem with routing in your network, as if not all packets actually are transferred, in case you are using UDP, you could try using TCP with the NSF server.

Another thing, you probably already know this, but just in case you don't, the Opencast Documentation has an example config for different webservers, that you could check whether there are any obvious differences: https://docs.opencast.org/r/14.x/admin/#configuration/https/nginx/

Katrin