Events ending up as empty lines in Admin UI after OC 11 reindexing
itz.feli...@gmail.com
/b53285e0-f9d7-4b3e-9d21-58445a20c2ca$ ls -lha 1
total 7.6M
drwxr-xr-x 2 800 800 4.0K Jun 16 2021 .
drwxr-xr-x 4 800 800 24 Jun 16 2021 ..
-rw-r--r-- 3 800 800 58K Jun 16 2021 09bd2649-7847-4c2d-91cb-51d59c943b46.png
-rw-r--r-- 2 800 800 843K Jun 16 2021 265179a3-437b-4fdb-8623-0ee35016022a.m4a
-rw-r--r-- 4 800 800 1.8M Jun 16 2021 46343293-9575-4f12-8324-5a0265fb7e7b.mp4
-rw-r--r-- 2 800 800 671K Jun 16 2021 dbeb76f7-beb0-4177-b91e-f20ecc63641f.ogg
-rw-r--r-- 2 800 800 826K Jun 16 2021 fafd9891-1b78-4073-89a5-b4d052406dde.mp3
-rw-r--r-- 1 800 800 7.3K Jun 16 2021 manifest.xml
-rw-r--r-- 1 800 800 1.4K Jun 16 2021 s-a7237e14-5b25-45bf-a97e-053a5709ce48.smil
-rw-r--r-- 4 800 800 1.8M Jun 16 2021 track-16.mp4
-rw-r--r-- 4 800 800 1.8M Jun 16 2021 track-17.mp4
Can missing DC catalogs somehow be generated from the manifest.xml (where at least a title is denoted)?
<mediapackage duration="56400" id="b53285e0-f9d7-4b3e-9d21-58445a20c2ca" start="2021-06-16T13:36:16Z" xmlns="http://mediapackage.opencastproject.org">
<title>Audioonly Tests (+Laud,-Vol,+Editor,3)</title>
<media>
<track id="fafd9891-1b78-4073-89a5-b4d052406dde" type="audioonly/audio">
<mimetype>audio/mpeg</mimetype>
<tags>
<tag>archive</tag>
<tag>mp3</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:fafd9891-1b78-4073-89a5-b4d052406dde:out.mp3</url>
<size>845668</size>
<checksum type="md5">6c659d7767b07149302a46c93563e98f</checksum>
<duration>52819</duration><audio id="audio-1">
<device />
<encoder type="MP3 (MPEG audio layer 3)" />
<channels>2</channels>
<samplingrate>44100</samplingrate>
<bitrate>128000.0</bitrate>
</audio>
<live>false</live>
</track>
<track id="dbeb76f7-beb0-4177-b91e-f20ecc63641f" type="audioonly/audio">
<mimetype>audio/ogg</mimetype>
<tags>
<tag>archive</tag>
<tag>opus</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:dbeb76f7-beb0-4177-b91e-f20ecc63641f:out.ogg</url>
<size>686632</size>
<checksum type="md5">16c0c1a6bb7c72e7b3422c5e910672fb</checksum>
<duration>52791</duration><audio id="audio-1">
<device />
<encoder type="Opus (Opus Interactive Audio Codec)" />
<channels>2</channels>
<samplingrate>48000</samplingrate>
</audio>
<live>false</live>
</track>
<track id="265179a3-437b-4fdb-8623-0ee35016022a" type="audioonly/audio">
<mimetype>audio/m4a</mimetype>
<tags>
<tag>archive</tag>
<tag>m4a</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:265179a3-437b-4fdb-8623-0ee35016022a:out.m4a</url>
<size>862804</size>
<checksum type="md5">01878143716ec44356699b75470bd023</checksum>
<duration>52808</duration><audio id="audio-1">
<device />
<encoder type="AAC (Advanced Audio Coding)" />
<framecount>2275</framecount>
<channels>2</channels>
<samplingrate>44100</samplingrate>
<bitrate>129207.0</bitrate>
</audio>
<live>false</live>
</track>
<track id="46343293-9575-4f12-8324-5a0265fb7e7b" type="audioonly/video">
<mimetype>video/mp4</mimetype>
<tags>
<tag>archive</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:46343293-9575-4f12-8324-5a0265fb7e7b:result.mp4</url>
<size>1804827</size>
<checksum type="md5">8f071967c3f00bc56b4c35f4a8732f6a</checksum>
<duration>56400</duration><audio id="audio-1">
<device />
<encoder type="AAC (Advanced Audio Coding)" />
<framecount>2275</framecount>
<channels>2</channels>
<samplingrate>44100</samplingrate>
<bitrate>129252.0</bitrate>
</audio><video id="video-1">
<device />
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10" />
<framecount>564</framecount>
<bitrate>131387.0</bitrate>
<framerate>10.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
<track id="track-16" type="presenter/work" ref="track:46343293-9575-4f12-8324-5a0265fb7e7b">
<mimetype>video/mp4</mimetype>
<tags>
<tag>archive</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:track-16:result.mp4</url>
<size>1804827</size>
<checksum type="md5">8f071967c3f00bc56b4c35f4a8732f6a</checksum>
<duration>56400</duration><audio id="audio-1">
<device />
<encoder type="AAC (Advanced Audio Coding)" />
<framecount>2275</framecount>
<channels>2</channels>
<samplingrate>44100</samplingrate>
<bitrate>129252.0</bitrate>
</audio><video id="video-1">
<device />
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10" />
<framecount>564</framecount>
<bitrate>131387.0</bitrate>
<framerate>10.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
<track id="track-17" type="presenter/source" ref="track:track-16">
<mimetype>video/mp4</mimetype>
<tags>
<tag>archive</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:track-17:result.mp4</url>
<size>1804827</size>
<checksum type="md5">8f071967c3f00bc56b4c35f4a8732f6a</checksum>
<duration>56400</duration><audio id="audio-1">
<device />
<encoder type="AAC (Advanced Audio Coding)" />
<framecount>2275</framecount>
<channels>2</channels>
<samplingrate>44100</samplingrate>
<bitrate>129252.0</bitrate>
</audio><video id="video-1">
<device />
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10" />
<framecount>564</framecount>
<bitrate>131387.0</bitrate>
<framerate>10.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
</media>
<metadata>
<catalog id="s-a7237e14-5b25-45bf-a97e-053a5709ce48" type="presenter/silence">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:s-a7237e14-5b25-45bf-a97e-053a5709ce48:smil.smil</url>
<checksum type="md5">1565ce19c67cbb10538d7ee6730df657</checksum>
</catalog>
</metadata>
<attachments>
<attachment id="09bd2649-7847-4c2d-91cb-51d59c943b46" type="audioonly/image">
<mimetype>image/png</mimetype>
<tags>
<tag>archive</tag>
<tag>waveform</tag>
</tags>
<url>urn:matterhorn:b53285e0-f9d7-4b3e-9d21-58445a20c2ca:1:09bd2649-7847-4c2d-91cb-51d59c943b46:waveform.png</url>
<checksum type="md5">49138213143fd087794cca394864b2b2</checksum>
</attachment>
</attachments>
<publications>
<publication id="6da679f1-83de-4620-90e7-5fb8287105f2" channel="engage-player">
<mimetype>text/html</mimetype>
<tags />
<url>https://staging-engage.opencast.example.com/play/b53285e0-f9d7-4b3e-9d21-58445a20c2ca</url>
<media />
<attachments />
<metadata />
</publication>
<publication id="e2a20e8c-0df3-4f2b-9a88-48db52bed5a4" channel="internal">
<tags />
<url>/admin-ng/index.html#/events/events/b53285e0-f9d7-4b3e-9d21-58445a20c2ca/tools/editor</url>
<media>
<track id="1d13a31c-962a-4506-8d4d-c37d5e98c86e" type="presenter/preview" ref="track:track-16">
<mimetype>video/mp4</mimetype>
<tags>
<tag>preview</tag>
</tags>
<url>https://staging-engage.opencast.example.com/static/mh_default_org/internal/b53285e0-f9d7-4b3e-9d21-58445a20c2ca/85d46957-639f-40ec-a6fd-c435536c2234/result.mp4</url>
<size>497278</size>
<checksum type="md5">8e8e3f17994bdacf8461b8472d99faa0</checksum>
<duration>56400</duration><audio id="audio-1">
<device />
<encoder type="AAC (Advanced Audio Coding)" />
<framecount>1138</framecount>
<channels>2</channels>
<samplingrate>22050</samplingrate>
<bitrate>64706.0</bitrate>
</audio><video id="video-1">
<device />
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10" />
<framecount>564</framecount>
<bitrate>7865.0</bitrate>
<framerate>10.0</framerate>
<resolution>640x360</resolution>
</video>
<live>false</live>
</track>
</media>
<attachments>
<attachment id="3cd9ab44-50c0-4bf9-a266-4fc9b46dddd9" type="presenter/audio+preview">
<mimetype>image/png</mimetype>
<tags>
<tag>preview</tag>
</tags>
<url>https://staging-engage.opencast.example.com/static/mh_default_org/internal/b53285e0-f9d7-4b3e-9d21-58445a20c2ca/9b92e7e0-9490-40db-a9f6-efedabca66d6/waveform.png</url>
</attachment>
<attachment id="3b899ab9-debf-426f-8b37-5359c3542b5d" type="presenter/video+preview" ref="track:track-16">
<mimetype>image/jpeg</mimetype>
<tags>
<tag>preview</tag>
</tags>
<url>https://staging-engage.opencast.example.com/static/mh_default_org/internal/b53285e0-f9d7-4b3e-9d21-58445a20c2ca/56040515-bc7e-45bb-a704-313b356efdf8/result_50_0p_preview.jpg</url>
<size>66723</size>
</attachment>
<attachment id="351ddacd-511f-459b-a595-598234fd27e6" type="presenter/waveform">
<mimetype>image/png</mimetype>
<tags>
<tag>preview</tag>
</tags>
<url>https://staging-engage.opencast.example.com/static/mh_default_org/internal/b53285e0-f9d7-4b3e-9d21-58445a20c2ca/29ed02da-c620-4bd4-b9a3-08c6ceb3ab5f/waveform.png</url>
</attachment>
</attachments>
<metadata />
</publication>
</publications>
</mediapackage>
Greg Logan
--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.
Maximiliano Lira Del Canto
Matthias Vollroth
Andreas Lindner
Hello everyone,
as Matthias already wrote, we have quite a lot of empty entries in our Admin – UI.
Our System:
We are currently running OC 11.8 and updated in September from OC 8.5.
We have 3 Server: admin – node, worker – node and a presentation node.
Our Problem:
These videos can be viewed with the paella player without problems.
Even the Metadata gets displayed inside the paella player.
But if you want to do anything to these videos except watching them, these entries do not work in any way, shape or form.
(Standalone Video Editor):
This is a big Problem for us, as we want to enable our lecturers to edit their videos by themselves.
But currently videos with empty entries in the admin – ui, cannot be edited at all.
We tested a lot of these videos and with all of them, only the preview video and the video title can be loaded.
Everything else will not be loaded (video – waveform, video – length, smil catalogue, metadata,…).
As the metadata fields inside the standalone video editor are empty, we tried to change the metadata and publish the video with the new information. After this we receive a 500 (Server Error).
ERROR:
POST https://<admin-node>/editor/a21a169f-7f21-4562-9a5d-ee1d6fda62fd/metadata.json 500 (Server Error)
This error does not leave an entry in our worker – log or in our admin – log, but does leave a warning in our presentation – node (see attachment).
2022-11-07T09:10:07,715 | WARN | (WebApplicationExceptionMapper:73) - javax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error
(Admin – UI):
In addition to the standalone video editor, changing information inside the admin – ui does not work either.
Neither Metadata changes, nor changes to the ACL can be applied, although we receive the “metadata saved” message.
But if we close the video details window and open it again, all changes are lost.
Republishing or deleting these videos does not work (“republish metadata”, “publish” and “delete” workflow).
As this is our production system, I cannot simply take one of these videos and run tests, but the one video that I could run some tests on, Opencast wrote an ERROR into the log:
2022-10-26T14:06:44,892 | ERROR | (DublinCoreUtil:100) - Unable to load metadata from catalogue 'https://<admin-node>/assets/assets/eccad833-b6ec-4007-bb54-45a999a82f5f/d759521b-c413-407f-a855-0230ac5b686b/6/dublincore.xml'
But I cannot be sure if this error is the real culprit, as the video was processed by a lecturer, and I only go this error after the "publish" workflow was started… (the whole error – log can be found in the attachments), and I cannot even be sure how the database entry of this video was looking like, as I cannot access this video anymore.
The only operation that does work is to retract a video (“retract” – workflow).
Further Investigation:
(Database):
These videos (with empty entries in the admin – ui) have a corresponding entry inside our database.
While these videos contain a similar number of mediapackage.xml files (compared to videos that used the same workflow, belong to the same series and come from the same source), these mediapackage.xml files themselves are shorter.
After checking these mefdiapackage.xml files, I found that they contain all the needed metadata for these videos.
It seems like the database should not be our problem.
(File System):
Because of this I continued with our filesystem.
As far as I
can tell, the video files are located in 4 directories:
/mh_default_org/api
/mh_default_org/engage-player
/mh_default_org/internal
/mh_default_org/oaipmh-default
These directories contain subdirectories, each with their own associated file.
(These subdirectories get saved in some mediapackage.xml for reference).
We found that /api, /engage-player and /oaipmh-default contain at least one doublincore.xml file.
But /internal is missing this doublincore.xml file.
Currently we are suspecting that this is the reason for these empty admin – ui entries as I suspect /internal may correlate to the admin – ui.
Our questions:
So, we still have the doublincore.xml files with all needed metadata, but we cannot get this data into the admin – ui.
Greg wrote that we would go through these recordings to fix the metadata, but I do not understand how to do this, as the workflows do not work (I could only test this with one video though, as I cannot simply risk disabling a video that may still be used).
Is there another way to change the metadata?
Or is it as simple as copying a directory from e.g. /mh_default_org/api/<uuid> to /mh_default_org/internal/<uuid>?
But these directories will then be missing inside the mediapackage.xml file in our database, and I would suspect this would not work?
Can anyone
help me with this problem?
Take care,
Andreas | TH Deggendorf
Dietmar Zenker
Andreas Lindner
sorry, I actually forgot to add that.
basically we have this structure:
-srv
I hope this helps.
Take care,
Andreas | TH Deggendorf
Andreas Lindner
We managed
to solve our problem.
So we have about 4000 videos that showed up as empty lines inside the admin-ui.
But the opencast media portal and the paella player could access the metadata, strangely enough neither the Opencast editor, nor the Standalone Editor did work.
If we wanted to run a workflow on one of these videos like “publish” or “republish-metadata” these workflows failed.
In addition, we could not use the api to get any metadata from any of the empty videos with /api/events/<video-id>
We always received
the following error (API, workflows, metadata):
Exception message WorkflowOperationException: org.opencastproject.assetmanager.api.AssetManagerException: An asset with checksum ec2ade4e91e4d277ca4b1988f03a4781 (md5) has already been archived but trying to copy or link asset StoragePath(orgId=mh_default_org, mpId=8ff51d53-a7b3-4b06-bd3b-3a8eb646a886, version=3, mpeId=5a4f80c4-0f65-4d2b-b55a-60617e241711) to it failed
We think this error first occurred during our update from opencast 8.5 to opencast 11.8.
After quite some investigation we concluded that this problem arose from the archive and the stored snapshots.
Let me explain:
Usually, the archive stores all snapshots that are created for one video.
As we had a very “custom” workflow sometimes these snapshots contain 6 video files. In addition, we saw that these snapshots were sometimes not referencing other snapshots, as some video files were edited.
In addition to that, many videos were in some way, shape or form changed and stored upwards of 10 snapshots.
Because of this we use an asset-delete workflow operation, to reduce our snapshots.
So, after updating opencast the index rebuild did not find the archive snapshot. Although the archive stored a snapshots, the rebuild somehow always searched for the wrong snapshot number.
(if the snapshot was named “4” the index rebuild looked for snapshot “3”)
This is due to the database, where the wrong snapshot is stored.
As we delete previous snapshots, this snapshot cannot be found.
And this produced our empty lines inside the admin – ui.
As these videos were published though, the media portal and the paella player could access the dublincore.xml and extract the metadata.
Our
solution:
Inside the Archive we rename the snapshot (“4” gets renamed to “3”).
After this we republish (“publish” workflow) the video. If you just want to have the metadata back, the “republish-metadata” workflow is insufficient, as the “publish” workflow pulls the video files back into the archives and rewrites the manifest.xml files.
So, with the “publish” workflow everything inside the admin – ui is fine, the publications work and the assets are mapped correct.