How to import videos from an existing Opencast instance

191 views
Skip to first unread message

Tom Keegasi

unread,
Oct 4, 2023, 8:13:56 AM10/4/23
to Opencast Users
Hello folks.

We'd like to migrate our existing Opencast videos from an outdated 8.5 instance to a fresh 14.4 installation we are currently setting up. Preferably we'd like to do this by re-importing existing videos (Episodes & Series) from the old installation into the new one. In doing so avoiding an in-place upgrade and starting fresh otherwise...

Of note is also that we have used the `publish-configure` workflow-operation extensively to publish videos to custom Opencast channels (eg. ilias, cms) that are then used by other plattforms (ILIAS, Content-Management-System) for embeding Opencast videos via plugins.

TL;DR Version:
We've found a potential solution for importing Videos with publications without having to re-encode, but we wonder if there is an easier way. We do this by fetching the MediaPackage from `[GET] /assets/episode/{EventUUID}` and moving all `<publication>` elements into the root of the MediaPackage.

Full Details:
So far we've learned a few things from the following scripts:
Starting with the `migrate-events-to-another-opencast` helper-script we ran into the problem that our events are currently not published to the engange-player/media-module and as such do not appear in neither the `[GET] /search/series.json` nor the `[GET] /search/episode.json` API output.
Instead, after some searching through other scripts and the REST documentation we've switched to using the `[GET] /api/series` and `[GET] /assets/episode/{EventUUID}` entpoints for fetching the list of series and event MediaPackages instead.

The next issue we ran into was that the example ingest workflow from the `migrate-events-to-another-opencast` helper-script did not seem to download any publications from our ingested MediaPackage, or more specifically the `ingest-download` workflow-operation did not seem to do so, as was verified by looking at the source instance access-log file. After some more digging we tried to add the `publication-to-workspace` workflow-operation in order to remedy this problem, but seemingly to no avail. Neither without, nor with this addition any content under the Mediapackage's `<publications>` element was downloaded from the old instance and thus isn't available to be republished by using a `publish-configure` workflow operation that we also added to the example ingest workflow.
So either we are using/understanding the `publication-channel-to-workspace` workflow operation wrong, or it isn't working as intended. You can see our current example Workflow definition and MediaPackage at the end.

While investigating things further, we also noticed that the MediaPackage that can be extracted from `[GET] /search/episode.json` for videos publised to the engange-player are slightly different from those we get from `[GET] /assets/episode/{EventUUID}`: The MediaPackage from the search endpoint contains an empty `<publications>` element and all (published) media, metadata and attachment elements seem to instead be listed inside the root of the MediaPackage instead.
Thus our current workaround consists of fetching a MediaPackage from the assets api, using an xml library to move all publications into the root of the MediaPackage and then throwing this file into the Ingest api, using our modified import workflow with appropriate `publishTo*` variables set from the available publication channels we detected in the MediaPackage during our other modifications.

In https://github.com/opencast/opencast/pull/3554 it is mentioned that since Opencast 11 'all publicationchannel metadata ist also stored in the mediapackage', but our MediaPackage is from Opencast 8.5, could this be our problem? We did quick compareison of MediaPackages from `[GET] /assets/episode/{EventUUID}` between Opencast 8.5 and 14.4 and did not notice any obvious differences though...

Our modified Ingest workflow:
---
id: import
title: Import and publish media from another Opencast
operations:
- id: publication-channel-to-workspace
description: Move publications into workspace
max-attempts: 1
exception-handler-workflow: partial-error
fail-on-error: 'false'
configurations:
- source-channel: "ilias"
- source-flavors: '*/*'
- source-tags: '*'

- id: ingest-download
description: Download external media package elements
max-attempts: 4
exception-handler-workflow: partial-error
fail-on-error: 'true'
configurations:
- delete-external: 'true'

- id: publish-engage
if: "${publishToEngage}"
fail-on-error: true
exception-handler-workflow: "partial-error"
description: "Publishing to Engage"
configurations:
- download-source-flavors: "dublincore/*,security/*,captions/*"
- download-source-tags: "engage-download"
- streaming-source-tags: "engage-streaming"
- check-availability: false

- id: publish-configure
if: "${publishToCms}"
fail-on-error: true
exception-handler-workflow: "partial-error"
description: "Publishing to CMS channel"
configurations:
- channel-id: cms
- mimetype: "application/json"
- download-source-flavors: "dublincore/*,security/*,captions/*"
- download-source-tags: "engage-download"
- streaming-source-tags: "engage-streaming"
- url-pattern: "${org_org_opencastproject_admin_ui_url}/api/events/${event_id}"
- with-published-elements: false
- check-availability: false

- id: publish-configure
if: "${publishToIlias}"
fail-on-error: true
exception-handler-workflow: "partial-error"
description: "Publishing to ILIAS channel"
configurations:
- channel-id: ilias
- mimetype: "application/json"
- download-source-flavors: "dublincore/*,security/*,captions/*"
- download-source-tags: "engage-download"
- streaming-source-tags: "engage-streaming"
- url-pattern: "${org_org_opencastproject_admin_ui_url}/api/events/${event_id}"
- with-published-elements: false
- check-availability: false

- id: snapshot
description: Archiving media
exception-handler-workflow: partial-error
fail-on-error: true
configurations:
- source-flavors: '*/*'

- id: cleanup
description: Cleaning up
fail-on-error: 'false'
configurations:
- delete-external: 'false'
- preserve-flavors: 'security/*'

A (stripped) example MediaPackage:
<?xml version="1.0"?>
<mediapackage xmlns="http://mediapackage.opencastproject.org" duration="267840" id="edd62312-a640-4407-9dc4-94cf4588369e" start="2023-09-15T09:53:00Z">
<title>Beispielvideo</title>
<series>8f718a94-6b72-40d2-9616-19ea4941a4ee</series>
<seriestitle>Beispielserie</seriestitle>
<creators>
<creator/>
</creators>
<media>
<track id="6039197b-f26d-4c3b-9c22-722aa8a40c6d" type="presentation/source">
<mimetype>video/mp4</mimetype>
<tags>
<tag>archive</tag>
</tags>
<size>543145897</size>
<checksum type="md5">c1fc49a664ff55449159d5ad796bff46</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>317354.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>1.5902081E7</bitrate>
<framerate>25.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
<track id="e42a75b3-d035-4aeb-8584-c4dee8de7dc7" type="presentation/prepared">
<mimetype>video/mp4</mimetype>
<tags>
<tag>archive</tag>
</tags>
<size>543145897</size>
<checksum type="md5">c1fc49a664ff55449159d5ad796bff46</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>317354.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>1.5902081E7</bitrate>
<framerate>25.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
</media>
<metadata>
<catalog id="e7c5588c-77c5-4fd7-bb42-9d2fbfd48209" type="dublincore/episode">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
<tag>engage-download</tag>
</tags>
<checksum type="md5">e5be1b8c9fcbfd4064a9c04bac3999e8</checksum>
</catalog>
<catalog id="e21b86c7-8ba8-48ce-b7d4-2db2fe78e27e" type="dublincore/series">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
<tag>engage-download</tag>
</tags>
<checksum type="md5">a6e634deab9b4a98137a98acaf87ef46</checksum>
</catalog>
</metadata>
<attachments>
<attachment id="security-policy-episode" type="security/xacml+episode">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
</tags>
<size>0</size>
<checksum type="md5">be895c01c2cf6ae73d86f899618cd510</checksum>
</attachment>
<attachment id="security-policy-series" type="security/xacml+series">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
</tags>
<size>0</size>
<checksum type="md5">c29424c6e3f613bfc47d19b83d8e63d6</checksum>
</attachment>
<attachment id="b4d86e7b-058e-4889-94cf-99be5c16fad6" type="presenter/player+preview">
<mimetype>image/png</mimetype>
<tags>
<tag>archive</tag>
<tag>engage-download</tag>
</tags>
<size>0</size>
<checksum type="md5">6ed3b37dc6266fb5b0539ccb07ad9773</checksum>
</attachment>
</attachments>
<publications>
<publication id="8d3ec4eb-f701-4924-90a8-4826481aec23" channel="internal">
<tags>
<tag>archive</tag>
</tags>
<media>
<track id="1216e4a3-bb12-43d8-a815-17f03ff2eda6" type="presentation/preview" ref="track:e42a75b3-d035-4aeb-8584-c4dee8de7dc7">
<mimetype>video/mp4</mimetype>
<tags>
<tag>preview</tag>
</tags>
<checksum type="md5">6dd52280348e34f63218216493353c83</checksum>
<duration>267866</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>5768</framecount>
<channels>2</channels>
<samplingrate>22050</samplingrate>
<bitrate>64490.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>153071.0</bitrate>
<framerate>25.0</framerate>
<resolution>640x360</resolution>
</video>
<live>false</live>
</track>
</media>
<attachments>
<!-- Elements of 'type=*preview' and 'type=*/waveform' have been stripped to improve readability -->
</attachments>
<metadata/>
</publication>
<publication id="6f2ce777-17a6-4f4a-a592-ba11f997ae13" channel="ilias">
<mimetype>application/json</mimetype>
<tags>
<tag>archive</tag>
</tags>
<media>
<track id="dd198599-fa36-43fd-a621-0cfb0222f02d" type="presentation/delivery" ref="track:ffa0acd1-ccea-487e-8b48-cd44514c73df">
<mimetype>video/mp4</mimetype>
<tags>
<tag>480p-quality</tag>
<tag>engage-download</tag>
<tag>engage-streaming</tag>
</tags>
<checksum type="md5">36ac7ab5979745f7e23abc744b75a49c</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>96576.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>307414.0</bitrate>
<framerate>25.0</framerate>
<resolution>854x480</resolution>
</video>
<live>false</live>
</track>
<track id="dd839e88-cb7b-4212-94fb-371f57b64eb7" type="presentation/delivery" ref="track:ffa0acd1-ccea-487e-8b48-cd44514c73df">
<mimetype>video/mp4</mimetype>
<tags>
<tag>360p-quality</tag>
<tag>engage-download</tag>
<tag>engage-streaming</tag>
</tags>
<checksum type="md5">be7e73053773ff41ec1589381d690084</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>64608.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>83292.0</bitrate>
<framerate>25.0</framerate>
<resolution>640x360</resolution>
</video>
<live>false</live>
</track>
<track id="6a2d5ce0-eb67-4aed-83f4-2468c3c2e9cb" type="presentation/source">
<mimetype>video/mp4</mimetype>
<tags>
<tag>archive</tag>
</tags>
<size>543145897</size>
<checksum type="md5">c1fc49a664ff55449159d5ad796bff46</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>317354.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>1.5902081E7</bitrate>
<framerate>25.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
<track id="0f6175f1-3665-4ea5-9863-0738220e1faa" type="presentation/delivery" ref="track:ffa0acd1-ccea-487e-8b48-cd44514c73df">
<mimetype>video/mp4</mimetype>
<tags>
<tag>720p-quality</tag>
<tag>engage-download</tag>
<tag>engage-streaming</tag>
</tags>
<checksum type="md5">5d6f0ab09fd24bd6047521e6011f59bb</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>129045.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>626391.0</bitrate>
<framerate>25.0</framerate>
<resolution>1280x720</resolution>
</video>
<live>false</live>
</track>
<track id="f04aaeb5-5f63-4f83-8b75-48b2eb2e47a1" type="presentation/delivery" ref="track:ffa0acd1-ccea-487e-8b48-cd44514c73df">
<mimetype>video/mp4</mimetype>
<tags>
<tag>1080p-quality</tag>
<tag>engage-download</tag>
<tag>engage-streaming</tag>
</tags>
<checksum type="md5">5fce0610a7830f0a9cd36be06d727154</checksum>
<duration>267840</duration>
<audio id="audio-1">
<device/>
<encoder type="AAC (Advanced Audio Coding)"/>
<framecount>12555</framecount>
<channels>2</channels>
<samplingrate>48000</samplingrate>
<bitrate>129045.0</bitrate>
</audio>
<video id="video-1">
<device/>
<encoder type="H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10"/>
<framecount>6695</framecount>
<bitrate>1570844.0</bitrate>
<framerate>25.0</framerate>
<resolution>1920x1080</resolution>
</video>
<live>false</live>
</track>
</media>
<attachments>
<!-- Elements of `type=*preview` have been stripped to improve readability -->
</attachments>
<metadata>
<catalog id="f61ec28c-f2de-4889-a38e-9ed9376c725e" type="dublincore/episode">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
<tag>engage-download</tag>
</tags>
</catalog>
<catalog id="6387935e-89ab-4327-8c86-89fb50712cc5" type="dublincore/series">
<mimetype>text/xml</mimetype>
<tags>
<tag>archive</tag>
<tag>engage-download</tag>
</tags>
<checksum type="md5">a6e634deab9b4a98137a98acaf87ef46</checksum>
</catalog>
</metadata>
</publication>
</publications>
</mediapackage>

Addendum:
For those that come across this issue and want to use the same approach:
We also slightly extended the example nginx configuration mentioned in the `migrate-events-to-another-opencast` helper script, by running a second nginx server-block under a different port using a slightly altered copy of our normal opencast reverse-proxy configuration:

server {
listen *:666 ssl http2;

# SSL and other configuration
[...]

location / {
proxy_set_header Authorization 'Basic <admin-account>';
proxy_set_header X-Requested-Auth '';
proxy_set_header X-Opencast-Matterhorn-Current-Job-Id '';
proxy_set_header X-Opencast-Matterhorn-Organization '';
proxy_set_header X-Opencast-Matterhorn-User '';

# Your normal opencast proxy settings
[...]
}

location /static/ {
alias /srv/opencast/downloads/;
sendfile on;
}
}

Where `<admin-account>` should be replaced by the basic-auth token of an account with admin-access, eg. `echo -n 'user:password' | base64`.

Best Regards,
Thomas

Greg Logan

unread,
Oct 11, 2023, 11:39:07 PM10/11/23
to us...@opencast.org
Hi Thomas,

That's quite the project you have, but you're on the right track.  I can't think of a better way to do things.

Part of your confusion about why entries are in some places and not others is that you are dealing with multiple sources of information.  The Asset Manager's files are not something you generally expose outside of Opencast's internal usage.  Likewise, the published files are generally things that can be re-generated and thus aren't necessarily in the Asset Manager.  But we use the same concept, and backing code to represent both - the MediaPackage.  I would not expect the search endpoints (which are user facing) to tell me things about stuff in the asset manager, and vice versa.  Your approach of mangling the two mediapackages together is the best that I can think of for what you're trying to do.

Alternatively, I believe the upgrade path from 8.x to 14 is fairly straightforward.  You'd need to do a number of migrations, but it might be easier than migrating to an entirely new Opencast.

G

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

Seeland, Per Pascal

unread,
Oct 12, 2023, 12:52:41 AM10/12/23
to us...@opencast.org

Hi,

 

just a note from our side. We did jump directly from 8.x to 13.x in March this year without any issues. It does take some time to rebuild the index for the 45k episodes, but otherwise it was a rather easy upgrade.

 

Cheers,

Pascal

Tom Keegasi

unread,
Oct 12, 2023, 3:26:43 AM10/12/23
to Opencast Users, Greg Logan
Hi Greg,

awesome and thank you for the information about different endpoints and use-cases like the Asset-Manager, that definitely
clear up a few confusions why there are different classes of MediaPackages and what they are intended for.

BR,
Thomas

Katrin Ihler

unread,
Nov 1, 2023, 12:42:24 PM11/1/23
to us...@opencast.org

Hi Thomas,

migration is a complex topic, but as I wrote at least two of the scripts you mentioned, feel free to ask me questions about specific details directly (though my memory is a bit fuzzy in places). This mail also reminded me that there's another migration script that I've been wanting to clean up and contribute for a while now, which does a similar thing as what you're trying to do (migrate archived and published data from one opencast to another).

Publication-channel-to-workspace won't do anything for you at that moment if I understand correctly, because there's not actually a publication available yet in the new system. What I've been doing for migrating publication data is ingest them along the rest of the archived assets via URLs pointing at the files on the old system, do ingest-download on all of those (you need to configure the IngestService so it can actually get at those files from your old system, so you need to put your credentials into the config) and then recreate the publication with those elements (ideally you can tell the elements for the archive vs publication apart with tags or flavors).

If you kick me hard enough over a longer period of time, I might actually create that PR... but I need to strip out a bunch of specific stuff yet and do at least some documentation.

Cheers,

Katrin

-- 
ELAN e.V.
Heger-Tor-Wall 12
D-49074 Osnabrück
https://elan-ev.de
Reply all
Reply to author
Forward
0 new messages