Help installing on Ubuntu 14.04

[Carsten Biemann]

Hi all,

I am trying to install Opencast 2.1.1 vom GIT on Ubuntu 14.04 LTS.

When running mvn clean install I get this error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on project matterhorn-composer-ffmpeg: There are test failures.

In the logs I found this more specific description:

07:29:07  INFO (AbstractCmdlineEncoderEngine:235) - Executing encoding command: ffmpeg -strict unofficial -i /home/biemann/matterhorn/modules/matterhorn-composer-ffmpeg/target/video8751947836541860890.mp4 -shortest -c copy /home/biemann/matterhorn/modules/matterhorn-composer-ffmpeg/target/video8751947836541860890_c8694515-e55e-4c40-afae-38dd4e54960e-work.mp4

[…]

07:29:07  INFO (FFmpegEncoderEngine:187) - /home/biemann/matterhorn/modules/matterhorn-composer-ffmpeg/target/video8751947836541860890.mp4: No such file or directory

So it seems that this test case isn't working due to the missing video-file. How can I correct or skip this test/problem?

Or – quite more easy – is there already a package to install Opencast with apt or does someone have a step-by-step installation guide for Ubuntu 14.04

Greetings

Carsten

[Lars Kiesow]

Hi Carsten,
my guess is that you did not properly install FFmpeg.
Regards,
Lars

[Carsten Biemann]

Hi Lars,

thanks for your answer. How can I check if I did (or did not) set up ffmpeg properly?

Tests with ffmpeg by myself on this machine work fine. Also the previous tests on the maven setup were passed without an error.

Also the error message says, the file cannot be found instead of a problem with ffmpeg itself.

How should I install ffmpeg? I used a self compiled ffmpeg 3.0.2 from the official homepage.

Who's familiar with Maven and know how to skip tests?

Regards from Frankfurt

Carsten

[Lars Kiesow]

Hi Carsten,
my guess is that file is the output of a previous FFmpeg call. If that
one failed, the file is missing for the next operation.
Can you give me the output of your ffmpeg (without any arguments)?

Btw, I've just compiles the latest r/2.1.x without a problem.

Also, there are no Debian packages, only RPMs so far.

Regards,
Lars

On Tue, 3 May 2016 05:44:29 -0700 (PDT)

[Dietmar Zenker]

Hi Carsten,

I get similar error messages when I build Opencast:

15:32:25  INFO (AbstractCmdlineEncoderEngine:235) - Executing encoding command: ffmpeg -strict unofficial -i /home/zenker/Opencast_Build/Opencast2.1.1/modules/matterhorn-composer-ffmpeg/target/video8888592358935577083.mp4 -shortest -c copy /home/zenker/Opencast_Build/Opencast2.1.1/modules/matterhorn-composer-ffmpeg/target/video8888592358935577083_825d8703-2a4a-465f-813b-a229fc81fa67-work.mp4
15:32:25  INFO (FFmpegEncoderEngine:187) - ffmpeg version 2.8.1 Copyright (c) 2000-2015 the FFmpeg developers
15:32:25  INFO (FFmpegEncoderEngine:187) - built with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-11)
15:32:25  INFO (FFmpegEncoderEngine:187) - /home/zenker/Opencast_Build/Opencast2.1.1/modules/matterhorn-composer-ffmpeg/target/video8888592358935577083.mp4: No such file or directory

But, despite this error message, the build process completes successfully with 0 errors in the final summary.

Does your build process interrupt at this step? Perhaps, other failures/issues are the primary cause of your unsuccessful build process?

Greetings from Idstein (I think you and Lars Peppel had visited me some years ago?),
Dietmar

[Carsten Biemann]

Hi,

here's the output of ffmpeg:

ffmpeg version 3.0.2 Copyright (c) 2000-2016 the FFmpeg developers

  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.1)

  configuration: 

  libavutil      55. 17.103 / 55. 17.103

  libavcodec     57. 24.102 / 57. 24.102

  libavformat    57. 25.100 / 57. 25.100

  libavdevice    57.  0.101 / 57.  0.101

  libavfilter     6. 31.100 /  6. 31.100

  libswscale      4.  0.100 /  4.  0.100

  libswresample   2.  0.101 /  2.  0.101

Hyper fast Audio and Video encoder

usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Is there any missing codec?

[Carsten Biemann]

Hi Dietmar,

the Maven process is interrupted at this task. I am not familiar with Maven, so I don't know how to extract the relevant infos (or skip needless tests).

Yes, we met, I think it was December 2014. :-)

Carsten

[Dietmar Zenker]

Hi Carsten,

although I don't think that this is a ffmpeg related problem: you should check if your ffmpeg supports the x264 encoder. Could you pls. verify that libx264 is listed when you call ffmpeg with the -encoders option:
ffmpeg -encoders

Greets from sunny Idstein,
Dietmar

[Carsten Biemann]

Thanks! This hint is good. x264 isn't supported resp. not listed.

Do you know how to install this under Ubuntu 14.04. I used apt to install the packages libx264-142, x264 and libx264-dev, but configuring ffmpeg from source have no chances / new supported codecs after this. So, I don't know how to enabled x264 for the ffmpeg 3.0.2 (in Ubuntu 14.04).

Frankfurt also sunny; hair style fits ;-)

Carsten

[Dietmar Zenker]

Hi Carsten,

if you install ffmpeg from the ubuntu repos via apt, you get probably a version without libx264 (for legal issues I think, and AFAIK ffmpeg has been replaced by libav in Ubuntu 14.04). Thus you have to compile it locally - installing all libx264 packages is a preliminary step for this. So the procedure is as follows:
 - download the ffmpeg source
 - configure it with a suitable profile that enables libx264:
   ./configure ... --enable-gpl --enable-libx264 etc.
- make
- make install

I recommend using the following profile of FFMPeg 2.8.1 which is automatically installed in CentOS as dependency during Opencast installation from the repos:

./configure --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --disable-static --enable-runtime-cpudetect --enable-gpl --enable-version3 --enable-postproc --enable-avfilter --enable-pthreads --enable-x11grab --enable-vdpau --disable-avisynth --enable-libdc1394 --enable-libvo-aacenc --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsoxr --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-bzlib --enable-libass --enable-libdc1394 --enable-libfreetype --enable-openal --enable-libopus --enable-libpulse --enable-libv4l2 --disable-debug --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --disable-stripping

Perhaps, you will be faced with some "missing libraries" errors during the make process, thus install all needed libraries and packages including the devel versions. Unfortunately, the names of the packages vary a bit between the Linux distibutions...

Hope that helps,
cheers Dietmar

[Lars Kiesow]

Hi,
don't bother including all of these libraries, most of them are only
necessary for very specific stuff and are usually not needed.

Also, use FFmpeg 3.0.2 as you might run into problems with old versions
and Opencast 2.2 (at lest old versions are not tested against).

The easiest way to get a working FFmpeg on Ubuntu is to use the static
binary builds of FFmpeg from:
http://johnvansickle.com/ffmpeg/

They should include all necessary libraries. Way easier than building
everything yourself.

Regards,
Lars

On Wed, 4 May 2016 03:42:32 -0700 (PDT)
Dietmar Zenker <dzenk...@online.de> wrote:

> Hi Carsten,
>
> if you install ffmpeg from the ubuntu repos via apt, you get probably
> a version without libx264 (for legal issues I think, and AFAIK ffmpeg
> has been replaced by libav in Ubuntu 14.04). Thus you have to compile
> it locally - installing all libx264 packages is a preliminary step
> for this. So the procedure is as follows:
> - download the ffmpeg source
> - configure it with a suitable profile that enables libx264:

> *./configure ... --enable-gpl --enable-libx264 etc.*

> - make
> - make install
>
> I recommend using the following profile of FFMPeg 2.8.1 which is
> automatically installed in CentOS as dependency during Opencast
> installation from the repos:
>

> *./configure --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64

> --mandir=/usr/share/man --enable-shared --disable-static
> --enable-runtime-cpudetect --enable-gpl --enable-version3
> --enable-postproc --enable-avfilter --enable-pthreads
> --enable-x11grab --enable-vdpau --disable-avisynth --enable-libdc1394
> --enable-libvo-aacenc --enable-libgsm --enable-libmp3lame
> --enable-libopencore-amrnb --enable-libopencore-amrwb
> --enable-libopenjpeg --enable-libsoxr --enable-libschroedinger
> --enable-libspeex --enable-libtheora --enable-bzlib --enable-libass
> --enable-libdc1394 --enable-libfreetype --enable-openal
> --enable-libopus --enable-libpulse --enable-libv4l2 --disable-debug
> --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid
> --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64

> -mtune=generic -fPIC' --disable-stripping*

[Carsten Biemann]

Thanks Dietmar for your help,

configuring ffmpeg with --enable-gpl --enable-libx264 wasn't enough. The compiled executable does not output the x264-support with the command-line-call ffmpeg -encoders

The config-flags from CentOS doesn't work on my Ubuntu 14.04 installation. I've tried to fix the inccorect flags, but it's enough to drive you up the wall. :-/

Carsten

[Carsten Biemann]

Hi Lars,

okay, I'll try this. An already compiled version with the necessary library is quite more convenient.

Carsten

[Corne]

Hi,

Building r/2.2.x on Ubuntu 14.04

I found this page quite helpful in building ffmpeg : Compile FFmpeg on Ubuntu, Debian, or Mint (http://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu)

The build fails on the "matterhorn-composer-ffmpeg" module test.

INFO (ComposerServiceImpl:1027) - Converting image attachment test image into video 396becb7-5890-49da-b5ba-94c33afe7d1c
INFO (AbstractCmdlineEncoderEngine:242) - Executing encoding command: ffmpeg -loop 1 -i /opt/matterhorn/modules/matterhorn-composer-ffmpeg/target/image2697958438786171926.jpg -c:v libx264 -r 25 -t 2.000 -pix_fmt yuv420p /opt/matterhorn/modules/matterhorn-composer-ffmpeg/target/image2697958438786171926_2_000_31634010-3284-439b-bd48-a54399cab69e-image-video.mp4

...
INFO (FFmpegEncoderEngine:187) - Unknown encoder 'libx264'

Which looks like the codec was not installed, although:

> ffmpeg -codecs | grep 264 

  ffmpeg version N-79753-g3395ad4 Copyright (c) 2000-2016 the FFmpeg developers

  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.1)

    configuration: 

    libavutil      55. 23.100 / 55. 23.100

    libavcodec     57. 38.100 / 57. 38.100

    libavformat    57. 36.100 / 57. 36.100

    libavdevice    57.  0.101 / 57.  0.101

    libavfilter     6. 44.100 /  6. 44.100

    libswscale      4.  1.100 /  4.  1.100

    libswresample   2.  0.101 /  2.  0.101

    D.V.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_vdpau ) 

So clearly it should work ?

Regards,
Corné

-c:v : Input video codec - it is a jpg (image) does not have a codec

-c:a : Output video

[Greg Logan]

Hi Corne,

What that is telling you is that h264 Decoding is supported (and that it's a Video codec that can operate in Lossy and lossLess), but the magic 'E' flag for encoding is not there.  Are you sure that it's using the correct ffmpeg version, and that you don't have another one installed elsewhere on your system that appears earlier on the path?

G

--
You received this message because you are subscribed to the Google Groups "Matterhorn Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to matterhorn-use...@opencast.org.

[Corne]

Thanks Greg,

Following a suggestion from Karen and using this script https://gist.github.com/xdamman/e4f713c8cd1a389a5917#file-install_ffmpeg_ubuntu-sh (attached)

(Which would be very helpful to have part of the source somewhere)

The codecs and ffmpeg configuration now looks like:

> ffmpeg -codecs | grep 264

  ffmpeg version git-2016-05-06-bd63ece Copyright (c) 2000-2016 the FFmpeg developers

  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.1)

  configuration: --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-version3

libavutil      55. 24.100 / 55. 24.100

libavcodec     57. 38.100 / 57. 38.100

libavformat    57. 36.100 / 57. 36.100

libavdevice    57.  0.101 / 57.  0.101

libavfilter     6. 45.100 /  6. 45.100

libswscale      4.  1.100 /  4.  1.100

libswresample   2.  0.101 /  2.  0.101

libpostproc    54.  0.100 / 54.  0.100

DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (encoders: libx264 libx264rgb )

And it continues to compile the modules and pass the test until it hits:

<testcase name="testEnrichment" classname="org.opencastproject.inspection.ffmpeg.MediaInspectionServiceImplTest" time="0.218">

    <failure message="expected:&lt;46&gt; but was:&lt;44&gt;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: expected:<46> but was:<44>

at org.junit.Assert.fail(Assert.java:88)

at org.junit.Assert.failNotEquals(Assert.java:743)

at org.junit.Assert.assertEquals(Assert.java:118)

at org.junit.Assert.assertEquals(Assert.java:555)

at org.junit.Assert.assertEquals(Assert.java:542)

at org.opencastproject.inspection.ffmpeg.MediaInspectionServiceImplTest.testEnrichment(MediaInspectionServiceImplTest.java:158)

]]></failure>

    <system-out><![CDATA[09:12:06  INFO (MediaInspector:329) - Successfully inspected track file:/opt/matterhorn/modules/matterhorn-inspection-service-ffmpeg/target/test-classes/test.mp4

]]></system-out>

  </testcase>

 AudioStream[] audioStreams = TrackSupport.byType(newTrack.getStreams(), AudioStream.class);
 assertEquals(46L, audioStreams[0].getFrameCount().longValue());

Corné

[Corne]

Follow Up:

Successfully Build r2.2x on Ubuntu 14.04.

ffprobe -show_format -show_streams -count_frames -of json test.mp4

...
"bit_rate": "10672",

"nb_frames": "46",

"nb_read_frames": "44",

"disposition": {
...

Changed profile test to reflect the the correct frame count number. if -count_frames is omitted then 46 would be the right answer.

...
"bit_rate": "10672",

"nb_frames": "46",

"disposition": {

...

Corné

[Rubén Pérez]

Hi,

For what is worth, I have the same problem with that test and have to adapt it or entirely ignore the tests in order to successfully build Opencast.


Regards

--
Rubén Pérez Vázquez

Universität zu Köln
Regionales Rechenzentrum (RRZK)
Weyertal 121, Raum 4.05
D-50931 Köln
✆: +49-221-470-89603

[Lars Kiesow]

Hi Rubén,
since you seem to be the only committer who can actually reproduce that
problem, can you have a look at it and (if necessary) fix it?
Regards,
Lars

[Karen Dolan]

Hi Corné & Rubén,

It might be just a matter of setting "accurateFrameCount" to false in the testEnrichment method. 

    if (accurateFrameCount)

      command.add("-count_frames");

…..

          /* Frame Count */

          obj = stream.get("nb_read_frames");

          if (obj != null) {

            vMetadata.setFrames(Long.parseLong((String) obj));

          }

- Karen

[Rubén Pérez]

Hi,

Lars, when we talked about this issue, you could not reproduce it. It all seems to be related with the ffmpeg version. Since I am not using that ffmpeg version, I did not look any further and simply "patched" the test for myself. I also have a static ffmpeg installation in ubuntu, and that may be the reason why the "nb_read_frames" amount is not correct.

Setting "accurateFrameCount" will probably make the test pass, but then we won't be testing the feature at all. Or, at least, not all of it.

--

[Karen Dolan]

Rubén,

Everything  sets the accurateFrameCount to false (configs [2], default values[3], the MediaInspectionServiceImpl’s activate method[4]) except for the MediaInspectionServiceImplTest [1].  Which seems a little weird. What’s stranger is that not everyone is having issues with that.

- Karen

Setting "accurateFrameCount" will probably make the test pass, but then we won't be testing the feature at all. Or, at least, not all of it.

It might be just a matter of setting "accurateFrameCount" to false in the testEnrichment method. 

[0]. From MediaInspector.java the constructor where the boolean is accepted

public MediaInspector(Workspace workspace, Parser tikaParser, String ffprobePath, boolean accurateFrameCount)

[1]. From: MediaInspectionServiceImplTest.java

 return some(new MediaInspector(workspace, new AudioParser(), binary, true)); <—test uses true!

[2]. From: org.opencastproject.inspection.ffmpeg.MediaInspectionServiceImpl.cfg

# Whether the calculation of the frames is accurate or not (default false)

accurate_frame_count=false

[3] From FFmpegAnalyzer.java

  /** Whether the calculation of the frames is accurate or not */

  private boolean accurateFrameCount;  <— which defaults to false

[4] From MediaInspectionServiceImpl

    inspector = new MediaInspector(workspace, tikaParser, ffprobeBinary, false); <— Impl uses false

[Corne]

Hi,

The test is written to test the code and the service, so would it not make more sense to test both cases? even though it might not be used it is a variable that can be (should be) tested.

[1]. From: MediaInspectionServiceImplTest.java

 return some(new MediaInspector(workspace, new AudioParser(), binary, true)); <—test uses true! 

 return some(new MediaInspector(workspace, new AudioParser(), binary, false)); // We test both cases

- Corné

[Corné]

Hi,

Created a Jira and Branch with the code that build successfully for this problem.

Jira: https://opencast.jira.com/browse/MH-11572

BitBucket: https://bitbucket.org/coosthuizen/opencast-2.2.x/branch/t/MH-11572-Inspection-Service-FFmpeg

Thanks,

Corné

[Carsten Biemann]

Hi,

thanks for the tip with the static binaries. They seem to do their job. ffmpeg -codecs is listing libx264 also for encoding. Great! :-)

Today I could try another maven run, but running into another test failure:

matterhorn-inspection-service-ffmpeg ............... FAILURE

Here some details of the error-XML-file:

-----

  <testcase name="testEnrichment" classname="org.opencastproject.inspection.ffmpeg.MediaInspectionServiceImplTest" time="0.495">

    <failure message="expected:&lt;46&gt; but was:&lt;44&gt;" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: expected:<46> but was:<44>

        at org.junit.Assert.fail(Assert.java:88)

        at org.junit.Assert.failNotEquals(Assert.java:743)

        at org.junit.Assert.assertEquals(Assert.java:118)

        at org.junit.Assert.assertEquals(Assert.java:555)

        at org.junit.Assert.assertEquals(Assert.java:542)

        at org.opencastproject.inspection.ffmpeg.MediaInspectionServiceImplTest.testEnrichment(MediaInspectionServiceImplTest.java:158)

]]></failure>

    <system-out><![CDATA[10:50:57  INFO (MediaInspector:320) - Successfully inspected track file:/home/biemann/matterhorn/modules/matterhorn-inspection-service-ffmpeg/target/test-classes/test.mp4

]]></system-out>

  </testcase>

-----

"44 instead of 46" as the only information isn't quite useful for me to find the reason or how to fix it.

Does anybody have an idea?

Regards

Carsten

Am Mittwoch, 4. Mai 2016 15:08:10 UTC+2 schrieb Lars Kiesow:

[Dietmar Zenker]

Hi Carsten,

this problem has been reported some posts in this thread before. But, this problem is new in 2.2.x which will not appear in 2.1.x. Which version are you trying to build??

Greetings,
Dietmar

[Corné]

Hi Carsten,

I've submitted a fix for this issue:

https://bitbucket.org/opencast-community/matterhorn/pull-requests/976/mh-11572-expanded-test-to-include-accurate/diff

Hope this helps,

Corné

[Carsten Biemann]

Hi Diemar, hi Corné,

I've checked out 2.1.1 from the GIT repo.

Regards

Carsten