D10 MXF rewrapping

[Phillip]

I have an NTSC 50Mbit D10 MXF file. First 50MB here:
http://lists.lpb.org/ffmbc/input.mxf

It has a single, four-channel audio track (pcm_s16le). When imported
into Media Composer 3.5.4, it only plays the second audio channel (3
and 4 are empty, so I'm not sure if they are correct). Video is
fine. The file plays all four channels correctly on an Omneon when
set to eVTR playback.

The original file is 3GB (there are 2TB total actually). I don't have
any smaller files to post, unfortunately, and this 50MB sample won't
actually import into Avid either (headers think it's still six-minute
clip) so I don't know how useful it will be for testing. Anyway, here
are the things I've tried:

Remux to another MXF file with:
ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -map_audio_channel
0:1:0:0:1:0 -map_audio_channel 0:1:1:0:2:0 -map_audio_channel
0:1:2:0:3:0 -map_audio_channel 0:1:3:0:4:0 -f mxf output.mxf -newaudio
-newaudio -newaudio

Fails to import in Avid: "Can not import this file into a bin. See
the documentation for more information about transferring media
files."

Trying a simple
ffmbc -i input.mxf -vcodec copy -acodec copy -f mxf output.mxf
Fails with the same error as above.

ffmbc -i input.mxf -vcodec copy -acodec copy -f mxf_d10 output.mxf
Fails to import, but the Avid gives a different error: "This import
file has invalid width/height"

So I fall back to trying Quicktime:
ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -vtag mx5n -vbsf
imxdump output.mov
ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -vtag AVmn -vbsf
imxdump output.mov
ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -vbsf imxdump
output.mov

All three of these import and produce correct audio, but the video is
solid white. This is what usually happens when Quicktime barfs on the
input. Wrong -vtag?

Am I missing something? Using -f mxf_d10 seems the closest. Maybe a
PAL/NTSC mismatch/assumption somewhere?

This is the debug info for input.mxf:

FFmpeg version SVN-r20416, Copyright (c) 2000-2009 Fabrice Bellard, et
al.
built on Oct 30 2009 04:05:33 with gcc 4.2.4
configuration: --enable-memalign-hack --prefix=/mingw --cross-
prefix=i686-mingw32- --cc=ccache-i686-mingw32-gcc --target-os=mingw32
--arch=i686 --cpu=i686 --enable-avisynth --enable-gpl --enable-
version3 --enable-zlib --enable-bzlib --enable-libgsm --enable-libfaad
--enable-pthreads --enable-libvorbis --enable-libtheora --enable-
libspeex --enable-libmp3lame --enable-libopenjpeg --enable-libxvid --
enable-libschroedinger --enable-libx264 --enable-libopencore_amrwb --
enable-libopencore_amrnb
libavutil 50. 3. 0 / 50. 3. 0
libavcodec 52.37. 1 / 52.37. 1
libavformat 52.39. 2 / 52.39. 2
libavdevice 52. 2. 0 / 52. 2. 0
libswscale 0. 7. 1 / 0. 7. 1
[NULL @ 0x3ee030]Probed with size=2048 and score=100
[mxf @ 0x3ee030]MAX_READ_SIZE:5000000 reached

Seems stream 0 codec frame rate differs from container frame rate:
59.94 (60000/1001) -> 29.97 (30000/1001)
Input #0, mxf, from 'input.mxf':
Duration: 00:06:57.15, start: 0.000000, bitrate: 1005 kb/s
Stream #0.0, 1001/30000: Video: mpeg2video, yuv422p, 720x512 [PAR
128:135 DAR 4:3], 1001/60000, 50000 kb/s, 29.97 tbr, 29.97 tbn, 59.94
tbc
Stream #0.1, 1001/30000: Audio: pcm_s16le, 48000 Hz, 4 channels,
s16, 3072 kb/s
Output #0, null, to '/dev/null':
Stream #0.0, 1/90000: Video: rawvideo, yuv422p, 720x512 [PAR
128:135 DAR 4:3], 1001/30000, q=2-31, 200 kb/s, 90k tbn, 29.97 tbc
Stream #0.1, 1/90000: Audio: pcm_s16le, 48000 Hz, 4 channels, s16,
3072 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Press [q] to stop encoding
[mpeg2video @ 0x1988330]vbv buffer: 9437184, bitrate:50000000
[mpeg2video @ 0x1988330]profile: 0, level: 5 vbv buffer: 9437184,
bitrate:50000000
[mpeg2video @ 0x1988330]sde w:720, h:512
[mpeg2video @ 0x1988330]GOP ( 0:00:05.[05]) closed_gop=1 broken_link=0
[mpeg2video @ 0x1988330]vbv_delay 3003, ref 0 type:1
[mpeg2video @ 0x1988330]qp:2 fc:15151515 I dc:2 pstruct:3 fdct:0
cmv:0 qtype:1 ivlc:1 rff:0
[mpeg2video @ 0x1988330]vbv buffer: 9437184, bitrate:50000000
[mpeg2video @ 0x1988330]profile: 0, level: 5 vbv buffer: 9437184,
bitrate:50000000
[mpeg2video @ 0x1988330]sde w:720, h:512
[mpeg2video @ 0x1988330]GOP ( 0:00:05.[06]) closed_gop=1 broken_link=0
[mpeg2video @ 0x1988330]vbv_delay 3003, ref 0 type:1
[mpeg2video @ 0x1988330]qp:1 fc:15151515 I dc:2 pstruct:3 fdct:0
cmv:0 qtype:1 ivlc:1 rff:0

Also there are way too many people named Phil/Phill/Phillip on this
list.

[Baptiste Coudurier]

Hi,

This is interesting.

> So I fall back to trying Quicktime:
> ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -vtag mx5n -vbsf
> imxdump output.mov
> ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -vtag AVmn -vbsf
> imxdump output.mov
> ffmbc -i input.mxf -vcodec copy -acodec pcm_s16le -vbsf imxdump
> output.mov
>
> All three of these import and produce correct audio, but the video is
> solid white. This is what usually happens when Quicktime barfs on the
> input. Wrong -vtag?

Yes, AVID IMX would be AVmp however muxer does not create necessary
atoms yet, so I guess it won't work.

Also, you don't need the -vbsf with AVID.

> Am I missing something? Using -f mxf_d10 seems the closest. Maybe a
> PAL/NTSC mismatch/assumption somewhere?

You have to use -f mxf_d10 to create a D-10 flavoured mxf.

Btw you log doesn't seem to indicate you are using ffmbc, but generic
ffmpeg, given the version number.

It seems your file is bottom field first which is incompatible with
D-10. D-10 is _always_ top field first, mandated by the specs. This
could explain why AVID complains, though it seems weird.

Can you cut a small sample using -f mxf_d10 and upload it somewhere ?

I understand the process needs serious documentation, I'm sorry about
that. I hope in the future FFmbc will have very exhaustive documentation.

Thanks.

[...]

--
Baptiste COUDURIER
Key fingerprint 8D77134D20CC9220201FC5DB0AC9325C5C1ABAAA
FFmpeg maintainer http://www.ffmpeg.org

[Phillip]

On Nov 13, 5:53 pm, Baptiste Coudurier <baptiste.coudur...@gmail.com>
wrote:

I browsed the code enough to see that mov_write_avid_tag() only gets
used for DNxHD, so I figured there was some special Avid proprietary
sauce missing.

> Also, you don't need the -vbsf with AVID.
>
> > Am I missing something?  Using -f mxf_d10 seems the closest.  Maybe a
> > PAL/NTSC mismatch/assumption somewhere?
>
> You have to use -f mxf_d10 to create a D-10 flavoured mxf.
>
> Btw you log doesn't seem to indicate you are using ffmbc, but generic
> ffmpeg, given the version number.

My ffmpeg build is newer, so I figured it might be more useful. I can
do a log with ffmbc if it makes a difference. All of my testing and
commandline examples were using ffmbc.

> It seems your file is bottom field first which is incompatible with
> D-10. D-10 is _always_ top field first, mandated by the specs. This
> could explain why AVID complains, though it seems weird.

That's unfortunate. I'll see what I can do about the next 4TB...

> Can you cut a small sample using -f mxf_d10 and upload it somewhere ?

I took the original 3GB mxf and remuxed it with
ffmbc -i fullfile.mxf -vcodec copy -acodec copy -f mxf_d10 output.mxf
Fails to import in the Avid: "This import file has invalid width/
height"

I then chopped the first 50MB with dd, and it's available here:
http://lists.lpb.org/ffmbc/remux-d10.mxf

[Phil Rhodes]

> [MXF grief]

Man, I hate MXF.

P

[Phillip]

I guess I should clarify that the goal is to write back to an XDCAM
deck also (if possible given the field order issue), but Avid
importing is more important.

Debug crashes on remux-d10.mxf using ffmpeg-r20416 and ffmbc:
ffmbc -debug 1 -i remux-d10.mxf -f null /dev/null 2> debug.txt


Here's debug using ffmbc on the original input.mxf for completeness:

FFmpeg version FFmbc-0.2, Copyright (c) 2000-2009 Fabrice Bellard, et
al.
configuration: --enable-memalign-hack --target-os=mingw32 --
arch=i686 --cpu=core2 --enable-static --enable-avisynth --enable-gpl --
enable-version3 --disable-shared --enable-zlib --enable-bzlib --enable-
w32threads --enable-libfaad --enable-static

libavutil 50. 3. 0 / 50. 3. 0

libavcodec 52.35. 0 / 52.35. 0
libavformat 52.38. 0 / 52.38. 0

libavdevice 52. 2. 0 / 52. 2. 0
libswscale 0. 7. 1 / 0. 7. 1

built on Oct 26 2009 19:25:35, gcc: 4.4.2 (x86.corei7.LPB)
[NULL @ 003edc40]Probed with size=2048 and score=100
[mxf @ 003edc40]All info found

Input #0, mxf, from 'input.mxf':
Duration: 00:06:57.15, start: 0.000000, bitrate: 1005 kb/s

Stream #0.0(und): Video: mpeg2video, yuv422p, 720x512 [PAR 128:135
DAR 4:3], tbc 2002/60000, 50000 kb/s, afps 29.970, tbf 1001/30000
Stream #0.1(und): Audio: pcm_s16le, 48000 Hz, 4 channels, s16,

3072 kb/s
Output #0, null, to '/dev/null':

Stream #0.0(und): Video: rawvideo, yuv422p, 720x512 [PAR 128:135
DAR 4:3], tbc 1001/30000, q=2-31, 200 kb/s, tbf 1/90000
Stream #0.1(und): Audio: pcm_s16le, 48000 Hz, 4 channels, s16,

3072 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Press [q] to stop encoding

[mpeg2video @ 01588070]vbv buffer: 9437184, bitrate:50000000
[mpeg2video @ 01588070]profile: 0, level: 5 vbv buffer: 9437184,
bitrate:50000000
[mpeg2video @ 01588070]sde w:720, h:512
[mpeg2video @ 01588070]GOP ( 0:00:05.[05]) closed_gop=1 broken_link=0
[mpeg2video @ 01588070]vbv_delay 3003, ref 0 type:1
[mpeg2video @ 01588070]qp:2 fc:15151515 I dc:2 pstruct:3 fdct:0

cmv:0 qtype:1 ivlc:1 rff:0

[mpeg2video @ 01588070]vbv buffer: 9437184, bitrate:50000000
[mpeg2video @ 01588070]profile: 0, level: 5 vbv buffer: 9437184,
bitrate:50000000
[mpeg2video @ 01588070]sde w:720, h:512
[mpeg2video @ 01588070]GOP ( 0:00:05.[06]) closed_gop=1 broken_link=0
[mpeg2video @ 01588070]vbv_delay 3003, ref 0 type:1
[mpeg2video @ 01588070]qp:1 fc:15151515 I dc:2 pstruct:3 fdct:0

cmv:0 qtype:1 ivlc:1 rff:0

[mpeg2video @ 01588070]vbv buffer: 9437184, bitrate:50000000
[mpeg2video @ 01588070]profile: 0, level: 5 vbv buffer: 9437184,
bitrate:50000000
[mpeg2video @ 01588070]sde w:720, h:512
[mpeg2video @ 01588070]GOP ( 0:00:05.[07]) closed_gop=1 broken_link=0
[mpeg2video @ 01588070]vbv_delay 3003, ref 0 type:1
[mpeg2video @ 01588070]qp:1 fc:15151515 I dc:2 pstruct:3 fdct:0

[Phillip]

Me too. It's supposed to be the ubiquitous file format, yet every
vendor needs something a little bit different.

Can't wait to see how much fun working with PBS's AS-03 flavor will
be...
https://secure.connect.pbs.org/conferences/technology/2009/Presentations/Oliver%20Morgan%20AS-03%20Presentation.pdf

[Phil Rhodes]

> Me too. It's supposed to be the ubiquitous file format, yet every
> vendor needs something a little bit different.

I was once asked to write an "opinion piece" on MXF, shortly after its
release (NAB '04, 05?) for HD magazine, which I foolishly refused to do
because I didn't think they'd publish the swearing...

...as far as I can see all it actually does is wrap up all the preexisting
interchange problems in a layer of extra complexity.

Is there anything that ffmbc can create that Avids will read that isn't MXF,
or is that a forlorn hope.

P

[Baptiste Coudurier]

Well I think I could make the IMX in .mov working :)

[Baptiste Coudurier]

On 11/13/09 4:25 PM, Phillip wrote:
> I guess I should clarify that the goal is to write back to an XDCAM
> deck also (if possible given the field order issue), but Avid
> importing is more important.
>
> Debug crashes on remux-d10.mxf using ffmpeg-r20416 and ffmbc:
> ffmbc -debug 1 -i remux-d10.mxf -f null /dev/null 2> debug.txt
>

Well, indeed something is really broken with this file.
Nor width nor height are set, no audio sample rate either.
I'm not sure how you managed to create it.

Did you cross compile ?
Can you compile it without arch, cpu, target-os ?
You don't need enable-version3 either I think

I should have win32 binaries disponible soon hopefully.

[Phil Rhodes]

> Well I think I could make the IMX in .mov working :)

Again please tell me how!

Is there -any- documentation on the new commandline switches anywhere?

I'm happy to contribute to docs and tests and so forth, I'd like to be a
decent user of free software, but I have literally zero information right
now.

P

PS - although to be fair I can't test on Avid as I don't have easy access to
one...

[Phillip]

On Nov 13, 10:42 pm, Baptiste Coudurier <baptiste.coudur...@gmail.com>
wrote:

> On 11/13/09 4:25 PM, Phillip wrote:
>
> > I guess I should clarify that the goal is to write back to an XDCAM
> > deck also (if possible given the field order issue), but Avid
> > importing is more important.
>
> > Debug crashes on remux-d10.mxf using ffmpeg-r20416 and ffmbc:
> > ffmbc -debug 1 -i remux-d10.mxf -f null /dev/null 2>  debug.txt
>
> Well, indeed something is really broken with this file.
> Nor width nor height are set, no audio sample rate either.
> I'm not sure how you managed to create it.
>
> Did you cross compile ?

No. Compiled on mingw with gcc 4.4.2 (Win2k8 x64).
Used the 32-bit toolchain from here (updated to current):
http://komisar.gin.by/mingw/

> Can you compile it without arch, cpu, target-os ?
> You don't need enable-version3 either I think

Everything's at work, so it'll have to wait for Monday.

FWIW, the same gcc 4.4.2 didn't have issues compiling
x264's weightp when that was committed earlier this week.
There was a rash of bad compiles out there from broken gcc's...
(Not that it still couldn't be broken gcc in this case.)

[Phillip]

On Nov 13, 11:03 pm, Phillip <pblu...@gmail.com> wrote:
> On Nov 13, 10:42 pm, Baptiste Coudurier <baptiste.coudur...@gmail.com>
> wrote:
>
> > On 11/13/09 4:25 PM, Phillip wrote:
>
> > > I guess I should clarify that the goal is to write back to an XDCAM
> > > deck also (if possible given the field order issue), but Avid
> > > importing is more important.
>
> > > Debug crashes on remux-d10.mxf using ffmpeg-r20416 and ffmbc:
> > > ffmbc -debug 1 -i remux-d10.mxf -f null /dev/null 2>  debug.txt
>
> > Well, indeed something is really broken with this file.
> > Nor width nor height are set, no audio sample rate either.
> > I'm not sure how you managed to create it.
>
> > Did you cross compile ?
>
> No.  Compiled on mingw with gcc 4.4.2 (Win2k8 x64).
> Used the 32-bit toolchain from here (updated to current):http://komisar.gin.by/mingw/
>
> > Can you compile it without arch, cpu, target-os ?
> > You don't need enable-version3 either I think
>
> Everything's at work, so it'll have to wait for Monday.
>
> FWIW, the same gcc 4.4.2 didn't have issues compiling
> x264's weightp when that was committed earlier this week.
> There was a rash of bad compiles out there from broken gcc's...
> (Not that it still couldn't be broken gcc in this case.)
>
>
>
> > I should have win32 binaries disponible soon hopefully.

With the new 0.3 binaries I can remux and import into the Avid
successfully using
ffmbc -i input.mxf -vcodec copy -acodec copy -f mxf_d10 remux2.mxf

I cannot, however, write the same remuxed file directly to a blank
XDCAM disc using the Sony PDZ-1 software. I'll guess that's because
of the incorrect (bottom first) field order with this particular file,
but has any ffmbc-generated output been tested with PDZ-1? I can
generate some IMX50 on XDCAM samples with a PDW-700 camera if it's
helpful to anyone.

Earlier you said that XDCAM requires separate mono audio tracks, but -
f mxf_d10 supports only one audio track. Are the separate mono audio
tracks only a requirement for long GOP HD operating points?

> > --
> > Baptiste COUDURIER
> > Key fingerprint                 8D77134D20CC9220201FC5DB0AC9325C5C1ABAAA

> > FFmpeg maintainer                                  http://www.ffmpeg.org- Hide quoted text -
>
> - Show quoted text -

[Baptiste Coudurier]

AFAIK, rewriting to disk has never been tested unfortunately. It's very
hard to debug that kind of problems :(
Not sure if it it because of field order, but you could test with a
correct top field first material.

> Earlier you said that XDCAM requires separate mono audio tracks, but -
> f mxf_d10 supports only one audio track. Are the separate mono audio
> tracks only a requirement for long GOP HD operating points?

Very, it's more like all flavour requires mono tracks except D-10 which
only supports one track 8 channels max per the specs.

[judi...@gmail.com]

yeah maybe some special Avid proprietary sauce missing.