MediaCodec InputBuffer expected (vs. actual) size

[Charley Robinson]

Hi All,

I'm testing MediaCodec for encoding H264/AVC on the Nexus 4 with the Qualcomm encoder, and running into trouble with the input format. The encoder is setup like this, so by my calculations, I should expect a single frame's pixel buffer to be of a specific size:

115200 == 320 (w) * 240 (h) * 12 (bits/pixel) / 8 (bits / byte)

Yet, a call to my input ByteBuffer's .remaining() function yields the curious value of:

118784 == ???

For context, we pull the buffer thusly:

ByteBuffer[] inputBuffers = mediaCodec.getInputBuffers();
int inputBufferIndex = mediaCodec.dequeueInputBuffer(0);
if (inputBufferIndex >= 0) {
  ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
  inputBuffer.clear();
  inputBuffer.remaining(); // != 115200 !?!?!?
  inputBuffer.put(pixelBuffer);
}

(Presumably) as a result, putting a 115200-byte pixel array into this mismatched input buffer produces a predictable kind of garbage in the resultant encoded stream: http://i.imgur.com/ZcwoU.png -- preview left (chroma planes flattened before encoding), decoded right. The same config setup cited above on the Nexus 7 using the Nvidia encoder (http://i.imgur.com/qketz.png) seems to work just fine, and there is no input buffer size mismatch for the same MediaFormat.

Questions:

Is there an error in my logic/approach?

How should one generally expect to predict the size of the codec input buffer, when Nexus devices behave inconsistently against the same MediaFormat configuration?

Thanks,
Charley

[Charley Robinson]

I disagree. The input buffer of the video encoder is meant solely for
consuming raw image (eg. bitmap) data. The yuv 4:2:0 format does not
contain vector information, perhaps you're thinking of the output of
the encoder? (eg. an encoded h.264 stream)

Although it may be flawed logic, the estimated buffer size calculation
was derived from the input buffer size that I was getting for several
other devices than the Nexus 4 (also in the Nexus family). An image
bitmap will always be of consistent size, but I was never able to get
this particular device's encoder to properly consume images from the
camera.

On Mon, Mar 4, 2013 at 2:51 PM, Botao Lin <lin28...@gmail.com> wrote:
> Hi Charley,
>
> I don't know how to solve your problem. But I do notice one error in your
> reasoning.
> --> The input buffer is used to stored compressed bit-stream.
> --> The formula you used, "115200 == 320* 240* 12/ 8", is to calculate
> size of the uncompressed image (the raw data).
>
> The data in the input buffer is relatively complicated, including, "encoded"
> pixel (color) value, motion vectors, etc. You might want to check out a
> video encode/decode book...
>
> Thanks,
> BT

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "android-platform" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/android-platform/awaNwgb6EbY/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to
> android-platfo...@googlegroups.com.
>
> To post to this group, send email to android-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/android-platform?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Martin Storsjö]

On Mon, 4 Mar 2013, Josh Green wrote:

> So from that, a couple of suggestions arise:
> �- The padding may be between each plane?�- The padding may only be at the
> start and/or end of the buffer? (So try using just the start and/or end of
> the buffer?)
> �- Is it possible the encoder has this extra space available but doesn't
> require it to be filled? (i.e. perhaps it supports detecting meta-data
> and/or extra stream parameters from a header if it exists)

These are pretty good guesses.

There's two or three different issues at work here:

- You can't simply assume you can take the camera buffer and copy it to
the MediaCodec, they use different color formats. The camera pictures
coming from the camera is normally NV21 (the old default) or YV12 (the new
standard-to-be which is buggy in a number of 4.0/4.1 devies). For what the
MediaCodec supports, you need to query
MediaCodecInfo.getCapabilitiesForType().colorFormats[]. Most encoders
either use COLOR_FormatYUV420SemiPlanar (which is NV12) or
COLOR_FormatYUV420Planar (which is I420), and convert the pixel data
accordingly. Neither of them match the camera formats directly. I420 is
like YV12 but with the two chroma planes swapped, and NV12 is like NV21
but with the chroma components reversed.

Copying YV12 to I420 is just copying three planes but switching the order
of the last two ones. Copying NV21 to NV12 is copying the luma plane and
swapping the order of each pair of chroma bytes. Copying YV12 to NV12
requires interleaving chroma components from the two chroma planes in YV12
to the single chroma plane in NV12.

The qualcomm encoder on Nexus4 takes NV12 input, so it looks like you're
feeding it YV12-formatted data.

- The YV12 data from the camera is specified to have a certain padding
(see the docs for the ImageFormat class), while the NV21 format data is
tightly packed. When copying the camera buffers to to MediaCodec and
converting from the camera formats to the encoder ones, this has to be
taken into account.

- Don't worry about the actual size of the input buffers, their size might
be rounded up to some larger size - just fill it with the amount of data
you have.

- Most encoders assume tightly packed input data to the encoders, as long
as the dimensions are multiples of 16. If the dimensions are not tightly
packed, some (Nexus 7) require you to implicitly pad the line sizes and
number of lines to the next multiple of 16. Others (Nexus S) don't require
you to do this.


So as long as you check MediaCodecInfo.CodecCapabilities.colorFormats[]
and convert the pixel data accordingly, and stay off non-multiple-of-16
dimensions, you should be safe in general. As long as you're not on
Samsung devices, because some of them interpret the color formats totally
differently, so you'd need to have special logic to work around this.

And as long as you're not running on the qualcomm encoder
OMX.qcom.video.encoder.avc. On this one, there's no extra padding at the
end of each line, and no extra padding lines at the end of planes. But the
chroma plane starts aligned at a 2048 byte boundary. This is not signalled
in the API in any way (and if I recall correctly, not all qualcomm
encoders assume this alignment, only the current/previous generation
IIRC).

... and as long as you're not running on some device with buggy drivers.
On certain manufacturers devices with the same qualcomm encoder as on
Nexus4, the MediaCodecInfo.getCapabilitiesForType() call takes 5 seconds
to return, which might not fit well in the codec startup. So on these you
can't call the proper querying method but simply has to assume that it
supports NV12 and run with that.

// Martin

[Chen Fanglin]

Hi Hai, Did you find out the reason why?

Fanglin

On Wednesday, March 27, 2013 7:46:00 PM UTC-4, long hai wrote:

I meet the similar problem to you. I want to decode a mp4 video file to get the raw video frame. Now I can get the right result on AVD, and know the format of  the raw video frame is YUV420p. But when I run the same code to decode the same video on nexus 4, the color format and buffer length are all wrong. It shows as follows:

On nexus 4: {height=180, what=1869968451, color-format=2141391875, slice-height=192, crop-left=0, width=320, crop-bottom=179, crop-top=0, mime=video/raw, stride=384, crop-right=319}

On AVD: {height=192, what=1869968451, color-format=19, slice-height=192, crop-left=0, width=320, crop-bottom=179, crop-top=0, mime=video/raw, stride=320, crop-right=319}

I found that the INFO_OUTPUT_BUFFERS_CHANGED is never called on nexus 4, but it will be called on AVD before INFO_OUTPUT_FORMAT_CHANGED. Have you found the answer and known why?

[long hai]

Hello Chen,

    I didn't find the exact reason why the video format is not same. But I found that the decoding method on AVD is not same to that on nexus 4. Nexus 4 uses the hardware decoding which is GPU decoding, while AVD is apparently the CPU based software decoding. So the output raw video format decoded by different methods may be different. Moreover, I knew that format 2141391875 is a HD video format specialized for Qualcomm products.