Decoding interlaced video through decoder,push

[Bernard Maassen]

As said we've run into a problem with decoding some video. The video is interlaced.
I've checked the sample PullPushDecoder and checked the following:

• compared VideoStreamInfo from output and input stream for the sample and our code, they are 100% the same
• compared the data that the sample is pushing into the decoder against what we push. Here there is a total mismatch. It seems that your code that reads the data file and retrieves samples from it using decoder.pull() is processing the data.

I got the feeling that the transcoder1 is "merging" the interlaced h264 frames into new h264 frames, instead of just delivering the raw h264 samples. And that the decoder.push() can't handle the interlaced frames.

I could create a sample of our code that shows the issue and send it to you, but would prefer to mail it, instead of posting it here.

Bernard

[Svilen Stoilov]

Hi Bernard,

 

The main purpose of the PullPushDecoder sample is to demonstrate the Transcoder::push() interface through the second transcoder.

The first transcoder is used just to split the input stream into frames.

However you may be right that the first transcoder actually transforms the data instead of just delivering the compressed frames.

This depends on the input/output StreamInfo of the first transcoder.

I believe that if the input stream is interlaced and the output StreamInfo is progressive (or even not specified explicitly) the first transcoder would decode, de-interlace and re-encode the video.

It may be that you are observing this effect.

 

But it would be best you if you can send us an input stream so that we can run it through the PullPushDecoder sample.

Or you may send us your test project if you have specific input/output settings.

Please use the support email: support at avblocks.com

 

Thanks,

Svilen

[sander.borsboom]

Hi Svilen,

Bernard got sick, so I took over fixing this bug.

I finally found out the problem: 

• The input stream has interlaceType (This case: "TopFieldFirst")
• We set the output stream to interlaceType "Progressive".
• We pushed each "half" frame separatly to the Transcoder.
• We flushed the transcoder after each push.

We were able to get a correct stream of decoder frames in two separate ways:

1. flush only every second frame, allowing the transcoder to build up a full (sort of deinterlaced?) frame.
   
2. Set the output stream to the same interlaceType as the input stream, resulting (it seems, didn't fully check yet) in one write for each two flushes with a full, but still interlaced frame.

So some questions (might have been already answered in the direct mail correspondance with Bernard)

1. Is the output we got without the fixes by design? With the output interlaceType set to progressive, we expected something like one correct (deinterlaced?) output frame for each two flushes instead of the one good frame for each IDR followed by grey difference images for each non-IDR.
2. Our old decoder didn't have any deinterlacing options, so we implemented our own approaches. Does AVBlocks have any deinterlacing options? When using option 1, it does seems to deinterlace. This is not a priority for us, just wondering if we are not missing some option somewhere.

Greetings,

Sander Borsboom

Op maandag 11 november 2013 15:04:54 UTC+1 schreef Svilen Stoilov:

[Svilen Stoilov]

Hi Sander,

First to your questions:
>> 1.Is the output we got without the fixes by design? With the output interlaceType set to progressive, we expected something like one correct (deinterlaced?) output frame for each two flushes instead of the one good frame for each IDR followed by grey difference images for each non-IDR.

I believe this is a bug. No grey frame should get out of the transcoder (or any internal decoder) under any circumstances and regardless of the methods you are calling.

>> 2.Our old decoder didn't have any deinterlacing options, so we implemented our own approaches. Does AVBlocks have any deinterlacing options? When using option 1, it does seems to deinterlace. This is not a priority for us, just wondering if we are not missing some option somewhere.

There is internal automatic deinterlacing. It comes into effect when the input is interlaced (top or bottom field first) and the output is progressive.
Now in AVBlocks 1.7.0 (to be released soon) we are adding options to specify the deinterlacing method.
We also plan to add an option for manual deinterlacing - on|off. This would be useful when the input video has a wrong scan type: interlaced instead of progressive and vice versa.

* Now about the grey frame bug
We'd like to explore and fix this case. You can help us a lot by providing the input stream and the code snippet that uses the transcoder.
In the past Bernard has sent a sample stream in the following format: a file containing the input video stream and another binary file describing the chunks (frames) that are received (or pushed to the transcoder).
Let's say that the input file is called "sample_video".
Then the other file "samples_video.frames" would contain the sizes (or offsets) of the frames that you receive.
Any format for frames/chunks description would work for us as long as you specify it.

* About transcoder flushing.
I don't understand why you need to flush the transcoder every (other) frame.
Transcoder::flush should be called when there's no more input data.
I think it's not guaranteed that Transcoder::push() and Transcoder::flush() calls can be alternated safely.
This is so because after Transcoder::flush() is called some internal encoders/decoders may do specific things for end-of-stream processing
and if more data is provided afterwards strange things may happen.
There's little buffering inside Transcoder.
If you push input data and nothing comes out of the output stream then almost certainly more input is needed for some reason (perhaps frames reordering for B-frames)
It would be helpful if you give more details about the need for flushing transcoder before end-of-stream.

Thanks,
Svilen

[Bernard Maassen]

Hi Svilen,

If you change the pullPushDecoder in the following way you can reproduce the problem:
- in the constructor of the ElemStream: add vsi->setInterlaceType( InterlaceType::TopFieldFirst ); such that the pulled data is still interlaced
- in the while loops if block decoderInitialized, add transcoder2->flush(); after res = transcoder2->push(index, data.get());

I've also attached the sample video that i used to reproduce the issue on the pullpush decoder

I will test today what happens on our side when we remove the flush, and move it to when we close the stream