Transcoder output moment

[sander.borsboom]

Hello,

as seen in another thread, we were manually flushing the Transcoder in a Pull-Push scenario to force it to write decoded frames after pushing an encoded frame. As we have understood, flush should be called only once, at the end of the stream, so we were wondering how we should achieve the following:

1. We configure a Transcoder with a custom Primo::Stream output stream that allows us to easily retrieve each write to it.
2. We receive an encoded H.264 frame from a camera (could be SPS, PPS, IDR or non-IDR).
3. We push the frame to the Transcoder.
4. Transcoder writes a decoded raw frame to the output stream or does nothing at all in the case of an undecodable frame (eg. SPS & PPS).
5. We retrieve the write (or NULL if nothing was written) and display it.

What seems to happen is:

• The first X frames (with the test data 15) are pushed, but nothing is written to output.
• For all following frames, a push into the Transcoder results in 1 write to the output stream of exactly 1 decoded frame.
• When the stream is finished, the following flush results in 15 separate writes of each exactly 1 decoded frame.

So the end result is that we have to wait for 15 frames before getting the first decoded frame. We would like to know if we are doing something wrong or if we could do something to remove this "buffering" behavior.

We have modified the PullPushDecoder sample to illustrate the behavior and I have attached the modified/added source files, example test data and the console output of running that test data. We modified the 32-bit version of the 1.7 OEM release.

With Kind Regards,

Sander Borsboom

[Svilen Stoilov]

Hi Sander,

thank you for the detailed description of this issue.

We easily reproduced it but we needed a little time to investigate the underlying problem and how it could be solved.

You are not doing anything wrong. This is how the AVC/H.264 decoder functions at the moment.

It buffers a number of decoded frames for possible frame reordering and potentially uses them as reference frames.

The number of buffered frames can vary from 1 to 16 and you are hitting the worst case.

In your sample video the delayed output is not required by the stream but the decoder chooses reliability and conformance than low-latency.

Changing the decoder to output the decoded frames as soon as possible and still be reliable and conformant in all cases is not trivial and it will take some time.

As a possible workaround we can suggest the following:

1. If you have control over the encoding parameters it will help if you use a lower format profile and level (as low as possible).

Please refer to http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Profiles for a short description of profiles and levels in AVC.

Consider your last sample video - it uses High Profile / Level 4.0. But I believe this is not needed actually.

It needs a Baseline profile with Level 2.2 in which case it will probably result in only 1 frame buffering in AVBlocks.

So if you don't use B-frames (I guess you don't) you should always use the Baseline profile when possible: for progressive video.

For interlaced video you have to use either Main or High profile but in this case it's important to use a lower format level.

In general lower resolution and frame rate requires lower format level (see the level table in the link above).

And a lower format level means less buffering in AVBlocks decoding.

Of course this all makes sense if you can control the profile and level of the encoded video.

It's even possible (supported by the AVC/H.264 standard) to explicitly specify the required number of buffered frames in the encoded video itself.

2. We can add a parameter in AVBlocks so that the AVC/H.264 decoder is instructed to work without buffering of decoded frames.

This may cause problems with videos that actually need buffering in order to be decoded.

But it's likely that your video streams (captured by your cameras) do not  need buffering.

So this solution may work for you but it has to be tested with real captured streams. Your video samples turn out to be very useful.

I have questions about your requirements / scenario. Are you aiming for 0 frame decoding latency?

I think that's not possible in certain cases. Is 1 frame decoding latency Ok for you?

Best Regards,

Svilen Stoilov

Developer Products Support

[Sander Borsboom]

Hello Svilen,

 

thank you for the detailed explanation, it helps a lot to clarify the underlying mechanism and reasons.

Our main requirement is to display frames as soon as possible, so if possible 0 latency, but in cases where this is not possible 1 frame would also be ok.

Some more details about our situation to underpin our reasoning:

• The normal framerates we get are very low: on average around 4.75 for h.264 and less than 5% higher than 10 fps. For the average camera, a delay of 16 frames would thus result a bit more than 3 second delay.
• We are unable to change the encoding parameters because of technical, quality and bandwidth reasons: our users want to have the highest quality for the lowest bandwidth, one of the main reasons behind their choice for low framerates. The reason for them is that they only have a limited amount of bandwidth and of course it helps for us/them too since 1 day of video takes less space. Technically, configuring advanced options such as setting the required number of frames to buffer are not given (and it seems not understood since this seems logical for them to do? See note about B-frames.)
• A sizable number of the cameras is movable by the viewer. Any delay between their actions and what they see is extra noticeable and frustrating for the user.
• B-frames, which I guess are the main reason behind the needed buffering are not used by the camera manufacturers as explained by Axis (One of the main ip camera pioneers):

• Axis network video products allow users to set the GOV (group of video) length, which determines how many P-frames should be sent before another I-frame is sent. By decreasing the frequency of I-frames (longer GOV), the bit rate can be reduced. To reduce latency, B-frames are not used. (http://www.axis.com/products/video/about_networkvideo/compression.htm)
  

So, option 1 is pretty much a no-go for us, but option 2 seems to be a good choice for our specific situation. Would you agree? 

If you agree on option 2, the question we will of course get is what ETA would be possible to implement this workaround/extra parameter? We can provide any number of test streams of different manufacturers and camera models for you to test with. Furthermore, after all our testing, this is our final open issue, and avblocks has a lot of priority, so we are very open to a "on your own risk" version with a quick fix to test with for other issues before a well tested/better engineered final fix.

With Kind Regards,

Sander Borsboom

2013/12/4 Svilen Stoilov via AVBlocks Support <avblocks-support+noreply-APn2wQe...@googlegroups.com>

--
You received this message because you are subscribed to the Google Groups "AVBlocks Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to avblocks-suppo...@googlegroups.com.
To post to this group, send email to avblocks...@googlegroups.com.
Visit this group at http://groups.google.com/group/avblocks-support.
For more options, visit https://groups.google.com/groups/opt_out.

--

Met vriendelijke groet / Kind regards,

Sander Borsboom
Technical manager video analytics

Cameramanager.com     Office:       +31(0)88-006.84.50 / +31(0)88-006.84.58 (direct)
Hogehilweg 19               Tech:        sup...@cameramanager.com
1101 CB Amsterdam      Email:       sander....@cameramanager.com
The Netherlands            Site:          www.cameramanager.com

[Svilen Stoilov]

Hi Sander,

we've been doing some tests and prototypes of the proposed fix to achieve decoding latency of 0 or 1 frames.

It looks like we'll be able to complete the fix till the end of this week and provide you with AVBlocks 1.7.1 on December 16.

So far we've tested with 2 sample videos coming from your cameras: one interlaced and one progressive stream.

Best Regards,

Svilen

[Sander Borsboom]

Hi Svilen,

We uploaded a number of test videos on tuesday, did you succeed in decoding them? I wasn't able to verify them because of some computer problems, so I was half expecting an email asking why stream X was weird/broken/etc.

Furthermore, can you verify that the release is still planned for the 16th? We are planning to release our client before the end of the year, so any possible changes in the schedule are important for us.

Greetings and thank you again for the help,

Sander Borsboom  

2013/12/10 Svilen Stoilov via AVBlocks Support <avblocks-support+noreply-APn2wQe...@googlegroups.com>

[Svilen Stoilov]

Hi Sander,

and thank you for these samples. We decode them properly with the proposed fix. We'll be able to achieve 1 frame/field latency. By the way some of your streams already contain a specific instruction to the decoder to buffer at most 1 frame (max_dec_frame_buffering). We'll add a parameter to AVBlocks so that you can explicitly set the required size of the decoding buffer.

I believe that we'll be able to release on December 16.

Best Regards,

Svilen

[VK]

We published AVBlocks 1.8 on the Licensing Site, which is accessible to customers with current licenses. Latest API docs are available on http://doc.avblocks.com/cpp/1.8/. The 1.8 demo versions will be released on www.avblocks.com in the next few days.  

[Svilen Stoilov]

Hi Sander,

There is a new parameter in AVBlocks 1.8.0: MaxDecFrameBuffering (the name is copied from the AVC/H.264 spec: max_dec_frame_buffering).

It controls the decoded picture buffer size of the decoder and indirectly affects the decoding latency.

Documentation:

http://doc.avblocks.com/cpp/1.8/namespaceprimo_1_1avblocks_1_1_param_1_1_decoder_1_1_video_1_1_h264_1_1_v_u_i.html#details

Here are some key notes about using this parameter:

• It must be set to the output pin (find sample code below)
• The supported and tested values are 0 and 1. A negative value is ignored.
• MaxDecFrameBuffering(0) achieves decoding latency of 0 frames  for progressive streams and 1 frame for interlaced streams.
• MaxDecFrameBuffering(1) achieves decoding latency of 1 frame both for progressive and interlaced streams.
• If the stream has B-frames (or more precisely the total number of reference frames is greater than 1) the decoding will still succeed but decoding latency will be greater than the required one. This is not valid for your samples. This is a safety behaviour.
• With MaxDecFrameBuffering(0|1) only one decoding thread is used. So it hurts performance and it must be used only when low decoding latency is the primary goal.

The following code can be used to set the required parameter to an output pin:

void setH264DecodingLatency(primo::avblocks::MediaPin* pin, int32_t desiredLatency)
{
 primo::avblocks::ParameterList* params ( primo::avblocks::Library::createParameterList() );
 primo::avblocks::IntParameter* maxDecFrameBuffering (primo::avblocks::Library::createIntParameter() );
 maxDecFrameBuffering->setName(primo::avblocks::Param::Decoder::Video::H264::VUI::MaxDecFrameBuffering);
 maxDecFrameBuffering->setValue(desiredLatency);
 params->add(maxDecFrameBuffering);
 pin->setParams(params);
 maxDecFrameBuffering->release();
 params->release();
}

If you have any questions about this feature I'll be happy to answer.

Best Regards,

Svilen