VP9 ROI support

[yeju...@intel.com]

Hi,

I verified that VP8 ROI (Region Of Insterest) feature with https://github.com/webmproject/libvpx/blob/master/examples/set_maps.c, but could not find a way to utilize VP9 ROI feature, could you help? thanks.

I modified set_maps.c a little to support VP9 ROI and see the key code following. The ROI regions are fixed in the code and i would expect that the video quality is always changed (better or worse) at the fixed locations. But, I see quality changes appear here and there, now and then. I even don't find a pattern.

static void set_roi_map_vp9(const vpx_codec_enc_cfg_t *cfg,

                        vpx_codec_ctx_t *codec) {

  unsigned int i, j;

  vpx_roi_map_t roi;

  memset(&roi, 0, sizeof(roi));

  roi.rows = (cfg->g_h + 7) / 8;

  roi.cols = (cfg->g_w + 7) / 8;

  roi.delta_q[0] = 0;

  roi.delta_q[1] = -61;

  roi.roi_map = (uint8_t *)malloc(roi.rows * roi.cols);

  for (i = 0; i < roi.rows; ++i) {

    for (j = 0; j < roi.cols; ++j) {

        if (j > roi.cols/2)

            roi.roi_map[i*roi.cols + j] = 0;

        else

            roi.roi_map[i*roi.cols + j] = 1;

    }

  }

  if (vpx_codec_control(codec, VP9E_SET_ROI_MAP, &roi))

    die_codec(codec, "Failed to set ROI map");

  free(roi.roi_map);

  //tried both with/without the following line

  // i tried this because of the sample code at https://github.com/webmproject/libvpx/blob/master/examples/vpx_temporal_svc_encoder.c line 844

  //vpx_codec_control(codec, VP9E_SET_AQ_MODE, 0/1);  //tried both 0 and 1

  {

      vpx_active_map_t map = { 0, 0, 0 };

      map.rows = (cfg->g_h + 15) / 16;

      map.cols = (cfg->g_w + 15) / 16;

      map.active_map = (uint8_t *)malloc(map.rows * map.cols);

      for (i = 0; i < map.rows * map.cols; ++i) map.active_map[i] = 1;

      //tried with/without the following lines

      //if (vpx_codec_control(codec, VP8E_SET_ACTIVEMAP, &map))

      //  die_codec(codec, "Failed to set active map");

      free(map.active_map);  

  }

}

[Jerome Jiang]

Hi,

Can you try using vpx_temporal_svc_encoder to verify VP9 ROI feature?

You can set ROI_MAP to 1 here

https://chromium.googlesource.com/webm/libvpx/+/refs/heads/master/examples/vpx_temporal_svc_encoder.c#31

And run the following command line:

examples/vpx_temporal_svc_encoder input.yuv output vp9 width height 1 framerate speed_setting 0 0 num_thread 0 bitrate

We'll change set_map.c to make it work for VP9 ROI.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Codec Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codec-devel...@webmproject.org.
To post to this group, send email to codec...@webmproject.org.
Visit this group at https://groups.google.com/a/webmproject.org/group/codec-devel/.
For more options, visit https://groups.google.com/a/webmproject.org/d/optout.

[yeju...@intel.com]

thanks for the quick reply.

I tried both 1 and 0 for ROI_MAP with the following command line, but don't find quality change which is excepted at the center of the frame (1/4w~3/4w, 1/4h~3/4h).

.../libvpx/build/examples$ ./vpx_temporal_svc_encoder 1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 10 0 0 1 0 250000  

Actually, i reviewed the code, and don't find meaningful difference between vpx_temporal_svc_encoder.c and my function set_roi_map_vp9() posted yesterday.

Thanks

Yejun

[Jerome Jiang]

Thanks for the feedback.

Actually there are series of patches[1] we didn't realize that overwrite segment ID to 0.

I have a pending CL[2] to set segment ID from ROI map.

You can try to run it again with my patch and let me know if you have further questions.

In VP9 we don't have speed 10. You should use speed 7 to speed 9 for realtime encoding.

[1]

https://chromium.googlesource.com/webm/libvpx/+/181dd52a23d8f0e192966bfa18e007554abb2799

https://chromium.googlesource.com/webm/libvpx/+/535e4039cd9d91e7e92a37f09853eda927bff05e

[2]https://chromium-review.googlesource.com/c/webm/libvpx/+/1489256

[yeju...@intel.com]

still does not work, i also observed something weird, not sure if my system's issue.

the command that i used: 

$ ./vpx_temporal_svc_encoder 1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 9 0 0 1 0 250000

the first thing i observed weird is that:

each frame is handled in second, but when I tried yesterday with 2d403737b88f76ce8410237a8eb682cc556ac302, it took ~1 minute to handle each frame.   I tried both ROI_MAP with 1 and 0.

the second thing is that:

after output.map.vp9_0.ivf is dumped, i tried to play it with ffplay on linux. The first several frames played as normal, then, no frame change on the screen for minutes, and then see the last several frames.

anything i can try? thanks.

[Jerome Jiang]

On Wed, Feb 27, 2019 at 12:58 AM <yeju...@intel.com> wrote:

still does not work, i also observed something weird, not sure if my system's issue.

the command that i used: 

$ ./vpx_temporal_svc_encoder 1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 9 0 0 1 0 250000

I used the park joy clip[1] and encoded with ROI_MAP 1 and I can see the segment ID in analyzer.

Could  you please let me know your observations? 250000Kbps is very high bitrate and it may be a bit difficult to see the difference.

It'll be easier if you use 2500Kbps for target bitrate. 

 

the first thing i observed weird is that:

each frame is handled in second, but when I tried yesterday with 2d403737b88f76ce8410237a8eb682cc556ac302, it took ~1 minute to handle each frame.   I tried both ROI_MAP with 1 and 0.

You were trying with speed 10 which is not supported by VP9. 

 

the second thing is that:

after output.map.vp9_0.ivf is dumped, i tried to play it with ffplay on linux. The first several frames played as normal, then, no frame change on the screen for minutes, and then see the last several frames.

I tried ffplay and I also had same problem as you. 

I decoded with vpxdec:

vpxdec --i420 output.map.vp9_0.ivf  -o out_roi.yuv

and out_roi.yuv can be played without problems.

And the Intel analyzer can decode the bitstream.

So I don't think it's libvpx issue. It might be ffplay's problem.

[yeju...@intel.com]

On Thursday, February 28, 2019 at 3:21:43 AM UTC+8, Jerome Jiang wrote:

On Wed, Feb 27, 2019 at 12:58 AM <yeju...@intel.com> wrote:

still does not work, i also observed something weird, not sure if my system's issue.

the command that i used: 

$ ./vpx_temporal_svc_encoder 1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 9 0 0 1 0 250000

I used the park joy clip[1] and encoded with ROI_MAP 1 and I can see the segment ID in analyzer.

Could  you please let me know your observations? 250000Kbps is very high bitrate and it may be a bit difficult to see the difference.

It'll be easier if you use 2500Kbps for target bitrate. 

thanks, i'v described my observations below :).        Just thought bitrate in bps, not Kbps, will change.

 

 

the first thing i observed weird is that:

each frame is handled in second, but when I tried yesterday with 2d403737b88f76ce8410237a8eb682cc556ac302, it took ~1 minute to handle each frame.   I tried both ROI_MAP with 1 and 0.

You were trying with speed 10 which is not supported by VP9. 

possible to add a check and report the issue if some one uses the invalid value?

 

 

the second thing is that:

after output.map.vp9_0.ivf is dumped, i tried to play it with ffplay on linux. The first several frames played as normal, then, no frame change on the screen for minutes, and then see the last several frames.

I tried ffplay and I also had same problem as you. 

I decoded with vpxdec:

vpxdec --i420 output.map.vp9_0.ivf  -o out_roi.yuv

and out_roi.yuv can be played without problems.

yes, this way works pretty well.

 

And the Intel analyzer can decode the bitstream.

So I don't think it's libvpx issue. It might be ffplay's problem.

 

 

I found that your patch has been pushed, and so tried with latest libvpx code with 503cb8e63a06ab62ca814bcf79ef9971bdaa745a, but still do not see the quality change with my eyes after changing ROI_MAP=1

here are my commands used:

$ ./examples/vpx_temporal_svc_encoder /work/media/ffmpeg/video/1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 9 0 0 1 0 2500 

$ ./vpxdec --i420  output.map.vp9_0.ivf  -o out_roi.yuv

$ /work/media/ffmpeg/build_vpxroi/ffplay -s 1920x1080 -i out_roi.yuv

[Jerome Jiang]

On Thu, Feb 28, 2019 at 11:58 PM <yeju...@intel.com> wrote:

On Thursday, February 28, 2019 at 3:21:43 AM UTC+8, Jerome Jiang wrote:

On Wed, Feb 27, 2019 at 12:58 AM <yeju...@intel.com> wrote:

still does not work, i also observed something weird, not sure if my system's issue.

the command that i used: 

$ ./vpx_temporal_svc_encoder 1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 9 0 0 1 0 250000

I used the park joy clip[1] and encoded with ROI_MAP 1 and I can see the segment ID in analyzer.

Could  you please let me know your observations? 250000Kbps is very high bitrate and it may be a bit difficult to see the difference.

It'll be easier if you use 2500Kbps for target bitrate. 

thanks, i'v described my observations below :).        Just thought bitrate in bps, not Kbps, will change.

 

 

the first thing i observed weird is that:

each frame is handled in second, but when I tried yesterday with 2d403737b88f76ce8410237a8eb682cc556ac302, it took ~1 minute to handle each frame.   I tried both ROI_MAP with 1 and 0.

You were trying with speed 10 which is not supported by VP9. 

possible to add a check and report the issue if some one uses the invalid value?

Yeah we can add that. We can map speed > 9 to speed 9 and report warning to users. 

 

 

the second thing is that:

after output.map.vp9_0.ivf is dumped, i tried to play it with ffplay on linux. The first several frames played as normal, then, no frame change on the screen for minutes, and then see the last several frames.

I tried ffplay and I also had same problem as you. 

I decoded with vpxdec:

vpxdec --i420 output.map.vp9_0.ivf  -o out_roi.yuv

and out_roi.yuv can be played without problems.

yes, this way works pretty well.

 

And the Intel analyzer can decode the bitstream.

So I don't think it's libvpx issue. It might be ffplay's problem.

 

 

I found that your patch has been pushed, and so tried with latest libvpx code with 503cb8e63a06ab62ca814bcf79ef9971bdaa745a, but still do not see the quality change with my eyes after changing ROI_MAP=1

here are my commands used:

$ ./examples/vpx_temporal_svc_encoder /work/media/ffmpeg/video/1080p_parkjoy_100frames.yuv output.map.vp9 vp9 1920 1080 1 30 9 0 0 1 0 2500 

$ ./vpxdec --i420  output.map.vp9_0.ivf  -o out_roi.yuv

$ /work/media/ffmpeg/build_vpxroi/ffplay -s 1920x1080 -i out_roi.yuv

I tried that again and I saw big difference between ROI and other area:

(A lot of details on one side of the red line)

 

After you run vpx_temporal_svc_encoder, it'll report encoding bitrate. Due to ROI's very low QP set in the example, the encoding bitrate will be much higher than the target (2500Kbps).

Could you please share your encoded bitstream or the output from the example?

Thanks.

[Peter Gusev]

Hi,

I've been following the thread and thank you for all the discussion posted, it's very informative and helpful in my explorations.

I have related two questions:

-- can someone point me to where I can read up more on ROI in VP9?

-- according to the source code, I see that for ROI to work, it has to be set up on the encoder side, is there a way to have encoder to produce (possibly using SVC) a stream where ROI selection happens on the decoder side (for example, for the area that falls outside ROI decoder gets only base layer, while for ROI -- all the enhancement layers too)?

Thank you in advance,

--

Peter Gusev

gpee...@gmail.com

+1 213 587-27-48

Research Scholar @ REMAP UCLA
Video streaming/ICN networks/Creative Coding/Interactive Media

dj peetonn

[yeju...@intel.com]

i checked again, and yes it works, thanks.

i have another two questions fro vp9, thanks.

1. it works only if aq_mode is 0?

I saw this line: vpx_codec_control(&codec, VP9E_SET_AQ_MODE, 0);

i tried other values and found it works with a little quality change (and less bitrate change), but the result is not as clear as value 0.

looks that my code should be like below, do you have any comment? thanks.

if (aq_mode != 0) {

   //output an warning for the result

}

//set the roi map ...

2. how to choose ref_frame?

there is code:

    memset(roi->ref_frame, -1, sizeof(roi->ref_frame));

    roi->ref_frame[1] = 1;

does this mean that the recommendation is roi->ref_frame[i] should be always 1, and the rest ref_frame should be -1?

(suppose i is the segment index that is set in roi_map)

[Jerome Jiang]

On Tue, Mar 5, 2019 at 6:12 PM <yeju...@intel.com> wrote:

i checked again, and yes it works, thanks.

i have another two questions fro vp9, thanks.

1. it works only if aq_mode is 0?

I saw this line: vpx_codec_control(&codec, VP9E_SET_AQ_MODE, 0);

i tried other values and found it works with a little quality change (and less bitrate change), but the result is not as clear as value 0.

looks that my code should be like below, do you have any comment? thanks.

if (aq_mode != 0) {

   //output an warning for the result

}

//set the roi map ...

AQ mode also uses segmentation to boost QP for part of the frame. However, in current implementation, AQ mode always uses segment 1 and 2 (the first 2).

Also we give AQ mode higher priority over segmentation:

https://chromium.googlesource.com/webm/libvpx/+/refs/heads/master/vp9/encoder/vp9_encoder.c#3935

We only apply ROI map when none of AQ mode is on.

2. how to choose ref_frame?

there is code:

    memset(roi->ref_frame, -1, sizeof(roi->ref_frame));

    roi->ref_frame[1] = 1;

does this mean that the recommendation is roi->ref_frame[i] should be always 1, and the rest ref_frame should be -1?

(suppose i is the segment index that is set in roi_map)

Setting ref_frame in ROI to 1 means forcing using last frame as reference frame.

However, in VP9 real-time mode, last frame and golden frame are both used as reference.

[Jerome Jiang]

Hi Peter,

Sorry for late reply.

Please see my inline comments and let me know if you have further questions.

On Fri, Mar 1, 2019 at 12:01 PM Peter Gusev <gpee...@gmail.com> wrote:

Hi,

I've been following the thread and thank you for all the discussion posted, it's very informative and helpful in my explorations.

I have related two questions:

-- can someone point me to where I can read up more on ROI in VP9?

The attached slides is from a conference where I gave talk about VP9 last year.

It covers more than ROI but Page 20 to 29 mostly covers the implementation and APIs used for ROI.

-- according to the source code, I see that for ROI to work, it has to be set up on the encoder side, is there a way to have encoder to produce (possibly using SVC) a stream where ROI selection happens on the decoder side (for example, for the area that falls outside ROI decoder gets only base layer, while for ROI -- all the enhancement layers too)?

Do you mean making ROI to work for SVC spatial layers?

E.g. the encoder is set to encode all spatial layers, and decoder decodes all spatial layers for ROI area and only base layer for area outside ROI?

[Peter Gusev]

Hi Jerome,

Sorry, I wasn't able to find slides attached to the message, could you please resend?

> Do you mean making ROI to work for SVC spatial layers?

> E.g. the encoder is set to encode all spatial layers, and decoder decodes all spatial layers for ROI area and only base layer for area outside ROI?

Yes, correct. And my interest is whether decoder can specify ROI (hopefully I'll find an answer for this in the slides).

Thanks,

[Jerome Jiang]

Oops I forgot to attach it.

Unfortunately we don't support specifying ROI on decoder side at the moment.

[yeju...@intel.com]

On Thursday, March 7, 2019 at 3:00:59 AM UTC+8, Jerome Jiang wrote:

On Tue, Mar 5, 2019 at 6:12 PM <yeju...@intel.com> wrote:

i checked again, and yes it works, thanks.

i have another two questions fro vp9, thanks.

1. it works only if aq_mode is 0?

I saw this line: vpx_codec_control(&codec, VP9E_SET_AQ_MODE, 0);

i tried other values and found it works with a little quality change (and less bitrate change), but the result is not as clear as value 0.

looks that my code should be like below, do you have any comment? thanks.

if (aq_mode != 0) {

   //output an warning for the result

}

//set the roi map ...

AQ mode also uses segmentation to boost QP for part of the frame. However, in current implementation, AQ mode always uses segment 1 and 2 (the first 2).

Also we give AQ mode higher priority over segmentation:

https://chromium.googlesource.com/webm/libvpx/+/refs/heads/master/vp9/encoder/vp9_encoder.c#3935

We only apply ROI map when none of AQ mode is on.

thanks, i'll change code like below:

if (aq_mode == 0) {

   // setup roi_map

} else {

  // print a warning that roi_map not enabled for current version

}

 

2. how to choose ref_frame?

there is code:

    memset(roi->ref_frame, -1, sizeof(roi->ref_frame));

    roi->ref_frame[1] = 1;

does this mean that the recommendation is roi->ref_frame[i] should be always 1, and the rest ref_frame should be -1?

(suppose i is the segment index that is set in roi_map)

Setting ref_frame in ROI to 1 means forcing using last frame as reference frame.

However, in VP9 real-time mode, last frame and golden frame are both used as reference.

I'm adding ROI path from FFmpeg to libvpx, and don't want to touch other logic.

For this case, which value do you recommended fro ref_frame, 1 (last frame)?  or just any value (0/1/2/3) except -1? 

And, is there a plan for the next libvpx release? I'm afraid i have to do version check for the ffmpeg+vp9 roi path.

btw, just fyi, this is my patch for ffmepg+vp8 roi, see at http://ffmpeg.org/pipermail/ffmpeg-devel/2019-March/240569.html

[Jerome Jiang]

If you don't wanna touch ROI for reference frames, you can just leave it and it'll be initialized to -1.

 

And, is there a plan for the next libvpx release? I'm afraid i have to do version check for the ffmpeg+vp9 roi path.

James Zern (jz...@google.com) can answer your questions related to libvpx release.

[James Zern]

On Thu, Mar 7, 2019 at 8:58 AM 'Jerome Jiang' via Codec Developers
<codec...@webmproject.org> wrote:
>
>
>
> On Wed, Mar 6, 2019 at 6:32 PM <yeju...@intel.com> wrote:

>> [...]

>>
>> And, is there a plan for the next libvpx release? I'm afraid i have to do version check for the ffmpeg+vp9 roi path.
>>
>
> James Zern (jz...@google.com) can answer your questions related to libvpx release.
>

We plan on making a release within the next 3-4 months.

[Yejun Guo]

Hi, 

I tried libvpx-vp9 roi in ffmpeg and found that ROI feature is only enabled with conditions:

1). none of AQ mode is on  (see above)

2). conditions at line 592, see below.

 577| static void apply_roi_map(VP9_COMP *cpi) {

 578|   VP9_COMMON *cm = &cpi->common;

 579|   struct segmentation *const seg = &cm->seg;

 580|   vpx_roi_map_t *roi = &cpi->roi;

 581|   const int *delta_q = roi->delta_q;

 582|   const int *delta_lf = roi->delta_lf;

 583|   const int *skip = roi->skip;

 584|   int ref_frame[8];

 585|   int internal_delta_q[MAX_SEGMENTS];

 586|   int i;

 587|   static const int flag_list[4] = { 0, VP9_LAST_FLAG, VP9_GOLD_FLAG,

 588|                                     VP9_ALT_FLAG };

 589|

 590|   // TODO(jianj): Investigate why ROI not working in speed < 5 or in non

 591|   // realtime mode.

 592+>  if (cpi->oxcf.mode != REALTIME || cpi->oxcf.speed < 5) return;

 593|   if (!roi->enabled) return;

 594|

 595|   memcpy(&ref_frame, roi->ref_frame, sizeof(ref_frame));

 596|

/work/media/libvpx/vp9/encoder/vp9_encoder.c 

Is there a libvpx API that I can use to query these conditions? for example, vpx_codec_roi_supported().

or can i query them one by one? for exmaple, vpx_codec_get_aq_mode/speed/mode...

thanks