How can I implement the long time reference mechanism for AV1 in RTC mode?

[klaus zhang]

hi,buddies

I'm using AV1 based on libaom in WebRTC library,and I want to implement the long time reference mechanism for the AV1 encoder,so far as I know,the only way is to make encoded frames reference golden frame only.

How can I do that ?Appreciated to have your answer.

[Marco Paniconi]

You should be able to do this with the reference control via set_svc_ref_frame_config, see the examples in example/svc_encoder_rtc.cc, there is an example there (use_rps_example) that may be useful.

[klaus zhang]

Thank you so much,I will try this by forcing SVC enalbed in the RTC mode.

[klaus zhang]

# create and config libaom encoder for manual frame reference

- create and init the encoder

'''
int LibaomAv1Encoder::InitEncode2(const VideoCodec* codec_settings,
                                  const Settings& settings) {
  if (codec_settings == nullptr) {
    RTC_LOG(LS_WARNING) << "No codec settings provided to "
                           "LibaomAv1Encoder.";
    return WEBRTC_VIDEO_CODEC_ERR_PARAMETER;
  }
  if (settings.number_of_cores < 1) {
    return WEBRTC_VIDEO_CODEC_ERR_PARAMETER;
  }
  RTC_LOG(LS_INFO) << __FUNCTION__ << "," << codec_settings->ToString();
  if (inited_) {
    RTC_LOG(LS_WARNING) << "Initing LibaomAv1Encoder without first releasing.";
    Release();
  }
  encoder_settings_ = *codec_settings;

  // Sanity checks for encoder configuration.
  const int32_t result = VerifyCodecSettings(encoder_settings_);
  if (result < 0) {
    RTC_LOG(LS_WARNING) << "Incorrect codec settings provided to "
                           "LibaomAv1Encoder.";
    return result;
  }
  if (encoder_settings_.numberOfSimulcastStreams > 1) {
    RTC_LOG(LS_WARNING) << "Simulcast is not implemented by LibaomAv1Encoder.";
    return result;
  }
  xabsl::string_view scalability_mode = encoder_settings_.ScalabilityMode();
  if (scalability_mode.empty()) {
    RTC_LOG(LS_WARNING) << "Scalability mode is not set, using 'NONE'.";
    scalability_mode = "NONE";
  }
  RTC_LOG(LS_INFO) << "Setting SVC mode to:" << scalability_mode;
  svc_controller_ = CreateScalabilityStructure(scalability_mode);
  if (svc_controller_ == nullptr) {
    RTC_LOG(LS_WARNING) << "Failed to set scalability mode "
                        << scalability_mode;
    return WEBRTC_VIDEO_CODEC_ERR_PARAMETER;
  }

  if (!SetSvcParams(svc_controller_->StreamConfig())) {
    return WEBRTC_VIDEO_CODEC_ERROR;
  }

  // Initialize encoder configuration structure with default values
  aom_codec_err_t ret =
      aom_codec_enc_config_default(aom_codec_av1_cx(), &cfg_, kUsageProfile);
  if (ret != AOM_CODEC_OK) {
    RTC_LOG(LS_WARNING) << "LibaomAv1Encoder::EncodeInit returned " << ret
                        << " on aom_codec_enc_config_default.";
    return WEBRTC_VIDEO_CODEC_ERROR;
  }

  Av1StrategyExperiment experiment = Av1StrategyExperiment::ParseSettings();
  // Overwrite default config with input encoder settings & RTC-relevant values.
  cfg_.g_w = encoder_settings_.width;
  cfg_.g_h = encoder_settings_.height;
  //[Issue 3265: Encoding speed get longer when value of
  // aom_codec_enc_cfg_t.g_threads
  // increased](https://bugs.chromium.org/p/aomedia/issues/detail?id=3265&q=reporter%3Aaspling.zhang%40gmail.com&can=1)
  cfg_.g_threads =
      NumberOfThreads(cfg_.g_w, cfg_.g_h, settings.number_of_cores);
  RTC_LOG(LS_INFO) << "number of threads used for av1 encoding is: "
                   << cfg_.g_threads << ",g_w:" << cfg_.g_w
                   << ",g_h:" << cfg_.g_h
                   << ",encoder_settings_.qpMax:" << encoder_settings_.qpMax;
  cfg_.g_timebase.num = 1;
  cfg_.g_timebase.den = kRtpTicksPerSecond;
  cfg_.rc_target_bitrate = encoder_settings_.maxBitrate;  // kilobits/sec.
  cfg_.g_input_bit_depth = kBitDepth;
  cfg_.kf_mode = AOM_KF_DISABLED;
  cfg_.rc_min_quantizer = kQpMin;
  cfg_.rc_max_quantizer = 63;
  // To improve the quality of screencast,we set a restriction on max quantizer.
  // As the trade-off it will slow down the encoding speed .
  if (codec_settings->mode == VideoCodecMode::kScreensharing &&
      experiment.max_qp_screencast() != -1) {
    cfg_.rc_max_quantizer = experiment.max_qp_screencast();
  }
  cfg_.g_usage = kUsageProfile;
  cfg_.g_error_resilient = 0;
  // Low-latency settings.
  cfg_.rc_end_usage = AOM_CBR;          // Constant Bit Rate (CBR) mode
  cfg_.g_pass = AOM_RC_ONE_PASS;        // One-pass rate control
  cfg_.g_lag_in_frames = kLagInFrames;  // No look ahead when lag equals 0.
  if (experiment.fastest_mode() && experiment.enable_buffer_optimization()) {
    // TODO(kluas):controlled by cloud config?
    cfg_.rc_undershoot_pct = 50;
    cfg_.rc_overshoot_pct = 1;
    cfg_.rc_buf_initial_sz = 600;
    cfg_.rc_buf_optimal_sz = 600;
    cfg_.rc_buf_sz = 1000;
  };

  // Creating a wrapper to the image - setting image data to nullptr. Actual
  // pointer will be set in encode. Setting align to 1, as it is meaningless
  // (actual memory is not allocated).
  frame_for_encode_ =
      aom_img_alloc(nullptr, AOM_IMG_FMT_I420, cfg_.g_w, cfg_.g_h, 1);

  // Flag options: AOM_CODEC_USE_PSNR and AOM_CODEC_USE_HIGHBITDEPTH
  aom_codec_flags_t flags = 0;

  // Initialize an encoder instance.
  ret = aom_codec_enc_init(&ctx_, aom_codec_av1_cx(), &cfg_, flags);
  if (ret != AOM_CODEC_OK) {
    RTC_LOG(LS_WARNING) << "LibaomAv1Encoder::EncodeInit returned " << ret
                        << " on aom_codec_enc_init.";
    return WEBRTC_VIDEO_CODEC_ERROR;
  }
  inited_ = true;

  aom_codec_control(&ctx_, AOME_SET_CPUUSED, 10);
  aom_codec_control(&ctx_, AV1E_SET_AQ_MODE, 3);
  aom_codec_control(&ctx_, AV1E_SET_GF_CBR_BOOST_PCT, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_CDEF, 1);
  aom_codec_control(&ctx_, AV1E_SET_LOOPFILTER_CONTROL, 1);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_WARPED_MOTION, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_OBMC, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_GLOBAL_MOTION, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_ORDER_HINT, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_TPL_MODEL, 0);
  aom_codec_control(&ctx_, AV1E_SET_DELTAQ_MODE, 0);
  aom_codec_control(&ctx_, AV1E_SET_COEFF_COST_UPD_FREQ, 3);
  aom_codec_control(&ctx_, AV1E_SET_MODE_COST_UPD_FREQ, 3);
  aom_codec_control(&ctx_, AV1E_SET_MV_COST_UPD_FREQ, 3);
  aom_codec_control(&ctx_, AV1E_SET_DV_COST_UPD_FREQ, 3);
  aom_codec_control(&ctx_, AV1E_SET_CDF_UPDATE_MODE, 1);

  // Settings to reduce key frame encoding time.
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_CFL_INTRA, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_SMOOTH_INTRA, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_ANGLE_DELTA, 0);
  aom_codec_control(&ctx_, AV1E_SET_ENABLE_FILTER_INTRA, 0);
  aom_codec_control(&ctx_, AV1E_SET_INTRA_DEFAULT_TX_ONLY, 1);
  aom_codec_control(&ctx_, AV1E_SET_AUTO_TILES, 1);
  aom_codec_control(&ctx_, AV1E_SET_MAX_CONSEC_FRAME_DROP_MS_CBR, INT_MAX);

  aom_codec_control(&ctx_, AV1E_SET_SVC_FRAME_DROP_MODE,
                    AOM_FULL_SUPERFRAME_DROP);
  aom_codec_control(&ctx_, AV1E_SET_POSTENCODE_DROP_RTC, 1);
  aom_codec_control(&ctx_, AV1E_SET_SVC_PARAMS, &*svc_params_);

  return WEBRTC_VIDEO_CODEC_OK;
}
'''

- use svc for configuration of frame references

'''
void LibaomAv1Encoder::SetSvcRefFrameConfig2( ) {
  const int slot_number = 8;  // slots 0 - 7.
  int last_idx = 0;
  // Moving index slot for last: 0 - (sh - 1)
  if (superframe_cnt_ > 1)
    last_idx = (superframe_cnt_ - 1) % slot_number;

  aom_svc_ref_frame_config_t ref_frame_config = {};
  for (int i = 0; i < 7; i++)
    ref_frame_config.reference[i] = 0;

  int slot_index = 3;  // slot for golden frame

  if (superframe_cnt_ > 0) {
    ref_frame_config.ref_idx[slot_index] = 3;
    ref_frame_config.reference[slot_index] = 1;
  }
  // Update every 8 frames
  if (superframe_cnt_ % slot_number == 0)
    ref_frame_config.refresh[3] = 1;
  aom_codec_err_t ret = aom_codec_control(&ctx_, AV1E_SET_SVC_REF_FRAME_CONFIG,
                                          &ref_frame_config);
  if (ret != AOM_CODEC_OK) {
    RTC_LOG(LS_WARNING) << "LibaomAv1Encoder::Encode returned " << ret
                        << " on control AV1_SET_SVC_REF_FRAME_CONFIG.";
  }
  aom_svc_ref_frame_comp_pred_t ref_frame_comp_pred;
  ref_frame_comp_pred.use_comp_pred[0] = 0;  // GOLDEN_LAST
  ref_frame_comp_pred.use_comp_pred[1] = 0;  // LAST2_LAST
  ref_frame_comp_pred.use_comp_pred[2] = 0;  // ALTREF_LAST
  ret = aom_codec_control(&ctx_, AV1E_SET_SVC_REF_FRAME_COMP_PRED,
                          &ref_frame_comp_pred);
  if (ret != AOM_CODEC_OK) {
    RTC_LOG(LS_WARNING) << "LibaomAv1Encoder::Encode returned " << ret
                        << " on control AV1E_SET_SVC_REF_FRAME_COMP_PRED.";
  }
}
'''

# result
- it crashes

# questions
My goal is that
1. encoder encodes frames,the first frame is a key frame and also be a golden frame
2. encoded frames is sent to the decoder and if it was decoded successfully,it sends back a feedback to the encoder to tell the encoder which frame has been decoded.
3. at encoder side,all frames only reference the golden frame
4. once receving the feedback of decoder,change the golden frame to that frame and use it as the only reference then.

with goal above,is it right for the code on setting frame reference manually above?How can I implement it into a right way? thanks very much. 

[Marco Paniconi]

Thanks for the test code , it looks you're setting the references correctly.  

I was able to patch your code from SetSvcRefFrameConfig2() into the examples/svc_encoder_rtc.cc and reproduce the failure in the av1 encoder.

Can you try this patch and let me know if still see issues: https://aomedia-review.googlesource.com/c/aom/+/200361

Thanks!

[klaus zhang]

Thank you so much! It works now.