[PATCH] vp9: Validate temporal_layer_id_per_spatial in SVC layer-id control

[Nishat Shabbir]

The VP9E_SET_SVC_LAYER_ID control copies the caller-supplied
vpx_svc_layer_id_t::temporal_layer_id_per_spatial[] values into the
encoder SVC state (svc->temporal_layer_id_per_spatial[]) without any
range checking, even though the scalar temporal_layer_id is validated
against ts_number_layers a few lines below.

In BYPASS temporal-layering mode with a ref-frame config set (via
VP9E_SET_SVC_REF_FRAME_CONFIG), set_flags_and_fb_idx_bypass_via_set_ref_frame_config()
copies temporal_layer_id_per_spatial[sl] into svc->temporal_layer_id,
which vp9_one_pass_svc_start_layer() then uses to index the fixed-size
svc->layer_context[VPX_MAX_LAYERS] array. An out-of-range value therefore
produces an out-of-bounds read/write during vpx_codec_encode().

Validate each per-spatial temporal id against ts_number_layers when the
control is set, returning VPX_CODEC_INVALID_PARAM, mirroring the existing
check on the scalar temporal_layer_id.
---
diff --git a/vp9/vp9_cx_iface.c b/vp9/vp9_cx_iface.c
index fc64343..7c1fb54 100644
--- a/vp9/vp9_cx_iface.c
+++ b/vp9/vp9_cx_iface.c
@@ -1918,6 +1918,14 @@ static vpx_codec_err_t ctrl_set_svc_layer_id(vpx_codec_alg_priv_t *ctx,
svc->temporal_layer_id = data->temporal_layer_id;
// Allow for setting temporal layer per spatial layer for superframe.
for (sl = 0; sl < cpi->svc.number_spatial_layers; ++sl) {
+ // Checks on valid temporal_layer_id_per_spatial input: it is used to index
+ // svc->layer_context[] in the encoder, so an out-of-range value results in
+ // an out-of-bounds access.
+ if (data->temporal_layer_id_per_spatial[sl] < 0 ||
+ data->temporal_layer_id_per_spatial[sl] >=
+ (int)ctx->cfg.ts_number_layers) {
+ return VPX_CODEC_INVALID_PARAM;
+ }
svc->temporal_layer_id_per_spatial[sl] =
data->temporal_layer_id_per_spatial[sl];
}
--
2.50.1 (Apple Git-155)

[James Zern]

Thanks for the patch. If you'd like to contribute this to the project please have a look at https://chromium.googlesource.com/webm/libvpx/+/refs/tags/v1.17.0-rc1/CONTRIBUTING.md. If there is a test case that can be added as a unit test to prevent regression, that would be helpful too.

You can also file a bug with more detail if needed by following the instructions here: https://chromium.googlesource.com/webm/libvpx/+/refs/tags/v1.17.0-rc1/README#238.

--
You received this message because you are subscribed to the Google Groups "Codec Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codec-devel...@webmproject.org.
To view this discussion visit https://groups.google.com/a/webmproject.org/d/msgid/codec-devel/178288811332.3881.10546743604055727875%40bugqore.com.

[Nishat Shabbir]

The VP9E_SET_SVC_LAYER_ID control copies the caller-supplied
vpx_svc_layer_id_t::temporal_layer_id_per_spatial[] values into the
encoder SVC state (svc->temporal_layer_id_per_spatial[]) without any
range checking, even though the scalar temporal_layer_id is validated
against ts_number_layers a few lines below.

In BYPASS temporal-layering mode with a ref-frame config set (via
VP9E_SET_SVC_REF_FRAME_CONFIG), set_flags_and_fb_idx_bypass_via_set_ref_frame_config()
copies temporal_layer_id_per_spatial[sl] into svc->temporal_layer_id,
which vp9_one_pass_svc_start_layer() then uses to index the fixed-size
svc->layer_context[VPX_MAX_LAYERS] array. An out-of-range value therefore
produces an out-of-bounds read/write during vpx_codec_encode().

Validate each per-spatial temporal id against ts_number_layers when the
control is set, returning VPX_CODEC_INVALID_PARAM, mirroring the existing
check on the scalar temporal_layer_id.
---

>From e203025f5467b239f75a274a277ca07569089722 Mon Sep 17 00:00:00 2001
From: Nishat Shabbir <nis...@bugqore.com>
Date: Thu, 2 Jul 2026 20:07:04 +0530
Subject: [PATCH] vp9: Validate temporal_layer_id_per_spatial in SVC layer-id
control

The VP9E_SET_SVC_LAYER_ID control copies the caller-supplied
vpx_svc_layer_id_t::temporal_layer_id_per_spatial[] values into the
encoder SVC state (svc->temporal_layer_id_per_spatial[]) without any
range checking, even though the scalar temporal_layer_id is validated
against ts_number_layers a few lines below.

In BYPASS temporal-layering mode with a ref-frame config set (via
VP9E_SET_SVC_REF_FRAME_CONFIG),
set_flags_and_fb_idx_bypass_via_set_ref_frame_config() copies
temporal_layer_id_per_spatial[sl] into svc->temporal_layer_id, which
vp9_one_pass_svc_start_layer() then uses to index the fixed-size
svc->layer_context[VPX_MAX_LAYERS] array. An out-of-range value therefore
produces an out-of-bounds read/write during vpx_codec_encode().

Validate each per-spatial temporal id against ts_number_layers when the
control is set, returning VPX_CODEC_INVALID_PARAM, mirroring the existing
check on the scalar temporal_layer_id.

Add an EncodeAPI unit test that exercises the new validation.
---
test/encode_api_test.cc | 50 +++++++++++++++++++++++++++++++++++++++++
vp9/vp9_cx_iface.c | 8 +++++++
2 files changed, 58 insertions(+)

diff --git a/test/encode_api_test.cc b/test/encode_api_test.cc
index 249db35..b4b764b 100644
--- a/test/encode_api_test.cc
+++ b/test/encode_api_test.cc
@@ -3361,6 +3361,56 @@ TEST(EncodeAPI, Vp9SvcEmptySuperframe) {
ASSERT_EQ(vpx_codec_destroy(&codec), VPX_CODEC_OK);
}

+// VP9E_SET_SVC_LAYER_ID must reject an out-of-range
+// temporal_layer_id_per_spatial value. An unchecked value is later used to
+// index the fixed-size svc->layer_context[] array during encode (in BYPASS
+// temporal-layering mode with a ref-frame config), causing an out-of-bounds
+// access.
+TEST(EncodeAPI, Vp9SvcSetLayerIdInvalidTemporalPerSpatial) {
+ vpx_codec_iface_t *const iface = vpx_codec_vp9_cx();
+ vpx_codec_enc_cfg_t cfg;
+ ASSERT_EQ(vpx_codec_enc_config_default(iface, &cfg, 0), VPX_CODEC_OK);
+
+ cfg.g_w = 320;
+ cfg.g_h = 240;
+ cfg.g_pass = VPX_RC_ONE_PASS;
+ cfg.g_lag_in_frames = 0;
+ cfg.rc_end_usage = VPX_CBR;
+ cfg.ss_number_layers = 2;
+ cfg.ts_number_layers = 1;
+ cfg.rc_target_bitrate = 400;
+ cfg.ss_target_bitrate[0] = 200;
+ cfg.ss_target_bitrate[1] = 200;
+ cfg.layer_target_bitrate[0] = 200;
+ cfg.layer_target_bitrate[1] = 400;
+
+ vpx_codec_ctx_t codec;
+ ASSERT_EQ(vpx_codec_enc_init(&codec, iface, &cfg, 0), VPX_CODEC_OK);
+ ASSERT_EQ(vpx_codec_control(&codec, VP9E_SET_SVC, 1), VPX_CODEC_OK);
+
+ vpx_svc_layer_id_t layer_id = {};
+ layer_id.spatial_layer_id = 0;
+ layer_id.temporal_layer_id = 0;
+
+ // In-range values (< ts_number_layers) are accepted.
+ layer_id.temporal_layer_id_per_spatial[0] = 0;
+ layer_id.temporal_layer_id_per_spatial[1] = 0;
+ ASSERT_EQ(vpx_codec_control(&codec, VP9E_SET_SVC_LAYER_ID, &layer_id),
+ VPX_CODEC_OK);
+
+ // Out-of-range (>= ts_number_layers) is rejected.
+ layer_id.temporal_layer_id_per_spatial[1] = 1;
+ ASSERT_EQ(vpx_codec_control(&codec, VP9E_SET_SVC_LAYER_ID, &layer_id),
+ VPX_CODEC_INVALID_PARAM);
+
+ // Negative is rejected.
+ layer_id.temporal_layer_id_per_spatial[1] = -1;
+ ASSERT_EQ(vpx_codec_control(&codec, VP9E_SET_SVC_LAYER_ID, &layer_id),
+ VPX_CODEC_INVALID_PARAM);
+
+ ASSERT_EQ(vpx_codec_destroy(&codec), VPX_CODEC_OK);
+}
+
// Encode one pass VBR with nonzero-lookahead, with resize and realtime mode.
// Test for the re-allocation of count_arf_frame_usage and
// count_lastgolden_frame_usage. Bug: 515433297.