Issue 1768 in webm: segfault in VP9 ResizeTest

640 views
Skip to first unread message

jz… via monorail

unread,
Jun 1, 2022, 1:01:37 AM6/1/22
to webm-d...@webmproject.org
Status: Available
Owner: ----
Labels: Type-Bug Pri-2
Components: libvpx

New issue 1768 by jz...@google.com: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768

v1.11.0-225-g8f56e1c07

This is related to the same setup as in issue #1767.

Currently only realtime is tested with VP9 [1], unlike VP8 which uses ONE_PASS_TEST_MODES. Matching that instantiation will result in intermittent segfaults.

This has only been observed under -fsanitize=integer and x86.

$ source tools/set_analyzer_env.sh integer
$ LDFLAGS="-fsanitize=integer --rtlib=compiler-rt -lgcc_s" ../configure --target=x86-linux-gcc --enable-ccache --enable-vp9-temporal-denoising --disable-dependency-tracking --enable-debug --disable-install-docs --disable-optimizations

[1] https://chromium.googlesource.com/webm/libvpx/+/8f56e1c074712ffa937dc48a14d4b01e378a170f/test/resize_test.cc#799
[2]
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==2662603==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0xf721cec0 (pc 0x089aeb55 bp 0xff8b3f38 sp 0xff8b3840 T2662603)
==2662603==The signal is caused by a READ memory access.
#0 0x89aeb55 in vpx_sad64x64x4d_avx2 vpx_dsp/x86/sad4d_avx2.c:97:15
#1 0x8bddec3 in vp9_diamond_search_sad_avx vp9/encoder/x86/vp9_diamond_search_sad_avx.c:196:7
#2 0x8b1c79a in full_pixel_diamond vp9/encoder/vp9_mcomp.c:2516:17
#3 0x8b1c079 in vp9_full_pixel_search vp9/encoder/vp9_mcomp.c:2877:13
#4 0x8cf7a25 in simple_motion_search vp9/encoder/vp9_encodeframe.c:3433:3
#5 0x8cf51bd in ml_predict_var_rd_paritioning vp9/encoder/vp9_encodeframe.c:3509:5
#6 0x8ced44d in rd_pick_partition vp9/encoder/vp9_encodeframe.c:4077:7
#7 0x8cc5d6f in encode_rd_sb_row vp9/encoder/vp9_encodeframe.c:4536:7
#8 0x8cc2bc7 in vp9_encode_sb_row vp9/encoder/vp9_encodeframe.c:5863:5
#9 0x8cc6099 in vp9_encode_tile vp9/encoder/vp9_encodeframe.c:5887:5
#10 0x8cf8fa3 in encode_tiles vp9/encoder/vp9_encodeframe.c:5900:7
#11 0x8cc92a3 in encode_frame_internal vp9/encoder/vp9_encodeframe.c:6117:9
#12 0x8cc6ef4 in vp9_encode_frame vp9/encoder/vp9_encodeframe.c:6277:5
#13 0x8b428d1 in encode_without_recode_loop vp9/encoder/vp9_encoder.c:4134:3
#14 0x8b3df64 in encode_frame_to_data_rate vp9/encoder/vp9_encoder.c:5522:10
#15 0x8b334d8 in Pass0Encode vp9/encoder/vp9_encoder.c:5793:3
#16 0x8b31496 in vp9_get_compressed_data vp9/encoder/vp9_encoder.c:7897:5
#17 0x8a68b14 in encoder_encode vp9/vp9_cx_iface.c:1422:20
#18 0x8881a13 in vpx_codec_encode vpx/src/vpx_encoder.c:208:13
#19 0x84c929c in libvpx_test::Encoder::EncodeFrameInternal(libvpx_test::VideoSource const&, unsigned long) test/encode_test_driver.cc:86:3
#20 0x84c8ee1 in libvpx_test::Encoder::EncodeFrame(libvpx_test::VideoSource*, unsigned long) test/encode_test_driver.cc:57:5
#21 0x84cabdf in libvpx_test::EncoderTest::RunLoop(libvpx_test::VideoSource*) test/encode_test_driver.cc:205:16
#22 0x85b26ad in (anonymous namespace)::ResizeTest_TestExternalResizeWorks_Test::TestBody() test/resize_test.cc:326:3
#23 0x85b30d8 in non-virtual thunk to (anonymous namespace)::ResizeTest_TestExternalResizeWorks_Test::TestBody() test/resize_test.cc
#24 0x8d8b50d in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/src/src/gtest.cc:2607:10
#25 0x8d72eb2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/src/src/gtest.cc:2643:14
#26 0x8d48b4a in testing::Test::Run() third_party/googletest/src/src/gtest.cc:2682:5
#27 0x8d49832 in testing::TestInfo::Run() third_party/googletest/src/src/gtest.cc:2861:11
#28 0x8d4a028 in testing::TestSuite::Run() third_party/googletest/src/src/gtest.cc:3015:28
#29 0x8d5d27f in testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/src/gtest.cc:5855:44
#30 0x8d8e69d in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/src/gtest.cc:2607:10
#31 0x8d75502 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/src/gtest.cc:2643:14
#32 0x8d5cd62 in testing::UnitTest::Run() third_party/googletest/src/src/gtest.cc:5438:10
#33 0x8664902 in RUN_ALL_TESTS() third_party/googletest/src/include/gtest/gtest.h:2490:46
#34 0x8663ade in main test/test_libvpx.cc:74:10
#35 0xf79e7904 in __libc_start_main csu/../csu/libc-start.c:332:16

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV vpx_dsp/x86/sad4d_avx2.c:97:15 in vpx_sad64x64x4d_avx2
==2662603==ABORTING

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

Git Watcher via monorail

unread,
Jun 1, 2022, 2:17:34 PM6/1/22
to webm-d...@webmproject.org

Comment #1 on issue 1768 by Git Watcher: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768#c1

The following revision refers to this bug:
https://chromium.googlesource.com/webm/libvpx/+/9d279c88c3e8873c114298d69e919bfef45a1dab

commit 9d279c88c3e8873c114298d69e919bfef45a1dab
Author: James Zern <jz...@google.com>
Date: Sat May 28 22:25:49 2022

resize_test: add TODO for ResizeTest instantiation for VP9

this should match VP8 and use ONE_PASS_TEST_MODES, but currently the
code will produce integer sanitizer warnings and may segfault under
certain conditions

Bug: webm:1767,webm:1768
Change-Id: I6482ff1862f19716fde3d57522591bc61d76a84f

[modify] https://crrev.com/9d279c88c3e8873c114298d69e919bfef45a1dab/test/resize_test.cc

jz… via monorail

unread,
Jun 1, 2022, 9:49:07 PM6/1/22
to webm-d...@webmproject.org
Updates:
Owner: jing...@google.com
Status: Assigned

Comment #2 on issue 1768 by jz...@google.com: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768#c2


> This has only been observed under -fsanitize=integer and x86.

This will show up in address sanitizer as well [1]. valgrind can catch the issue in x86-64 too [2]. From a quick look it seems the address offsets are not scaled to the smaller reference frame, causing is_mv_in() to succeed where it possibly shouldn't. This code was carried over to av1, so it's possible the issue exists there as well.

[1]
AddressSanitizer:DEADLYSIGNAL
=================================================================
==3022783==ERROR: AddressSanitizer: SEGV on unknown address 0xf61796c0 (pc 0x090c4a38 bp 0xffee8618 sp 0xffee85f0 T0)
==3022783==The signal is caused by a READ memory access.
#0 0x90c4a38 in sad vpx_dsp/sad.c:27:57
#1 0x90c4962 in vpx_sad64x64_c vpx_dsp/sad.c:50:12
#2 0x92cdc44 in vp9_diamond_search_sad_c vp9/encoder/vp9_mcomp.c:2166:15
#3 0x92d3f88 in full_pixel_diamond vp9/encoder/vp9_mcomp.c:2516:17
#4 0x92d36b3 in vp9_full_pixel_search vp9/encoder/vp9_mcomp.c:2877:13
#5 0x95daa1f in simple_motion_search vp9/encoder/vp9_encodeframe.c:3433:3
#6 0x95d5cd3 in ml_predict_var_rd_paritioning vp9/encoder/vp9_encodeframe.c:3509:5
#7 0x95c6da2 in rd_pick_partition vp9/encoder/vp9_encodeframe.c:4077:7
[2]
==3013309== Invalid read of size 16
==3013309== at 0x554864: sad (sad.c:27)
==3013309== by 0x554864: vpx_sad64x64_c (sad.c:50)
==3013309== by 0x5CDC6B: vp9_diamond_search_sad_c (vp9_mcomp.c:2166)
==3013309== by 0x5CEC02: full_pixel_diamond (vp9_mcomp.c:2516)
==3013309== by 0x5CEC02: vp9_full_pixel_search (vp9_mcomp.c:2877)
==3013309== by 0x6944A3: simple_motion_search (vp9_encodeframe.c:3433)
==3013309== by 0x6944A3: ml_predict_var_rd_paritioning.constprop.0 (vp9_encodeframe.c:3509)
==3013309== by 0x69B4E0: rd_pick_partition (vp9_encodeframe.c:4077)
==3013309== by 0x6A7BF3: encode_rd_sb_row (vp9_encodeframe.c:4536)
==3013309== by 0x6AA655: vp9_encode_sb_row (vp9_encodeframe.c:5863)
==3013309== by 0x6AA655: vp9_encode_tile (vp9_encodeframe.c:5887)
==3013309== by 0x6AA655: encode_tiles (vp9_encodeframe.c:5900)
==3013309== by 0x6AA655: encode_frame_internal (vp9_encodeframe.c:6117)
==3013309== by 0x6AAB94: vp9_encode_frame (vp9_encodeframe.c:6277)
==3013309== by 0x5DAD93: encode_without_recode_loop (vp9_encoder.c:4134)
==3013309== by 0x5DAD93: encode_frame_to_data_rate (vp9_encoder.c:5522)
==3013309== by 0x5DC3E2: Pass0Encode (vp9_encoder.c:5793)
==3013309== by 0x5DC3E2: vp9_get_compressed_data (vp9_encoder.c:7897)
==3013309== by 0x5A6882: encoder_encode (vp9_cx_iface.c:1422)
==3013309== by 0x503C9F: vpx_codec_encode (vpx_encoder.c:208)
==3013309== Address 0x940cc20 is 9 bytes after a block of size 14,423 alloc'd
==3013309== at 0x483F7B5: malloc (vg_replace_malloc.c:381)
==3013309== by 0x505111: vpx_memalign (vpx_mem.c:62)
==3013309== by 0x505111: vpx_malloc (vpx_mem.c:70)
==3013309== by 0x505111: vpx_calloc (vpx_mem.c:76)
==3013309== by 0x5D78CD: alloc_frame_mvs (vp9_encoder.c:3333)
==3013309== by 0x5D78CD: set_frame_size (vp9_encoder.c:3760)
==3013309== by 0x5D81F3: encode_without_recode_loop (vp9_encoder.c:3880)
==3013309== by 0x5D81F3: encode_frame_to_data_rate (vp9_encoder.c:5522)
==3013309== by 0x5DC3E2: Pass0Encode (vp9_encoder.c:5793)
==3013309== by 0x5DC3E2: vp9_get_compressed_data (vp9_encoder.c:7897)
==3013309== by 0x5A6882: encoder_encode (vp9_cx_iface.c:1422)
==3013309== by 0x503C9F: vpx_codec_encode (vpx_encoder.c:208)
==3013309== by 0x30FAC7: libvpx_test::Encoder::EncodeFrameInternal(libvpx_test::VideoSource const&, unsigned long) (encode_test_driver.cc:86)

jz… via monorail

unread,
Jun 1, 2022, 9:53:31 PM6/1/22
to webm-d...@webmproject.org

Comment #3 on issue 1768 by jz...@google.com: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768#c3

This occurs at frame 269 where the scaled size is 80x60 from a 320x240 source.

mar… via monorail

unread,
Oct 20, 2022, 2:40:23 PM10/20/22
to webm-d...@webmproject.org
Updates:
Cc: ji...@google.com
Owner: mar...@google.com

Comment #6 on issue 1768 by mar...@google.com: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768#c6

(No comment was entered for this change.)

Git Watcher via monorail

unread,
Jan 17, 2023, 9:05:09 PM1/17/23
to webm-d...@webmproject.org

Comment #7 on issue 1768 by Git Watcher: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768#c7


The following revision refers to this bug:
https://chromium.googlesource.com/webm/libvpx/+/71d01660cc40306c2c7c80c8ed510e520a0c4b93

commit 71d01660cc40306c2c7c80c8ed510e520a0c4b93
Author: Marco Paniconi <mar...@google.com>
Date: Sat Jan 14 03:46:10 2023

Fix to segfault for external resize test in vp9

Failure occurs for 1 pass non-realtime mode at speed 0.
Due to speed feautre rd_ml_partition.var_pruning, which
doesn't check for scaled reference in simple_motion_search().

Bug: webm:1768

Change-Id: Iddcb56033bac042faebb5196eed788317590b23f

[modify] https://crrev.com/71d01660cc40306c2c7c80c8ed510e520a0c4b93/test/resize_test.cc
[modify] https://crrev.com/71d01660cc40306c2c7c80c8ed510e520a0c4b93/vp9/encoder/vp9_encodeframe.c

mar… via monorail

unread,
Jan 18, 2023, 1:50:02 PM1/18/23
to webm-d...@webmproject.org
Updates:
Status: Fixed

Comment #8 on issue 1768 by mar...@google.com: segfault in VP9 ResizeTest
https://bugs.chromium.org/p/webm/issues/detail?id=1768#c8


(No comment was entered for this change.)

Reply all
Reply to author
Forward
0 new messages