VP9 encoder crash

[Mihail Balikov]

Hello,

Very rarely and randomly our application crashes into libvpx/vp9 encoder.

Finally, we were able to track it down.

CRASH:

vp9_pick_inter_mode()

vp9_int_pro_motion_estimation()

vpx_int_pro_col_sse2()

_mm_load_si128(0x000000594b4d931f) --> read access violation

The problem is that the memory address 0x000000594b4d931f is not 16bytes aligned.

------

https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_load_si128&expand=3310

__m128i _mm_load_si128 (__m128i const* mem_addr)

Description

Load 128-bits of integer data from memory into dst. mem_addr must be aligned on a 16-byte boundary or a general-protection exception may be generated.

---------

THE ISSUE:

(1) When using RESIZE_DYNAMIC and VPX_CBR the encoder at some random point decides to scale down the input image from 320:240 to 240:180. This generates x_scale_fp = y_scale_fp = 21845

vp9_setup_scale_factors_for_frame()

x_scale_fp = get_fixed_point_scale_factor(320, 240); -> return 21845

y_scale_fp = get_fixed_point_scale_factor(240, 180); -> return 21845

(2) Which can lead to memory unaligment in:

vp9_pick_inter_mode()

     find_predictors()

  vp9_setup_pred_block()

setup_pred_plane(..., stride, mi_col, mi_row, scale, subsampling_x, subsamplig_y)

before setup_pred_plane() -> dst[0].buf = 0x000000594b4c3220

after setup_pred_plane()    -> dst[0].buf = 0x000000594b4da71f (unaligned)

stride = 640

mi_row = 14

mi_col = 12

subsampling_x = subsampling_y = 1

scale.x_scale_fp = 21845

scale.y_scale_fp = 21845

scale funcitons: scaled_x() and scaled_y()

offset = 0x000000594b4da71f - 0x000000594b4c3220 = 95487 bytes

----- simple math from setup_pred_plane()

x = (8 * mi_col) >> subsampling_x = 96

y = (8 * mi_row) >> subsampling_y = 112

scaled_x = (x * x_scale_fp) >> 14 = (96 * 21845) >> 14 = 127

scaled_y = (y * y_scale_fp) >> 14 = (112 * 21845) >> 14 = 149

offset = scaled_y * stride + scaled_x = 149 * 640 + 127 = 95487

================

The application is using libvpx 1.6.1 on Windows x64 build with Visual Studio 2017.

The same code paths exist in latest libvpx.

regards,

Mihail Balikov

[James Zern]

Hi,

On Sat, Feb 16, 2019 at 10:43 PM Mihail Balikov <mbal...@gmail.com> wrote:
>
> Hello,
>
> Very rarely and randomly our application crashes into libvpx/vp9 encoder.
> Finally, we were able to track it down.
>

Thanks for providing the detail in this report, would you mind filing
a bug for this [1]?

[1] https://bugs.chromium.org/p/webm/issues/entry?template=Codec+Defect+Report+%28libvpx%29

[Mihail Balikov]

done

--
You received this message because you are subscribed to the Google Groups "Codec Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codec-devel...@webmproject.org.
To post to this group, send email to codec...@webmproject.org.
Visit this group at https://groups.google.com/a/webmproject.org/group/codec-devel/.
For more options, visit https://groups.google.com/a/webmproject.org/d/optout.