Re: Float to integer conversion problem in libvpx1.1.0 of VP8 Encoder

[John Koleszar]

On Thu, May 9, 2013 at 9:35 AM, <gbala...@gmail.com> wrote:
>
> Hi,
> I m using the libvpx1.1.0 code for VP8 Encoder and i m using the generic-gnu as target which uses gcc compiler for compiling.
>
> In "vp8_initialize_rd_consts()" function of Rdopt.c,
>
> void vp8_initialize_rd_consts(VP8_COMP *cpi, int Qvalue)
> {
> int q;
> int i;
> double capped_q = (Qvalue < 160) ? (double)Qvalue : 160.0;
> double rdconst = 2.80;
>
> vp8_clear_system_state(); //__asm emms;
>
> cpi->RDMULT = (int)(rdconst * (capped_q * capped_q));
>
> ........
>
> }
>
>
> The above line which i have highlighted in red has some issues.
>
> for eg . If the Qvalue = 55 , which is being converted to double which is capped_q = 55.00. So the above expression will be :
>
> cpi->RDMULT = (int ) ( 2.80 * ( 55.0 * 55.0))
>
> Ideally the Multiplication ( 2.80 * ( 55.0 * 55.0)) should result in 8470.00 .
>
> But when it is typecasted to "int" and stored in cpi->RDMULT, it results in 8469. I m using a different compiler which gives 8470 after typecasting also.
> Because of this the threshold values for all the 20 Modes of interprediction varies and the encoding modes for inter prediction also varies. ie. instead of NEWMV i m getting NEARMV as the best encoding mode.
>
>
> Please give me some suggestions regarding this ..... Is this type of typecasting is done intentionally or it is a bug ?
>
>
> Regards,
> G.Balaji
>

Hi,

Nice job tracking this down. In this case, I'd say the right thing to
do is add 0.5 before the cast so that the value is rounded
consistently. The discrepancy you're seeing could either be a
fast-math optimization done by the compiler, or could vary based on
the floating point width on the target. On x86 for instance, you'll
sometimes get different results if the code uses the x87 floating
point path (80 bit) vs the SSE2 path (64 bit). libvpx generally
disables the 80 bit path on x86, but you won't get that behavior if
you're building for generic-gnu. 2.80 isn't exactly representable, so
it doesn't surprise me that you're seeing this kind of variation.

I've added a bug to our issue tracker here, or if you want to send a
patch I'd be happy to apply it:
https://code.google.com/p/webm/issues/detail?id=576

Thanks,

John