Issue decoding WebP images with semi-transparent pixels on Android via JNI

[Marc Reichelt]

Hi there!

I am working on a backport library that enables Android developers to decode WebP images on Android versions older than 4.3:

https://github.com/mreichelt/webp-android-backport-library

I want to use it myself in a cartoon application I recently released: https://play.google.com/store/apps/details?id=de.nichtlustig.android

After some work the decoding works fine with regular images (without transparency) and with images that have fully transparent pixels, as my unit tests show.

However, I identified a problem with images that contain pixels with semi-transparent colors. For example, a pixel in a WebP image with the color #21000000 (ARGB) gets decoded as #21080017 instead (I tracked the problem in an issue and a unit test).

From here on I don't know where to look anymore. I tested all the WebPDecodeXXXXInto variants, but only WebPDecodeRGBAInto showed promising results.

I think a reason might be that I recently switched from the WebPDecodeRGBA method to WebPDecodeRGBAInto, where I use pre-allocated Bitmaps to decode into. The reason I switched was that with the first variant I needed to hold the bitmap data at least twice in memory (decoded from WebP lib and finally needed to convert to Bitmap in Java code), which was bad for performance and sometimes even lead to native crashes.

The code I use now looks like this (full file here):

jboolean decodeRGBAIntoInternal(JNIEnv * env, AndroidBitmapInfo info, jbyteArray encoded, jobject bitmap) {
    size_t output_buffer_size;
    uint8_t* result;
    void *pixels;
    int ret;
    jbyte* encoded_array;
    jsize encoded_length;
    int width = 0;
    int height = 0;


    output_buffer_size = info.height * info.stride;
    encoded_array = (*env)->GetByteArrayElements(env, encoded, NULL);
    encoded_length = (*env)->GetArrayLength(env, encoded);
    ret = WebPGetInfo(encoded_array, encoded_length, &width, &height);


    if (!ret) {
        LOGE("unable to get webp info");
        return JNI_FALSE;
    }
    if (info.width != width || info.height != height) {
        LOGE("webp size %dx%d does not match bitmap size %dx%d", width, height, info.width, info.height);
        return JNI_FALSE;
    }


    if ((ret = AndroidBitmap_lockPixels(env, bitmap, &pixels)) < 0) {
        LOGE("AndroidBitmap_lockPixels() failed ! error=%d", ret);
        return JNI_FALSE;
    }


    result = WebPDecodeRGBAInto(encoded_array, encoded_length, (uint8_t*) pixels, output_buffer_size, info.stride);


    AndroidBitmap_unlockPixels(env, bitmap);


    (*env)->ReleaseByteArrayElements(env, encoded, encoded_array, JNI_ABORT);


    return result ? JNI_TRUE : JNI_FALSE;
}

Does anyone have an idea what I am doing wrong? In which direction should I look?

Thank you in advance & regards

Marc Reichelt

PS: Keep up the good work on WebP - I love it! :)

[Marc Reichelt]

Hi everyone,

Romain Guy wrote that Android expects the Bitmap data to be in pre-multiplied form:

https://plus.google.com/+MarcReichelt/posts/YQdtsxkBq3o

From then on I found out the webp library has a MODE_rgbA. So I quickly tried to create a WebPDecodergbAInto method (notice the small 'rgb' letters) that uses this mode MODE_rgbA instead of MODE_RGBA, and seems to apply the pre-multiplication.

This seems to solve my test case with the single pixel. But more complex examples, like 'testLine' or 'testComplexRoundImage' still fail. 'testLine':

Pixels at (5, 0) have different colors. expected:<#870[202]00> but was:<#870[000]00>

So there seems to be something else I have to do. Any ideas?

And does anyone know if there is a method to access the static function DecodeIntoRGBABuffer externally? Because otherwise I will have to copy the whole webp library into my Github project…

Thanks a lot & regards

Marc

[Pascal Massimino]

Hi Marc,

On Sun, Dec 27, 2015 at 6:21 PM, Marc Reichelt <mcrei...@gmail.com> wrote:

Hi everyone,

Romain Guy wrote that Android expects the Bitmap data to be in pre-multiplied form:

https://plus.google.com/+MarcReichelt/posts/YQdtsxkBq3o

From then on I found out the webp library has a MODE_rgbA. So I quickly tried to create a WebPDecodergbAInto method (notice the small 'rgb' letters) that uses this mode MODE_rgbA instead of MODE_RGBA, and seems to apply the pre-multiplication.

This seems to solve my test case with the single pixel. But more complex examples, like 'testLine' or 'testComplexRoundImage' still fail. 'testLine':

Pixels at (5, 0) have different colors. expected:<#870[202]00> but was:<#870[000]00>

how did you generate the reference PNG image that you are comparing to in the unit-test?

The same pre-multiplication formula as libwebp must be used.

In libwebp, we use: x = (int)(X * alpha / 255.) to pre-multiply.

So there seems to be something else I have to do. Any ideas?

And does anyone know if there is a method to access the static function DecodeIntoRGBABuffer externally? Because otherwise I will have to copy the whole webp library into my Github project…

Hmm... this function is only a helper function for the visible public ones (WebPDecodeXXXInto).

Even if we could make it public (maybe we need a WebPDecodergbAInto variant after all), this wouldn't be backward compatible

with older libwebp libraries shipped on Android < 0.4.3... (if you're using them). Your best shot would be using the advanced decoding

API (which pretty much amounts to re-rewriting DecodeIntoRGBABuffer()'s code, but this API isn't mapped in the JNI yet.

hope it helps,

skal/

 

--
You received this message because you are subscribed to the Google Groups "WebP Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webp-discuss...@webmproject.org.
To post to this group, send email to webp-d...@webmproject.org.
Visit this group at https://groups.google.com/a/webmproject.org/group/webp-discuss/.
For more options, visit https://groups.google.com/a/webmproject.org/d/optout.

[Marc Reichelt]

Hi skal,

 

how did you generate the reference PNG image that you are comparing to in the unit-test?

I generated them with the official dwebp 0.5.0 decoder - to have comparable results. I am aware of Android using an older version of the WebP library - that's why I am not using it to compare the images. :)

 

The same pre-multiplication formula as libwebp must be used.

In libwebp, we use: x = (int)(X * alpha / 255.) to pre-multiply.

Ah, that seems to be the problem. I looked for how Android pre-multiplies the colors, and I found some sources:

https://android.googlesource.com/platform/development/+/master/samples/ApiDemos/src/com/example/android/apis/graphics/BitmapPixels.java#65

https://github.com/google/skia/blob/master/include/core/SkColorPriv.h#L544

https://github.com/google/skia/blob/master/include/core/SkMath.h#L175

The formula used is:

static inline U8CPU SkMulDiv255Round(U16CPU a, U16CPU b) {
    SkASSERT(a <= 32767);
    SkASSERT(b <= 32767);
    unsigned prod = a*b + 128;
    return (prod + (prod >> 8)) >> 8;
}

which is called like this:

static inline
SkPMColor SkPremultiplyARGBInline(U8CPU a, U8CPU r, U8CPU g, U8CPU b) {
    if (a != 255) {
        r = SkMulDiv255Round(r, a);
        g = SkMulDiv255Round(g, a);
        b = SkMulDiv255Round(b, a);
    }
    return SkPackARGB32(a, r, g, b);
}

This means Android uses a slightly different formula to premultiply the values. Which also explains why my code works, but has *slighty* different values.

Now, I think I have three options:

1. I leave the code as it is right now, and live with the result being slightly different - because the visual outcome is good, and the WebP function to decode & premultiply is likely highly optimized for speed. But it would be nice to have working unit tests.
2. I write my own premultiply implementation that copies the Android one, which is not likely to be so highly optimized as the WebP one. I have to do *another* O(n) operation after all.
3. Contribute to the WebP implementation to use the Android pre-multiplication formula - though I don't know if this is the right way to go, because it could break results for existing users of the rgbA variants. Or maybe it wouldn't break - at least these variants are private up until now and don't seem to be used, so it would be an internal change.

I think I will try out option (2) - but maybe you guys are interested in option (3)?

 

So there seems to be something else I have to do. Any ideas?

And does anyone know if there is a method to access the static function DecodeIntoRGBABuffer externally? Because otherwise I will have to copy the whole webp library into my Github project…

Hmm... this function is only a helper function for the visible public ones (WebPDecodeXXXInto).

Even if we could make it public (maybe we need a WebPDecodergbAInto variant after all), this wouldn't be backward compatible

with older libwebp libraries shipped on Android < 0.4.3... (if you're using them). Your best shot would be using the advanced decoding

API (which pretty much amounts to re-rewriting DecodeIntoRGBABuffer()'s code, but this API isn't mapped in the JNI yet.

I am not using the old WebP libs anyway because the whole point of my WebP backport library is to bring the features of the new WebP versions to older versions of Android. Including decoding images with alpha channel on Android versions down to 2.3. :)

I am not using the SWIG bindings in my current version of webp-backport. I started to use them in the beginning, but got native crashes pretty quick. I didn't try them out much more, and found a way to use just the functions I need and write the needed JNI bindings myself. Which might not be such a good idea at a second look - but for now it seems to work. :)

I can't say how thankful I am for your help - thanks a lot! And if we meet in real life I'd like to invite you to a beer! :)

Best regards

Marc

[Pascal Massimino]

Hi Marc,

That could work. Now, reverting to RGBA output (not rgbA), you also save an O(n) operation within libwebp (even if it's an optimized one). So, that's a wash, to some extent...

 

1. Contribute to the WebP implementation to use the Android pre-multiplication formula - though I don't know if this is the right way to go, because it could break results for existing users of the rgbA variants. Or maybe it wouldn't break - at least these variants are private up until now and don't seem to be used, so it would be an internal change.

I think I will try out option (2) - but maybe you guys are interested in option (3)?

At some point, i think i experimented with a similar formula than Skia. But then reverted to the one we have now, because it's easier and faster to implement (esp. in SSE2).

But! the code still has a trace Skia's variant[*]. In alpha_processing.c:210 for instance, you have the comment:

// ... For bit-wise equivalence to (int)(x * a / 255. + .5),
// one can use instead: (x * a * 65793 + (1 << 23)) >> 24

 

So, if you have WebP's code around, you can simply modify the "#if 1" a line alpha_processing.c:212 to get Skia's behaviour.

SSE2 variant should be straightforward to adapt, but as I said, there's an extra add, which makes it slightly slower for a minor

diff in precision.

 

So there seems to be something else I have to do. Any ideas?

And does anyone know if there is a method to access the static function DecodeIntoRGBABuffer externally? Because otherwise I will have to copy the whole webp library into my Github project…

Hmm... this function is only a helper function for the visible public ones (WebPDecodeXXXInto).

Even if we could make it public (maybe we need a WebPDecodergbAInto variant after all), this wouldn't be backward compatible

with older libwebp libraries shipped on Android < 0.4.3... (if you're using them). Your best shot would be using the advanced decoding

API (which pretty much amounts to re-rewriting DecodeIntoRGBABuffer()'s code, but this API isn't mapped in the JNI yet.

I am not using the old WebP libs anyway because the whole point of my WebP backport library is to bring the features of the new WebP versions to older versions of Android. Including decoding images with alpha channel on Android versions down to 2.3. :)

Thanks a lot for doing this! This is quite useful.

 

I am not using the SWIG bindings in my current version of webp-backport. I started to use them in the beginning, but got native crashes pretty quick. I didn't try them out much more, and found a way to use just the functions I need and write the needed JNI bindings myself. Which might not be such a good idea at a second look - but for now it seems to work. :)

I can't say how thankful I am for your help - thanks a lot! And if we meet in real life I'd like to invite you to a beer! :)

Deal! :)

hope it helps,

skal/

[*] here, a small proggy to play with formulae:

#include <stdio.h>

int main() {
  for (unsigned b = 0; b < 256; ++b) {
    for (unsigned a = 0; a < 256; ++a) {
      const unsigned prod = a * b + 128;
      const unsigned v0 = (prod + (prod >> 8)) >> 8;
      const unsigned v1 = (int)(a * b / 255. + .5);
      const unsigned v2 = (b * a * 65793 + (1 << 23)) >> 24;
      if (v0 != v1 || v0 != v2) {
        printf("!! a=%d b=%d  -> v= %d / %d / %d\n", a, b, v0, v1, v2);
        return -1;
      }
    }
  }
  printf("OK!\n");
  return 0;
}

[Marc Reichelt]

Hey Pascal,

At some point, i think i experimented with a similar formula than Skia. But then reverted to the one we have now, because it's easier and faster to implement (esp. in SSE2).

But! the code still has a trace Skia's variant[*]. In alpha_processing.c:210 for instance, you have the comment:

// ... For bit-wise equivalence to (int)(x * a / 255. + .5),
// one can use instead: (x * a * 65793 + (1 << 23)) >> 24

 

So, if you have WebP's code around, you can simply modify the "#if 1" a line alpha_processing.c:212 to get Skia's behaviour.

SSE2 variant should be straightforward to adapt, but as I said, there's an extra add, which makes it slightly slower for a minor

diff in precision.

That was exactly what I needed! Thanks to you all the unit tests are running through now and I can finally close https://github.com/mreichelt/webp-android-backport-library/issues/18 :)

I just released version 0.4.6 of my library with the new premultipy method. Awesome! :)

Thanks a lot for doing this! This is quite useful.

Thanks! I have to admit I am doing this because I need it myself, but that's also a good incentive for me to keep improving. ;)

 

Thanks again for your help & see you around

Marc