Missing resources (l10n) on Android

[Václav Brožek]

Hi all, especially l10n and Android experts!

When investigating http://crbug.com/658902 I traced a crash down to Chrome instances missing resources (demonstrating as ResourceBundle::GetLocalizedString returning an empty string for a valid ID). While the crash was in multi-platform code, the vast majority, if not all, of recent crashes were on Android.

I landed a patch to handle this gracefully, because I understand out guide on CHECKs as advising against crashes except for security vulnerabilities. But running without UI strings hardly seems like a usable state of Chrome, even if it does not crash. To better understand the issue, does anybody know:

• What reasons could lead to the resources not being available?
• What could have made this so Android specific?
• What is the expected frequency of the resources missing (to compare with the crash rate on the particular bug)?

Any other insights or comments welcome.

Thanks!

Vaclav

[Andrew Grieve]

Stab in the dark, but Android is the only platform that uses a compiler warning-based resource ID whitelist (see enable_resource_whitelist_generation). If there is a bug in the whitelist generation, then that might cause a resource missing issue.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Andrey Logvinov]

Hi Vaclav and all,

In Yandex Browser for Android, we are experiencing a similar problem with mismatched version of v8 snapshot (unlike Chromium, we also extract v8 snapshot from the apk file). We use the same ResourceExtractor.ExtractTask.getApkVersion(), which relies on PackageInfo.lastUpdateTime, to determine if the application has been updated and we need to extract resources again. In our production logs, we are seeing a very peculiar behavior: the APK version code changes but PackageInfo.lastUpdateTime nevertheless remains the same so we fail to re-extract the resources, including the v8 snapshot, and we crash on snapshot version check. Even more strangely, the APK version sometimes changes down, not up. It is as though something silently replaces the APK file while leaving everything else, including the last update timestamp, intact.

We could not reproduce this behavior in our development environment so we presently have no idea why this would happen, but we are going to try to fix this by checking both version code and last update timestamp to determine if the application version has changed.

Could your problem also be related to resource extraction? I unfortunately have no access to http://crbug.com/658902 so I only have the information from your message.

Could perhaps someone on Android team suggest why this kind of behavior would be possible?

Kind regards,

Andrey

[Andrey Logvinov]

Hi Vaclav and all,

In Yandex Browser for Android, we are experiencing a similar problem with mismatched version of v8 snapshot (unlike Chromium, we also extract v8 snapshot from the apk file). We use the same ResourceExtractor.ExtractTask.getApkVersion(), which relies on PackageInfo.lastUpdateTime, to determine if the application has been updated and we need to extract resources again. In our production logs, we are seeing a very peculiar behavior: the APK version code changes but PackageInfo.lastUpdateTime nevertheless remains the same so we fail to re-extract the resources, including the v8 snapshot, and we crash on snapshot version check. Even more strangely, the APK version sometimes changes down, not up. It is as though something silently replaces the APK file while leaving everything else, including the last update timestamp, intact.

We could not reproduce this behavior in our development environment so we presently have no idea why this would happen, but we are going to try to fix this by checking both version code and last update timestamp to determine if the application version has changed.

Could your problem also be related to resource extraction? I unfortunately have no access to http://crbug.com/658902 so I only have the information from your message.

Could perhaps someone on Android team suggest why this kind of behavior would be possible?

Kind regards,

Andrey

On Friday, November 25, 2016 at 7:25:01 PM UTC+3, vabr wrote:

[Václav Brožek]

Andrew,

Thanks for the pointer, I did not know about the whitelist generation. However, if that failed, then all builds should be affected, not just a handful of instances, shouldn't they?

Andrey,

Sorry to hear about those issues. I'm afraid I have no good answers, as I struggle to find similar ones myself.

http://crbug.com/658902 is restricted because of the crash info. I believe that there is no additional information on that bug relating it to the resource loading, other than I already stated here. The exact numbers of crash instances are on the internal dashboard, so I guess they are confidential. What I can share is that the crash seems to be present even before 54, and is not the most frequent crash.

Cheers,

Vaclav

--

[Andrew Grieve]

Sounds like what Andrey described would explain this crash. We extract the localization resources, so if it was out of date, then resource IDs could be missing. This could also explain the mismatched resource reported in https://bugs.chromium.org/p/chromium/issues/detail?id=658523 (sorry, also internal-only, but it's just another report of resource IDs being wrong, which is fixed after re-installing chrome).

Andrey - if you find that checking versionCode works around this issue, I'd happily accept a patch adding such a check :)

[Andrey Logvinov]

Great, as soon as we have some confirmation that this indeed works, I will submit a patch.

 

-- 

Kind regards,

Andrey Logvinov

+7 (903) 600-56-97

alog...@yandex-team.ru

 

 

 

29.11.2016, 00:30, "Andrew Grieve" <agr...@chromium.org>:

[Torne (Richard Coles)]

Is there a specific reason why you have to extract the resources instead of loading them from APK as regular chrome does? Not having this kind of version mismatch issue is just one of the advantages of doing that..

[Andrey Logvinov]

- agrieve@, vabr@

 

AFAIK this is to keep the initial download size to a minimum (it matters because Yandex Browser does not come pre-installed like Chrome). That way, the snapshot can be compressed with LZMA algorithm along with other resources. But I agree that this is a disadvantage in regard to version coherency.

 

-- 

Kind regards,

Andrey Logvinov

+7 (903) 600-56-97

alog...@yandex-team.ru

 

 

 

29.11.2016, 16:31, "Torne (Richard Coles)" <to...@chromium.org>:

[Torne (Richard Coles)]

The Play Store compresses APKs while they're sent over the wire, though possibly not as efficiently as LZMA. So at least in that context, raw APK size is not the best thing to measure for download size, since they aren't linearly related.

[Andrey Logvinov]

For what it's worth, we have reproduced this behavior with Yandex Browser in the following scenario:

 

1. Obtain root on the device

2. Install version A of the browser

3. Move the browser app to SD card (we don't allow it in the manifest, but with root privileges it is still possible)

4. Remove the SD card

5. Install version B of the browser with different v8 snapshot version as compared to A.

 

For some reason, PackageInfo.lastUpdateTime is not changed in this scenario, so resources are not extracted from the version B, instead, extracted resources from version A remain (the extracted resources are part of app data which is not part of ASEC container so it is not moved to SD card).

 

And this line from logcat looks interesting, because it describes what seems to be happening:

 

12-09 14:37:27.972 I/PackageManager( 820): Package com.yandex.browser codePath changed from /mnt/asec/com.yandex.browser-1/pkg.apk to /data/app/com.yandex.browser-1.apk; Retaining data and using new

 

An alternative scenario is to install version A on SD card 1, remove SD card 1, insert SD card 2, install version B on SD card 2, then remove SD card 2 and put SD card 1 back in.

 

We don't know if this is exactly what happens in production, but from the user feedback we have seen our users rooting their devices and moving the browser and other apps to SD card, so this is not unrealistic.

 

Could Chrome experience missing resource problems for the same reason?

 

-- 

Kind regards,

Andrey Logvinov

+7 (903) 600-56-97

alog...@yandex-team.ru

 

 

 

29.11.2016, 00:30, "Andrew Grieve" <agr...@chromium.org>:

Sounds like what Andrey described would explain this crash. We extract the localization resources, so if it was out of date, then resource IDs could be missing. This could also explain the mismatched resource reported in https://bugs.chromium.org/p/chromium/issues/detail?id=658523 (sorry, also internal-only, but it's just another report of resource IDs being wrong, which is fixed after re-installing chrome).

[Andrew Grieve]

Chrome certainly could be. I think the best way for us to test is to just do the speculative fix and see if crash report go down.

[Andrew Grieve]

Filed a bug for this here:

https://bugs.chromium.org/p/chromium/issues/detail?id=673458

With the initial bug (http://crbug.com/658902) fixed, I'm not actually sure if there's any crash reports to track for a fix though.