Linking against Android's system libraries

[Diego Pino]

Hi,

I'm exploring a way to build ChromiumTestShell for Android linking against some system libraries. I checked build/common.gypi and saw that, at this moment, it's possible to build ChromiumTestShell for Android using some system libraries, basically libstlport, libicu and libexpat. However, it's only possible to use system libraries when building Chromium as a WebView.

build/common.gypi:1381

# When building as part of the Android system, use system libraries
# where possible to reduce ROM size.
# TODO(steveblock): Investigate using the system version of sqlite.
'use_system_sqlite%': 0, # '<(android_webview_build)',
'use_system_expat%': '<(android_webview_build)',
'use_system_icu%': '<(android_webview_build)',
'use_system_stlport%': '<(android_webview_build)',

In any case, I modified some .gyp scripts and managed to build Chromium as a browser using Android's system libraries. Using this approach, the size of the ROM gets reduced considerably but this change should be benefitial for reducing RAM consumption too.

Now here comes my concern. The problem of using Android's libraries is that Chromium's and Android's versions of these libraries differ. In the case of stlport, both use version 5.0, so nothing to worry about. In the case of libexpat, Chromium uses 2.0.1 but Android uses 2.1.0. In addition, Chromium's version has two patches applied that fix a bug in Windows. With regard to libicu, both versions differ enormously. Chromium uses icu 4.6 while Android uses version 49.1.1 (versions number enumeration changed since 4.7). Chromium's icu comes with many patches applied that fix bugs in the upstream 4.6 version. Fortunately, these patches where proposed to be applied upstream and all of them have made it, but some are applied in versions higher that 49.1.1, so those patches are not available on Android's icu. In addition to that, Chromium's icu has other changes in the source code.

I wonder if those building Chromium as a WebView for Android don't have the same concerns of using different versions of some libraries. I'm afraid that things may crash or not work due to this change. On the other hand, I think the benefits of building against system libraries are important, but maybe apart from this potential compatibility issues there are other drawbacks I'm failing to see. I'd appreciate any comment or feedback on this. Thanks!

Diego

[Torne (Richard Coles)]

On 11 November 2013 15:50, Diego Pino <pino...@gmail.com> wrote:

Hi,

I'm exploring a way to build ChromiumTestShell for Android linking against some system libraries. I checked build/common.gypi and saw that, at this moment, it's possible to build ChromiumTestShell for Android using some system libraries, basically libstlport, libicu and libexpat. However, it's only possible to use system libraries when building Chromium as a WebView.

build/common.gypi:1381

# When building as part of the Android system, use system libraries
# where possible to reduce ROM size.
# TODO(steveblock): Investigate using the system version of sqlite.
'use_system_sqlite%': 0, # '<(android_webview_build)',
'use_system_expat%': '<(android_webview_build)',
'use_system_icu%': '<(android_webview_build)',
'use_system_stlport%': '<(android_webview_build)',

In any case, I modified some .gyp scripts and managed to build Chromium as a browser using Android's system libraries. Using this approach, the size of the ROM gets reduced considerably but this change should be benefitial for reducing RAM consumption too.

Now here comes my concern. The problem of using Android's libraries is that Chromium's and Android's versions of these libraries differ. In the case of stlport, both use version 5.0, so nothing to worry about. In the case of libexpat, Chromium uses 2.0.1 but Android uses 2.1.0. In addition, Chromium's version has two patches applied that fix a bug in Windows. With regard to libicu, both versions differ enormously. Chromium uses icu 4.6 while Android uses version 49.1.1 (versions number enumeration changed since 4.7). Chromium's icu comes with many patches applied that fix bugs in the upstream 4.6 version. Fortunately, these patches where proposed to be applied upstream and all of them have made it, but some are applied in versions higher that 49.1.1, so those patches are not available on Android's icu. In addition to that, Chromium's icu has other changes in the source code.

Actually it's *much* worse than this: different versions of Android have *different* versions of the system libraries, with different patches, and vendors often upgrade/customise/patch those libraries as well, so even two devices with the same version of Android may have different library versions. Chrome for Android has to work on all devices from ICS onward, so cannot depend on any of this. None of these libraries are part of the NDK and therefore there's no guarantee they even exist *at all* on an Android system, and they should never be used directly by applications.

I wonder if those building Chromium as a WebView for Android don't have the same concerns of using different versions of some libraries. I'm afraid that things may crash or not work due to this change. On the other hand, I think the benefits of building against system libraries are important, but maybe apart from this potential compatibility issues there are other drawbacks I'm failing to see. I'd appreciate any comment or feedback on this. Thanks!

In the WebView build we have made sure that the library versions match, or that they are close enough that we can expect them to work: for example, Android 4.4 uses exactly the same skia version as the Chromium M30 branch that the Android 4.4 webview is based on. We can do this because we control both sides and can upgrade/patch libraries in both Chromium and the Android system if we need to. The WebView is shipped in the system image so there is no risk of the library being incompatible at runtime if it wasn't at build time. Third party apps can't and shouldn't rely on this.

Don't use non-NDK libraries on Android. It just makes your app very fragile.

Diego

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[sanjoy pal]

On Monday, November 11, 2013 9:20:52 PM UTC+5:30, Diego Pino wrote:

Hi,

I'm exploring a way to build ChromiumTestShell for Android linking against some system libraries. I checked build/common.gypi and saw that, at this moment, it's possible to build ChromiumTestShell for Android using some system libraries, basically libstlport, libicu and libexpat. However, it's only possible to use system libraries when building Chromium as a WebView.

build/common.gypi:1381

# When building as part of the Android system, use system libraries
# where possible to reduce ROM size.
# TODO(steveblock): Investigate using the system version of sqlite.
'use_system_sqlite%': 0, # '<(android_webview_build)',
'use_system_expat%': '<(android_webview_build)',
'use_system_icu%': '<(android_webview_build)',
'use_system_stlport%': '<(android_webview_build)',

In any case, I modified some .gyp scripts and managed to build Chromium as a browser using Android's system libraries. Using this approach, the size of the ROM gets reduced considerably but this change should be benefitial for reducing RAM consumption too.

Hi Diego,

I am also trying to build Chrome using Android's system libraries. 

As you have mentioned that you are able to build Chromium as a browser using Android's system libraries, 

are you able to link more system libraries othar than icu, sltport, expat, skia, jpeg which are also used by android_webview_build?

Trying to understand what are the challenges you have faced while linking to other libraries.

If possible can you please share the changes you have made to achive the ROM reduction?

Thank you,

Sanjoy Pal

[Ravi Kasibhatla]

Hi Torne,

As Diego pointed, when we are using the system libraries, there is reduction in the ROM size of native webview library (libwebviewchromium.so), but I had a query about the impact of using system libraries on performance/RAM.

You pointed in another thread (removing of xslt posted on blink-dev - https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/zIg2KC7PyH0%5B76-100-false%5D), splitting into multiple libraries can have its own pitfalls in terms of performance/size.

From the above referred discussion, I am quoting couple of your explanations with regards to splitting into multiple libraries and its impact on loading and execution:

         "ELF-based systems use position independant code for shared libraries, which means that there *aren't* relocations for each individual call and it's *not* possible to have the loader fix up the calls directly, there is always an indirection via the relocated dynamic symbols. Patching callsites directly is slow and expensive (both in disk IO and in RAM usge, since this makes a lot of pages dirty) ..." 

         "Splitting code into more libraries is very likely to increase the total memory usage; relocation tables do not work the way you think. The references in a single library are coalesced into a single location (the indirection referred to in the previous section), but references across multiple libraries can't be coalesced, so having more libraries means you have more relocations that refer to the same symbols, taking up more space."

         "... because of paging the data is *not* loaded in memory until needed: relocation processing causes it to be needed immediately, and thus relocation causes more disk IO. It is absolutely not true that all the data must be loaded from disk anyway;"

Just wanted to know from you, in light of these *reasons*, will there be impact (in case of Chromium WebView), since we are loading multiple libraries on runtime unlike Chromium itself where all code is part of single library. If yes, how much could the impact be?

Regards,

Ravi Kasibhatla.

[Torne (Richard Coles)]

The extra memory cost of having more cross-library references is very small unless you take splitting the binary up to an extreme level; my point in that thread was just to emphasise that splitting into more libraries is not a memory *saving* as the poster I was replying to believed. The cost isn't something that anyone needs to worry about under normal circumstances.

The saving of being able to use an existing library which is preloaded by the zygote and shared by other processes is huge (it makes depending on the library almost "free"), which is why we do it for the WebView; it is a memory saving as well as a ROM size saving, and you have nothing to worry about.

My earlier points in this thread about only being able to rely on non-NDK libraries if you are part of the OS image still applies, though; seriously, don't do this unless you are building a custom Android image with your code included; even if it looks like it works on the devices you have handy to test on, it will go wrong in all sorts of obvious and/or subtle ways on some other device you haven't tried. :)

--
Torne (Richard Coles)
to...@google.com

[Eric Wang]

Hi Diego,

在 2013年11月11日星期一 UTC+8下午11:50:52，Diego Pino写道：

[Eric Wang]

Hi Diego,

    My task is "try to rebuild the browser package linking to system libraries instead of bundling its own copies of libjpeg, libpng, zlib, ", but I don't know how to do it, could you help me about it?

Thank you 

Eric

    

在 2013年11月11日星期一 UTC+8下午11:50:52，Diego Pino写道：

Hi,

[Lei Zhang]

Presumably this is on Linux? You should start your own thread instead
of replying to a different thread about Android. I don't think Diego
cares about your problem.

To help answer your question, look here:
https://chromium.googlesource.com/chromium/src/+/master/build/linux/unbundle/README

[Eric Wang]

Yes, maybe it's too easy for them, I'll try to get info from the wiki then.

Thanks Lei.

在 2015年12月4日星期五 UTC+8上午10:22:01，Lei Zhang写道：

[Eric Wang]

Lei,

在 2015年12月4日星期五 UTC+8上午10:41:54，Eric Wang写道：

[Eric Wang]

在 2015年12月4日星期五 UTC+8下午2:02:09，Eric Wang写道：

Lei,

As I'm a rookie on android and chromium, so could you give me more help explicitly?
I could not get the key point on the wiki, :(

Thanks

Eric