Solution for Breaking change, Linux Build chromeos=1 and HarfBuzz linkage?

[Dominik Röttsches]

Hi,

while working on important performance and correctness improvements in text, fonts & layout in Blink we are dependent on being able to use newest versions of the HarfBuzz text shaping library. HarfBuzz is maintained by Behdad Esfahbod, who works for Google, and we need a close loop between Blink and HarfBuzz for these improvements. Previously, we had an #ifdef in Blink to keep some form of compatibility with very old versions, but this is no longer viable, since we're moving an important function to a HarfBuzz built-in implementation. This is discussed in issue https://bugs.chromium.org/p/chromium/issues/detail?id=589340.

Background: There are two types of linkage to HarfBuzz in Chrome:

* static linking into the Chrome executable, which happens on all other build configs except CrOS and

* dynamic linkage against the system version, which happens on CrOS.

On CrOS itself, a reasonably new version of HarfBuzz is available. On a host Ubuntu 14.04 the version of HarfBuzz is quite old and not sufficiently recent for this work.

So, the CL we landed (already reverted) caused a conflict with building Chrome on a Linux/(Goo|U)buntu host system with chromeos=1 because this build expects HarfBuzz to be up-to-date on the Ubuntu host system. This expectation is met on CrOS but not on the Ubuntu host system.

For this build type to be supported we need a new version of HarfBuzz to be installed on the host system that builds the chromeos=1 build, equivalent to the one used by Chrome/Blink, or at least equivalent to the one used on the CrOS environment.

In https://bugs.chromium.org/p/chromium/issues/detail?id=589342 we suggested for affected developers to install such a version manually. The reaction has been that this is too high a burden, and ideally should be automated. An additional suggestion has been to make a newer HarfBuzz version for Ubuntu version as a package that can be installed. Is there someone who owns the chromeos=1 build and could help with that?

We believe that expecting linkage to work and execution against the old system version is problematic to say the least, and we need to find a better solution here. Keeping backwards compatibility in the font code is unfortunately not possible anymore and causes too much overhead.

Hoping to find consensus and ideally getting some help from the chromeos=1 build community, thanks,

Dominik

[Dirk Pranke]

I don't know if anyone in particular really owns the chromeos=1 code.

However, here's my suggestion:

We relatively recently switched desktop linux builds to build against checked-in sysroots by default, though packagers and other developers

who really don't want this can turn it off.

Maybe we should create another sysroot that reflects the dependencies we expect on CrOS, even if the actual CrOS builds don't use it?

-- Dirk

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Nico Weber]

sbc is working on getting cros=1 builds working with the sysroot. It's somewhat involved, sadly. But I agree, that seems like the best approach.

[Christian Biesinger]

In this specific situation, wouldn't this fail at runtime though? Or do we do store the sysroot path in the binary for so search paths? (I forgot the name of that rpath option....)

-Christian

[Dirk Pranke]

Yes, we should still pick up the correct shared libraries from the sysroot via RPATH.

(and, yeah, I saw the note from sbc@ after I sent my reply :).

-- Dirk

[Albert Bodenhamer]

My team is likely the biggest user of chromeos=1.

Is this something we could include in install-build-deps?

Alternatively, could we fire a LOG(FATAL) in chromeos=1 builds if the incorrect version of harfbuzz is installed (along with instructions for fixing it)?

You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Dirk Pranke]

We can (and probably should) install a chromeos sysroot if we see chromeos=1 in the GYP_DEFINES;

to do so should be easy, assuming we have a working sysroot we can pull down from somewhere,

by modifying //build/linux/sysroot_scripts/install-sysroot.py .

It wouldn't surprise me if sbc@ was already planning to make that so, but if not, you or someone on

your team could do so.

-- Dirk

[Dominik Röttsches]

Hi again, and thanks to everyone who came up with proposals on how to address this!

sbc@, looks like you're working on switching the chromeos=1 to using sysroot by default, could you CC me on a bug or CC me on a CL?

I would think that once this lands can just reland our CL which depends on a newer HarfBuzz. That's great, so big thanks for taking this one on and making managing our dependencies much better!

Is there some documentation on how/when the sysroots are updated/regenerated? This would be good to know once we need to push out a new HarfBuzz version.

Dominik

(sry, resent from @chromium.org account)

[Dominik Röttsches]

Hi,

since Sam asked me for clarification (in a separate thread) on which HarfBuzz version is needed and explained about what is contained in the sysroot:

He explained that the sysroot mirrors debian/jessie for example. Looking at Debian Jessie, it has HarfBuzz it has HarfBuzz 0.9.37, which is a version that at least links and builds with our latest changes, but it still does not have performance improvements and other fixes that we need and this gap might cause breakages in the future when we again rely on newer HarfBuzz fixes and features.

HarfBuzz is actually part of the Chrome repository and is statically linked into the Chrome executable on all other platforms except ChromeOS. 

jshin@ has updated HarfBuzz for ChromeOS in the past using ebuild files in the media-libs/ directory, compare:

https://bugs.chromium.org/p/chromium/issues/detail?id=517603

So, I would say, somehow the sysroot should follow this version from the ebuild files when building chromeos=1 - Could it somehow contain that version?

What else can we do to fix the Linux host chromeos=1 build to not rely on a dated version? What are the ChromeOS release bots building against?

I remember I've tried  chromeos=1 & use_system_harfbuzz=0 but IIRC this build fails as well, since the gtk=>pango dependency still links against system HarfBuzz and linking fails due to duplicate symbols.

Albert, would you know someone who could spend some time investigating this?

Thanks,

Dominik

[Albert Bodenhamer]

We're currently in the middle of CY and don't have a lot of spare cycles, but I'll see what I can do.

 

Thanks,

Dominik

[Albert Bodenhamer]

+Achuith Bhandarkar can you work with Dominik and Sam to figure out a path forward?

On Fri, Mar 4, 2016 at 8:57 AM Dominik Röttsches <dr...@google.com> wrote:

Hi Sam, Albert,

I took another closer look at the problem and how the sysroot images
are generated.

I would propose the following steps to find a compromise here that
allows us to land our change without #ifdefs in our code:

1) Land Sam's change sysroot default on for chromeos=1 build
https://codereview.chromium.org/1585463002

2) Then update wheezy sysroot images to contain harfbuzz 0.9.36, which
is the wheezy version, or pin them to 1.0.1 from unstable. An example
CL on how to update the sysroot can be found here:
https://bugs.chromium.org/p/chromium/issues/detail?id=576807

When the linux=1 && chromeos=1 buid uses the sysroot by default, this
will fix our linkage issues against versions of HarfBuzz before
0.9.28.

Still, chromeos=1 buils on Linux hosts with an old HarfBuzz version
such as 0.9.36 in the sysroot will face expose a set of layout and
text bugs, which do not occur on ChromeOS as the HarfBuzz version
there is not as out-of-date. However, this may be acceptable for a
developer-only build. These regressions could be partially mitigated
by pinning to the latest version from unstable (1.0.1), plus pinging
the Debian maintainer of libharfbuzz package to roll a new release
reflecting an even newer the upstream version 1.2.3. In fact, latest
HarfBuzz 1.2.3 is needed for correct font fallback behavior and
showing Sinhala, and Devanagari script languages correctly and is
practically the only reference version for correct layout and text.

Sam, if you could do 1) and help with uploading new wheezy sysroot
images as part of 2), I am happy to prepare a CL adding HarfBuzz to
the wheezy sysroot. WDYT?

I hope we can resolve the issue for now, until the next ABI break at least.

Theoretical alternatives to this approach:
* Define a new sysroot type to be used only for the chromeos=1 Linux
build type which has a custom .deb of HarfBuzz intergrated. This .deb
would either have to a) be built locally from third_party/harfbuzz-ng
or b) coming from the debian package repository. a) would require a
lot of build scripts work and adding debian/rules files to HarfBuzz
upstream, b) would require a roundtrip via the Debian maintainer to
get update versions each time. Overall, this seems overkill for fixing
the linkage issue - given that for local development perhaps

Dominik

[Dominik Röttsches]

(sorry, reposting from @chromium.org so that it reaches the group)

[Dominik Röttsches]

Ping thestig@, Sam says in a comment to his CL that you might be able to help landing this, once the sysroot images are updated to Debian Jessie?

Do we have a tracking bug for this update? Who is taking care of updating the sysroot images? This is now blocking landing this CL, and in turn, blocking us from making important performance progress in fonts & layout.

Thanks,

Dominik

[Lei Zhang]

Uh, I'm also out of office for a while. Sam left some instructions [1]
- is that enough that someone else can give it a whirl? It would be
good to spread the knowledge anyway.

[1] https://codereview.chromium.org/1761123002/

[Dominik Röttsches]

(sending this to chromium-dev as well, sorry, I will try to learn switching my sender account when posting to chromium-dev)

Since the sysroot and default switching to sysroot images is slowed down by Sam being out, and it will only get us half the way anyway, since the HarfBuzz version would still be controlled by the Debian release cycle and not fully by us, I explored another alternative (thanks to Raphael Kubo da Costa for that suggestion):

Building HarfBuzz into a shared_library / loadable_module, a wip CL is here:

https://codereview.chromium.org/1783443003

However, I currently can not get this to link, I would appreciate someone's help who is familiar with managing dependencies and link paths in GN. Dirk, Phajdan, could you perhaps take a look at the CL?

Thanks,

Dominik

[Dirk Pranke]

I feel like we've gone off the rails here somewhere, but maybe it's just that I've lost track of things.

Correct me if I'm wrong, but here's my understanding:

1) In general, we should always use the version of harfbuzz in //third_party

2) On an actual chromeos build, in order to reduce the footprint of chrome, we want to use the system

harfbuzz. The CrOS build folks will ensure that the system harfbuzz (as found in a sysroot) is at least 

as new as the one in chromium's //third_party.

So, it seems like we should decouple use_system_harfbuzz from is_chromeos and make the CrOS build

system set use_system_harfbuzz explicitly. Failing that, we can use a package config script to check the version

if is_chromeos is true (which is what we do today).

Is there some reason that doesn't work, apart from not having a cros sysroot installed by default for "fake"

chromeos=1 builds? 

If not, I'd rather spend time getting the cros sysroot working than spend time trying to come up with a weird

build cycle.

Since we control what the sysroot looks like for CrOS (not Debian), I don't understand your comment about

the Debian release cycle ...

Apart from all of that, I doubt all of chromium-dev really cares about this at this point. Feel free to follow up with

me off-list (or in a bug) and cc the other interested parties.

-- Dirk

[Achuith Bhandarkar]

chromeos=1 is a configuration used by a good number of chromeos developers on a daily basis, at least everyone in chromeos ui and chromeos enterprise. This isn't a weird, rarely used setup. 

So far, the instructions to build linux have sufficed for the chromeos=1 configuration.

It doesn't seem acceptable to add a number of manual steps for all chromeos developers with the error condition being a mysterious compile failure. 

[Dirk Pranke]

Perhaps I didn't make myself clear :(

The problem, as I understand it, is that, today, when you set chromeos=1 we bypass the 

thirdparty harfbuzz and attempt to use the "system" harfbuzz (i.e., the one in the 

checked-in linux64 sysroot), which is too old.

One possible fix for this would be to decouple the two switches, so chromeos=1 would

not automatically imply using the system harfbuzz, and so using the linux64 sysroot

would still work fine; the side effect of this would be that in a real CrOS build (i.e., one

using cros_sdk etc.), you'd have to also set use_system_harfbuzz. This is presumably not

a problem, because the real CrOS builds already set a whole bunch of other variables

explicitly.

Another alternative would be to make setting chromeos=1 cause the build to switch to

using a new checked-in sysroot that matches the CrOS dependencies (i.e., contains newer

stuff than the linux64 desktop), and then the "system" harfbuzz (i.e., the one in the CrOS

sysroot) would be okay. The risk here would be that using a CrOS sysroot might cause

some tests to break if you were to actually try to run them in a normal linux build (i.e., not

on a CrOS device). We'd have to check that ...

In no situation are we adding more manual steps, or breaking configurations.

Does that make sense?

-- Dirk

[Nico Weber]

Decoupling the switches sounds like a great idea to me. Regular cros builds already set tons of gyp defines, adding one more seems fine (https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild#261).

The sysroot idea was mentioned above, sounds like that's somewhat involved.