libchromeos (and policy/metrics): using shared libraries only
Mike Frysinger
library, read on; if you don't, then reply with +1 ;)
does anyone have objections to converting libchromeos to a shared
library, or any other ideas as to why we'd *need* a static library and
could *not* utilize a shared one ? as part of the libchrome SLOT
work, we hit problems with static libraries that also need libbase.
if they weren't static libraries, then life would be good.
for the SLOT background logic, consider this: we have libchrome-85154
(the current snapshot in our tree) and we want to add libchrome-122837
(a much newer version). all packages currently in the tree are
linking against libchrome-85154, but we want to start migrating one by
one to the new version. otherwise we have to do atomic commits for
all packages in the tree and make sure they're all tested ahead of
time. this is time consuming and requires cross-coordination of many
developers. which is why we're ~40000 svn revs behind the current
chromium source tree (almost 1 year at this point)!
http://crosbug.com/16623
now why is libchromeos a problem ? code in there uses stuff from
libchrome and since we only produce a static libchromeos.a, we need to
link against the libchrome version that libchromeos was built against
whenever anyone wants to use libchromeos.
consider the shill package: it uses code from both these packages. so
today, it'd do:
g++ -o shill .... -lbase-85154 -lchromeos -lbase-85154 ....
the first -lbase is shill itself needing libchrome while the second
one is pulled in indirectly via the libchromeos pkg-config file. this
is perfectly fine.
but now let's say we want to update shill to libchrome-122837.
someone does all the hard work and then they try to link:
g++ -o shill .... -lbase-122837 -lchromeos -lbase-85154 ....
this is very bad. we can hit many issues, some of which would
manifest themselves as link failures while others could be runtime
misbehavior. if the shill upgrade is incomplete, it could pull in old
symbols and it'd be hard to notice, or if symbols move between object
files, we could easily hit multiple definition errors as the shill
code tries to pull objects in from libbase-122837.a while
libchromeos.a causes stuff to get pulled in from libbase-85154.a, or
if the structures passed between funcs changes, and we pull some
symbols from libbase-122837 while others come from libbase-85154, we
could easily get random runtime memory corruption/crashes.
you might say "well, let's just update all the packages that need
libchromeos simultaneously". trouble is, of the ~23 packages that use
libchrome, more than half also use libchromeos, and the ones that
don't are fairly "small". so SLOT-ing libchrome without addressing
libchromeos doesn't gain us that much in terms of easing developer
pain.
so my solution is to convert libchromeos to a shared library only (and
to drop the libpolicy.a and libmetrics.a that we currently ship since
both of those are already installing/linking shared libraries). this
way the libchromeos code base can pull in the stuff it needs from
libchrome statically and then we control the symbols exported by that
to only be the API we've developed ourselves. the libchrome version
that libchromeos is built against no longer matters.
some might cry performance foul because a shared library implies all
the code must be PIC. however, i'd note that we already build both
libchrome and libchromeos with -fPIC manually added, and all non-PIC
code in the entire ChromeOS system is built as PIE (due to our
hardening work). so we aren't talking basic code generation issues
here like non-PIC/PIE speed versus PIC.
further, while static linking of the libchromeos library would cause
only the objects used to get pulled in (vs a shared library mapping
the entire thing in), we're talking about ~120KiB of *utility* code
here (i.e. not performance critical), and further, once the first
daemon gets loaded that needs this, the shared library gets, well,
shared with all others thus saving on the disk load and .text init.
so there could even be a performance gain ...
thus my conclusion performance wise is that we're talking about "system noise".
other ideas that have come to mind but i've disqualified:
- merge libchrome and libchromeos into a single .a
deal breaker: we're sourcing multiple git live repos in a single ebuild
- disallow base/ usage inside of libchromeos
deal breaker: base/ provides a lot of useful code which we'd be on
the hook for re-implementing and thus wasting developer cycles
- manually copy source files that we use from libchrome into libchromeos
deal breaker: we have to manually sync things which wastes developer
cycles (and we still have symbol collision issues unless we get fancy
and rewrite the copied symbols at build/link time which is fragile
when you talk about C++)
- SLOT libchromeos so there is a 1:1 coordination between libchrome SLOTs
combo breaker: cros-wrokon doesn't support more than 1 stable ebuild
per $PN; supporting uniqueness on a CAT/PN:SLOT basis vs just CAT/PN
will take a bit of work to make sure we don't screw things up on the
automatic-upgrade script side of things, but that still wouldn't solve
the local developer workflow who run `cros_workon start libchromeos`.
further, we'd have to add #ifdef's to libchromeos so that it'd support
multiple libchrome versions simultaneously in a single git source
tree. all of this leads to unhappy developers.
-mike
Jonathan Kliegman
- manually copy source files that we use from libchrome into libchromeos
deal breaker: we have to manually sync things which wastes developer
cycles (and we still have symbol collision issues unless we get fancy
and rewrite the copied symbols at build/link time which is fragile
when you talk about C++)
- SLOT libchromeos so there is a 1:1 coordination between libchrome SLOTs
combo breaker: cros-wrokon doesn't support more than 1 stable ebuild
per $PN; supporting uniqueness on a CAT/PN:SLOT basis vs just CAT/PN
will take a bit of work to make sure we don't screw things up on the
automatic-upgrade script side of things, but that still wouldn't solve
the local developer workflow who run `cros_workon start libchromeos`.
further, we'd have to add #ifdef's to libchromeos so that it'd support
multiple libchrome versions simultaneously in a single git source
tree. all of this leads to unhappy developers.
-mike
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en
Mike Frysinger
> On Tue, Feb 21, 2012 at 4:55 PM, Mike Frysinger <vap...@chromium.org> wrote:
>> - disallow base/ usage inside of libchromeos
>> deal breaker: base/ provides a lot of useful code which we'd be on
>> the hook for re-implementing and thus wasting developer cycles
>
> While I'm somewhat new to this problem and missing background/history, from
> having to deal with this today I think this is the best choice.
i was speaking specifically to libchromeos, but i guess you're talking
about the whole tree ?
> We deprecate base/ usage. Existing packages can continue to use the
> existing one, but any new features should just be implemented in libchromeos
> directly.
i think this would be duplication in effort that wouldn't solve the
underlying problem (see below)
> Code can be copied or cherry-picked from chrome/base if needed, but trying
> to rely on non-library and unstable code from another project doesn't seem
> practical.
i don't think this would solve the network effect. we still have a
lot of packages that need to have their API coordinated when someone
wants to upgrade. for example, shill uses a task queue API that was
rewritten between 85154 and 122837. so we cherry pick in 85154 (and
everything it uses since base/ likes to re-use itself). then another
project uses that API. but shill wants to update to 122837. we
cherry pick in the new API (and everything it uses), and now we have
to make sure all other packages that might have been using the old API
gets updated. so were back to square one.
further, this has the problem of basically cherry-picking parallel
trees. it's really hard to extract small portions of the libchrome
API and mix between versions.
> Unless we can get a commitment from some group of people to garden and keep
> the code in base/ fresh (and the packages that depend on it) , we'll just
> end up in this same place in another half year just with other moving bits
> thrown on top to add to the confusion.
the biggest problem with libchrome today is that we get only one
version for the whole tree. so if we want to update to a newer rev,
we have to have every project test that new rev, do their CLs to work
with it, and then have every package get merged at once (source git
repo and ebuild update). if something goes wrong, we now have to roll
back commits across many projects.
with the SLOT work, now people can drop in a new libchrome version for
free -- nothing in the tree will use it by default. then if someone
wants to update their project to the newer snapshot, we have two
commits that have to be done together -- one for the 9999 ebuild and
one for the source git repo. this is a lot easier for a single
developer/small group to coordinate since it's the code they're
familiar with.
-mike
Jonathan Kliegman
On Tue, Feb 21, 2012 at 17:11, Jonathan Kliegman <kli...@chromium.org> wrote:
> On Tue, Feb 21, 2012 at 4:55 PM, Mike Frysinger <vap...@chromium.org> wrote:
>> - disallow base/ usage inside of libchromeosi was speaking specifically to libchromeos, but i guess you're talking
>> deal breaker: base/ provides a lot of useful code which we'd be on
>> the hook for re-implementing and thus wasting developer cycles
>
> While I'm somewhat new to this problem and missing background/history, from
> having to deal with this today I think this is the best choice.
about the whole tree ?
> We deprecate base/ usage. Existing packages can continue to use thei think this would be duplication in effort that wouldn't solve the
> existing one, but any new features should just be implemented in libchromeos
> directly.
underlying problem (see below)
> Code can be copied or cherry-picked from chrome/base if needed, but tryingi don't think this would solve the network effect. we still have a
> to rely on non-library and unstable code from another project doesn't seem
> practical.
lot of packages that need to have their API coordinated when someone
wants to upgrade. for example, shill uses a task queue API that was
rewritten between 85154 and 122837. so we cherry pick in 85154 (and
everything it uses since base/ likes to re-use itself). then another
project uses that API. but shill wants to update to 122837. we
cherry pick in the new API (and everything it uses), and now we have
to make sure all other packages that might have been using the old API
gets updated. so were back to square one.
further, this has the problem of basically cherry-picking parallel
trees. it's really hard to extract small portions of the libchrome
API and mix between versions.
> Unless we can get a commitment from some group of people to garden and keepthe biggest problem with libchrome today is that we get only one
> the code in base/ fresh (and the packages that depend on it) , we'll just
> end up in this same place in another half year just with other moving bits
> thrown on top to add to the confusion.
version for the whole tree. so if we want to update to a newer rev,
we have to have every project test that new rev, do their CLs to work
with it, and then have every package get merged at once (source git
repo and ebuild update). if something goes wrong, we now have to roll
back commits across many projects.
with the SLOT work, now people can drop in a new libchrome version for
free -- nothing in the tree will use it by default. then if someone
wants to update their project to the newer snapshot, we have two
commits that have to be done together -- one for the 9999 ebuild and
one for the source git repo. this is a lot easier for a single
developer/small group to coordinate since it's the code they're
familiar with.
-mike
Chris Masone
Jonathan Kliegman
We already had this discussion on the Systems team (on which pretty much all of the consumers of this code work) and decided that the work that Mike is doing is the most sensible way forward; I'd really rather avoid re-hashing the same discussion we had 6 months ago.
Mike Frysinger
> On Tue, Feb 21, 2012 at 5:31 PM, Mike Frysinger wrote:
>> On Tue, Feb 21, 2012 at 17:11, Jonathan Kliegman wrote:
>> > On Tue, Feb 21, 2012 at 4:55 PM, Mike Frysinger wrote:
>> >> - disallow base/ usage inside of libchromeos
>> >> deal breaker: base/ provides a lot of useful code which we'd be on
>> >> the hook for re-implementing and thus wasting developer cycles
>> >
>> > While I'm somewhat new to this problem and missing background/history,
>> > from
>> > having to deal with this today I think this is the best choice.
>>
>> i was speaking specifically to libchromeos, but i guess you're talking
>> about the whole tree ?
>
> I was referring to libcros specifically, sorry. I needed to add a function
> to it and then saw it had been added a month ago. So was trying to figure
> out how to get that code accessible to the power_manager project.
i heard whispers that libcros should die and anything left worth
keeping should merge with libchromeos
> There would be some code duplication but how much of base do we actually use
> now? At this point, any code written in the last 9 months has either been
> duplicated or wasn't needed.
the reason i got involved in this was to get rid of gtk. our old
snapshot needs it, but the latest does not.
wrt other people, it could be that things got duplicated, but i think
i've heard that some stuff was simply delayed in the hopes that some
other sucker would do the update. i guess i'm the sucker now.
> The SLOT work looks like it could help and might make my complaints
> irrelevant. But I'm still a bit worried - I tried to see what happens by
> setting to tip of tree as an option for what I was doing and libchrome
> doesn't even compile as is. It feels like we need to either have someone
> committed to keeping the code at least compilable or we should just move on
> from it.
yes, i have some more ideas on how to improve this situation
my end goal is to automate as much as possible to keep devs focusing
on the areas they actually care about (their sub-projects). anything
that involves copying & pasting and manual intervention sounds like
surrender :).
-mike
Mike Frysinger
> tl;dr: if you know of a reason why libchromeos *must* be a static
> library, read on; if you don't, then reply with +1 ;)
ok, i should have done experiments before starting this thread, but
here's a nuance that prevents libbase from being cleanly linked into a
shared library and an executable simultaneously: not all of libbase
are pure utility functions; some include state.
an easy example: if we look at the logging module, it has "PathString
*log_file_name". this allows someone to direct all output to logger
to a file. when we create libchromeos.so, it gets its own copy of
libbase (and thus internal state), so it has its own internal
"log_file_name". when we link an application that uses libchromeos
and the logging module from libbase, the app gets its own copy of
libbase and thus its own "log_file_name". now when the app calls the
logger func to set the file path, they set the application's copy of
"log_file_name". but when any code inside of libchromeos.so tries to
utilize the logger, they don't have the log file set, so things get
routed incorrectly.
unfortunately, we've already crossed the point where this would be a
problem. libpolicy.so from libchromeos links in libbase and has its
own copy of logger and friends. the reason this hasn't noticed
immediately is that libpolicy.so ends up exporting these symbols as
part of its ABI, so any applications that do "-lpolicy -lbase" get the
logger symbols via libpolicy.so. if the app were to do "-lbase
-lpolicy" though, we'd hit the situation i described above.
libchaps.so appears to do the same thing -- it links in libbase and
exports the symbols it got as part of its ABI.
libmetrics.so is similar, but it manages to skate by because that ends
up linking to libpolicy.so first, so it uses the copy of symbols that
libpolicy.so is exporting.
in practice, it seems we've already got a few apps with this bad
state. if i start with a fresh chroot and build an x86-alex board and
run this magic incantation (basically it says locate all ELF's that
have a copy of the logging "log_file_name" and are linked against
lib{metrics}.so):
$ scanelf -F'#s%F %N' -yq -s+_ZN7logging13log_file_nameE -N
libmetrics.so ./{usr/,}*bin/*
$ scanelf -F'#s%F %N' -yq -s+_ZN7logging13log_file_nameE -N
libpolicy.so ./{usr/,}*bin/*
$ scanelf -F'#s%F %N' -yq -s+_ZN7logging13log_file_nameE -N
libchaps.so ./{usr/,}*bin/*
i find these broken programs:
chromeos-base/cashew (/usr/sbin/cashewd)
chromeos-base/chromeos-cryptohome (/usr/sbin/cryptohome)
chromeos-base/chromeos-cryptohome (/usr/sbin/cryptohomed)
chromeos-base/chromeos-cryptohome (/usr/sbin/cryptohome-path)
chromeos-base/chromeos-login (/sbin/keygen)
chromeos-base/chromeos-login (/sbin/session_manager)
chromeos-base/chromeos-wm (/usr/bin/chromeos-wm)
chromeos-base/crash-reporter (/sbin/crash_reporter)
chromeos-base/cromo (/usr/sbin/cromo)
chromeos-base/metrics (/usr/bin/metrics_client)
chromeos-base/metrics (/usr/bin/metrics_daemon)
chromeos-base/power_manager (/usr/bin/backlight-tool)
chromeos-base/power_manager (/usr/bin/powerd)
chromeos-base/power_manager (/usr/bin/powerm)
chromeos-base/power_manager (/usr/bin/suspend_delay_sample)
chromeos-base/shill (/usr/bin/shill)
chromeos-base/update_engine (/usr/bin/update_engine_client)
chromeos-base/update_engine (/usr/sbin/update_engine)
we can't just say "well, if you're going to use libchrome, then don't
use the logger module and instead always use the chromeos logger
module" because a lot of the base headers implicitly call the logger
module (just grep for DCHECK and CHECK macros). my emphasis here has
been and continues to be "make
i have an even crazier proposal, but i'll play with this locally first
to verify it'll work
-mike
Mike Frysinger
> we can't just say "well, if you're going to use libchrome, then don't
> use the logger module and instead always use the chromeos logger
> module" because a lot of the base headers implicitly call the logger
> module (just grep for DCHECK and CHECK macros). my emphasis here has
> been and continues to be "make
rest of the thought here was letting devs focus on their work and make
libchrome{,os} usage as simple as possible
> i have an even crazier proposal, but i'll play with this locally first
> to verify it'll work
ok, my crazier idea has passed some unittests, built & booted, and i
can browse the web, so here's what i've got:
- libchrome builds shared libraries (SLOT-ed ebuilds) and no static libs
- cros-workon libchromeos ebuild depends on all the libchrome SLOT's,
and then builds libchromeos/libpolicy shared libraries for each
supported libchrome SLOT (and no static libs) in a single ebuild
- parallel_emerge will rebuild any pkgs when their direct
dependencies get updated (this is how it works now and is required for
sanity with any static lib usage)
if i were to create a single libchrome shared library, this would have
to link against all 3rd party libraries that any object in there uses.
that means gtk, gdk, libevent, libdl, and more would always get
loaded for any app that uses libchrome which is clearly undesirable.
my solution is to have libchrome create multiple shared libraries.
the current list:
- libbase-gtk-85268.so - stuff that requires gtk/gdk/etc...
- libbase-event-85268.so - stuff that requires libevent/etc...
- libbase-dl-85268.so - stuff that requires libdl/etc...
- libbase-core-85268.so - stuff that only requires glib/pthread/librt
since pretty much everything that uses libbase uses stuff that
requires glib/pthread/librt, i've set that as the baseline for the
"core". if the base code requires any additional libs, it gets split
out. but all of this are internal details that external libs
shouldn't have to care about, so the ebuild also creates a
libbase-85268.so linker script:
GROUP ( AS_NEEDED ( -lbase-core-85268 -lbase-gtk-85268
-lbase-event-85268 -lbase-dl-85268 ) )
this tells the linker to link against the shared libs only when they
would actually get used without "polluting" the linking steps of other
apps by trying to do -Wl,--as-needed -lbase-.... -Wl,--no-as-needed in
the pkg-config file.
i haven't done this split for libchromeos, but i'm thinking i should
because it has one small object that ends up pulling in libcprecpp,
and another small object that ends up pulling in openssl (for
cryptohome stuff).
the downside to the one-libchromeos-to-many-libchrome is that
libchromeos might have to do some #ifdef checking in order to support
multiple libbase's simultaneously, but based on how much code
libchromeos actually uses from base/, i suspect this won't be a big
deal in practice. time will tell of course.
now the other projects select the SLOT of libchrome that they're using
and do something like:
pkg-config --libs --cflags libchrome-85268 libchromeos-85268
attempting to do include/link paths yourself is deprecated. you must
go through pkg-config.
the end result is that programs that depend on libchrome{,os} can
update independently of any other package so long as it doesn't link
against a library (static or shared) that also needs libchrome{,os}.
otherwise, that shared library induces a loop that requires all
packages using it to get upgraded together.
for example, the chaps package creates libchaps.so which links against
libchrome. chromeos-cryptohome also links against libchaps.so, so if
either chaps or chromeos-cryptohome want to update to a newer
libchrome version, they have to update to the same version at the same
time.
the biggest hassle appears to be consumers of libmetric.so. looks
like that creates a loop of 12 pkgs. but this is still half the size
of the current loop, and the other half are freed. if we find this
reduced set is still too many to manage, it could be addressed by
libmetrics.so doing what libchromeos.so does -- depend on all the
libchrome SLOTs that consumers of libmetrics cares about, and then
build a copy of libmetrics for each one. i suggest we put that off
for now and see what kind of improvement we get from what i have now.
note, for the purposes of *building*, these packages can update
independently. they'll just end up with multiple copies of libchrome
being loaded at runtime which is obviously a bad thing. this gives us
some breathing room in terms of not having to land a bunch of CL's
simultaneously and breaking the build, but they should ultimately get
to the same point. i'll be adding some checks to our build_image code
to catch this. as a matter of policy, i think we should say "no
release should ship with more than one version of libchrome". this
gives independent developer groups breathing room to upgrade once we
select the version we want to ship the next release with.
if anyone has any thoughts before i start dumping CL's, now's the time ;)
-mike
Mike Frysinger
brave reviewers who are willing to weird their +2 stick :)
especially if you're fluent in scons (you poor poor soul) ...
libchrome ebuild to build as shared+SLOT:
https://gerrit.chromium.org/gerrit/15415
libchromeos ebuild to build against all libchrome SLOTs:
https://gerrit.chromium.org/gerrit/17016
libchromeos source builds as shared libs:
https://gerrit.chromium.org/gerrit/16763
-mike
Mike Frysinger
http://dev.chromium.org/chromium-os/packages/libchrome
hopefully if i get hit by a (cat) bus, or someone else feels like improving things, all the information (back story and current state) is captured succinctly for them to get from 0 to fast.
feedback/improvements welcomed!
-mike