[gentoo-dev] does v8 shared library make sense with current upstream approach?

["Paweł Hajdan, Jr."]

I'd like to get your feedback and opinion about removing shared v8
library package from Gentoo. It's currently used by www-client/chromium,
dev-db/drizzle, dev-db/mongodb, dev-lang/v8cgi and sci-geosciences/osgearth.

net-libs/nodejs switched back to bundled v8 a long time ago:

25 Feb 2013; Patrick Lauer +nodejs-0.6.21-r2.ebuild,
+nodejs-0.8.20.ebuild:
Version bump for 0.6 and 0.8 that also disables shared v8 as our v8
maintainers remove all compatible versions

Some bugs for reference:

https://bugs.gentoo.org/show_bug.cgi?id=417879
https://bugs.gentoo.org/show_bug.cgi?id=420995
https://bugs.gentoo.org/show_bug.cgi?id=471582
https://bugs.gentoo.org/show_bug.cgi?id=477300
https://bugs.gentoo.org/show_bug.cgi?id=484786

From mongodb upstream <https://jira.mongodb.org/browse/SERVER-10282> :
"compiling with versions of v8 other than what is included is not
currently supported."

I'd like maintainers of all packages depending on dev-lang/v8 to make
their packages use bundled v8 copy instead (I can file bugs for that,
let's discuss here whether it should be done).

For now V8 upstream gives no guarantees about API/ABI stability and
actually breaks it very often
(<http://upstream-tracker.org/versions/v8.html>). Having a shared
library so closely tied to packages (which results in frustrating
blockers, since v8 is updated often and chromium is synchronized with
that) is not really much different from everyone bundling the library.
I'd like that to improve, but for now it's time for a pragmatic solution
to this IMHO.

Paweł

[Peter Stuge]

"Paweł Hajdan, Jr." wrote:
> "compiling with versions of v8 other than what is included is not
> currently supported."

..

> For now V8 upstream gives no guarantees about API/ABI stability and
> actually breaks it very often

I agree that it isn't worth the effort to try to offer V8 as a
library then.

As soon as upstream stabilizes one API/ABI I think it would be wise
to make a package providing that as a library.

Not everybody understands it but actually it isn't a library if there
isn't a stable API/ABI.


//Peter

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

FYI - Spidermonkey is in the exact same situation -- upstream develops
with the expectation that projects will embed the code or at best
bundle the lib. They also completely break API with every major
version bump (ie, every 6 weeks). Fortunately they accepted patches
that support installing multiple versions concurrently, and so I've
started slotting it in the tree.

IMO, just like spidermonkey, yes we should still try and keep libs as
system-installed rather than bundling. Just because upstream doesn't
think it's the "right" idea and doesn't support it, doesn't mean we
shouldn't continue to push for this paradigm. That said, I don't know
anything about v8 and if it would be feasible to slot it, and
ultimately, it's going to be up to the dev's maintaining both v8 and
its rdeps, since they're the ones that need to do the work...


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iF4EAREIAAYFAlI/iikACgkQ2ugaI38ACPAQcQD+PicDLQX4e2TsZv5wuAKlVKGW
rjNhGjeE4Eet/So9xqQBAJzDUp5AeiZqmRpzCxzQoi5OOorYfRnTZMDU9elgcDVP
=CfAi
-----END PGP SIGNATURE-----

[Dirkjan Ochtman]

On Mon, Sep 23, 2013 at 2:24 AM, Ian Stakenvicius <a...@gentoo.org> wrote:
> FYI - Spidermonkey is in the exact same situation -- upstream develops
> with the expectation that projects will embed the code or at best
> bundle the lib. They also completely break API with every major
> version bump (ie, every 6 weeks). Fortunately they accepted patches
> that support installing multiple versions concurrently, and so I've
> started slotting it in the tree.

On the other hand, I'm assuming non-core Mozilla projects and external
projects rely only on ESR releases of Spidermonkey, such that the API
only changes every 30 weeks or so?

Cheers,

Dirkjan

[Michał Górny]

Dnia 2013-09-22, o godz. 17:17:53
""Paweł Hajdan, Jr."" <phajd...@gentoo.org> napisał(a):

> I'd like maintainers of all packages depending on dev-lang/v8 to make
> their packages use bundled v8 copy instead (I can file bugs for that,
> let's discuss here whether it should be done).
>
> For now V8 upstream gives no guarantees about API/ABI stability and
> actually breaks it very often
> (<http://upstream-tracker.org/versions/v8.html>). Having a shared
> library so closely tied to packages (which results in frustrating
> blockers, since v8 is updated often and chromium is synchronized with
> that) is not really much different from everyone bundling the library.
> I'd like that to improve, but for now it's time for a pragmatic solution
> to this IMHO.

If this trend continues, I think we should work on some technical way
of tracking bundled libraries, e.g. for security issues. Like ebuilds
listing the corresponding Gentoo packages, like:

QA_BUNDLES="dev-foo/bar-1.2.3"

--
Best regards,
Michał Górny

["Paweł Hajdan, Jr."]

On 9/22/13 5:24 PM, Peter Stuge wrote:
> "Paweł Hajdan, Jr." wrote:
>> "compiling with versions of v8 other than what is included is not
>> currently supported."
> ..
>> For now V8 upstream gives no guarantees about API/ABI stability and
>> actually breaks it very often
>
> I agree that it isn't worth the effort to try to offer V8 as a
> library then.

Perfect.

> As soon as upstream stabilizes one API/ABI I think it would be wise
> to make a package providing that as a library.

Yes, and I'll be trying to move upstream in that direction.

> Not everybody understands it but actually it isn't a library if there
> isn't a stable API/ABI.

Thanks for mentioning that. Totally agreed. At least applicable to a
shared library.

Paweł

[Peter Volkov]

В Вт, 24/09/2013 в 11:46 -0700, "Paweł Hajdan, Jr." пишет:

> On 9/22/13 5:24 PM, Peter Stuge wrote:
> > "Paweł Hajdan, Jr." wrote:
> >> "compiling with versions of v8 other than what is included is not
> >> currently supported."
> > ..
> >> For now V8 upstream gives no guarantees about API/ABI stability and
> >> actually breaks it very often
> >
> > I agree that it isn't worth the effort to try to offer V8 as a
> > library then.
>
> Perfect.

But could you comment on how hard it'll be to slot v8 shared library?
Keeping libraries bundled is a security nightmare.

--
Peter.

[Diego Elio Pettenò]

On Tue, Sep 24, 2013 at 7:46 PM, "Paweł Hajdan, Jr." <phajd...@gentoo.org> wrote:

Perfect.

Seriously? "Perfect" because one person agrees with you?

Sigh.

Diego Elio Pettenò — Flameeyes
flam...@flameeyes.eu — http://blog.flameeyes.eu/

["Paweł Hajdan, Jr."]

On 9/25/13 1:16 AM, Peter Volkov wrote:
> But could you comment on how hard it'll be to slot v8 shared library?
> Keeping libraries bundled is a security nightmare.

Slotting v8 would be hugely impractical and rather a misuse of SLOTs.

Slotting makes sense when there are at most 2-3 major versions, each
with its own release series, that are incompatible, but packages can
depend on one or another series. Thing gtk2 vs. gtk3 for example, or
maybe some Java libraries.

With v8 API and ABI breaks can (and we've seen that) be introduced even
between patch version increments like a.b.c.x vs. a.b.c.y. Trying to
slot that would be totally equivalent to bundling at the cost of
increased complexity (more custom changes compared to upstream - also
headers would need to be handled somehow, currently we don't have a good
way for that, especially one that would be consistent across distros).

Finally, note that v8 upstream only supports the latest stable v8.
Slotting would require us to keep old, _known_ to be vulnerable versions
of v8 in the tree. Backporting of patches isn't always
possible/feasible, and even then for complex and fast moving software
it's not guaranteed to fix all the issues (some security bugs might have
been fixed in more recent versions without realizing security implications).

At least with bundling upstream _may_ track v8 security vulnerabilities
and do something with their copy. With slotting we'd have _known_
vulnerable v8 versions right there in the tree. That'd be a security
nightmare.

Please let me know if you have more questions. At this moment I'm
confident slotting does not address the problem. More deeper, upstream
changes should be made, and I'll be working on that but it's going to
take time.

Paweł

["Paweł Hajdan, Jr."]

On 9/25/13 2:44 AM, Diego Elio Pettenò wrote:
> On Tue, Sep 24, 2013 at 7:46 PM, "Paweł Hajdan, Jr."
> <phajd...@gentoo.org>wrote:
>> Perfect.
>
> Seriously? "Perfect" because one person agrees with you?
>
> Sigh.

Look at the API breaks and talk to v8 upstream - if you have a better
solution to actual bugs that people report against Gentoo, I'm all for it.

Note that I've spent and keep spending time on unbundling what's
possible from chromium. I'm not saying bundled is good or fine, but in
the current situation of v8 I honestly think trying to ship a shared
library offers us *no* advantages and actually creates problems, mainly
because v8 was not designed to be used as a shared library and breaks
API/ABI even before patch version bumps like a.b.c.x -> a.b.c.y.

Paweł

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

It would at minimum make sense to have a chromium-bundled v8 and a
system v8, if you're not doing that already. Mozilla's do that now;
they won't work with a shared libmozjs (indeed, they embed it
statically into libXul, which is also internal and package-specific).

However, if it's possible to keep the rest of the tree using one
system package of v8 (or very small subset), and just maintain
that(those) via security backports, would that be viable?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iF4EAREIAAYFAlJDCPYACgkQ2ugaI38ACPCbEgD/ZLCT9XFwk2Ara+G0gRQTas7P
Wp78He716eSWD9nuaA8BAJlvk7SgBgCpYMORMYhsC1UlhWRLUNYDBf6NlUPDw/3x
=hKKg
-----END PGP SIGNATURE-----

["Paweł Hajdan, Jr."]

On 9/25/13 9:01 AM, Ian Stakenvicius wrote:
> However, if it's possible to keep the rest of the tree using one
> system package of v8 (or very small subset), and just maintain
> that(those) via security backports, would that be viable?

In the current state of v8, no.

Latest security-supported v8 is defined as one used by stable chromium.

Security backports are not supported by upstream, and are not always
even possible with a fast-changing codebase.

A good test for this type of questions is look at some of the bugs below:

https://bugs.gentoo.org/show_bug.cgi?id=417879
https://bugs.gentoo.org/show_bug.cgi?id=420995
https://bugs.gentoo.org/show_bug.cgi?id=471582
https://bugs.gentoo.org/show_bug.cgi?id=477300
https://bugs.gentoo.org/show_bug.cgi?id=484786

and try to post fixes for them. If you or anyone else can do that, I'm
happy to take them and change my opinion (note that I'd apply some
quality standards to the patches, not just look whether they happen to
work for now).

I actually really hope to improve this in the long term (as said
before), and we can definitely revisit this in the future. For now I'd
like to address real problems that affect users.

Paweł

[Chris Stankevitz]

On Sun, Sep 22, 2013 at 5:17 PM, "Paweł Hajdan, Jr."
<phajd...@gentoo.org> wrote:
> I'd like to get your feedback and opinion about removing shared v8
> library package from Gentoo.

The three "inside the box" options require hope:

1. Use share lib. Hope upstream package devs code to whichever V8 API
is used by Gentoo.

2. Bundle. When security problems are fixed, hope upstream package
devs update to the API used in the latest V8.

3. Use slots. Hope V8 security problems are "back ported".

When packages use V8 they put security conscious people in an awkward
"hope" position. It would be nice if packages recognized this and
added switches to disable V8. Then we could use option 1 or 2 and
fail ("disable v8 use flag") when upstream doesn't stay on top of
things.

An "outside the box" option might be to bundle... but somewhere tag
insecure versions of V8. Packages that only work with insecure
versions of V8 require the user to assert an "insecure" use flag or
keyword.

Chris

[Richard Yao]

One idea is to make a new bundled-lib category for virtual packages and
then make virtual package for each library that is bundled.

["Paweł Hajdan, Jr."]

On 10/6/13 12:05 AM, Chris Stankevitz wrote:
> On Sun, Sep 22, 2013 at 5:17 PM, "Paweł Hajdan, Jr."
> <phajd...@gentoo.org> wrote:
>> I'd like to get your feedback and opinion about removing shared v8
>> library package from Gentoo.
>
> The three "inside the box" options require hope:
>
> 1. Use share lib. Hope upstream package devs code to whichever V8 API
> is used by Gentoo.

This is not happening, and there is a good history and evidence of it.
Upstream package devs code to the V8 API they bundle.

Even then, V8 API changes every 6 weeks. It's pretty short time for most
projects to adapt. And it's not like they only change 1-2 things,
sometimes fundamental parts of the API are almost rewritten.

For an example read
<https://groups.google.com/d/msg/v8-users/MUq5WrC2kcE/Z3LyOmELzD0J>.

Note that I'm working with upstream and it seems to slowly make some
improvements, e.g.
<https://groups.google.com/d/msg/v8-users/jq8k9s4xEu8/N-es0or3uz4J>.

> 2. Bundle. When security problems are fixed, hope upstream package
> devs update to the API used in the latest V8.

I think this is where we're at. Actually it's more tricky since I know
e.g. node.js developers sometimes say the security holes don't apply to
them and don't update. They may be right, but upstream V8 says only
latest stable V8 is security supported, which makes sense to me.

> 3. Use slots. Hope V8 security problems are "back ported".

How is that different from bundling? When an old version of V8 has known
vulnerabilities it has to be removed from the tree.

Feel free to "try" to backport, it's just not that easy with project
moving as fast as V8. You'd pretty much have to have the same
understanding of the code that V8 upstream developers have, and at that
point you could probably help solve the API/ABI stability problems in a
more direct way.

Paweł