-optimise and binary compatibility

[Mark Harrah]

It looks like -optimise is designed so that it doesn't affect binary compatibility. Going off of Miguel's inliner walkthrough[1],

* the inliner only inlines methods from an external library that are annotated with @inline
* the inliner only inlines methods that call publicly accessible methods

This might require that the implementations of @inline methods don't change, but otherwise the inliner doesn't cause problems for binary compatibility, right?

-Mark

[1] http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/2011Q4/Inliner.pdf

[Vlad Ureche]

Hi Mark,

Short answer: Currently, it won't affect binary compatibility. It might affect it in the 2.10 release.

Long answer: Whenever the inlined code needs access to a private/protected member, the inliner makes it public so it can be accessed from anywhere. But since there's no contract saying the member must be public, recompiling parts of the program or recompiling files in a different order may cause the members not to be made public and thus impact binary compatibility, especially since the inlining algorithm is sensitive to ordering.

For now, member public-ization is off because of this bug, but Miguel has been looking at ways to restore it. I think he can give you more up to date information on what's going to happen in 2.10.

HTH,
Vlad

[martin odersky]

On Sat, May 12, 2012 at 5:37 PM, Mark Harrah <dmha...@gmail.com> wrote:

It looks like -optimise is designed so that it doesn't affect binary compatibility.  Going off of Miguel's inliner walkthrough[1],

* the inliner only inlines methods from an external library that are annotated with @inline
* the inliner only inlines methods that call publicly accessible methods

This might require that the implementations of @inline methods don't change, but otherwise the inliner doesn't cause problems for binary compatibility, right?

Unfortunately it does cause problems. The inliner might have to give certain private fields and methods public access, so that the code in question can be inlined (if one does not do that, almost nothing can be inlined). If you then compile that code again without optimization, a client that contains the inlined methods and that's not compiled with them will break.

 -- Martin

[Mark Harrah]

Ah, sorry. I'm not worried about compiling with -optimise and then without it and expecting binary compatibility. I agree that is unlikely and probably unreasonable. I'd like to turn on -optimise and know that I can make the same compatibility guarantees as if I weren't using it.

That is, I (mostly) know how to preserve binary compatibility without -optimise. If I'm using -optimise, what else do I have to worry about?

-Mark

> -- Martin

[Mark Harrah]

On Sat, 12 May 2012 19:15:45 +0200
Vlad Ureche <vlad....@gmail.com> wrote:

> On Sat, May 12, 2012 at 5:37 PM, Mark Harrah <dmha...@gmail.com> wrote:
>
> > It looks like -optimise is designed so that it doesn't affect binary
> > compatibility. Going off of Miguel's inliner walkthrough[1],
> >
> > * the inliner only inlines methods from an external library that are
> > annotated with @inline
> > * the inliner only inlines methods that call publicly accessible methods
> >
> > This might require that the implementations of @inline methods don't
> > change, but otherwise the inliner doesn't cause problems for binary
> > compatibility, right?
> >
> > -Mark
> >
> > [1]
> > http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/2011Q4/Inliner.pdf
> >
>
> Hi Mark,
>
> Short answer: Currently, it won't affect binary compatibility. It might
> affect it in the 2.10 release.
>
> Long answer: Whenever the inlined code needs access to a private/protected
> member, the inliner makes it public so it can be accessed from anywhere.
> But since there's no contract saying the member must be public, recompiling
> parts of the program or recompiling files in a different order may cause
> the members not to be made public and thus impact binary compatibility,
> especially since the inlining algorithm is sensitive to ordering.
>
> For now, member public-ization is off because of this

> bug<https://groups.google.com/forum/?hl=en&fromgroups#%21topic/scala-internals/yGmOkBn9Gmk>,

> but Miguel has been looking at ways to restore it. I think he can give you
> more up to date information on what's going to happen in 2.10.

Thanks for the response. I forgot about that thread. Indeed, I am looking for what Paul talks about: some way to tell the optimizer that I care about binary compatibility. If that means I have to manually adjust method accessibility, perhaps I want that. I'm not really sure what I need, since I've not really used -optimise in practice yet (but would like to, hence the question) and I'm not too familiar with the implications on binary compatibility (since I haven't done that either, but would like to).

-Mark

> HTH,
> Vlad

[Vlad Ureche]

On Sat, May 12, 2012 at 7:38 PM, Mark Harrah <dmha...@gmail.com> wrote:

Thanks for the response.  I forgot about that thread.  Indeed, I am looking for what Paul talks about: some way to tell the optimizer that I care about binary compatibility.  If that means I have to manually adjust method accessibility, perhaps I want that.  I'm not really sure what I need, since I've not really used -optimise in practice yet (but would like to, hence the question) and I'm not too familiar with the implications on binary compatibility (since I haven't done that either, but would like to).

That's tricky. It would be the equivalent of tracking all the code we've already inlined and making sure that the interface it expects is respected. I don't even know if that's possible: once you defined a field in your class and accessed it from an @inline method, it must stay there forever (even if you remove it from the source code, the compiler must add it back to respect the interface).

A weaker guarantee we could provide is that as long as you don't change the code of the @inline method and the members it accesses, the interface stays the same. In that case, it seems we can do it: Adriaan's idea of adding public accessors that the inlined code can use should do the trick. Somebody needs to code it up and check the performance cost of accessors does not overweight the benefits.

Vlad

[Mark Harrah]

I would think that keeping the implementation of inlined methods the same is good practice for binary compatibility anyway. It may not always cause LinkageErrors, but a client would be running different code (the inlined old version) than the library expects it to run (the new version).

I figured that there was a correct way to use -optimise and preserve binary compatibility because the standard library does that. It might be that the standard library is special and commits get reverted if they caused the optimizer to break the library's binary compatibility. I'd just like to know in advance so that I don't enable -optimise and then get stuck.

-Mark

> Vlad

[Miguel Garcia]

Mark,

For now I'm focusing on avoiding IllegalAccessError when a program (compiled against optimized libraries) runs with their non-optimized version (or any combination thereof). Progress looks promising, as discussed in https://issues.scala-lang.org/browse/SI-5442

With that approach in place, I don't see any binary compatibility issues resulting from -optimized or not. But, in case I'm overlooking some use case, please comment on it at the bug report above.

Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

[Mark Harrah]

On Sun, 3 Jun 2012 03:01:36 -0700 (PDT)
Miguel Garcia <miguel...@tuhh.de> wrote:

> Mark,
>
> For now I'm focusing on avoiding IllegalAccessError when a program
> (compiled against optimized libraries) runs with their non-optimized
> version (or any combination thereof). Progress looks promising, as
> discussed in https://issues.scala-lang.org/browse/SI-5442
>
> With that approach in place, I don't see any binary compatibility issues
> resulting from -optimized or not. But, in case I'm overlooking some use
> case, please comment on it at the bug report above.

That seems reasonable to me. Thanks for your work on this.

-Mark