llgo is self hosting

[Peter Collingbourne]

Hi all,

As of commit d06327c741c08a1cd3ab50c7f62b8964b01fefc9 the libgo branch of llgo is self hosting. What that means is that it can build itself and all the Go code that it depends on, including the standard library, and the resulting binary can also build itself and all its dependencies to produce an identical binary which passes all the same standard library tests as a gc-built binary. This is a pretty significant milestone for any compiler and to my knowledge llgo is the first Go compiler to accomplish this milestone.

Thanks are in order to Andrew and the other llgo contributors, but this couldn't have been possible without the excellent go.tools libraries mostly developed by Robert Griesemer and Alan Donovan, and the libgo standard library mostly maintained by Ian Lance Taylor which of course is based on the community maintained Go standard library.

Peter

[Andrew Wilkins]

On Tue, May 6, 2014 at 12:33 PM, 'Peter Collingbourne' via llgo-dev <llgo...@googlegroups.com> wrote:

Hi all,

As of commit d06327c741c08a1cd3ab50c7f62b8964b01fefc9 the libgo branch of llgo is self hosting. What that means is that it can build itself and all the Go code that it depends on, including the standard library, and the resulting binary can also build itself and all its dependencies to produce an identical binary which passes all the same standard library tests as a gc-built binary. This is a pretty significant milestone for any compiler and to my knowledge llgo is the first Go compiler to accomplish this milestone.

Thank you very much Peter. I'm ecstatic that llgo is finally in good shape, in no small part due to all of your recent contributions.

While I was away last week I managed to compile libgo using clang and llgo, albeit very roughly hewn; I was eventually able to link a hello world program. I will continue on with this, so that we may implement LTO. It would be nice to have OS X support back again, too.

I have started toying with an escape analysis package built on top of go/ssa. Nothing much to show yet.

Thanks are in order to Andrew and the other llgo contributors, but this couldn't have been possible without the excellent go.tools libraries mostly developed by Robert Griesemer and Alan Donovan, and the libgo standard library mostly maintained by Ian Lance Taylor which of course is based on the community maintained Go standard library.

Absolutely, llgo is happily built on strong foundations. I look forward to gccgo/libgo receiving more love as llgo adoption and contribution increases.

Peter

--
You received this message because you are subscribed to the Google Groups "llgo-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to llgo-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Andrew Wilkins

http://awilkins.id.au

[minux]

On Tue, May 6, 2014 at 12:33 AM, 'Peter Collingbourne' via llgo-dev <llgo...@googlegroups.com> wrote:

As of commit d06327c741c08a1cd3ab50c7f62b8964b01fefc9 the libgo branch of llgo is self hosting. What that means is that it can build itself and all the Go code that it depends on, including the standard library, and the resulting binary can also build itself and all its dependencies to produce an identical binary which passes all the same standard library tests as a gc-built binary. This is a pretty significant milestone for any compiler and to my knowledge llgo is the first Go compiler to accomplish this milestone.

Thanks are in order to Andrew and the other llgo contributors, but this couldn't have been possible without the excellent go.tools libraries mostly developed by Robert Griesemer and Alan Donovan, and the libgo standard library mostly maintained by Ian Lance Taylor which of course is based on the community maintained Go standard library.

Great news! Congratulations!

The llgo project is growing at amazing speed. Thank you all!

So what's the next step?

[Andrew Wilkins]

My priorities are:

    1. get libgo building with clang: first as an archive of object files, then as an LLVM bitcode archive so we get LTO/WPO for optimal performance.

    2. escape analysis

    3. PNaCl (there are some upstream prereqs here, though, such as zero-cost exception support in PNaCl itself).

I'd like to see MacOS support up there too, but I do not have the hardware to work on it.

Cheers,

Andrew

[Richard Musiol]

Congratulations on self hosting! That's a major milestone.

What's the problem with OS X? I might look into it.

Richard

--

[Peter Collingbourne]

As for me:

1. Fix most remaining gc and libgo test suite failures. Some of them are relatively obscure, so they're a lower priority.
2. I also want to hook some of the sanitizers up to llgo so that we can have things like race checking.

Peter

--

[r.ekl...@gmail.com]

Wow! You've all done an amazing job and congratulations on reaching this important milestone.

/u

[Andrew Wilkins]

On Tue, May 6, 2014 at 5:18 PM, Richard Musiol <ma...@richard-musiol.de> wrote:

Congratulations on self hosting! That's a major milestone.

What's the problem with OS X? I might look into it.

AFAIK, neither gccgo nor libgo have support for OS X. I don't know what is missing specifically.

There is an open bug on GCC: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46986

[Andrew Wilkins]

On Tue, May 6, 2014 at 5:18 PM, Richard Musiol <ma...@richard-musiol.de> wrote:

Congratulations on self hosting! That's a major milestone.

What's the problem with OS X? I might look into it.

AFAIK, neither gccgo nor libgo have support for OS X. I don't know what is missing specifically.

There is an open bug on GCC: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46986

[Kelsey Hightower]

Awesome job! Let me know if you still need OS X hardware for testing. I can help get you a Mac Mini or Laptop if that would help. 

 

Cheers,

Andrew

[Andrew Wilkins]

It would, and that's very generous. I'm hesitant to accept as I don't know how much time I'll have for it.

If you have something I (or another contributor?) can log into remotely that'd be helpful.

Cheers,

Andrew

[Atom Symbol]

On Tuesday, May 6, 2014 6:33:06 AM UTC+2, Peter Collingbourne wrote:

Hi all,

As of commit d06327c741c08a1cd3ab50c7f62b8964b01fefc9 the libgo branch of llgo is self hosting. What that means is that it can build itself and all the Go code that it depends on, including the standard library, and the resulting binary can also build itself and all its dependencies to produce an identical binary which passes all the same standard library tests as a gc-built binary. This is a pretty significant milestone for any compiler and to my knowledge llgo is the first Go compiler to accomplish this milestone.

I agree that self hosting is an important milestone. On the other hand it isn't as important as in the distant past where existed no compilers and no high level languages. Nowadays there is an abundance of choices one can make when choosing an implementation language for a new compiler.

In relation to this topic, the Tulgo compiler takes a very different approach compared to llgo: the Tulgo compiler itself is implemented in a mature programming language (Java) which has tools around it (most importantly: IDE, profiler, debugger). The compiler runs in the cloud and the end-user interacts with it through a lean command-line front-end implemented in Go (https://github.com/tul-project/tul). This potentially enables for the compiler implementation to internally switch the implementation language without the user noticing anything (not that I plan to do so). Also, the front-end has a much lower update frequency compared to the compiler itself. The networked architecture enables all users to immediately benefit from updates to the compiler without the need to recompile the front-end. Note: Tulgo is a work in progress and can currently compile the raytrace-simple.go benchmark only.

[Andrew Wilkins]

On Sun, May 18, 2014 at 4:35 PM, Atom Symbol <0xe2.0x...@gmail.com> wrote:

On Tuesday, May 6, 2014 6:33:06 AM UTC+2, Peter Collingbourne wrote:

Hi all,

As of commit d06327c741c08a1cd3ab50c7f62b8964b01fefc9 the libgo branch of llgo is self hosting. What that means is that it can build itself and all the Go code that it depends on, including the standard library, and the resulting binary can also build itself and all its dependencies to produce an identical binary which passes all the same standard library tests as a gc-built binary. This is a pretty significant milestone for any compiler and to my knowledge llgo is the first Go compiler to accomplish this milestone.

I agree that self hosting is an important milestone. On the other hand it isn't as important as in the distant past where existed no compilers and no high level languages. Nowadays there is an abundance of choices one can make when choosing an implementation language for a new compiler.

Certainly you don't need to write the compiler in its target language, as gccgo has also shown. Still, there are benefits that are timeless: improvements to the compiler output improve the compiler itself (increasing motivation to improve the compiler); and most important to me, it provides a very valuable proof of consistency (Peter's bootstrapping script performs a binary comparison of the executables built at stages two and three).

 

In relation to this topic, the Tulgo compiler takes a very different approach compared to llgo: the Tulgo compiler itself is implemented in a mature programming language (Java) which has tools around it (most importantly: IDE, profiler, debugger). The compiler runs in the cloud and the end-user interacts with it through a lean command-line front-end implemented in Go (https://github.com/tul-project/tul). This potentially enables for the compiler implementation to internally switch the implementation language without the user noticing anything (not that I plan to do so). Also, the front-end has a much lower update frequency compared to the compiler itself. The networked architecture enables all users to immediately benefit from updates to the compiler without the need to recompile the front-end. Note: Tulgo is a work in progress and can currently compile the raytrace-simple.go benchmark only.

It's an interesting approach, and certainly with merit; there are pros and cons here too, though. First, users are at the mercy of the network; second, you must establish yourself as a trusted party. I think the latter is a much more difficult issue when dealing with proprietary software. OTOH, in addition to the benefits you've noted, you may be able to do some more interesting optimisations given a large enough corpus of code.

Seems llgo is not producing very efficient code for raytrace-simple.go at the moment. I haven't had any time for llgo lately, but I'll look into this at some point...

[Atom Symbol]

On Monday, May 19, 2014 3:15:29 AM UTC+2, Andrew Wilkins wrote:

On Sun, May 18, 2014 at 4:35 PM, Atom Symbol <0xe2.0x...@gmail.com> wrote:

On Tuesday, May 6, 2014 6:33:06 AM UTC+2, Peter Collingbourne wrote:

Hi all,

As of commit d06327c741c08a1cd3ab50c7f62b8964b01fefc9 the libgo branch of llgo is self hosting. What that means is that it can build itself and all the Go code that it depends on, including the standard library, and the resulting binary can also build itself and all its dependencies to produce an identical binary which passes all the same standard library tests as a gc-built binary. This is a pretty significant milestone for any compiler and to my knowledge llgo is the first Go compiler to accomplish this milestone.

I agree that self hosting is an important milestone. On the other hand it isn't as important as in the distant past where existed no compilers and no high level languages. Nowadays there is an abundance of choices one can make when choosing an implementation language for a new compiler.

Certainly you don't need to write the compiler in its target language, as gccgo has also shown. Still, there are benefits that are timeless: improvements to the compiler output improve the compiler itself (increasing motivation to improve the compiler);

I agree that letting the compiler compile itself is nice. Since Tulgo is written in Java, that language would be Java without GUI.

and most important to me, it provides a very valuable proof of consistency (Peter's bootstrapping script performs a binary comparison of the executables built at stages two and three).

 

In relation to this topic, the Tulgo compiler takes a very different approach compared to llgo: the Tulgo compiler itself is implemented in a mature programming language (Java) which has tools around it (most importantly: IDE, profiler, debugger). The compiler runs in the cloud and the end-user interacts with it through a lean command-line front-end implemented in Go (https://github.com/tul-project/tul). This potentially enables for the compiler implementation to internally switch the implementation language without the user noticing anything (not that I plan to do so). Also, the front-end has a much lower update frequency compared to the compiler itself. The networked architecture enables all users to immediately benefit from updates to the compiler without the need to recompile the front-end. Note: Tulgo is a work in progress and can currently compile the raytrace-simple.go benchmark only.

It's an interesting approach, and certainly with merit; there are pros and cons here too, though. First, users are at the mercy of the network; second, you must establish yourself as a trusted party. I think the latter is a much more difficult issue when dealing with proprietary software. OTOH, in addition to the benefits you've noted, you may be able to do some more interesting optimisations given a large enough corpus of code.

- Yes, establishing trust might turn out to be the biggest non-technical problem. I hope that Tulgo's feature set will be enough for people to overcome the non-technical limitations.

- Another benefit that came to my mind recently is that of offline analysis: The compiler service compiles the Go source code and returns the binary object to the user. At the same time the compiler starts an offline analysis of the code that can run for an extended period of time. When finished with the analysis the compiler service emails the user with a notice saying that an optimized version of the binary object is available for download. I suppose there will be a -foffline-analysis compiler switch to start this process. I plan for this feature to appear in Tulgo in year 2015+. There is also the topic of offline analysis with a human performing some parts of the analysis.

Seems llgo is not producing very efficient code for raytrace-simple.go at the moment. I haven't had any time for llgo lately, but I'll look into this at some point...

raytrace-simple.go contains expressions that are taking the address of a variable and pass that address to a method that is invoked via an indirect CALL instruction. At compile-time, whole program analysis is required to determine that the variable actually does not escape to heap. Tulgo is able to determine it at runtime (not at compile-time) thanks to reference counting. Without reference counting the allocated memory can be freed later by a garbage collection.