How does Go can compile so quickly?

[Archos]

Ago some time, when I installed Rust, I remember that it took a long time. I've measured the time now in linux/amd64.

Rust took 50 minutes in compiling a few commands and libraries:

    install: /usr/local/bin/rustc
    install: /usr/local/bin/cargo
    install: /usr/local/bin/rustdoc
    install: /usr/local/lib/librustrt.so
    install_lib: /usr/local/lib/libcore-*.so
    install_lib: /usr/local/lib/libstd-*.so
    install_lib: /usr/local/lib/librustc-*.so
    install_lib: /usr/local/lib/libsyntax-*.so
    install: /usr/local/lib/librustllvm.so
    install: /usr/local/share/man/man1/rustc.1
    install: /usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib/librustrt.so
    install_lib: /usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib/libcore-*.so
    install_lib: /usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib/libstd-*.so
    install_lib: /usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib/librustc-*.so
    install_lib: /usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib/libsyntax-*.so
    install: /usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib/libmorestack.a

In change, Go took only 2 minutes in building the compiler, the C/Go bootstrap tool, all packages and commands, testing packages and another tests.

Robert Griesemer, one of designers of Go, comes from Wirth's school of compilers. So, is Go using the compiler technique that Niklaus Wirth describes in this book[1]?

[1]: http://www.ethoberon.ethz.ch/WirthPubl/CBEAll.pdf

[Rémy Oudompheng]

On 2012/10/13 Archos <raul...@sent.com> wrote:
> In change, Go took only 2 minutes in building the compiler, the C/Go
> bootstrap tool, all packages and commands, testing packages and another
> tests.
>
> Robert Griesemer, one of designers of Go, comes from Wirth's school of
> compilers. So, is Go using the compiler technique that Niklaus Wirth
> describes in this book[1]?

The Go compiler is made of a small number of files, and it is not
written in C++. That alone can justify why it is quick to compile.
It just does what ordinary compilers do: parse source code and then
translate that to a sequence of CPU instructions. It does that in a
quite straightforward way. There is no reason for that to be long, and
I don't think it's a matter of style or technique.
The question is rather: why are other compilers slow?, which has
various possible answers.

Rémy.

[Job van der Zwan]

It builds its own bootstrap compiler first (that is what the "C bootstrap tool" is, isn't it?), then compiles the Go toolchain with that, right? Might that also have something to do with it?

[Rémy Oudompheng]

On 2012/10/13 Job van der Zwan <j.l.van...@gmail.com> wrote:
> It builds its own bootstrap compiler first (that is what the "C bootstrap
> tool" is, isn't it?), then compiles the Go toolchain with that, right? Might
> that also have something to do with it?

The bootstrap tool "dist" is a small C program that replaces a shell
script that used to build the toolchain. It doesn't compile anything
it self.
The toolchain is built with GCC. There is a C compiler in the built-in
distribution, but it's used to compile the runtime library, not the
toolchain.

Rémy.

[Matt Joiner]

http://stackoverflow.com/questions/2976630/why-does-go-compile-quickly

There are a few good blog posts that I can't seem to find offhand, that discuss how Go's compilation model allows it to check a few 100 files per module, rather than the 10000-100000s checked by a typical C++ source file. I'd start with looking at these algorithmic improvements before looking at parsimony in the tool chain.

Rémy.

--

[Remi Gillig]

I think the part you are talking about is the last section of this page : http://doc.cat-v.org/bell_labs/pikestyle

For the rest, you can have a lot insight on the Go compilers by reading the Plan9 C compilers papers :

http://doc.cat-v.org/plan_9/4th_edition/papers/compiler

http://doc.cat-v.org/bell_labs/new_c_compilers/

[Jim Teeuwen]

The most important difference with the likes of GCC, is that Go has a sane and very simple dependency model. The compiler needs to compile each referenced package once, and only once. Whereas GCC and friends will happily parse the same header hundreds, even thousands of times over for no reason except stupid design. This creates an exponential slowdown and leads to builds taking hours.

[ron minnich]

The Plan 9 compiler and linker are underappreciated gems. We ported
the amd64 compiler/linker to be native on unix-like systems. That took
almost no time at all. We can build all of plan 9, all libraries, all
commands, all kernels, in under 2 minutes.

They reward study.

ron

[Daniel Jo]

Curious, but is there any fundamental reason why the popular C/C++ compilers (like GCC) operate the way that they do? Surely it must be possible for them to be altered to compile programs in a much saner way without violating the language specs.

-Daniel

--

[Sebastien Binet]

On Sun, Oct 14, 2012 at 7:02 PM, Daniel Jo <ost...@gmail.com> wrote:
> Curious, but is there any fundamental reason why the popular C/C++ compilers
> (like GCC) operate the way that they do? Surely it must be possible for them
> to be altered to compile programs in a much saner way without violating the
> language specs.

IIRC, the fundamental issue is the preprocessor.
the content of a header file might change depending on some
preprocessor macro's evaluation.

so the compiler has to redo everything again.
well, actually, not everything, and that's what the clang guys are
trying to achieve with the module system (which is slightly different
than the module system c++1y might have. one day.)

see:
https://github.com/boostcon/cppnow_presentations_2012/blob/master/wed/modules_aspen2012.pdf

-s

[Ziad Hatahet]

There are many things going on behind the scenes when you build the Rust compiler. As far as I know, it is self hosting (unlike Go), and it ships with the LLVM. There is a discussion about this on the Rust mailing list here: https://mail.mozilla.org/pipermail/rust-dev/2012-October/002456.html

--
Ziad

--
 
 

[Ian Lance Taylor]

On Sun, Oct 14, 2012 at 4:29 AM, Jim Teeuwen <jimte...@gmail.com> wrote:
> The most important difference with the likes of GCC, is that Go has a sane
> and very simple dependency model. The compiler needs to compile each
> referenced package once, and only once. Whereas GCC and friends will happily
> parse the same header hundreds, even thousands of times over for no reason
> except stupid design. This creates an exponential slowdown and leads to
> builds taking hours.

That's a bit unfair. The problem is not with GCC; it is with the
language that GCC is compiling. It is quite difficult to avoid
recompiling C++ header files multiple times. The C++ language
committee is looking into adding modules to the language to make this
easier. Without modules, C++ code changes behaviour based on the set
of names in scope at the time that the code is compiled. This can be
addressed in a limited manner by precompiling headers, which GCC does
in fact support, but the approach is not usable for every project.

Ian

[Paulo Pinto]

Agreed.

Any language with modules should compile really fast, when compared
with the 70's
toolchains of C and C++.

Anyone old enough can surely remember how Turbo Pascal used to compile
blazingly
fast in the 80's. Embarcadero's web site (actual owners) state 34,000
lines/minute for
Turbo Pascal 5.5 (1989).

http://edn.embarcadero.com/article/20803

Additionally the linker wasn't a clunky 70's compatible UNIX linker,
rather integrated with the compiler.

This is common to almost any modern language with modules system.

The only way C++, thus GCC, can improve, is by having modules. But
this is going to take awhile, because
they are planned for C++17 (assuming this date is met).

--
Paulo

On Oct 15, 5:39 am, Ian Lance Taylor <i...@google.com> wrote:

[Rob Pike]

Go doesn't compile particularly quickly. It's just that it isn't slow.

Standards have fallen.

-rob

[Paulo Pinto]

On my personal experience, the problem is usually one of scale.

Many module based languages compile relatively fast, specially when
compared
to C and C++ toolchains.

But when you start doing enterprise level software, with thousands of
libraries,
for each layer devised by the architect of the month, the build times
sky rocket,
regardless of fast a given compiler is.

Is there any data you can share regarding Go on this subject?

--
Paulo

[roger peppe]

I think the speed of the plan 9 C compiler was also helped
by the way that the include files were structured in that
environment - each library had exactly one include file,
and no include file included another include file.

Obviously this solution is not possible while still
maintaining exact ANSI-C library compatibility.

It did have a disadvantage too (the dependencies
of a library were exposed).

> --
>
>

[Ian Lance Taylor]

On Sun, Oct 14, 2012 at 11:38 PM, Paulo Pinto <paulo....@gmail.com> wrote:
> On my personal experience, the problem is usually one of scale.
>
> Many module based languages compile relatively fast, specially when
> compared
> to C and C++ toolchains.
>
> But when you start doing enterprise level software, with thousands of
> libraries,
> for each layer devised by the architect of the month, the build times
> sky rocket,
> regardless of fast a given compiler is.
>
> Is there any data you can share regarding Go on this subject?

At enterprise scale, the most important time for developers is
incremental compilation time. Go's advantage here is that the build
system only needs to recompile a package if the exported data of an
imported package has changed. There are many changes you can make to
a package P that do not change P's export data, and thus do not
require the recompilation of packages that import P. Thus many
typical changes to a package require only recompiling that specific
package, and relinking the executable. This general approach should
work for any module-based language, of course.

I don't have any specific data to share but in practice at Google the
overhead of the build system outweighs the compilation time for large
Go programs. (The build system is not written in Go.)

Ian

[Kevin Gillette]

I agree that, especially since building the _compilers_ is usually regarded as a one-time process, time required to bootstrap the system isn't particularly important or relevant. What would be (barely) relevant is the time it takes to build equivalent programs, including concurrent programs.

On Monday, October 15, 2012 10:41:22 AM UTC-6, Jens Alfke wrote:

On Saturday, October 13, 2012 1:05:24 PM UTC-7, Archos wrote:

Ago some time, when I installed Rust, I remember that it took a long time. I've measured the time now in linux/amd64.

Rust took 50 minutes in compiling a few commands and libraries:

Apologies for jumping in late (and slightly off-topic), but it seems no one's answered the specific question of why building Rust takes so long compared to Go. There's a thread on rust-dev that was created in response to this one, in which the question was answered in detail:

https://mail.mozilla.org/pipermail/rust-dev/2012-October/002462.html

The short answer is that the Rust makefile first builds LLVM, and LLVM is a huge C++ project that takes a _long_ time to compile. I checked out and built Rust last night, out of curiosity, and at least 95% of the build time was LLVM and Clang — the time spent on Rust itself was just a few minutes at the beginning and end, and that includes building three copies of the compiler targeted for different platforms.

I think the takeaway is that building the compiler for language X is not a good benchmark of compilation speed of X :)

--Jens

[Archos]

El lunes, 15 de octubre de 2012 17:41:22 UTC+1, Jens Alfke escribió:

On Saturday, October 13, 2012 1:05:24 PM UTC-7, Archos wrote:

Ago some time, when I installed Rust, I remember that it took a long time. I've measured the time now in linux/amd64.

Rust took 50 minutes in compiling a few commands and libraries:

Apologies for jumping in late (and slightly off-topic), but it seems no one's answered the specific question of why building Rust takes so long compared to Go. There's a thread on rust-dev that was created in response to this one, in which the question was answered in detail:

https://mail.mozilla.org/pipermail/rust-dev/2012-October/002462.html

The short answer is that the Rust makefile first builds LLVM, and LLVM is a huge C++ project that takes a _long_ time to compile. I checked out and built Rust last night, out of curiosity, and at least 95% of the build time was LLVM and Clang — the time spent on Rust itself was just a few minutes at the beginning and end, and that includes building three copies of the compiler targeted for different platforms.

I think the takeaway is that building the compiler for language X is not a good benchmark of compilation speed of X :)

Anyway, the build time can be speed up, and now Graydon Hoare has marked it as a priority for the next version:

https://github.com/mozilla/rust/wiki/Note-0.5-priorities

[Larry Clapp]

On Monday, October 15, 2012 12:41:22 PM UTC-4, Jens Alfke wrote:

On Saturday, October 13, 2012 1:05:24 PM UTC-7, Archos wrote:

Ago some time, when I installed Rust, I remember that it took a long time. I've measured the time now in linux/amd64.

Rust took 50 minutes in compiling a few commands and libraries:

Apologies for jumping in late (and slightly off-topic), but it seems no one's answered the specific question of why building Rust takes so long compared to Go. There's a thread on rust-dev that was created in response to this one, in which the question was answered in detail:

https://mail.mozilla.org/pipermail/rust-dev/2012-October/002462.html

That was a neat read, thanks for posting it.

There was a neat hat-tip to Rob in there, too: "Secondly, Rob Pike has probably implemented more compilers than all of the Rust team combined, so there's the issue of the Rust compiler simply not being written as cleverly as the Go compiler."

-- L

[Rob Pike]

Thanks for the vote of confidence but Ken Thompson wrote the Go
compilers, not me. I've written a number of interpreters but only a
couple of actual compilers and none for a general-purpose language.

-rob