Update on new Go linker?

[Ben Hoyt]

Is there a follow-up doc to http://golang.org/s/better-linker that describes what's been done, the progress that's been made, whether a new object format is in fact being used, etc? I see a lot of "cmd/link: ..." commits, and it sounds like the new linker will definitely be the default for 1.15, but I'm wondering if there's a summary doc available?

Also, I'm curious how the dev.link branch development works? Is this the "new linker" branch, and commits on this branch or merged into master when they're ready? Given that the new cmd/link is already merged into master, why is development on a branch still necessary?

-Ben

[Than McIntosh]

Hi,

[Austin and others please chime in if you have more to add or need to correct anything here]

The not-yet-written 1.15 release notes will include a section on new linker, stay tuned for that.

Branch development: yes, "dev.link" is the branch being used for the linker modernization work. This branch gets merged into master at various points when things are stable.  This sort of feature branch is used periodically by the team when folks are working on a development effort that spans multiple Go release cycles. Doing things this way helps to keep the master branch stable.

> Given that the new cmd/link is already merged into master, why is development on a branch still necessary?

I think the short answer is that although 1.15 will use a version of the new linker, we're not finished yet with all the changes that we want to make. The 1.15 linker will have a lot of improvements (reduced memory usage, faster linking, better internal concurrency, new object file format, indexed symbol representation, etc), but there are still a number of things outlined in http://golang.org/s/better-linker that haven't been completed yet (for example, use of DWARF 5) and will be pushed out to 1.16.

Cheers, Than

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/4d7a7b2f-ca48-4950-ae28-3773c6685deb%40googlegroups.com.

[Austin Clements]

Thanks Than!

The quick summary of what's been done is that we (whereby "we" I really mean Than, Jeremy, and Cherry! All I did was ask a lot of annoying questions) converted the linker over to a new object format that uses pre-resolved symbol indexes as described in better-linker and also made many other improvements along the way like parallelizing key phases and removing a lot of unnecessary I/O synchronization. In order to best build on all of the past work on the linker, we did this conversion as a "wavefront", with a phase that converted from the new representation to the old representation that we pushed further and further back in the linker. We're not done yet: that conversion phase is still there, though exactly when it happens and what it does depends on the platform. For amd64 ELF platforms, it's quite late and does relatively little. For other platforms, it's not quite as far back and does more, so the wins aren't as big yet. Either way, there's more to look forward to for 1.16. :)

To expand on the branch question, we used a branch for a few reasons: mostly, the new linker work is a long-haul multi-person effort; also, for most of its development, the new linker wasn't yet an improvement over the old linker. We started merging it into master just during the past month once it reached the point where it was at least on par with the old linker on all our benchmarks. At that point we kept developing on the branch because we were still regularly breaking things and didn't want to break master for everyone. We would merge to master at points where it was stable (which was still fairly regular). Work is still ongoing and will continue to happen on the branch at this point.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CA%2BUr55FHtwkNfBn1OMDRF5Pqzp1pFXM3eAwLgD2ddSB-1Z%3D8sA%40mail.gmail.com.

[Daniel Martí]

On Mon, May 04, 2020 at 09:18:44 -0400, 'Austin Clements' via golang-dev wrote:
> Work is still ongoing and will continue to happen on the branch at
> this point.

Quick follow-up question - will merges to master continue during the
freeze, given that there's a flag to go back to the old linker? Or will
they pile up until the tree reopens for 1.16?

This is assuming that work on the new linker continues during the
freeze, which might not be a correct assumption. Maybe it pauses
entirely to focus on stability and fixes for master.

[Austin Clements]

Good questions. We are planning to continue work on the linker during the freeze on dev.link—at a bare minimum, there are still a lot of loose ends that we should tie up while they're still fresh—though stabilization will take priority. We are not planning to merge dev.link to master during the freeze, though we will continue merging master into dev.link so it should be easy to land when the tree reopens. The feature flag is really intended to be a heavy-handed emergency revert strategy in case something turns out to be seriously broken. We wouldn't want to break something by continuing development on master during the freeze and then have to use it.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/20200504144720.rr5vgxr7c7pfjryx%40carbon.localhost.

[Ben Hoyt]

These responses were very informative -- thanks!

On Tuesday, May 5, 2020 at 3:09:21 AM UTC+12, Austin Clements wrote:

Good questions. We are planning to continue work on the linker during the freeze on dev.link—at a bare minimum, there are still a lot of loose ends that we should tie up while they're still fresh—though stabilization will take priority. We are not planning to merge dev.link to master during the freeze, though we will continue merging master into dev.link so it should be easy to land when the tree reopens. The feature flag is really intended to be a heavy-handed emergency revert strategy in case something turns out to be seriously broken. We wouldn't want to break something by continuing development on master during the freeze and then have to use it.

On Mon, May 4, 2020 at 10:47 AM Daniel Martí <mvdan@...> wrote:

On Mon, May 04, 2020 at 09:18:44 -0400, 'Austin Clements' via golang-dev wrote:
> Work is still ongoing and will continue to happen on the branch at
> this point.

Quick follow-up question - will merges to master continue during the
freeze, given that there's a flag to go back to the old linker? Or will
they pile up until the tree reopens for 1.16?

This is assuming that work on the new linker continues during the
freeze, which might not be a correct assumption. Maybe it pauses
entirely to focus on stability and fixes for master.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.