About Golang internal link future development direction and plan

[Wei....@arm.com]

I'm investigating Golang internal link for potential improvement on arm64 port.

But I find that internal link is only enabled for very few cases such as: (!iscgo && buildmode==exe  && GOOS==linux).

Even for amd64 port, internal link doesn't support PIE now (It seems to support PIE before).

So my question is: what's the future development direction and plan of Golang internal link?

More dependence on external linker (such as gcc) or less dependence on external one (even on dependence at last)?

[Brad Fitzpatrick]

The internal linking is not going away. It's faster and means users don't need to have a C compiler installed on their machine.

But nobody's working on it as their main priority. It's been getting gradually cleaned up, though, as it was one of the last autoconverted-from-C programs to get much love. Any changes to make the internal linker supported in more places (where linux/amd64 etc work already) would almost certainly be accepted.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Hudson-Doyle]

On 8 January 2018 at 19:35, <Wei....@arm.com> wrote:

I'm investigating Golang internal link for potential improvement on arm64 port.

But I find that internal link is only enabled for very few cases such as: (!iscgo && buildmode==exe  && GOOS==linux).

What you're seeing here is that the people who worked on the arm64 support mostly came from a Linux background where having a host linker installed doesn't seem like a big deal.

 

Even for amd64 port, internal link doesn't support PIE now (It seems to support PIE before).

So my question is: what's the future development direction and plan of Golang internal link?

More dependence on external linker (such as gcc) or less dependence on external one (even on dependence at last)?

I'm not aware of any real plans here. I've had this silly idea for a while to implement ELF/PE/MachO linkers in Go, convert cmd/link to always operate in what is now external link mode and invoke the bundled host linker if the cgo usage is simple enough. I think this would lead to cleaner code, but it would probably be slower, unfortunately.

Cheers,

mwh

[Wei....@arm.com]

Fast building speed is important for Golang and we can incrementally improve internal link for it.

I just have two basic questions:

1. Why CGO requires external link? Is there any technical issues or considerations to require it or just due to a lot of coding work?

2. PIE was once supported by internal link for linux/amd64 but switch to external link last year. Is any plan to switch it back to internal link for linux/amd64 in the future?

Thanks

Wei Xiao

[Ian Lance Taylor]

On Tue, Jan 9, 2018 at 2:36 AM, <Wei....@arm.com> wrote:
> Fast building speed is important for Golang and we can incrementally improve
> internal link for it.
> I just have two basic questions:
> 1. Why CGO requires external link? Is there any technical issues or
> considerations to require it or just due to a lot of coding work?

It's coding work, and maintenance, and bug fixes for unusual use cases
that will rarely arise. The most complex issue I can think of is that
system linkers have a lot of code just for C++ global
constructors/destructors and C++ exception handling information. If
we don't use external linking, we need to write our own versions of
that code, and keep it up to date.

> 2. PIE was once supported by internal link for linux/amd64 but switch to
> external link last year. Is any plan to switch it back to internal link for
> linux/amd64 in the future?

It's certainly desirable, but I'm not aware of anybody actively working on it.

Ian

> On Tuesday, January 9, 2018 at 3:55:03 AM UTC+8, Michael Hudson-Doyle wrote:
>>
>> On 8 January 2018 at 19:35, <Wei....@arm.com> wrote:
>>>
>>> I'm investigating Golang internal link for potential improvement on arm64
>>> port.
>>> But I find that internal link is only enabled for very few cases such as:
>>> (!iscgo && buildmode==exe && GOOS==linux).
>>
>>
>> What you're seeing here is that the people who worked on the arm64 support
>> mostly came from a Linux background where having a host linker installed
>> doesn't seem like a big deal.
>>
>>>
>>> Even for amd64 port, internal link doesn't support PIE now (It seems to
>>> support PIE before).
>>>
>>> So my question is: what's the future development direction and plan of
>>> Golang internal link?
>>> More dependence on external linker (such as gcc) or less dependence on
>>> external one (even on dependence at last)?
>>
>>
>> I'm not aware of any real plans here. I've had this silly idea for a while
>> to implement ELF/PE/MachO linkers in Go, convert cmd/link to always operate
>> in what is now external link mode and invoke the bundled host linker if the
>> cgo usage is simple enough. I think this would lead to cleaner code, but it
>> would probably be slower, unfortunately.
>>
>> Cheers,
>> mwh
>>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to golang-dev+...@googlegroups.com.

[Michael Hudson-Doyle]

On 9 January 2018 at 23:36, <Wei....@arm.com> wrote:

Fast building speed is important for Golang and we can incrementally improve internal link for it.

I just have two basic questions:

1. Why CGO requires external link? Is there any technical issues or considerations to require it or just due to a lot of coding work?

How arm64 specific are you being here? On other arches,  cgo internal linking only implements what is needed for the uses of cgo in the standard library. Extending that to cover all possible uses of cgo would be a mountain of work, it would basically be reimplementing an entire system linker.

On arm64 specifically, there is cgo internal linking at all. Bringing it up to the standard of other arches wouldn't be a huge amount of work, you'd just need to implement support for the relocations that the host compiler emits when compiling os/user and net (and none of the ugly c++ stuff Ian mentions). I'm still not going to do it myself though :-)

 

2. PIE was once supported by internal link for linux/amd64 but switch to external link last year. Is any plan to switch it back to internal link for linux/amd64 in the future?

I guess it can be switched back when the bugs that caused it to be disabled get fixed. I don't know if anyone is planning on working on that.

Cheers,

mwh

 

Thanks

Wei Xiao

On Tuesday, January 9, 2018 at 3:55:03 AM UTC+8, Michael Hudson-Doyle wrote:

On 8 January 2018 at 19:35, <Wei....@arm.com> wrote:

I'm investigating Golang internal link for potential improvement on arm64 port.

But I find that internal link is only enabled for very few cases such as: (!iscgo && buildmode==exe  && GOOS==linux).

What you're seeing here is that the people who worked on the arm64 support mostly came from a Linux background where having a host linker installed doesn't seem like a big deal.

 

Even for amd64 port, internal link doesn't support PIE now (It seems to support PIE before).

So my question is: what's the future development direction and plan of Golang internal link?

More dependence on external linker (such as gcc) or less dependence on external one (even on dependence at last)?

I'm not aware of any real plans here. I've had this silly idea for a while to implement ELF/PE/MachO linkers in Go, convert cmd/link to always operate in what is now external link mode and invoke the bundled host linker if the cgo usage is simple enough. I think this would lead to cleaner code, but it would probably be slower, unfortunately.

Cheers,

mwh

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Wei....@arm.com]

I will go through the details with some test cases.

As for "implement support for the relocations that the host compiler emits when compiling os/user and net (and none of the ugly c++ stuff Ian mentions)",

Do you mean building Golang itself with "GO_EXTLINK_ENABLED=0" ?

Thanks

Wei Xiao

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Ian Lance Taylor]

On Wed, Jan 10, 2018 at 2:43 AM, <Wei....@arm.com> wrote:
>
> I will go through the details with some test cases.
> As for "implement support for the relocations that the host compiler emits
> when compiling os/user and net (and none of the ugly c++ stuff Ian
> mentions)",
> Do you mean building Golang itself with "GO_EXTLINK_ENABLED=0" ?

No, simply that (*tester).internalLink is changed to return true on
arm64, and all the tests continue to pass.

Ian

[Ian Lance Taylor]

Sorry, forgot to say that that method is in cmd/dist/test.go.

Ian

[Wei....@arm.com]

Thanks for the detailed information!

From my understanding, we need to do at least following things to make internal link support cgo for arm64:

1. improve ldelf to recognise relocation types of aarch64 elf

2. convert aarch64 elf relocation types into internal relocation types (i.e objabi.R_xxx)

3. prepare GOT and PLT items for dynamic linking

4. handle arm64 specific internal relocation types

As for supporting PIE with internal link, we need to handle TLS and related runtime stuffs.

Is there anything missing?

[Wei....@arm.com]

As for Golang internal link, it seems that we only need to consider Local Exec and Initial Exec TLS.

Is there any scenario for General Dynamic or Local Dynamic TLS?

Thanks

Wei Xiao

[Ian Lance Taylor]

On Wed, Jan 31, 2018 at 2:47 AM, <Wei....@arm.com> wrote:
>
> As for Golang internal link, it seems that we only need to consider Local
> Exec and Initial Exec TLS.
> Is there any scenario for General Dynamic or Local Dynamic TLS?

We don't currently expect to support those when doing an internal
link, no. For shared libraries we expect to use the external linker.

Ian

> On Wednesday, January 10, 2018 at 11:32:35 PM UTC+8, Ian Lance Taylor wrote:
>>
>> On Wed, Jan 10, 2018 at 7:31 AM, Ian Lance Taylor <ia...@golang.org>
>> wrote:
>> > On Wed, Jan 10, 2018 at 2:43 AM, <Wei....@arm.com> wrote:
>> >>
>> >> I will go through the details with some test cases.
>> >> As for "implement support for the relocations that the host compiler
>> >> emits
>> >> when compiling os/user and net (and none of the ugly c++ stuff Ian
>> >> mentions)",
>> >> Do you mean building Golang itself with "GO_EXTLINK_ENABLED=0" ?
>> >
>> > No, simply that (*tester).internalLink is changed to return true on
>> > arm64, and all the tests continue to pass.
>>
>> Sorry, forgot to say that that method is in cmd/dist/test.go.
>>
>> Ian
>

[eric...@arm.com]

Hi, since Wei is currently out of arm, I am doing this work now. Currently we have added support of internal linking mode for:

1,  go toolchain,

2, pie build mode for the pure go program. 

Because it involves some TLS stuff, I'd like to ask what do you think about adding internal link mode support for using TLS variables in cgo? We have written some code to implement section .tdata, but not much. If you think this is unnecessary, then I will not spend time on it.

在 2018年1月8日星期一 UTC+8下午2:35:29，Wei....@arm.com写道：

[Ian Lance Taylor]

On Wed, Nov 21, 2018 at 2:07 AM, <eric...@arm.com> wrote:
>
> Hi, since Wei is currently out of arm, I am doing this work now. Currently
> we have added support of internal linking mode for:
> 1, go toolchain,
> 2, pie build mode for the pure go program.
>
> Because it involves some TLS stuff, I'd like to ask what do you think about
> adding internal link mode support for using TLS variables in cgo? We have
> written some code to implement section .tdata, but not much. If you think
> this is unnecessary, then I will not spend time on it.

I'm not sure I understand the question, but in general it's fine to
support TLS variables in cmd/link. All you should need to add is
enough to support runtime/cgo, which basically means supporting the
errno variable if that is a TLS variable. You shouldn't need to
support TLS variables in arbitrary cgo code; programs that use
arbitrary cgo code are going to use the external linker anyhow.

Ian

> 在 2018年1月8日星期一 UTC+8下午2:35:29，Wei....@arm.com写道：
>>
>> I'm investigating Golang internal link for potential improvement on arm64
>> port.
>> But I find that internal link is only enabled for very few cases such as:
>> (!iscgo && buildmode==exe && GOOS==linux).
>> Even for amd64 port, internal link doesn't support PIE now (It seems to
>> support PIE before).
>>
>> So my question is: what's the future development direction and plan of
>> Golang internal link?
>> More dependence on external linker (such as gcc) or less dependence on
>> external one (even on dependence at last)?
>

[eric...@arm.com]

Thanks. Got it.

在 2018年11月22日星期四 UTC+8上午6:49:52，Ian Lance Taylor写道：