Why syscall, not dlopen

[ph...@bolthole.com]

Howdy folks,

I'm curious about an internal design decision.

Seems like there are two categories of use of ASM in the Go language.

One is for CPU based primatives such as atomic operations. 

Those are actually fairly portable.

THen, there are OS-specific hacks. Mostly around implementing Syscall.

Which is often just a way to hack into native libc type calls.

Except... they seem almost completely unneccesary to me, seeing as how dlopen() has been around for a very long time now, and is literally made to do exactly that.

So, what I'm saying is, for UNIX class of operating systems (and possibly even other ones), seems like using dlopen() is actually a much cleaner, portable way to do things, for the backend runtime to access neccessary OS-level functions.

Heck, its even part of the POSIX standard!!

http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html

[Steven Hartland]

Syscall doesn’t touch libc it’s kernel level calls only

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bryan C. Mills]

Well, for now.

(https://golang.org/issue/17490)

[Philip Brown]

On Sat, Apr 7, 2018 at 3:07 AM, Steven Hartland <ste...@multiplay.co.uk> wrote:
> Syscall doesn’t touch libc it’s kernel level calls only

yes, I know that, but that isnt the point.
most, if not all, uses of Syscall in Go, are to do things that are
normally done with standard libc calls, but Go simply hasnt gotten
around to implementing a normal function for.

Example: It's common to use using Syscall to essentially do fcntl(),
because until this week, there was no other way to invoke fcntl()
Noone had gotten around to implementing a Go function for it.

But it should be fairly straightfoward, if tedious. to implement every
single POSIX standard libc call, for EVERY POSIX compliant OS. in one
go.
By using dlopen as an intermediary.

Nice, clean, no asm hacks.

[ron minnich]

On Sat, Apr 7, 2018 at 7:50 AM Philip Brown <ph...@bolthole.com> wrote:

By using dlopen as an intermediary. 

Nice, clean, no asm hacks.

One person's clean is another person's not so clean. Using dlopen forces a dependency on libc, and further you force me to do dynamic linking, which I absolutely do not want to do, especially for the case where we embed Go-based userland in firmware, a la u-root and NERF.

I do NOT want to require "standard" libc at any point, especially if it is glibc. I like it the way it is now. For one thing, the security people like the fact that I can build a pure Go-based userland, without bringing in any C-based support of any kind. It would be a terrible loss were Go to ever change away from having that capability.

ron

 

[ron minnich]

And, I forgot to mention, with your assumptions that everyone has dlopen and libc, you're ignoring the case of systems like Plan 9, which do not use dynamic linking (the correct choice in my view).

[Philip Brown]

I didnt say "every system", I said "every POSIX compliant system".

[Ian Lance Taylor]

It is a feature that on most systems Go is capable of generating a
statically linked binary that requires no other support. This is
particularly useful when building Docker containers.

Ian

[Philip Brown]

On Sat, Apr 7, 2018 at 10:35 AM, Ian Lance Taylor <ia...@golang.org> wrote:
> It is a feature that on most systems Go is capable of generating a
> statically linked binary that requires no other support. This is
> particularly useful when building Docker containers.

Hmm.
I'm kinda getting a vibe, that the Go language has currently nestled itself into
a kind of niche market, of embedded type applications, and there is no
interest in additionally promoting it as a more popular, general case
language.
Which is a shame, because i've enjoyed programming with it in my
short experience so far, more than other languages.
(and I've been through quite a few in my 20 years of programming)

[Brad Fitzpatrick]

On Thu, Apr 12, 2018 at 10:17 AM, Philip Brown <ph...@bolthole.com> wrote:

On Sat, Apr 7, 2018 at 10:35 AM, Ian Lance Taylor <ia...@golang.org> wrote:
> It is a feature that on most systems Go is capable of generating a
> statically linked binary that requires no other support.  This is
> particularly useful when building Docker containers.

Hmm.
I'm kinda getting a vibe, that the Go language has currently nestled itself into
 a kind of niche market, of embedded type applications, and there is no
interest in additionally promoting it as a more popular, general case
language.

That's an interesting conclusion to jump to starting from this design decision, and not one I think is commonly shared by others on this list.

[Ian Lance Taylor]

On Thu, Apr 12, 2018 at 10:17 AM, Philip Brown <ph...@bolthole.com> wrote:

I'm sorry, I'm not sure what you mean. What we are talking about here
is a tiny detail: how the Go runtime and syscall packages call into
the kernel. Currently they make directly system calls. You are
suggesting that they should call into libc instead. Which option is
used makes absolutely no difference to 95% of Go programmers. There
is no advantage either way. Where there is a clear advantage to the
libc approach, as on Solaris, we use it. In what way does this
suggest that Go is in a niche market?

For that matter I'll note that gccgo, an alternative Go compiler, does
use libc calls consistently, as you suggest. That makes it
particularly clear that this is not a feature of Go, the language, but
merely a minor implementation detail.

Ian

[Philip Brown]

On Thu, Apr 12, 2018 at 10:35 AM, Ian Lance Taylor <ia...@golang.org> wrote:
> For that matter I'll note that gccgo, an alternative Go compiler, does
> use libc calls consistently, as you suggest. That makes it
> particularly clear that this is not a feature of Go, the language, but
> merely a minor implementation detail.

an interesting point.
However, I thought that gccgo does not support the nice conventions of
"go build", "go get", "go test", and all the other full command
options mentioned
at https://golang.org/cmd/go/

To me, that integrated environment is a major part of the appeal of the
language.
If gccgo does not support all those things, then to me it is not a full
implementation, and not appealing.

[Ian Lance Taylor]

gccgo supports all of those. It comes with its own version of the "go" command.

Ian

[Philip Brown]

On Thu, Apr 12, 2018 at 11:18 AM, Ian Lance Taylor <ia...@golang.org> wrote:
> gccgo supports all of those. It comes with its own version of the "go" command.

Nice!
I guess it's a little under documented. I'll check it out.
Thanks

[ron minnich]

On Thu, Apr 12, 2018 at 10:17 AM Philip Brown <ph...@bolthole.com> wrote:

I'm kinda getting a vibe, that the Go language has currently nestled itself into
 a kind of niche market, of embedded type applications, and there is no
interest in additionally promoting it as a more popular, general case
language.

Here's your initial statement: "THen, there are OS-specific hacks. Mostly around implementing Syscall. Which is often just a way to hack into native libc type calls."

This statement is technically wrong but also philosophically wrong for Unix. 

It's technically wrong because a language runtime that has its own system call interface is not "often just a way to hack into native libc system calls"; there are all kinds of reasons for a language runtime to call system calls directly, and not use libc. That's why there is a system call interface in the first place. 

Go has its own reasons for calling system calls directly (good reasons in my view); other languages have had other reasons. In at least one case I am involved in, the ability build a statically linked binary, using only the Go toolchain, entirely free of libc, is considered a real security feature. 

Your statement is philosophically wrong because restriction of system call usage to libc is certainly not the unix model; lots of early unix code was written in assembly, so calling libc made no sense. Linux has a guarantee on the system call ABI for the kernel that it has honored quite well. Plan 9 has had a stable kernel ABI for almost 30 years. It's reasonable to provide a stable ABI for system calls.

The original library for Unix back in the day was .. the kernel. Hence the name system call: we treated them as just another variety of library call, which had a slightly different calling convention. 

I'm sorry that Apple has lost its understanding of Unix, but that's an issue for Apple, not Go.

But hey, you don't have to believe *me*. The person who wrote much of the original Go toolchain, which was built on the Plan 9 C toolchain, also co-invented Unix. If calling system calls directly, not via libc, was good enough for the co-inventor of Unix, it's good enough for me.

ron

[Brad Fitzpatrick]

https://golang.org/doc/install/gccgo#Using_gccgo mentions that both of the methods of building that are supporting, including the go command.

File a bug if you find documentation omissions.

[Philip Brown]

On Thu, Apr 12, 2018 at 11:44 AM, Brad Fitzpatrick <brad...@golang.org> wrote:
> https://golang.org/doc/install/gccgo#Using_gccgo mentions that both of the
> methods of building that are supporting, including the go command.
>
> File a bug if you find documentation omissions.

I am happy to make some suggestions for the page.

Looking at the page, however, it is not immediately apparent how or
where to give feedback for it.
That is to say, sites that are readily accepting of feedback on their
webpages, usually have a fairly directly, friendly form to give
feedback on that particular page.
(Since Go is backed by google, I think it's fair to make the reference
that most xx.google.com sites, have a "send feedback" link on every
page that makes this sort of thing relatively painless ;)

[Brad Fitzpatrick]

You've filed bugs and code already, so you've already found:

https://golang.org/project/

https://golang.org/project/#howto

https://golang.org/doc/contribute.html 

[Jan Mercl]

On Thu, Apr 12, 2018 at 10:39 PM Philip Brown <ph...@bolthole.com> wrote:

> I am happy to make some suggestions for the page.

Here's the page and section: https://github.com/golang/go/blob/191efbc419d7e5dec842c20841f6f716da4b561d/doc/gccgo_install.html#L183

Here is the issue tracker: https://github.com/golang/go/issues

--

-j

[Philip Brown]

So, what I think you are saying, is:

1. the documentation on the website, is just an automatic upload of
what's in the source code tree
2. there is no particular person, or subgroup, responsible for good
techwriting for Go documentation
3. which means that documentation is dependant on PROGRAMMERS feeling
like writing/improving the
documentation. Seeing as how usually, only programers are willing to
directly deal with source version control, etc.

That's... unfortunate. IMO, any major software project should have
some designated tech writer(s). especially one backed by a company
such as google.
Oh well

[andrey mirtchovski]

> That's... unfortunate. IMO, any major software project should have
> some designated tech writer(s). especially one backed by a company
> such as google.

nobody has volunteered to be such a person. would you like to?

[Ian Lance Taylor]

On Thu, Apr 12, 2018 at 3:20 PM, Philip Brown <ph...@bolthole.com> wrote:
>
> So, what I think you are saying, is:
>
> 1. the documentation on the website, is just an automatic upload of
> what's in the source code tree

Approximately yes. The website is itself a Go program, there is no
separate automatic upload.

> 2. there is no particular person, or subgroup, responsible for good
> techwriting for Go documentation

There is Rob Pike, but other than that, yes.

> 3. which means that documentation is dependant on PROGRAMMERS feeling
> like writing/improving the
> documentation. Seeing as how usually, only programers are willing to
> directly deal with source version control, etc.

And yet I think the Go docs are fairly good.

> That's... unfortunate. IMO, any major software project should have
> some designated tech writer(s). especially one backed by a company
> such as google.

That is an easy mistake to make. Google is a very large company, but
Go is a very tiny part of it, and Go is not a source of revenue.
While I think it would be awfully nice if Google hired a tech writer
to work on the Go documentation, that's actually not near the top of
the list of things I would ideally like Google to hire people to work
on. (I work at Google myself, and I do appreciate the fact that the
company lets me work on a fun project like Go.)

I fully agree that the gccgo docs are less than ideal, that is partly
a result of the complex nature of gccgo, and partly simply my fault.

Ian

> On Thu, Apr 12, 2018 at 1:49 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
>>
>>
>> On Thu, Apr 12, 2018 at 1:39 PM, Philip Brown <ph...@bolthole.com> wrote:
>>>
>>> On Thu, Apr 12, 2018 at 11:44 AM, Brad Fitzpatrick <brad...@golang.org>
>>> wrote:
>>> > https://golang.org/doc/install/gccgo#Using_gccgo mentions that both of
>>> > the
>>> > methods of building that are supporting, including the go command.
>>> >
>>> > File a bug if you find documentation omissions.
>>>
>>>
>>>
>>> I am happy to make some suggestions for the page.
>>>
>>> Looking at the page, however, it is not immediately apparent how or
>>> where to give feedback for it.
>>> That is to say, sites that are readily accepting of feedback on their
>>> webpages, usually have a fairly directly, friendly form to give
>>> feedback on that particular page.
>>> (Since Go is backed by google, I think it's fair to make the reference
>>> that most xx.google.com sites, have a "send feedback" link on every
>>> page that makes this sort of thing relatively painless ;)
>>
>>
>> You've filed bugs and code already, so you've already found:
>>
>> https://golang.org/project/
>> https://golang.org/project/#howto
>> https://golang.org/doc/contribute.html
>>
>

[Ralph Corderoy]

Hi Philip,

> That's... unfortunate. IMO, any major software project should have
> some designated tech writer(s). especially one backed by a company
> such as google. Oh well

This has wandered well away from the dlopen implementation question now,
and seems better suited for golang-gonuts list where you might find more
willing to debate and critique the decisions behind documentation
production. :-)

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

[Dan Kortschak]

It's my recollection that this did actually exist for a while. I also
seem to remember that it resulted in fairly poorly structured issues
being filed against the problems people found.

[Michael Jones]

RE: "That's... unfortunate. IMO, any major software project should have some designated tech writer(s)"

The literal API is covered completely, though arguably not so deeply as UNIX man-page style.

There are examples, but maybe you want more or different kinds of examples.

There is a language specification

There are best practices.

There are blogs (Dave Cheney's among the best)

There are books at Amazon

There are books online for download

There are guides to installing and building compiler source.

If there was a designated tech writer, or a hundred, what would you want them to write about? It would be helpful to know. Maybe answers like "something quite different than the above" or maybe "like one of those, but different in detail."

There are lapses (I'd like to see medium sized apps built as exemplars: the shell in Go, "ngininx in Go", curl in go, grep in go, ...) but I don't know if code samples are what a tech writer would really produce with beauty. Please share.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[matthe...@gmail.com]

That's... unfortunate. IMO, any major software project should have 
some designated tech writer(s). especially one backed by a company 
such as google. 

The command details could completely change if a good idea comes along; my opinion is now isn’t the right time to write comprehensive implementation documentation.

I personally have been happy with the language and standard library stuff but cmd code could be better presented. Maybe having a final implementation architecture with formal documentation could be a Go 2 goal, although there’s optimization pressure that may cause future architecture changes.

Will Go ever be done? I'd like compiler tinkering to be as accessible as possible without relying on the Internet.

Matt