x/sys/unix: problem adding code to support fgetattr function on solaris

[ksh...@racktopsystems.com]

I am writing a Go wrapper around the fgetattr function and I am running into a problem getting it to compile.  I have the same problem using Go 1.5 or Go built from the latest pull of the master branch.  In the following, I will use the setup I have for running with the latest master build.

I am running on omnios 151015

I have the following environment variables exported:

    GOARCH=amd64
    GOOS=solaris
    GOPATH=$HOME/go_stuff/sub_repos/go_master
    GOROOT=/usr/local/go_master

    PATH=/usr/local/go_master/bin:/opt/gcc-5.1.0/bin:/opt/omni/bin:/usr/gnu/bin:/usr/bin:/usr/sbin:/sbin

I have the x/sys repo in $HOME/go_stuff/sub_repos/go_master/src/golang.org/x/sys.  When I cd into $HOME/go_stuff/sub_repos/go_master/src/golang.org/x/sys/unix and run "go test", everything works fine.

    $ go test

    PASS
    ok golang.org/x/sys/unix 0.008s

I have added a file named attr_solaris.go in x/sys/unix that just contains the following lines:

    // +build amd64,solaris

    package unix

    /*

    */

    import "C"

Running "go test" now results in the following error:

    $ go test

    # golang.org/x/sys/unix

    ./asm.s: Assembler messages:

    ./asm.s:9: Error: no such instruction: `text ·use(SB),NOSPLIT,$0'

    FAIL    golang.org/x/sys/unix [build failed]

When I look at x/sys/unix/asm.s, line 9 is slightly different from what is in the error message

    TEXT ·use(SB),NOSPLIT,$0

I don't know why the error message has "text" in lower case when the source file has it in upper case.  I have tried specifying -v -x and -work in order to see what might be happening.

    $ go test -v -x -work

    WORK=/tmp/go-build126758309

    mkdir -p $WORK/golang.org/x/sys/unix/_test/golang.org/x/sys/

    mkdir -p $WORK/golang.org/x/sys/unix/_test/_obj_test/

    cd /export/home/kim/go_stuff/sub_repos/go_master/src/golang.org/x/sys/unix

    CGO_LDFLAGS="-g" "-O2" /usr/local/go_master/pkg/tool/solaris_amd64/cgo -objdir $WORK/golang.org/x/sys/unix/_test/_obj_test/ -importpath golang.org/x/sys/unix -- -I $WORK/golang.org/x/sys/unix/_test/_obj_test/ attr_solaris.go

    gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -print-libgcc-file-name

    gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/golang.org/x/sys/unix/_test/_obj_test/ -g -O2 -o $WORK/golang.org/x/sys/unix/_test/_obj_test/_cgo_main.o -c $WORK/golang.org/x/sys/unix/_test/_obj_test/_cgo_main.c

    gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/golang.org/x/sys/unix/_test/_obj_test/ -g -O2 -o $WORK/golang.org/x/sys/unix/_test/_obj_test/_cgo_export.o -c $WORK/golang.org/x/sys/unix/_test/_obj_test/_cgo_export.c

    gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/golang.org/x/sys/unix/_test/_obj_test/ -g -O2 -o $WORK/golang.org/x/sys/unix/_test/_obj_test/attr_solaris.cgo2.o -c $WORK/golang.org/x/sys/unix/_test/_obj_test/attr_solaris.cgo2.c

    gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/golang.org/x/sys/unix/_test/_obj_test/ -g -O2 -o $WORK/golang.org/x/sys/unix/_test/_obj_test/asm.o -c ./asm.s

    # golang.org/x/sys/unix

    ./asm.s: Assembler messages:

    ./asm.s:9: Error: no such instruction: `text ·use(SB),NOSPLIT,$0'

    $WORK/golang.org/x/sys/unix/_test/unix.test -test.v=true

    FAIL    golang.org/x/sys/unix [build failed]

It appears to be using the asm.s file that already exists in x/sys/unix and not using a generated one so I am at a loss as to why the case of "text" is not upper case.

I actually had some Go code in attr_solaris.go originally and it showed the exact same symptoms.  I have reduced the code to the smallest sample that still shows the problem.

Am I doing something wrong or is there a bug in the toolchain?

Kim

[Ian Lance Taylor]

On Fri, Aug 28, 2015 at 3:21 PM, <ksh...@racktopsystems.com> wrote:
>
> I have added a file named attr_solaris.go in x/sys/unix that just contains
> the following lines:
>
> // +build amd64,solaris
>
>
> package unix
>
>
> /*
>
> */
>
> import "C"
>
>
> Running "go test" now results in the following error:
>
> $ go test
>
> # golang.org/x/sys/unix
>
> ./asm.s: Assembler messages:
>
> ./asm.s:9: Error: no such instruction: `text ·use(SB),NOSPLIT,$0'

When you import "C" into a file that is being built by the go tool,
you change the way that .s files are handled. In a build without any
import "C", .s files are passed to the Go assembler. In a build with
import "C", .s files are passed to the system assembler via GCC.

The system assembler does not understand the text directive.

In other words, the x/sys/unix package can not use cgo.

Ian

[Rob Pike]

The message

    ./asm.s:9: Error: no such instruction: `text ·use(SB),NOSPLIT,$0'

implies that the TEXT func-declaring word is spelled in lower case. It should be upper case.

-rob

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Lance Taylor]

On Fri, Aug 28, 2015 at 5:11 PM, Rob Pike <r...@golang.org> wrote:
> The message
>
> ./asm.s:9: Error: no such instruction: `text ·use(SB),NOSPLIT,$0'
>
> implies that the TEXT func-declaring word is spelled in lower case. It
> should be upper case.

It is upper case in the file, it's bring printed as lower case because
this message is coming from the GNU assembler, which lower cases the
instruction field as it reads the line.

Ian

[Rob Pike]

Oh. Well yes, then, that won't work.

-rob

[Aram Hăvărneanu]

Yes, you can't use cgo in syscall or x/sys.

What are you trying to do? It's possible to add support for whatever
it is that you need, without cgo.

--
Aram Hăvărneanu

[Kim Shrier]

I am wanting to call fgetattr to get extended file attributes. I have Go wrappers around the nvlist code I will need to step through the list of attributes. I would think that support to get to fgetattr, fsetattr, getattrat, and setattrat would belong in x/sys/unix. I was also wondering if the nvlist code would also belong in x/sys/unix.

Kim

[Ian Lance Taylor]

On Sun, Aug 30, 2015 at 8:58 AM, Kim Shrier <ksh...@racktopsystems.com> wrote:
> I am wanting to call fgetattr to get extended file attributes. I have Go wrappers around the nvlist code I will need to step through the list of attributes. I would think that support to get to fgetattr, fsetattr, getattrat, and setattrat would belong in x/sys/unix. I was also wondering if the nvlist code would also belong in x/sys/unix.

nvlist makes sense for C, but it really doesn't for Go. I think you
should write Go API for fgetattr and friends that uses Go concepts,
not anything like nvlist.

Ian

[Kim Shrier]

I am not intending to expose nvlist in the wrapper I put around fgetattr but I do have to walk the list to populate a Go map, which I am intending to be the return value of the Go fgetattr function, so I have Go wrappers I have written for the nvlist functions. Since solaris uses nvlist for a lot of things, I was thinking that these wrapper functions should be generally available.

Kim

[Ian Lance Taylor]

On Sun, Aug 30, 2015 at 6:40 PM, Kim Shrier <ksh...@racktopsystems.com> wrote:
> I am not intending to expose nvlist in the wrapper I put around fgetattr but I do have to walk the list to populate a Go map, which I am intending to be the return value of the Go fgetattr function, so I have Go wrappers I have written for the nvlist functions. Since solaris uses nvlist for a lot of things, I was thinking that these wrapper functions should be generally available.

I don't understand when Go code would use nvlist functions. If they
are useful for functions in x/sys/unix, then, sure. Otherwise, I
don't see the point.

Ian

[Kim Shrier]

I also have Go wrapper functions around the libzfs functions and many of them have nvlist parameters. You can see the function declarations in /usr/include/libzfs.h. The Go functions I have written do not expose nvlist but they have to deal with nvlists, either creating them so they can be passed to the C functions, or deciphering them and turning them into Go maps or slices. Rather than repeat the code to convert from Go types to C types and vice-versa, I have encapsulated it into Go wrapper functions around the C nvlist functions.

What I am doing is providing a layer of Go functions to access a lot of solaris specific functionality so that application programmers do not need to know how to interface to C code and can live in a Go universe. I have permission to contribute some of this code to the Go community if it is useful. I was thinking that some of this would make sense to put into x/sys but now I am thinking it should go into its own packages outside of x/sys. Particularly since it appears that I can’t use cgo for code that is in x/sys.

Kim

[Aram Hăvărneanu]

Much of what you say, including libzfs stuff makes sense to live in
x/sys. As I mentioned already, you don't need cgo, so there are no
problems.

--
Aram Hăvărneanu

[Kim Shrier]

Maybe I just don’t understand something. How can I provide access to these C functions from Go without using cgo?

Kim

[Ian Lance Taylor]

As far as I can tell these C functions are only useful to Go code that
is already using cgo. I don't understand why any code not already
using cgo would want to use these functions. I think it follows that
a program that is already using cgo can simply call the functions
directly. Can you describe why a Go program would want to call these
functions directly without simply calling via cgo?

Ian

[Kim Shrier]

This is the context for which I am developing this code.

Right now, we have a number of programs that manipulate and
report on ZFS datasets and pools that are written in Python.
These programs shell out to various existing commands and
collect the output from stdout so it can be scraped for the
information needed. I hope the following doesn’t sound
condescending, I am just trying to accurately describe the
situation. Most of the programmers writing these programs
started out doing HTML, CSS, and a little Javascript. They
have progressed to using Python for the server side code.
They really are not used to working with a compiled language
and Go is a bit of a stretch for them. The current plan is to
migrate a majority of the Python code to Go and stop shelling
out to get the work done.

I have experience with C and assembly language so I am
writing the “scary” code that bridges from the Go universe to
the functionality they want access to, that is exposed in the
various C libraries on Solaris. In other words, I am writing all
the cgo code and everybody else is only writing Go and using
the functions my code provides.

I was thinking that the cgo code I am writing would be useful
to others and I thought that x/sys was the logical place to put it.
I think I don’t understand enough about the Go toolchain to pull
this off but I would like to remedy that situation.

Kim

[Ian Lance Taylor]

On Mon, Aug 31, 2015 at 11:09 AM, Kim Shrier <ksh...@racktopsystems.com> wrote:
>
>> On Aug 31, 2015, at 10:31 AM, Ian Lance Taylor <ia...@golang.org> wrote:
>>
>> As far as I can tell these C functions are only useful to Go code that
>> is already using cgo. I don't understand why any code not already
>> using cgo would want to use these functions. I think it follows that
>> a program that is already using cgo can simply call the functions
>> directly. Can you describe why a Go program would want to call these
>> functions directly without simply calling via cgo?
>

> This is the context for which I am developing this code.
>
> Right now, we have a number of programs that manipulate and
> report on ZFS datasets and pools that are written in Python.
> These programs shell out to various existing commands and
> collect the output from stdout so it can be scraped for the
> information needed. I hope the following doesn’t sound
> condescending, I am just trying to accurately describe the
> situation. Most of the programmers writing these programs
> started out doing HTML, CSS, and a little Javascript. They
> have progressed to using Python for the server side code.
> They really are not used to working with a compiled language
> and Go is a bit of a stretch for them. The current plan is to
> migrate a majority of the Python code to Go and stop shelling
> out to get the work done.
>
> I have experience with C and assembly language so I am
> writing the “scary” code that bridges from the Go universe to
> the functionality they want access to, that is exposed in the
> various C libraries on Solaris. In other words, I am writing all
> the cgo code and everybody else is only writing Go and using
> the functions my code provides.
>
> I was thinking that the cgo code I am writing would be useful
> to others and I thought that x/sys was the logical place to put it.
> I think I don’t understand enough about the Go toolchain to pull
> this off but I would like to remedy that situation.

That all sounds great. But I still don't understand why any code not
already using cgo would want to use the nvlist functions.

At this point I can't tell whether it makes for your library to live
in x/sys or not. If it requires cgo, it clearly can't live in
x/sys/unix. Perhaps you should set up your library as a go gettable
library to start with, and we can consider whether it makes sense to
move it into x/sys.

Ian

[Kim Shrier]

> On Aug 31, 2015, at 1:37 PM, Ian Lance Taylor <ia...@golang.org> wrote:
>
> That all sounds great. But I still don't understand why any code not
> already using cgo would want to use the nvlist functions.

It’s true, any code wanting to use nvlist functions would already be using
cgo. I have 3 different packages that interface to various C libraries
that all need to deal with nvlist. Rather than just repeat the code to
handle the type conversions in each package, it seemed better to do it
once in a package dedicated to interfacing to nvlist and let my other
packages pass Go data types into the nvlist functions and get Go data
types out of the nvlist functions. It just seems cleaner that way.

> At this point I can't tell whether it makes for your library to live
> in x/sys or not. If it requires cgo, it clearly can't live in
> x/sys/unix. Perhaps you should set up your library as a go gettable
> library to start with, and we can consider whether it makes sense to
> move it into x/sys.

That makes the most sense to me now. I am going to redo some of
my code in light of what I have learned from this conversation and see
what I can do to get it placed in a public repository.

Thanks for your time and patience.

Kim

[Aram Hăvărneanu]

Kim Shrier <ksh...@racktopsystems.com> wrote:
> Maybe I just don’t understand something. How can I provide access
> to these C functions from Go without using cgo?

The Solaris port essentially "provides access to some C functions",
specifically the system libc, without using the user-visible cgo
mechanism (it does use the internal mechanism, however).

See the way //go:cgo_import_dynamic mechanism is used in the runtime,
syscall, and x/sys.

--
Aram Hăvărneanu

[Aram Hăvărneanu]

Ian Lance Taylor <ia...@golang.org> wrote:

> As far as I can tell these C functions are only useful to Go code that
> is already using cgo. I don't understand why any code not already
> using cgo would want to use these functions. I think it follows that
> a program that is already using cgo can simply call the functions
> directly. Can you describe why a Go program would want to call these
> functions directly without simply calling via cgo?

It is not true that these C functions are only useful to Go code
that is already using cgo, why would that be the case? These functions
are useful to a large class of Go code that runs on Solaris, and
potentially FreeBSD as well. It would be very useful if that Go
code can be cross-compiled easily, like mlst other Go code.

--
Aram Hăvărneanu

[Ian Lance Taylor]

You didn't really answer my final question, though. I'm asking
because I don't believe that system calls are constructing those
memory structures using malloc. My assumption is that they are being
done by libc functions. So I don't understand why Go code wouldn't
call the system calls directly and construct Go structures rather than
nvlist structures.

Ian

[Aram Hăvărneanu]

On Tue, Sep 1, 2015 at 7:43 PM, Ian Lance Taylor <ia...@golang.org> wrote:
> So I don't understand why Go code wouldn't
> call the system calls directly and construct Go structures rather than
> nvlist structures.

Sure, that's what it should do.

--
Aram Hăvărneanu

[Kim Shrier]

In the case of wrapping libzfs, libzfs is not just a thin veneer over the system
calls. libzfs is over 20K lines of code. If I were to just make system calls from
my Go code, I would be reimplementing a lot of libzfs functionality in Go.
I wouldn’t reimplement the nvlist stuff but without actually evaluating the
size of the other code in libzfs, I would guess that the effort would be
larger than I have time for and would introduce errors.

It never occurred to me to just do system calls so I am going to see if
my intuitions are correct in the amount of effort to go that route.

Kim

[Kim Shrier]

It turns out the the system calls do return nvlist structures. Just looking at
fgetattr(), it calls xattr_openat() which calls openat() which is a syscall not
currently implemented in syscall_solaris.go so it doesn’t show up in
zsyscall_solaris_amd64.go.

The fact that openat() isn’t in syscall_solaris.go seems to be a simple thing
to remedy. So assuming that syscall was in place, after openat() returns,
fgetattr() calls cgetattr() which calls read() on the file handle returned by
openat(). The data returned by read() is a packed nvlist structure of file
attributes. This packed nvlist is expanded to an unpacked nvlist and
returned to the user.

So the system calls involved for fgetattr do not explicitly know about
nvlist structures, but the read() system call does returned a nvlist that
has been marshaled into a form that allows it to be read and written
as an array of bytes. Since Solaris stores the raw information for
extended file attributes as a nvlist, I see no way to escape having Go
code that has to deal with nvlists.

Kim

[Aram Hăvărneanu]

On Tue, Sep 1, 2015 at 10:48 PM, Kim Shrier <ksh...@racktopsystems.com> wrote:
> So the system calls involved for fgetattr do not explicitly know about
> nvlist structures, but the read() system call does returned a nvlist that
> has been marshaled into a form that allows it to be read and written
> as an array of bytes. Since Solaris stores the raw information for
> extended file attributes as a nvlist, I see no way to escape having Go
> code that has to deal with nvlists.

Sure, but this is in no way different than Go code having to know the
layout of, say, some target-specific socket structures. The Go code
has to know the layout of the data in order to expose it to
higher-level Go code, but it doesn't have to call existing C functions
from libc to process it.

Also, I will add again, that on Solaris at least (and with a very
small effort on other ELF platforms too), any library can be called
from Go without cgo, including libzfs, so you don't need to
reimplement libzfs in Go.

--
Aram Hăvărneanu