where is my lib? cgo and c libraries

[Peter Kleiweg]

A compiled program that was written in pure Go is self-contained. It runs without external dependencies. A program using external C libraries, when compiled, can still miss the right information to be run.

I don't like that, and I think it could be done better.

I `go get` a package that depends on a C ibrary. I have that C library, not in a standard location, but the package uses pgk-config to find the location of my libraries, and it installs without problem.

I write a program that uses the package. It builds without error. But when I try to run the program, I get an error: library not found.

I can fix that in two ways.

1. I set LD_LIBRARY_PATH to point to the directory with the C library before I run the program.

2. When I use `go build`, I add an option that says to include the path to the C library in the code:

-ldflags="-r /path/to/lib"

I think that when a package gets installed, the path to C libaries should be recorded in the package. And when I build a program with that package, that path info should be set in the binary automaticly. And it should also work for packages importing packages that use C libraries.


--
Peter Kleiweg
my Go programming cookbook: http://www.let.rug.nl/~kleiweg/go/

[minux]

i thought you have filed an issue for this:
https://code.google.com/p/go/issues/detail?id=5224
and i think it's a bug in cmd/cgo.

[Peter Kleiweg]

Op zaterdag 6 april 2013 13:54:07 UTC+2 schreef minux het volgende:

No, this is something different. The bug is that the program won't build, new in Go tip.

What I am refering  to here is that the program builds, but won't run. This has always been this way.

[Ian Lance Taylor]

Now that we use external linking, this is going to be pushed into the
space of the external linker. External linkers historically do not
add -L options to the shared library search path, because it leads to
a problem that is the reverse of the one you cite. At runtime, the
program can wind up searching many directories unnecessarily. This is
particularly bad when those directories are mounted using NFS or FUSE
or some such technology.

From Go's perspective it would be easy to say "whenever we pass a -L
option to the system linker, also pass a -Wl,-rpath option." It would
also be easy for the gcc driver to do this, and it would be easy for
an ELF linker to do this. The question, then, is whether the go tool
should behave differently from those older tools.

Ideally pkg-config would provide the right answer. It does have
support for this. So one question to ask would be why pkg-config
isn't doing the right thing for the library in question.

Ian

[Peter Kleiweg]

Op zaterdag 6 april 2013 15:52:05 UTC+2 schreef Ian Lance Taylor het volgende:

Now that we use external linking, this is going to be pushed into the

What's that, external linking?

 

space of the external linker.  External linkers historically do not
add -L options to the shared library search path, because it leads to
a problem that is the reverse of the one you cite.  At runtime, the
program can wind up searching many directories unnecessarily.  This is
particularly bad when those directories are mounted using NFS or FUSE
or some such technology.

From Go's perspective it would be easy to say "whenever we pass a -L
option to the system linker, also pass a -Wl,-rpath option."  It would
also be easy for the gcc driver to do this, and it would be easy for
an ELF linker to do this.  The question, then, is whether the go tool
should behave differently from those older tools.

Yes, do it differently. Go was meant to make life easier. It doesn't use download, extract, configure, make, make install, it uses go get, which is enough most of the time. I say, make it work too in those cases where now it doesn't work because of missing paths to C libraries.

 

For the user it's simple: a program should work, or it is useless. So, make it work, without the need of special actions by the user.

So yes, pass that option to the linker. If it is not necessary, the linker drops it anyway. If it is necessary to run the program, than add it. 

Ideally pkg-config would provide the right answer.  It does have
support for this.  So one question to ask would be why pkg-config
isn't doing the right thing for the library in question.

Because it is also used to build binary distributions of programs?

[Ian Lance Taylor]

On Sat, Apr 6, 2013 at 10:43 AM, Peter Kleiweg <pkle...@xs4all.nl> wrote:
> Op zaterdag 6 april 2013 15:52:05 UTC+2 schreef Ian Lance Taylor het
> volgende:
>
>> Now that we use external linking, this is going to be pushed into the
>
>
> What's that, external linking?

On tip, when using cgo, which is the only case in which a shared
library can arise, the Go linker will invoke the system linker to
finish the link.

>> space of the external linker. External linkers historically do not
>> add -L options to the shared library search path, because it leads to
>> a problem that is the reverse of the one you cite. At runtime, the
>> program can wind up searching many directories unnecessarily. This is
>> particularly bad when those directories are mounted using NFS or FUSE
>> or some such technology.
>>
>> From Go's perspective it would be easy to say "whenever we pass a -L
>> option to the system linker, also pass a -Wl,-rpath option." It would
>> also be easy for the gcc driver to do this, and it would be easy for
>> an ELF linker to do this. The question, then, is whether the go tool
>> should behave differently from those older tools.
>
>
> Yes, do it differently. Go was meant to make life easier. It doesn't use
> download, extract, configure, make, make install, it uses go get, which is
> enough most of the time. I say, make it work too in those cases where now it
> doesn't work because of missing paths to C libraries.
>
> For the user it's simple: a program should work, or it is useless. So, make
> it work, without the need of special actions by the user.

As I tried to explain, if we make that change, it will work for a
different set of people and fail for a different set of people. It's
not as though the change would be right for everybody. It might be
the right choice for Go, but it's not going to be the right choice for
everybody.

> So yes, pass that option to the linker. If it is not necessary, the linker
> drops it anyway. If it is necessary to run the program, than add it.

You need to think about the case in which the library is in different
places at link time and at run time, which is common when distributing
binary programs.

>> Ideally pkg-config would provide the right answer. It does have
>> support for this. So one question to ask would be why pkg-config
>> isn't doing the right thing for the library in question.
>
>
> Because it is also used to build binary distributions of programs?

pkg-config is used to provide the correct set of options to include a
shared library in your program. It provides -L options to use. So it
could also provide -Wl,-rpath options.

Ian

[Peter Kleiweg]

Op zondag 7 april 2013 00:46:44 UTC+2 schreef Ian Lance Taylor het volgende:

On Sat, Apr 6, 2013 at 10:43 AM, Peter Kleiweg <pkle...@xs4all.nl> wrote:
> Op zaterdag 6 april 2013 15:52:05 UTC+2 schreef Ian Lance Taylor het
> volgende:

[...] 

 

>> From Go's perspective it would be easy to say "whenever we pass a -L
>> option to the system linker, also pass a -Wl,-rpath option."  It would
>> also be easy for the gcc driver to do this, and it would be easy for
>> an ELF linker to do this.  The question, then, is whether the go tool
>> should behave differently from those older tools.
>
>
> Yes, do it differently. Go was meant to make life easier. It doesn't use
> download, extract, configure, make, make install, it uses go get, which is
> enough most of the time. I say, make it work too in those cases where now it
> doesn't work because of missing paths to C libraries.
>
> For the user it's simple: a program should work, or it is useless. So, make
> it work, without the need of special actions by the user.

As I tried to explain, if we make that change, it will work for a
different set of people and fail for a different set of people.  It's
not as though the change would be right for everybody.  It might be
the right choice for Go, but it's not going to be the right choice for
everybody.

If it is right for Go, just do it. What is right for other programming languages is irrelevant. 

 

> So yes, pass that option to the linker. If it is not necessary, the linker
> drops it anyway. If it is necessary to run the program, than add it.

You need to think about the case in which the library is in different
places at link time and at run time, which is common when distributing
binary programs.

Distro-builders are the tiniest of minorities of Go users, and capable enough to deal with unusual situations.

The vast majority are the actual users, who do `go get` and `go build`, and expect it to work. Because that is what Go promised us.

At the moment, in some cases, it doesn't work. This needs fixing.

 

>> Ideally pkg-config would provide the right answer.  It does have
>> support for this.  So one question to ask would be why pkg-config
>> isn't doing the right thing for the library in question.
>
>
> Because it is also used to build binary distributions of programs?

pkg-config is used to provide the correct set of options to include a
shared library in your program.  It provides -L options to use.  So it
could also provide -Wl,-rpath options.

At the moment you are building a program that imports a package that was build with the help of pkg-config,  pkg-config is already out of the picture. The package should now have all the necessary link info. You are working with a Go package now, and Go packages don't do pkg-config, like they don't do headers files or configuration scripts. So the package should contain all the information necessary for building against that package, and pass that information on to the build process to create an executable that just runs.

[Ian Lance Taylor]

On Sun, Apr 7, 2013 at 1:29 PM, Peter Kleiweg <pkle...@xs4all.nl> wrote:
> Op zondag 7 april 2013 00:46:44 UTC+2 schreef Ian Lance Taylor het volgende:
>

>> pkg-config is used to provide the correct set of options to include a
>> shared library in your program. It provides -L options to use. So it
>> could also provide -Wl,-rpath options.
>
>
> At the moment you are building a program that imports a package that was
> build with the help of pkg-config, pkg-config is already out of the
> picture. The package should now have all the necessary link info. You are
> working with a Go package now, and Go packages don't do pkg-config, like
> they don't do headers files or configuration scripts. So the package should
> contain all the information necessary for building against that package, and
> pass that information on to the build process to create an executable that
> just runs.

The linker flags specified by pkg-config are recorded in the Go
package built using cgo. Those flags are then used when the final
link is done. This can apply to both -L and -Wl,-rpath options.

In other words, if pkg-config lists both -L and -Wl,-rpath options,
then everything should work.

Ian

[Peter Kleiweg]

Op maandag 8 april 2013 22:01:12 UTC+2 schreef Ian Lance Taylor het volgende:

The linker flags specified by pkg-config are recorded in the Go
package built using cgo.  Those flags are then used when the final
link is done.  This can apply to both -L and -Wl,-rpath options.

In other words, if pkg-config lists both -L and -Wl,-rpath options,
then everything should work.

I tested that. It doesn't work. 

[Ian Lance Taylor]

That sounds like a bug. Which version of Go did you test? Current
tip or 1.0.3?

Ian

[Peter Kleiweg]

Op maandag 8 april 2013 22:26:13 UTC+2 schreef Ian Lance Taylor het volgende:

1.0.3

I also tested it a while back with an earlier version of Go 1. 

[Ian Lance Taylor]

I would hope that this would work as of revision 16465 submitted on
March 28.

If you find out that it doesn't, please file an issue. Thanks.

Ian

[Peter Kleiweg]

Op maandag 8 april 2013 22:36:16 UTC+2 schreef Ian Lance Taylor het volgende:

In tip, using pkg-config doesn't work at all. See issue 5224.

 

[Ian Lance Taylor]

Oh, right. There is an outstanding CL to fix that:
https://codereview.appspot.com/8465043/ .

Ian

[Peter Kleiweg]

Op maandag 8 april 2013 23:16:33 UTC+2 schreef Ian Lance Taylor het volgende:

That's another bug, not specific to pkg-config. Issue 5224 is a bug only with pkg-config.

[Ian Lance Taylor]

I would expect that that CL will fix the pkg-config issue too.

Ian

[Peter Kleiweg]

Op maandag 8 april 2013 23:37:15 UTC+2 schreef Ian Lance Taylor het volgende:

On Mon, Apr 8, 2013 at 2:30 PM, Peter Kleiweg <pkle...@xs4all.nl> wrote:
>
>
> Op maandag 8 april 2013 23:16:33 UTC+2 schreef Ian Lance Taylor het
> volgende:
>>
>> On Mon, Apr 8, 2013 at 2:02 PM, Peter Kleiweg <pkle...@xs4all.nl> wrote:

 

[...] 

 

>> > In tip, using pkg-config doesn't work at all. See issue 5224.
>>
>> Oh, right.  There is an outstanding CL to fix that:
>> https://codereview.appspot.com/8465043/ .
>
>
> That's another bug, not specific to pkg-config. Issue 5224 is a bug only
> with pkg-config.

I would expect that that CL will fix the pkg-config issue too.

The other issue is fixed. And it didn't fix this issue.

[hhz...@gmail.com]

"Go means to make life easier, however, it goes other direction.