cgo on windows

[Russ Cox]

What is the status of cgo on Windows (issue 1741)?
I know that there are some issues with missing symbols.
Alex has reported missing chkstk, which I assume is for
gcc's stack buffer overflow checks, and I have tried to
run cgo on my Windows 7 install and gotten errors about
missing _assert.

What is the official way to use cgo? I know you need mingw
installed, but I do have mingw installed and that does not
seem to be enough.

What else is necessary? If we can collect that information here,
then we can turn it into a document.

Thanks.
Russ

[minux]

On Thu, Mar 8, 2012 at 4:21 AM, Russ Cox <r...@golang.org> wrote:

What is the status of cgo on Windows (issue 1741)?
I know that there are some issues with missing symbols.
Alex has reported missing chkstk, which I assume is for

I've got a workaroud for this, CL 5786043.

I can enable misc/cgo/test on Windows now.

But I can only test on windows/386, could anyone please help me

test on windows/amd64?

gcc's stack buffer overflow checks, and I have tried to
run cgo on my Windows 7 install and gotten errors about
missing _assert.

What's your test program?

Right now, I find msic/cgo/life can be passed, but misc/cgo/stdio has failed.

(seems cgo doesn't detect that "stdio" is in fact a macro? or missing include

file?)

[brainman]

On Thursday, March 8, 2012 7:21:05 AM UTC+11, rsc wrote:
> What is the status of cgo on Windows (issue 1741)?

About misc/cgo tests. There are 4 tests:

1) life - working now, we just need little batch file so windows gobuilder can exercise it; I will do that, if no one bits me to it;

2) test - failed for me

C:\MinGW\go\misc\cgo\test>go test
# testmain
C:\DOCUME~1\brainman\LOCALS~1\Temp\go-build185558154\_\C_\MinGW\go\misc\cgo\test\_test/_/C_/MinGW/go
/misc/cgo/test.a(C:\DOCUME~1\brai)(.text): __chkstk: not defined
__chkstk(0): not defined
FAIL _/C_/MinGW/go/misc/cgo/test [build failed]

it used to work, but we had some Makefile magic to link particular libs to it. I will look at http://codereview.appspot.com/5786043/ to see if it fixes it;

3) stdio - never worked before; from what I remember, it was to do with how mingw gcc was treating stdin/stdout/stderr variables - 8l could not find them at the end;

6) testso - never worked before; I didn't look properly, but I am not sure, if it can be done with mingw (I am not expert here);

> ..., and I have tried to

> run cgo on my Windows 7 install and gotten errors about
> missing _assert.
>

Show us you error. My experience was that different versions of mingw gcc (including 32 or 64 bit) did matter in how things worked: different lib locations and like.

> What is the official way to use cgo?  I know you need mingw
> installed, but I do have mingw installed and that does not
> seem to be enough.

You should not need nothing but mingw gcc. Like I said, I can build and run misc/cgo/life.

Alex

[Daniel Theophanes]

I'm using TMD-gcc 64-bit to compile go.  Go appears to work fine, but I get the same errors as rsc (assert not found) when I try to compile life.  When I remove the refs to assert, it compiles but it panics at runtime/proc.c:221.

Alex, what version of mingw are you using (32-bit, not TDM)?

[brainman]

On Thursday, March 8, 2012 1:13:10 PM UTC+11, Daniel Theophanes wrote:

Alex, what version of mingw are you using (32-bit, not TDM)?

32 bit:

$ gcc --version
gcc.exe (tdm-1) 4.5.2
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

64 bit:

$ gcc --version
gcc.exe (GCC) 4.7.0 20111212 (experimental)
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Alex

[Daniel Theophanes]

Thanks.  I switched over to gcc 32-bit, re-compiled the tree, and life works now with gcc (GCC) 4.6.1.

The 64-bit version that wasn't working was: gcc (tdm64-1) 4.6.1.

And sweet, with cgo working properly, looks like I may have a working go windows service :)

-Daniel

[minux]

OK, I found the real cause for the missing symbols (e.g., __chkstk/_assert).

Normally mingw gcc static link some static library behind our back, so we have two options:

1. force gcc to dynamic linking libgcc/libmingw, etc., then cgo will find the appropriate library for __chkstk.

I've tried this approach, but found out that despite -shared-libgcc, gcc will always static link __chkstk

from libgcc, could anybody tell me any option to really force it to be dyanmic linked?

As a side note, because libgcc.dll won't be installed on normal Windows machines, I don't think this

is the correct way to go.

2. we need a way include libgcc.a/libmingw.a into our linker.

This is the only route we can go now.

Any opinions?

[Joseph Poirier]

AFAIK, __chkstk is from the MSVC runtime not gcc. Is __chkstk being
resolved with the dynamic linking you mention because there's a msvc
library being pulled in? You can suppress __chkstk with the /Gs option
under MSVC but no idea what the gcc equivalent is, or if there is one.

http://support.microsoft.com/kb/100775

-joe

[Joseph Poirier]

I just found a reference to this option -mno-stack-arg-probe

[minux]

Sorry about that. But this detail doesn't matter here. The real problem is:

gcc are doing some static linking behind our back, and what's worse,

some function is defined in these static libraries (that is not all functions

are imp_* function for dlls [which cmd/ld can generate by itself]).

I think this is a major impedance mismatch problem, and I cannot think of

any elegant solution (except from parse linker map files).

[Russ Cox]

On Thu, Mar 8, 2012 at 10:07, minux <minu...@gmail.com> wrote:
> Sorry about that. But this detail doesn't matter here. The real problem is:
> gcc are doing some static linking behind our back, and what's worse,
> some function is defined in these static libraries (that is not all
> functions
> are imp_* function for dlls [which cmd/ld can generate by itself]).
>
> I think this is a major impedance mismatch problem, and I cannot think of
> any elegant solution (except from parse linker map files).

If we pass -fpic then there should be no static linking involved.
Unfortunately, we don't pass -fpic on Windows because then
the compiler complains that it is redundant.

[Joseph Poirier]

Note sure if this is helpful but from > gcc -v --help in ld.exe
section it lists

-Bdynamic, -dy, -call_shared Link against shared libraries

[minux]

Let me give a concrete example:

long long f(long long x, long long y) { return x / y; }

This simple function compiled by gcc on 32-bit platforms will pull in __divdi3 from libgcc.a.

What do we do to support linking this function into Go executables?

On Linux/FreeBSD/Mac OS X, this is pretty easy, we can dynamic link in libgcc.so,

because we can be sure it is available (but sad truth is, currently the user have to

explicitly handle this, we don't pass -shared-libgcc to gcc).

But on Windows, we can't do that because we can't be sure if the user of this cgo 

program have libgcc.dll available.

To make matters worse, mingw automatically link in some non-standard static libraries, 

like libmingw32 (still worse, they do not even have a dynamic counterpart).

You can see the original http://weekly.golang.org/misc/cgo/life/Makefile for some "ugly"

workarounds for this problem.

In general, to solve this problem, we need the ability to link in some (gcc) static library (this

is the easy part), and a way to find out which one needs to be linked in on Windows (which,

I think, is quite difficult).

[Russ Cox]

Where is libgcc.a? What is in it? What is its license?
Can we just copy it into runtime/cgo.a and be done with it?

Russ

[minux]

On Fri, Mar 9, 2012 at 1:00 AM, Russ Cox <r...@golang.org> wrote:

Where is libgcc.a?  What is in it?  What is its license?

Some helper functions for gcc when the target architecture doesn't provide native support

for some operation gcc needs (for example, long long divide on i386 systems)

We can get the location of it easily: gcc -print-libgcc-file-name

Its license is (for recent gcc version) GPL v3 with GCC Runtime Library Exception.

IMO, we can copy it into runtime/cgo.a, but as I'm not an lawyer, I think we should

ask iant about this.

Can we just copy it into runtime/cgo.a and be done with it?

Oh, licensing issues aside, this is a good idea. On windows, we can even copy in libmingw32.a

and its friends. I will experiment this right now.

(As they are meant to be static linked, I assume their license terms are more permissive

than GPL, I will look into it)

[Ian Lance Taylor]

Russ Cox <r...@golang.org> writes:

> Where is libgcc.a? What is in it? What is its license?
> Can we just copy it into runtime/cgo.a and be done with it?

libgcc.a is under a GPL+exception license. Copying it and maintaining
the copy would be a pain because it is built in a funny way.

__chkstk is always statically linked because it is called in a special
way that preserves the argument registers. Dynamically link it might
cause the dynamic call to clobber those registers, which would of course
break the program.

I'm not sure what the command is that is failing. For cgo's purposes
all we really need to know is that __chkstk is special. So one approach
is to just make it special: recognize it in cgo. Another approach is to
explicitly link against -lgcc; since cgo already assumes that gcc is
available, it seems to me that that should always be fine.

Ian

[minux]

On Fri, Mar 9, 2012 at 1:35 AM, Ian Lance Taylor <ia...@google.com> wrote:

Russ Cox <r...@golang.org> writes:

> Where is libgcc.a?  What is in it?  What is its license?
> Can we just copy it into runtime/cgo.a and be done with it?

libgcc.a is under a GPL+exception license.  Copying it and maintaining

I think by copy rsc means we first 'ar x /usr/lib/libgcc.a' and then 'go tool pack grc ../runtime/cgo.a *.o'

the copy would be a pain because it is built in a funny way.

__chkstk is always statically linked because it is called in a special
way that preserves the argument registers.  Dynamically link it might
cause the dynamic call to clobber those registers, which would of course
break the program.

Thanks for the clarification. Is there any other functions in libgcc.a that is

special?

[Joseph Poirier]

Something doesn't add up... Is there a mismatch between the function
and object name (too many underscores)?

-joe

[minux]

On Fri, Mar 9, 2012 at 1:00 AM, Russ Cox <r...@golang.org> wrote:

Where is libgcc.a?  What is in it?  What is its license?
Can we just copy it into runtime/cgo.a and be done with it?

I've tried this on Linux/amd64. Here's my result:

1. unpack libgcc.a and integrate them into runtime/cgo.a

mkdir t

cd t

ar x `gcc -m64 -print-libgcc-file-name`

go tool pack grc $GOROOT/pkg/linux_amd64/runtime/cgo.a *.o

2. test with this simple program:

package main

/*

int f(int x){return __builtin_popcount(x);} // it will call __popcountdi2

*/

import "C"

func main() {

        println(C.f(100))

}

Unfortunately, when 'go build test.go', I got these:

# command-line-arguments

/tmp/go-build232027963/command-line-arguments.a(t.cgo2.o)(.text): __popcountdi2: not defined

__popcount_tab(0): not defined

__fixunsxfti(0): not defined

__floattixf(0): not defined

__fixunssfti(0): not defined

__fixunsdfti(0): not defined

fflush(0): not defined

__clz_tab(0): not defined

__popcountdi2(0): not defined

I even tried to add only _popcountsi2.o and _popcount_tab.o, but still no success.# command-line-arguments

/tmp/go-build344259576/command-line-arguments.a(t.cgo2.o)(.text): __popcountdi2: not defined

__popcount_tab(0): not defined

__popcountdi2(0): not defined

I think it's too late to fix these error in cmd/ld before Go 1, but I will try if it is the right thing to do.

Or, may be I just missed somthing?

[Joseph Poirier]

Would making mingw gcc link against the msvc libraries (I know it's
possible) instead of its own help?

[minux]

On Fri, Mar 9, 2012 at 3:33 AM, Joseph Poirier <jdpo...@gmail.com> wrote:

Would making mingw gcc link against the msvc libraries (I know it's

possible) instead of its own help?

For functions in libgcc, I think the answer is no, because gcc and msvc surely

need different helper function and naming conventions.

As shown in http://codereview.appspot.com/5786043/, the __chkstk problem has a

workaround. But the general problem of using libgcc and other libmingw32 functions

still doesn't solved.

[Joseph Poirier]

Linking mingw-built C programs against msvcrt.dll isn't a problem, but
like you said, libgcc would be required if gcc extensions (built-ins)
were used.

[Joseph Poirier]

To compile an executable it's just: gcc test.c -nostdlib -lmsvcrt -o test.exe
But I think main has to be named _main, it's been a while.

[Ian Lance Taylor]

minux <minu...@gmail.com> writes:

> Thanks for the clarification. Is there any other functions in libgcc.a that
> is
> special?

I wouldn't be surprised but I don't know of any others.

Ian

[Joseph Poirier]

FYI

cgo/test passes when including msvcrt with the CL
http://codereview.appspot.com/5786043/

a = append(a, "-lmsvcrt", "-mno-stack-arg-probe", "-fno-stack-check",
"-fno-stack-protector")

-joe

[Joseph Poirier]

I played around with this for several hours, changed some code, and
managed to get mingw-gcc-msvcrt working for about 60% of what I
compiled. :( O well.

[minux]

On Fri, Mar 9, 2012 at 8:07 AM, Joseph Poirier <jdpo...@gmail.com> wrote:

cgo/test passes when including msvcrt with the CL

http://codereview.appspot.com/5786043/

a = append(a, "-lmsvcrt", "-mno-stack-arg-probe", "-fno-stack-check",
"-fno-stack-protector")

I think -lmsvcrt is not needed here. If you delete it, misc/cgo/test could still

pass the tests.

[minux]

On Fri, Mar 9, 2012 at 4:15 AM, Joseph Poirier <jdpo...@gmail.com> wrote:

Linking mingw-built C programs against msvcrt.dll isn't a problem, but

like you said, libgcc would be required if gcc extensions (built-ins)
were used.

Correction, not necessarily gcc extension (built-ins), for example:

long long f(long long x, long long y) { return x / y; }

Compile this function on i386 machine will pull in __divdi3 from libgcc,

because i386 doesn't have 64-bit divide instructions, so gcc must use

a function to provide it, and this function resides in libgcc.

[minux]

I think it is a general issue not only on Windows, so I filed issue 3261.

[Joseph Poirier]

Makes sense.

I realized after I posted that cgo/test passed with just the CL applied.

Being able to use the msvcrt dll on a windows machine rather then
having to include mingw-gcc libs would be nice but it's not worth the
trouble at this point, nor does it solve the existing problem. I
didn't quite understand what was happening until I started
experimenting, sorry for all the previous noise.

-joe