Build kubernetes with gollvm

Yuan Ting

unread,

Jul 9, 2019, 3:05:20 AM7/9/19

to golang-nuts

Hi, I'm working on converting some Go projects to llvm IR for static analysis. When I build kubernetes with gollvm, an unexpected error occurred as :

I0709 10:48:29.158892   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.map_StatusDetails is not copyable
I0709 10:48:29.158899   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.map_TypeMeta is not copyable
I0709 10:48:29.158906   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.map_UpdateOptions is not copyable
I0709 10:48:29.158915   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.scheme is not copyable
I0709 10:48:29.158922   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.skipGenerated is not copyable
I0709 10:48:29.158929   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.sovGenerated is not copyable
I0709 10:48:29.158937   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.sozGenerated is not copyable
I0709 10:48:29.158944   16573 deepcopy.go:266] Type k8s.io/apimachinery/pkg/apis/meta/v1.valueToStringGenerated is not copyable
F0709 10:48:29.165681   16573 deepcopy.go:866] Hit an unsupported type invalid type.
!!! Error in ./hack/run-in-gopath.sh:33
  Error in ./hack/run-in-gopath.sh:33. '"${@}"' exited with status 255
Call stack:
  1: ./hack/run-in-gopath.sh:33 main(...)
Exiting with status 1
Makefile.generated_files:152: recipe for target 'gen_deepcopy' failed
make[1]: *** [gen_deepcopy] Error 1
Makefile:523: recipe for target 'generated_files' failed
make: *** [generated_files] Error 2

The commands I have used are

go get -d k8s.io/kubernetes # In this step I'm using the main Go compiler
cd $GOPATH/src/k8s.io/kubernetes
KUBE_VERBOSE=4 DBG_CODEGEN=1 make GOFLAGS="-x -work"     # In this step I switched to gollvm

my gollvm version is

go version go1.12.2 gollvm LLVM 9.0.0svn linux/amd64

The fatal error raised at https://github.com/kubernetes/kubernetes/blob/master/vendor/k8s.io/gengo/examples/deepcopy-gen/generators/deepcopy.go#L866

Is this an error caused by using gollvm (while the main Go compiler works fine), or an error due to other misalignments?

Thanks.

Than McIntosh

unread,

Jul 9, 2019, 1:10:55 PM7/9/19

to Yuan Ting, golang-nuts

Hi,

I'll take a look later this morning; off the top of my head I'm not sure what the issue might be.

Question: are you building with modules enabled?

Thanks ,Than

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c9572f0e-1db2-4407-8b68-bd904c928699%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Yuan Ting

unread,

Jul 9, 2019, 1:32:29 PM7/9/19

to golang-nuts

Thank you for your help. I didn't add any flags about modules when compiling gollvm or compiling k8s.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c9572f0e-1db2-4407-8b68-bd904c928699%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

Yuan Ting

unread,

Jul 9, 2019, 1:54:13 PM7/9/19

to golang-nuts

In addition, I also failed to build etcd.

$ git clone https://github.com/etcd-io/etcd.git
$ cd etcd/ && ./build

go build: when using gccgo toolchain, please pass linker flags using -gccgoflags, not -ldflags

# go.etcd.io/etcd/etcdserver/api/rafthttp

#0 0x000055ecc4cfa34a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6c34a)

#1 0x000055ecc4cf8124 llvm::sys::RunSignalHandlers() (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6a124)

#2 0x000055ecc4cf8262 SignalHandler(int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6a262)

#3 0x00007fbbd49d5890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)

#4 0x000055ecc44f0940 Export::type_index(Type const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x262940)

#5 0x000055ecc44f0c31 Export::write_type(Type const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x262c31)

#6 0x000055ecc449f2e1 Named_type::do_export(Export*) const (/home/yt/LLVMsvn/install/bin/llvm-goc+0x2112e1)

#7 0x000055ecc44f10ce Export::write_type_definition(Type const*, int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x2630ce)

#8 0x000055ecc44f1390 Export::write_types(int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x263390)

#9 0x000055ecc44fcea0 Export::export_globals(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Package*, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Package*> > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Package*, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Package*> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Import_init_set const&, Bindings const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x26eea0)

#10 0x000055ecc4457047 Gogo::do_exports() (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1c9047)

#11 0x000055ecc4441a90 go_parse_input_files(char const**, unsigned int, bool, bool) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1b3a90)

#12 0x000055ecc442ee69 gollvm::driver::CompileGoImpl::invokeFrontEnd() (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1a0e69)

#13 0x000055ecc4436ee5 gollvm::driver::CompileGoImpl::performAction(gollvm::driver::Compilation&, gollvm::driver::Action const&, llvm::SmallVector<gollvm::driver::Artifact*, 3u> const&, gollvm::driver::Artifact const&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1a8ee5)

#14 0x000055ecc442950f gollvm::driver::Driver::processAction(gollvm::driver::Action*, gollvm::driver::Compilation&, bool) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19b50f)

#15 0x000055ecc44296af gollvm::driver::Driver::processActions(gollvm::driver::Compilation&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19b6af)

#16 0x000055ecc43cf403 main (/home/yt/LLVMsvn/install/bin/llvm-goc+0x141403)

#17 0x00007fbbd386db97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0

#18 0x000055ecc442130a _start (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19330a)

Stack dump:

0. Program arguments: /home/yt/LLVMsvn/install/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=/tmp/go-build979597534=/tmp/go-build -gno-record-gcc-switches -fgo-pkgpath=go.etcd.io/etcd/etcdserver/api/rafthttp -o $WORK/b192/_go_.o -I $WORK/b192/_importcfgroot_ etcdserver/api/rafthttp/coder.go etcdserver/api/rafthttp/doc.go etcdserver/api/rafthttp/http.go etcdserver/api/rafthttp/metrics.go etcdserver/api/rafthttp/msg_codec.go etcdserver/api/rafthttp/msgappv2_codec.go etcdserver/api/rafthttp/peer.go etcdserver/api/rafthttp/peer_status.go etcdserver/api/rafthttp/pipeline.go etcdserver/api/rafthttp/probing_status.go etcdserver/api/rafthttp/remote.go etcdserver/api/rafthttp/snapshot_sender.go etcdserver/api/rafthttp/stream.go etcdserver/api/rafthttp/transport.go etcdserver/api/rafthttp/urlpick.go etcdserver/api/rafthttp/util.go

I have no idea of this failure (may be caused by CGO_ENABLED=0 in etcd/build?), please take a look together.

Thanks.

On Tuesday, July 9, 2019 at 9:10:55 PM UTC+8, Than McIntosh wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c9572f0e-1db2-4407-8b68-bd904c928699%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Tuesday, July 9, 2019 at 9:10:55 PM UTC+8, Than McIntosh wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

Than McIntosh

unread,

Jul 9, 2019, 2:03:11 PM7/9/19

to Yuan Ting, golang-nuts

That stack trace looks a lot like

https://github.com/golang/go/issues/32778

which was fixed last week. What vintage is your gollvm?

Thanks, Than

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cbdb54a2-4989-4989-8298-690b1e5a9f10%40googlegroups.com.

Yuan Ting

unread,

Jul 10, 2019, 12:03:51 AM7/10/19

to golang-nuts

My gollvm version is

commit 29005f52b3501c489cb1653506cd479d5a178e98 (HEAD -> master, origin/master, origin/HEAD)
Author: Cherry Zhang <...>
Date:   Sat Jun 29 00:25:46 2019 -0400

    bridge: support builtin memset
    
    Change-Id: I7321f57e0d58c0ff5c3a19f7cbf5721fabbf1263
    Reviewed-on: https://go-review.googlesource.com/c/gollvm/+/184439
    Reviewed-by: Than McIntosh <...>

gofrontend version is

commit 7a8e10be0ddb8909ce25a264d03b24cee4df60cc (HEAD -> master, origin/master, origin/HEAD)
Author: Cherry Zhang <...>
Date:   Wed Jul 3 15:55:19 2019 -0400

    compiler: optimize 0,1,2-case select statement
    
    For a select statement with zero-, one-, or two-case with a
    default case, we can generate simpler code instead of calling the
    generic selectgo. A zero-case select is just blocking the
    execution. A one-case select is mostly just executing the case. A
    two-case select with a default case is a non-blocking send or
    receive. We add these special cases for lowering a select
    statement.
    
    Change-Id: I519d246a4a5ba6871bb303160bba1ec1e3074bd0
    Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/184998
    Reviewed-by: Ian Lance Taylor <...>

I retry to pull gollvm from https://go.googlesource.com/gollvm but it tells me I'm already up to date.

Thanks.

Than McIntosh

unread,

Jul 10, 2019, 11:56:03 AM7/10/19

to Yuan Ting, golang-nuts

OK, thanks for checking on that.

Sounds like this is a new problem (not too surprising, since this is an area of the compiler that is undergoing a lot of changes in recent weeks; tip is a bit unstable).

I have my hands full with a couple of other bugs that I am juggling right now; filed issue https://github.com/golang/go/issues/33020 to track.

Cheers, Than

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ae1c6a36-b5ec-4ca8-b488-3e4bbe4ac775%40googlegroups.com.

Yuan Ting

unread,

Jul 10, 2019, 12:38:45 PM7/10/19

to golang-nuts

Thank you, I will continue to pay attention to it. Hope those bugs can be solved :)

Thank you again for your patience.

Than McIntosh

unread,

Jul 17, 2019, 4:21:46 PM7/17/19

to Yuan Ting, golang-nuts

Hi again,

Regarding building https://github.com/etcd-io/etcd.git with gollvm -- I have a fix for the compiler problem you ran into, but in my testing I've run into other problems that are not amenable to compiler fixes. Specifically, it looks as though some of the dependent packages for etcd are not compatible with gccgo/gollvm:

/tmp/go/pkg/mod/github.com/modern-go/refl...@v1.0.1/type_map.go:42: error: undefined reference to 'reflect.typelinks'
/tmp/go/pkg/mod/github.com/modern-go/refl...@v1.0.1/type_map.go:74: error: undefined reference to 'reflect.typelinks'
/tmp/go/pkg/mod/github.com/modern-go/refl...@v1.0.1/type_map.go:78: error: undefined reference to 'reflect.resolveTypeOff'
/tmp/go/pkg/mod/golang.org/x/s...@v0.0.0-20180909124046-d0be0721c37e/unix/gccgo.go:50: error: undefined reference to 'gccgoRealSyscallNoError'
/tmp/go/pkg/mod/golang.org/x/s...@v0.0.0-20180909124046-d0be0721c37e/unix/gccgo.go:23: error: undefined reference to 'gccgoRealSyscallNoError'
/tmp/go/pkg/mod/golang.org/x/s...@v0.0.0-20180909124046-d0be0721c37e/unix/gccgo.go:30: error: undefined reference to 'gccgoRealSyscall'
/tmp/go/pkg/mod/golang.org/x/s...@v0.0.0-20180909124046-d0be0721c37e/unix/gccgo.go:37: error: undefined reference to 'gccgoRealSyscall'
/tmp/go/pkg/mod/golang.org/x/s...@v0.0.0-20180909124046-d0be0721c37e/unix/gccgo.go:44: error: undefined reference to 'gccgoRealSyscall'
/tmp/go/pkg/mod/golang.org/x/s...@v0.0.0-20180909124046-d0be0721c37e/unix/gccgo.go:55: error: undefined reference to 'gccgoRealSyscall'
error: ld returned 1 exit status

The vendored copy of golang.org/x.sys seems to be targeting gccgo symbols that no longer exist (e.g. "gccgoRealSyscall"), and the "modern-go/reflect2" package is using unsafe to reach into the reflect package to access symbols that are only present in the main Gc version of the runtime... without work done to fix these issues the application is not going to be buildable with gollvm.

Thanks, Than

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f0b029e6-ac56-42bf-9909-e4a5db8c930e%40googlegroups.com.

Yuan Ting

unread,

Jul 18, 2019, 1:50:16 PM7/18/19

to golang-nuts

Hi Than

Thank you for your fix. I have some curious about the difference between the main Go compiler and gollvm. I read gollvm's readme and know that the main difference comes from the efficiency of GC. Is the other difference causes the symbol undefined reference as you mentioned above? Or some bugs (or library version skew) that will be fixed in the future.

Thanks, Ting

Ian Lance Taylor

unread,

Jul 18, 2019, 2:55:39 PM7/18/19

to Yuan Ting, golang-nuts

On Thu, Jul 18, 2019 at 6:50 AM Yuan Ting <yuan...@ict.ac.cn> wrote:
>
> Thank you for your fix. I have some curious about the difference between the main Go compiler and gollvm. I read gollvm's readme and know that the main difference comes from the efficiency of GC. Is the other difference causes the symbol undefined reference as you mentioned above? Or some bugs (or library version skew) that will be fixed in the future.

The gc toolchain has a different assembler that does not use the same
syntax as standard assembler for a platform
(https://golang.org/cmd/asm). GoLLVM uses the standard assembler for
whatever platform it is being used on. So Go packages that use
assembler code, like golang.org/x/sys, have to use different code to
support gc and GoLLVM. That is part of your problem. (That code does
exist in current versions of golang.org/x/sys; the build above is
showing an older version of the package; it's quite possible that
current versions would work).

The gc toolchain has a special approach for unifying type descriptors
that uses a runtime-specific symbol runtime.typelinks. GoLLVM does
not have that symbol. I don't know what github.com/modern-go/reflect2
is, but it seems to be using unsafe mechanisms to reach into the
runtime to examine that symbol. That won't work with GoLLVM. The
reflect package in general is closely tied to the compiler, so it's
possible that whatever modern-go/reflect2 is, it will not work with
GoLLVM.

Ian

Yuan Ting

unread,

Jul 19, 2019, 1:26:06 AM7/19/19

to golang-nuts

Thank you. The explanation really makes sense. I think I should modify the source code of etcd, or change to another project that is not so hackful.

Yuan Ting

unread,

Jul 19, 2019, 8:11:26 AM7/19/19

to golang-nuts

Sorry to have troubled you again, I found that many famous Go projects also import github.com/modern-go/reflect2 , such as kubernetes and moby. According to the second paragraph in your explanation, these popular projects will also failed in compilation. I think this may be a big challenge for gccgo/gollvm's promotion.

To solve this compiler-sensitive problem, the way I can come up with is replacing all unsafe mechanisms as mentioned above in reflect2 to safe or standard mechanisms (and may go against the purpose of reflect2). I'm not familiar with runtime, but is it possible to support the same symbol in gccgo/gollvm style?

Best,

Ting

On Thursday, July 18, 2019 at 10:55:39 PM UTC+8, Ian Lance Taylor wrote:

Ian Lance Taylor

unread,

Jul 19, 2019, 2:30:32 PM7/19/19

to Yuan Ting, golang-nuts

On Fri, Jul 19, 2019 at 1:11 AM Yuan Ting <yuan...@ict.ac.cn> wrote:
>
> Sorry to have troubled you again, I found that many famous Go projects also import github.com/modern-go/reflect2 , such as kubernetes and moby. According to the second paragraph in your explanation, these popular projects will also failed in compilation. I think this may be a big challenge for gccgo/gollvm's promotion.
>
> To solve this compiler-sensitive problem, the way I can come up with is replacing all unsafe mechanisms as mentioned above in reflect2 to safe or standard mechanisms (and may go against the purpose of reflect2). I'm not familiar with runtime, but is it possible to support the same symbol in gccgo/gollvm style?

Simply supporting runtime.typelinks is not enough. The reflect
package is closely tied to the compiler implementation. The reflect
package is not identical in the gc library and the GoLLVM library. If
people need reflect2 to work with GoLLVM, then the only workable
option is for the reflect2 authors to write a version of their package
that uses the gccgo build tag to build a version that works with
GoLLVM. Perhaps you could help them with that. Sorry this isn't very
helpful. I don't really understand why people are using the package.

Ian

Yuan Ting

unread,

Jul 20, 2019, 2:32:03 AM7/20/19

to golang-nuts

OK, I will check out the reflect2 package and take a look. Thanks a lot for your direction!

Best,

Ting

Ivan Serdyuk

unread,

Jul 29, 2020, 3:56:11 AM7/29/20

to golang-nuts

Yuan,

are you still working on gollvm based port of k8s?

Ivan

Yuan Ting

unread,

Jul 29, 2020, 6:09:57 AM7/29/20

to golang-nuts

Hi Ivan,

Yes, I still try to build famous projects by gollvm but there are still some troubles (such as https://groups.google.com/d/msg/golang-nuts/2fIk7hbbYJc/zIh3B4tbAAAJ )

The trouble mentioned here is gccgo compatibility of reflect2 (a package imported by k8s). The developers of reflect2 give an imperfect solution (https://github.com/modern-go/reflect2/pull/10) to avoid linkage errors. I think this patch may help to build k8s.

Ivan Serdyuk

unread,

Sep 26, 2020, 11:33:15 AM9/26/20

to Yuan Ting, Brendan Burns, golang-nuts

Brendan,

could you review this and express your vision against the usage of

https://godoc.org/k8s.io/kubernetes/third_party/forked/golang/reflect

https://golang.org/pkg/reflect/

https://github.com/modern-go/reflect2

packages?

What is required to patch, within the reflect2 package, so Kubernetes could be compiled via gollvm?

Ivan

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1fd95c73-6e30-4089-bad4-b21c261d788a%40googlegroups.com.

nanzi yang

unread,

Jan 29, 2024, 2:21:30 AM1/29/24

to golang-nuts

Hello Yuan Ting:
I am trying to build Kubernetes with gollvm, and I am glad you have made some progress. Did you finish compiling Kubernetes with gollvm and generating IR? If it is true, could you show me some practical steps so I can compile the Kubernetes to LLVM IR files? Looking forward to your reply.