Building on DragonFly BSD

[Venkatesh Srinivas]

Hi,

I've convinced go to build on DragonFly BSD - the toolchain runs and
produces executables, but the executables run on Linux, not DragonFly
(there's no runtime yet). I built with GOOS=linux, GOARCH=386. This
process (and this email) is a bit messy at the minute, but I hope its
of some use.

Changes required:
- Modified src/all.bash, {src,src/cmd}/make.bash, and
{src,src/cmd}/clean.bash to not require bash.
- Switch from using 'make' to 'gmake' in a number of places (primarily
{src,src/cmd}/make.bash
- Stubbed out a src/libmach/dragonfly.c
- Remove cov and prof from make.bash, as they depend on a real libmach.
- Switch from 'bison' to 'yacc' in src/cmd/cc/Makefile and
src/cmd/gc/Makefile. I was unable to get src/cmd/8a to build when I
used the system yacc instead of bison. I punted on making it work and
used Inferno's iyacc (iyacc -D1 -d a.y) instead to build a.y; using
Inferno's yacc meant I needed to remove the header comment in a.y, as
// is not a valid comment character.
- In src/pkg/runtime/linux/signals.h, there is "#define C SigCatch".
8c wasn't buying this, so I replaced C with SigCatch in the table.

What's left before running go apps natively: (possibly of interest for
anyone doing any other ports as well)
- src/libmach/dragonfly.c is currently a stub. It needs to implement
these functions: attachproc()/detachproc(), procthreadpids(),
procnotes(), ctlproc(), proctextfile(), and procstatus().

A good starting point would be
http://swtch.com/usr/local/plan9/src/libmach/FreeBSD.c. I don't know
if this is required to run programs; I doubt it. It is required for
cov and prof, however. Here is a start of one:
http://grex.org/~vsrinivas/src/libmach-dragonfly.c

- arch-neutral runtime (pkg/runtime/dragonfly/*).
thread.c will need to provide lock() / unlock(), noteclear() /
notewakeup() / notesleep(), newosproc(), and minit().
newosproc() could just wrap rfork(), though the system libthread
doesn't (it uses lwp_create() instead). minit() just needs to set up a
signal stack. signals.h and os.h look fairly ordinary; sigtab[] is
just

Maybe thread.c ought use pthreads instead of the OS-specific
threading? We can use pthread_mutex for lock/unlock, pthread_create
for newosproc...

- arch-specific runtime (pkg/runtime/dragonfly/386/*)
defs.h is autogenerated. signal.c provides the C-half of the signal
handlers, which mostly print registers & stacks in bad cases. sys.s
will have to provide wrappers for all of the system calls we use and
to provide a caught-signal handler. Oh joy.

- 8l (src/cmd/8l/obj.c) needs to learn how to generate BSD-friendly
ELF files. -H7 (currently default on Linux) produces Linux ELFs. I
have no idea what's involved with this.

- we need a src/libcgo/dragonfly_386.c, which would know where in the
TLS to stick ThreadStart.m and ThreadStart.g. This might not be
mandatory, we might need this only if we are using libcgo?

- ???

Am I missing anything?

Happy hacking,
-- vs

[Russ Cox]

On Wed, Nov 11, 2009 at 04:04, Venkatesh Srinivas <m...@acm.jhu.edu> wrote:
> I've convinced go to build on DragonFly BSD - the toolchain runs and
> produces executables, but the executables run on Linux, not DragonFly
> (there's no runtime yet). I built with GOOS=linux, GOARCH=386. This
> process (and this email) is a bit messy at the minute, but I hope its
> of some use.

Nice. I realize you were going for quick and functional, but there
are some long term things that I'd suggest for a solid port to DragonFly.

> Changes required:
> - Modified src/all.bash, {src,src/cmd}/make.bash, and
> {src,src/cmd}/clean.bash to not require bash.

Long term, I think it makes sense to require bash for the install process.
Saying /bin/sh is tempting but frustratingly non-standard on Linux,
especially since the most recent Debian/Ubuntu systems have started
shipping sh == dash instead of sh == bash. I never want to debug
another dash/bash inconsistency again. The problem with saying /bin/sh
is that it will work for all of us and then someone will come along with
yet another different shell installed as /bin/sh and break for them.
Bash, for all its faults, is much more reliably the same.

> - Switch from using 'make' to 'gmake' in a number of places (primarily
> {src,src/cmd}/make.bash

Long term, we need to figure out a better way here too.
On Linux, there is typically no gmake binary. To get GNU make
you just run "make". The solution is probably to use $(MAKE)
in the Makefiles (do both versions set that automatically?)
and do a small bit of sniffing in make.bash, etc.

> - Stubbed out a src/libmach/dragonfly.c
> - Remove cov and prof from make.bash, as they depend on a real libmach.

It's fine to leave libmach/dragonfly.c for far down the road.
If you stub out a bit more of the functions, you should be able
to compile cov and prof and get binaries that just fail when
they run (presumably printing an error about libmach not
being implemented). Then you won't need to edit the make.bash.

> - Switch from 'bison' to 'yacc' in src/cmd/cc/Makefile and
> src/cmd/gc/Makefile. I was unable to get src/cmd/8a to build when I
> used the system yacc instead of bison. I punted on making it work and
> used Inferno's iyacc (iyacc -D1 -d a.y) instead to build a.y; using
> Inferno's yacc meant I needed to remove the header comment in a.y, as
> // is not a valid comment character.

For the same reasons as in the discussion of bash above, it is
likely better to just require people to install bison to build.
We used to use yacc here, but like /bin/sh, yacc has different
meanings on different systems, and it is amazingly frustrating
to try to keep up with them all.

> - In src/pkg/runtime/linux/signals.h, there is "#define C SigCatch".
> 8c wasn't buying this, so I replaced C with SigCatch in the table.

Perhaps this is an artifact of using yacc/iyacc? It certainly works
in the Linux build.

> What's left before running go apps natively: (possibly of interest for
> anyone doing any other ports as well)
> - src/libmach/dragonfly.c is currently a stub. It needs to implement
> these functions: attachproc()/detachproc(), procthreadpids(),
> procnotes(), ctlproc(), proctextfile(), and procstatus().
>
> A good starting point would be
> http://swtch.com/usr/local/plan9/src/libmach/FreeBSD.c. I don't know
> if this is required to run programs; I doubt it. It is required for
> cov and prof, however. Here is a start of one:
> http://grex.org/~vsrinivas/src/libmach-dragonfly.c

It's not required to run programs, only to run cov and prof.

> - arch-neutral runtime (pkg/runtime/dragonfly/*).
> thread.c will need to provide lock() / unlock(), noteclear() /
> notewakeup() / notesleep(), newosproc(), and minit().
> newosproc() could just wrap rfork(), though the system libthread
> doesn't (it uses lwp_create() instead). minit() just needs to set up a
> signal stack. signals.h and os.h look fairly ordinary; sigtab[] is
> just
>
> Maybe thread.c ought use pthreads instead of the OS-specific
> threading? We can use pthread_mutex for lock/unlock, pthread_create
> for newosproc...

Because of the different calling conventions and binary formats,
the Go binaries don't link against the standard system libraries,
which means it can't use the pthread library implemementations here.
The amount of code here is very small.

> - arch-specific runtime (pkg/runtime/dragonfly/386/*)
> defs.h is autogenerated. signal.c provides the C-half of the signal
> handlers, which mostly print registers & stacks in bad cases. sys.s
> will have to provide wrappers for all of the system calls we use and
> to provide a caught-signal handler. Oh joy.

It's not too bad: once you've done one, you've done them all.

> - 8l (src/cmd/8l/obj.c) needs to learn how to generate BSD-friendly
> ELF files. -H7 (currently default on Linux) produces Linux ELFs. I
> have no idea what's involved with this.

It is likely only a matter of changing the OS type in the ELF header.

> - we need a src/libcgo/dragonfly_386.c, which would know where in the
> TLS to stick ThreadStart.m and ThreadStart.g. This might not be
> mandatory, we might need this only if we are using libcgo?

Right, you can skip libcgo on the first time through.
It is only need when using cgo (the FFI support).

Russ

[Ben Bullock]

I was trying to build Go on FreeBSD so I'm chipping in here rather
than starting a new thread.

On Nov 12, 4:19 pm, Russ Cox <r...@golang.org> wrote:

> > Changes required:
> > - Modified src/all.bash, {src,src/cmd}/make.bash, and
> > {src,src/cmd}/clean.bash to not require bash.
>
> Long term, I think it makes sense to require bash for the install process.

In this case, it would help a lot if there was some way to specify the
location of bash. Right now it's necessary to do something like

find . -name "*.bash" | xargs perl -pi -e 's:#!/bin/bash:#!usr/local/
bin/bash:'

at the top of the build directory.

> Saying /bin/sh is tempting but frustratingly non-standard on Linux,

What about having a configure script in /bin/sh which dies if it can't
find bash, and the actual work scripts in bash?

> > - Switch from using 'make' to 'gmake' in a number of places (primarily
> > {src,src/cmd}/make.bash
>
> Long term, we need to figure out a better way here too.
> On Linux, there is typically no gmake binary.  To get GNU make
> you just run "make".  The solution is probably to use $(MAKE)
> in the Makefiles (do both versions set that automatically?)
> and do a small bit of sniffing in make.bash, etc.

Again this is something that is easy to fix.

find . -name "*.bash" | xargs perl -pi -e 's:\bmake:gmake:'
find . -name "*.bash" | xargs perl -pi -e 's:\bgmake
\.bash:make.bash:'

fixed all the make/gmake problems, and if one thought about it
somewhat, perhaps one of these substitutions could be eliminated.

> For the same reasons as in the discussion of bash above, it is
> likely better to just require people to install bison to build.

Why can't you just distribute the C output file from Bison with the Go
distribution so that it doesn't need to be rebuilt by everyone? Unless
Bison is going to make a different output for different OSes or
configurations?

[Russ Cox]

> Why can't you just distribute the C output file from Bison with the Go
> distribution so that it doesn't need to be rebuilt by everyone? Unless
> Bison is going to make a different output for different OSes or
> configurations?

At this stage in Go's development we are trying to keep the
amount of precomputed packaging to a minimum, so that
there are fewer pieces to keep in sync in the repository.
Your suggestion makes a lot of sense once we get to a point
when we have more substantial release milestones, but
it's very early yet. For now, we hope that it isn't too much
trouble for people to install bison, which is a very common
and small program.

Russ

[Ian Lance Taylor]

I think the above are good suggestions; do you want to try to write
some patches and contribute them? In my experience, it tends to be
easier for the people encountering these problems to write patches
than it is for others to work out what the right patches should be.

>> For the same reasons as in the discussion of bash above, it is
>> likely better to just require people to install bison to build.
>
> Why can't you just distribute the C output file from Bison with the Go
> distribution so that it doesn't need to be rebuilt by everyone? Unless
> Bison is going to make a different output for different OSes or
> configurations?

The main issue here is remembering to update the generated output when
the input changes. If we can make that automatic somehow then I think
this would be fine.

Ian

[Russ Cox]

> I think the above are good suggestions; do you want to try to write
> some patches and contribute them?  In my experience, it tends to be
> easier for the people encountering these problems to write patches
> than it is for others to work out what the right patches should be.

There's already a patch in flight that tries to address these problems.
If you wait a day, the build process at least should be fixed on those
systems. There may even be a FreeBSD implementation. It seems
to be very close.

http://codereview.appspot.com/user/dho

Russ

[Ben Bullock]

On Nov 15, 5:03 am, Russ Cox <r...@golang.org> wrote:

> There's already a patch in flight that tries to address these problems.
> If you wait a day, the build process at least should be fixed on those
> systems.  There may even be a FreeBSD implementation.  It seems
> to be very close.

The build scripts seem to be working now, but the build breaks down
here:

--------------------

%%%% making cov %%%%

/home/ben/software/go/src/cmd/cov
gmake: *** No rule to make target `install-freebsd', needed by
`install'. Stop.

---------------------

I am using the following FreeBSD (output of "uname -a"):

FreeBSD mikan 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Sun May 17 20:22:54
JST 2009 root@:/usr/obj/usr/src/sys/GENERIC i386

I am also having some problems with Mercurial on FreeBSD like this:

hg clone -r release https://go.googlecode.com/hg/ $GOROOT
abort: could not import module thread!
Exception AttributeError: "'httpsrepository' object has no attribute
'urlopener'" in <bound method httpsrepository.__del__ of
<mercurial.httprepo.httpsrepository object at 0x28541b4c>> ignored

This Mercurial is 1.3.1 installed via "make install" in /usr/ports/
devel/mercurial. My ports are up to date and googling for the above
gibberish doesn't give anything. Is anyone else seeing this problem?

It seems to work on Linux so I made a tarball from there.

[Russ Cox]

>> There's already a patch in flight that tries to address these problems.
>> If you wait a day, the build process at least should be fixed on those
>> systems.  There may even be a FreeBSD implementation.  It seems
>> to be very close.
>
> The build scripts seem to be working now, but the build breaks down
> here:
>
> --------------------
>
> %%%% making cov %%%%
>
> /home/ben/software/go/src/cmd/cov
> gmake: *** No rule to make target `install-freebsd', needed by
> `install'.  Stop.

The FreeBSD implementation is not in yet, only the fixes
to allow the build to run without assuming bash is /bin/bash.
If you'd like a notification when the FreeBSD implementation is in,
please follow http://code.google.com/p/go/issues/detail?id=166.

Russ