Re: code review 6060047: linnix: an inferno/nix hybrid for bootstrapping Nix on ... (issue 6060047)

[rmin...@gmail.com]

I hate to carp, but I'm going to have to. This is a 1000-file patch that
by its own admission does not work, meaning reviewers can not test it.

I understand the goal is to make NIX more accessible to non-Plan 9
users. I think that's a good goal. I just don't see how this approach
gets us there.

I'd like to propose a different approach. Shells all provide the same
basic primitives, as do make commands. They can spawn processes,
redirect IO, set up pipes. So take a standard NIX source tree, and aim
for the lowest common denominator. Create mkfiles that gmake can process
as well as mk. Or at least make mkfiles that look like:
</$objtype/mkfile
<make.common
and
include $objtype/makefile
include make.common

(This is what Russ proposed when I tried to put mkfiles into Go. He made
it clear it was not acceptable to have two entirely parallel sets of
recipes, unless the one simply included the other. He was right.)

Same with shell scripts. Get rid of constructs that are not common to sh
and rc. If you've got a for loop over 5 directories, which we do, then
just break it out into five sets of commands. Yes, ugly, but I think two
parallel sets of mkfiles/makefile and rc scripts/sh scripts is going to
be much harder to deal with.

Failing that, I think this set of changes needs to come in gradually, in
working form, in a way that we can test it. This patch is more than I
can understand. Sorry, I'm slow.

Every file you add should be considered another barrier to acceptance in
other communities. Hence, for every file you add, you might want to show
how you are removing a file.

This is a neat idea, but it needs another iteration, and I think it has
to be added in an incremental manner in a way in which everything always
keeps working. That was an original goal of NIX anyway -- everything
keeps working.

Another approach, which I'm more comfortable with, would be to change 6?
so they build under both gcc and Plan 9 -- easy, been done, see Go! --
and then use those to build the rest of the binaries. I still think that
can work.


http://codereview.appspot.com/6060047/

[Noah Evans]

That was the initial stab. There's a new version that works. If I get
permission to release to the wild, then I'll bitbucket it and then we
can think about it how to deal with it from there.

As for getting rid of the sh/rc dependencies, I think at the end of
the day you should probably include rc in the distro and then do
MKSHELL=rc (suggested by Russ).

At the end of the day I think I'd propose something more radical,
which is doing away with mk and shells entirely and doing the build
system in go. Much of the complexity in the Nix build system is in
mk/parse, which is not very accessible. Go templates make this a lot
easier. For dependency analysis and compiling much of the "go" command
could be repurposed there.

Noah

[rmin...@gmail.com]

On 2012/04/26 07:38:35, npe wrote:
> That was the initial stab. There's a new version that works.

Then we definitely want to abandon this patch. It's huge and if it's
just a stab I think it will hurt too much.

> As for getting rid of the sh/rc dependencies, I think at the end of
> the day you should probably include rc in the distro and then do
> MKSHELL=rc (suggested by Russ).

sounds good. This seems as though it would be a non-hurtful change to
the current source. Why not start here, with that
simple change, and see how far we get?

> At the end of the day I think I'd propose something more radical,
> which is doing away with mk and shells entirely and doing the build
> system in go. Much of the complexity in the Nix build system is in
> mk/parse, which is not very accessible. Go templates make this a lot
> easier. For dependency analysis and compiling much of the "go" command
> could be repurposed there.

if we're doing that then why add these 1000 files? This is too large a
change if it's temporary IMHO.

ron

http://codereview.appspot.com/6060047/

[noah....@gmail.com]

*** Abandoned ***

http://codereview.appspot.com/6060047/

[Noah Evans]

Good point. Gone.

It was easier initially to just do things the inferno way. I was able
to get around most of the mkfiles by just transliterating the shell
code into /bin/sh. I'm not sure if the plan9port rc or inferno rcsh is
the best way to go on the shell though.

Akshat, what is the critical path on go these days?

Noah

[Charles Forsyth]

actually, the Inferno ones (which Noah was using, I think) do just that.

[Noah Evans]

I was. Is that what you were really saying Ron? ie. compile 6[lca]
with gcc/2c? Or did you mean something different?

Noah

[Akshat Kumar]

On 26 April 2012 12:37, Noah Evans <noah....@gmail.com> wrote:
> Akshat, what is the critical path on go these days?

Once my tsemacquire and fmt patches are
accepted to Plan 9, one only needs two
CLs: 6031056 and 5608059 for a Go that
builds and runs. I can go on about what
needs to be done, but for your purposes,
that should be all you need in order to use
the build system.

On 26 April 2012 13:41, Charles Forsyth <charles...@gmail.com> wrote:
> actually, the Inferno ones (which Noah was using, I think) do just that.

Which is probably why the Go team chose the
Inferno suite as a basis for their own.

[Noah Evans]

Did you submit the patches through patch(1) already? If so how long
have you been waiting?

Charles might know a little bit about the Inferno compilers and how
they got into go. ;)

Noah

[Akshat Kumar]

On 26 April 2012 14:22, Noah Evans <noah....@gmail.com> wrote:
> Did you submit the patches through patch(1) already? If so how long
> have you been waiting?

Yes, I did; I'm working on a few things
jmk needs. :-)

> Charles might know a little bit about the Inferno compilers and how
> they got into go. ;)
>
> Noah

I know quite well that he knows this. :-)
I was just giving an observation for others.

[Noah Evans]

Hah. What is your timeframe on the go stuff? That might change my
plans considerably.

Noah


On Thu, Apr 26, 2012 at 11:35 PM, Akshat Kumar

[Devon H. O'Dell]

Op 26 april 2012 17:40 heeft Noah Evans <noah....@gmail.com> het
volgende geschreven:

> Hah. What is your timeframe on the go stuff? That might change my
> plans considerably.

I haven't been following the Plan 9 Go port much, but happy to help
with it if anything's needed.

--dho

[Christopher Nielsen]

On Thu, Apr 26, 2012 at 14:48, Devon H. O'Dell <devon...@gmail.com> wrote:
> Op 26 april 2012 17:40 heeft Noah Evans <noah....@gmail.com> het
> volgende geschreven:
>> Hah. What is your timeframe on the go stuff? That might change my
>> plans considerably.
>
> I haven't been following the Plan 9 Go port much, but happy to help
> with it if anything's needed.
>
> --dho

I've only been glancing at the Plan 9 go port, but the same is true
for me. If you need any help, give a holler.

--
Christopher Nielsen
"They who can give up essential liberty for temporary safety, deserve
neither liberty nor safety." --Benjamin Franklin
"The tree of liberty must be refreshed from time to time with the
blood of patriots & tyrants." --Thomas Jefferson

[ron minnich]

On Thu, Apr 26, 2012 at 1:56 PM, Noah Evans <noah....@gmail.com> wrote:
> I was. Is that what you were really saying Ron? ie. compile 6[lca]
> with gcc/2c? Or did you mean something different?

I meant change plan 9 6? so they build with gcc, and I favor doing it
via the plan9ports universe since it's something linux people might
have less trouble with; plus, the plan9ports libraries are mostly like
standard plan 9 but work well in linux.

So the path I had in mind:
1. install plan9ports
2. run a script that builds 6? -- these + plan9ports give you a
toolchain that can produce native binaries. use the plan9ports
9c command to do this -- I tried this today and it gets a lot of the
way there, using proper -I switches to 9c.
3. cd /sys/src; mk all
4. cd /sys/src/nix/k10; mk

My main reason for this path was to minimize changes to the current
nix/plan 9 source tree, such that it continues to build as now on Plan
9 but can also build under Linux/OSX. Maybe you could do this same
kind of approach but use inferno as your toolkit. My impression was
you had to make many minor changes to the plan 9 mkfiles/scripts and
I'd like to avoid that.

I'm sorry if I was too hard on your patch, I was just utterly
overwhelmed by it.

thanks

ron

[Bakul Shah]

On Thu, 26 Apr 2012 18:07:44 PDT ron minnich <rmin...@gmail.com> wrote:
>
> I meant change plan 9 6? so they build with gcc, and I favor doing it
> via the plan9ports universe since it's something linux people might
> have less trouble with; plus, the plan9ports libraries are mostly like
> standard plan 9 but work well in linux.
>
> So the path I had in mind:
> 1. install plan9ports
> 2. run a script that builds 6? -- these + plan9ports give you a
> toolchain that can produce native binaries. use the plan9ports
> 9c command to do this -- I tried this today and it gets a lot of the
> way there, using proper -I switches to 9c.

gcc-4.6 has a -fplan9-extensions option which might be worth
trying. Don't know if there are any GPL issues for the
resulting binaries.

[Noah Evans]

You have to change the mkfiles no matter what, especially mk/parse.
There's no getting around that in a Posix environment without 9vx or
maybe chroot. There are too many plan 9 specific assumptions.

I thought about the plan9ports method but what that does is really
gives new users two problems (installing plan9ports and then compiling
6c and ar(?) then nix). Plan 9 ports is also huge, which makes it
harder for the curious user to figure out what is going on under the
hood.

What makes the inferno method nice is that it's fully self contained.
Change your mkconfig, do a makemk.sh, mk installall and cd nix-os/ &&
mk all and you are done. Outside of go, it really is the shortest
path.

Noah

[Charles Forsyth]

The Reply-To defaults for this conversation seem messed up.

I've restricted it to the list.

As I mentioned to Noah in e-mail, for a commercial project about 11 years ago, I created a

cross-compilation environment (which is what this is) for Plan 9

that would allow the whole of Plan 9 (not just the kernel) to be built on Solaris, Linux, various other

Unix systems, and Windows. I don't think MacOSX existed, but if it did, it was also supported.

The use of the phrase "inferno/nix hybrid" has undoubtedly caused some confusion.

It isn't a hybrid of operating systems. It is creating a cross-compilation environment.

It is nothing to do with Inferno proper. It is borrowing Inferno's cross-compilation environment,

with some (presumably) necessary changes for the new application.

Many of the files in the patch are not needed to provided the cross-compilation environment.

It would be better to separate the cross-compilation aspect from the thing to be compiled.

For instance, libc, os, are copies of those Nix components, possibly somehow modified,

but should not be in this package.

The key bits are the utils from Inferno (which is Inferno's cross-compilation environment)

and supporting libraries such as lib9, and libbio. Compared to the go environment let alone plan9ports,

it is tiny: a handful of directories, and essentially the same source files as on Plan 9

for nm, size, libmach, and the compiler suites. It gets smaller still if it's restricted to 8?, 6?, q? and 5?

which are the only compiler suites likely to be used. That's easy: just leave the others out.

(It's not worthwhile subdividing libmach.)

It is straightforward to parcel it as a separate package, and the patch probably does most

of what is required. I know it is possible because as I mentioned I previously did it.

Unfortunately, I discarded it once the project was finished and didn't keep a copy on Plan 9

(so it's not in our dump), or if I did, I filed it cleverly and couldn't find it for Noah to look at.

The compilers and auxiliary programs can be compiled not only by gcc, but by the Microsoft compiler suite, including the free download. With a further small but annoying set of changes, LLVM (clang) should also work.

[Noah Evans]

Thanks for clearly saying what I've been failing to explain.

Sent from my iPhone

[Charles Forsyth]

Ron mentioned one possibility (plan9ports) that wasn't available when I did my earlier cross-development environment for Plan 9.

It's a good question whether that might now be an easier option for this project.

I didn't even consider suggesting it in my discussion with Noah, and if I had thought

of it, I might have dismissed it at the time as obviously over the top for this application.

Perhaps that would have been short-sighted.

Ignoring Windows support, which might not matter anyway, or could be defined away as "uninteresting",

plan9ports has got the advantage of having many of the right programs

(because it has almost every Plan 9 program).

In particular, it has got mk and rc, and the correct version of awk.

It hasn't got the compiler tools, and auxiliary programs such as ar, nm, size, and so on.

It's also important that those are built using 32-bit compilers (even on amd64).

Apart from cross-compilation support,in my earlier work, I found I needed to add the

equivalent of Inferno's $ROOT or Plan 9 port's own environment variable(s) to say

where the source is stored, and change the Plan 9 mkfiles (I think I used sam -d) to

add $P9ROOT or whatever at appropriate points to allow the programs to be built in

a directory inside the larger host file system. Something similar would seem to be needed

for nix.

There's a further possibility, but since it just occurred to me,

I don't know whether it would work well (or at all):

adapt 9vx to run Plan 9 commands under Plan 9 inside the Nix root.

With that, apparently nothing much would need to change. 9vx could run out of the root,

so you'd only need the one additional host executable, for 9vx itself.

I don't know how weird it would be to use that way, though.