Makefrag-user-lib

[Davide Libenzi]

Currently we use that for user/ builds, but that assumes a static library being built.

I introduced two more targets in master.

One is 'snc', which is the Akaros simple netcat (should build only for Akaros), and the other one is krpfo2perf, which is the kprof to Linux perf file format converted (which should build for bot Linux and Akaros).

Currently these two targets have their own Makefiles, called by make-C, from the master Makefile.

Ideally we could have a Makefrag-libs and Makefrag-apps in the root directory, one for libraries and one for apps (or one handling both).

On the root directory, so that it could be used from tools/ as well.

It leaves open how to have a target build for both Linux and Akaros...

[Kevin Klues]

I'd personally rather see an external framework that can be told how
to build libs and apps on a per lib/app basis and install them into
the akaros kfs via make magic. Something like:

akaros-pkg-builder/
├── pkg/
├── sysroot/
├── .gitignore
├── .config
└── Makefile

Where we use ncurses to select packages we know how to build, and we
have per-package Makefiles in the 'pkg' directory which the top level
Makefile can invoke to build and install a specific package. Packages
would then be installed into the 'sysroot' directory and recursively
copied into $AKAROS_ROOT/kern/kfs once all selected packages were
built. We could also remove them from kfs via the reverse operation.

I wouldn't store the source for any of these packages in this repo
directly, but I would store patches to upstream libs/apps in here
(kind of like we do for busybox in the akaros repo). So per-package
Makefiles would know how to fetch the appropriate source from wherever
it is hosted, apply any relevant patches, build the source, and
install it into 'sysroot'.

As a concrete example, if busybox were moved out into this framework:

akaros-pkg-builder/
├── pkg
│ └── busybox
│ ├── akaros-patches
│ │ └── 1.17.3
│ │ ├── 0001-busybox-perror-with-errstr.patch
│ │ ├── 0002-busybox-create-errstr.patch
│ │ ├── 0003-busybox-rm-rewinddir.patch
│ │ ├── 0004-busybox-shell-resource.patch
│ │ ├── 0005-bb-echo-perror.patch
│ │ ├── 0006-busybox-major-minor-makedev.patch
│ │ └── 0007-echo-buffering.patch
│ ├── busybox-1.17.3-akaros
│ │ ├── ...
│ │ ├── ...
│ │ ├── ...
│ ├── upstream-patches
│ └── 1.17.3
│ ├── busybox-1.17.3-dnsd.patch
│ ├── busybox-1.17.3-iproute.patch
│ ├── busybox-1.17.3-ls.patch
│ ├── busybox-1.17.3-sort.patch
│ └── busybox-1.17.3-unicode.patch
│ ├── .gitignore
│ ├── busybox-1.17.3.tar.bz2
│ ├── defconfig-1.17.3
│ └── Makefile
├── sysroot
├── .config
├── .gitignore
└── Makefile

The only requirement being that any folders under pkg have a Makefile
in them. After that, they can contain their own .gitignore to decide
what should be ignored and what should be kept as part of the repo.
For busybox, the Makefile, all of the patches, and the defconfig would
become part of the repo, while everything else would be ignored.

By convention, the per-pkg Makefile will need to have 'make' 'make
install', and 'make uninstall' targets. Under the hood it will invoke
the actual Makefile of the package (or whatever build system it uses)
to build the package before installing/uninstalling it into 'sysroot'.
Invoking 'make', 'make install', or 'make uninstall' on the top level
Makefile will simply iterate through the directories in pkg that have
been selected in the ncurses config and forward those commands onto
them.

This is actually similar to what we do currently within the Akaros
tree using the top level akaros Makefile. I'm just proposing to move
it to an external repo and start expanding it.

Thoughts?

> --
> You received this message because you are subscribed to the Google Groups
> "Akaros" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akaros+un...@googlegroups.com.
> To post to this group, send email to aka...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
~Kevin

[Davide Libenzi]

Overachiever! ☺

Why would you not store the sources?

You are describing what is that distributions do (source packages contain the tar.gz of the original source, plus distro patches), but they do store the tar.gz.

Also, what about in-tree pass and libs?

Moreover, I find it nice that there is a common make machinery, which emits a consistent output while building.

If every package implements its own make system, this is lost.

IMHO the build configuration for packages should be declaring, per target:

1) Target type (exe, lib, so, ...)

2) The files to build (with possible "glob" spec which resorts to the actual *.c,*.S,...)

3) CFLAGS options. Global, with possible per-file override.

4) LDFLAGS

5) What to install and where

The a master make machinery can read that, and do the proper building.

BTW, Busybox is not a very nice example, as even if there is nothing new to build, it does a bunch of things.

[Davide Libenzi]

OTOH, the advantage of using external packages makefiles, fed with proper CC, LD, AS, ... configs, is that we do not have to rewrite them in terms of new configuration ones.

It all depends on how many packages we intend to import from the wild.

[Kevin Klues]

> Overachiever!
> Why would you not store the sources?

Because I'm picturing some of these packages being actively developed
elsewhere (e.g. dorpbear-akaros).

> You are describing what is that distributions do (source packages contain the tar.gz of the original source, plus distro patches), but they do store the tar.gz.

> Also, what about in-tree apps and libs?

We could keep some in there with the nice one-line compiling, but it's
not necessarily appropriate for everything.

> Moreover, I find it nice that there is a common make machinery, which emits a consistent output while building.

I agree. I am the one who wrote it that way :)

> If every package implements its own make system, this is lost.
> IMHO the build configuration for packages should be declaring, per target:
>
> 1) Target type (exe, lib, so, ...)
> 2) The files to build (with possible "glob" spec which resorts to the actual *.c,*.S,...)
> 3) CFLAGS options. Global, with possible per-file override.
> 4) LDFLAGS
> 5) What to install and where

I agree, except if we want to start porting over a wide variety of
packages, it's intractable to port all of their makefiles to follow
this pattern. Standardizing on a way to get them installed in kfs (or
even through a 9p mount point) is important though.

> BTW, Busybox is not a very nice example, as even if there is nothing new to build, it does a bunch of things.

Sure, but it will always do that unless we change something in the
busybox makefile itself (which isn't unreasonable).

[Davide Libenzi]

Oh, I agree. If the target is mass import, it is not practical to rewrite all the makefiles.

At that point, if makefiles are reasonably following the standard, they will read env for CC, LD, CFLAGS, CXXFLAGS, PREFIX, ..., so we can feed them the proper values.

All autoconf/automake based ones (a lot of OSS stuff), are following that standard.

We could feed them, instead of Akaros gcc binaries directly, a wrapper script which parses the cmdline and does the clean output.

[Kevin Klues]

Yeah, that seems like the way to go. Parsing wouldn't be that hard either, I imagine.

--
~Kevin

[Davide Libenzi]

I saw other projects have a "compile" or "link" wrapper scripts, which do that.

cmake (n.1 truly multi-platform make system IMO) is one, for example.

[Barret Rhoden]

On 2015-11-20 at 17:51 Kevin Klues <klu...@gmail.com> wrote:
> I'd personally rather see an external framework that can be told how
> to build libs and apps on a per lib/app basis and install them into
> the akaros kfs via make magic. Something like:
>
> akaros-pkg-builder/
> ├── pkg/
> ├── sysroot/
> ├── .gitignore
> ├── .config
> └── Makefile
>
> Where we use ncurses to select packages we know how to build, and we
> have per-package Makefiles in the 'pkg' directory which the top level
> Makefile can invoke to build and install a specific package. Packages
> would then be installed into the 'sysroot' directory and recursively
> copied into $AKAROS_ROOT/kern/kfs once all selected packages were
> built. We could also remove them from kfs via the reverse operation.

This is a little broader in scope from what I was asking Davide for
(which was the ability to make snc or kprof2perf in a stand-alone
manner).

But, since we're on the topic, this is basically a package management
system. Instead of rolling our own, we should try to use an existing
one. There's an excellent one that already exists for Gentoo (a
source-based distro) called portage. It's a tree of packages and
instructions for building them that doesn't contain the source. It has
local patches. You can specify CFLAGS and do cross compilations and all
sorts of things. It can build from git repos.

I think the right answer is to try and use portage to build packages
for akaros. We'd need to modify or write the ebuilds, for which there
are numerous instructions online. This also doesn't seem like a high
priority, esp considering apps like busybox are closely coupled with
the kernel/runtime.

Barret

[Kevin Klues]

> This also doesn't seem like a high
> priority, esp considering apps like busybox are closely coupled with
> the kernel/runtime.

I just know that there's been lots of times I wanted to try and port
something over, but didn't bother because we didn't have a
standardized way of doing this. I assume I'm not alone.

[Kevin Klues]

Also, I already threw together the simple pkg-manager based on what I
proposed here. It seems to work for me, but I won't bother pushing it
upstream if it's not what you want.

--
~Kevin

[Davide Libenzi]

As I have an "open ticket" ☺ for making the build of two binaries more explicit, what is the plan for this?

Is it a short term thing, so I should hold my ticket, or is it more a long term plan?