Upcoming switch to 3.3.0 and ABI break

[Lionel Sambuc]

Hello everyone,


The MINIX team would like to give some heads up about what is on the short term roadmap.

As you all know, we have been importing a lot of the NetBSD userland, as we feel what makes MINIX special is not the tools one can use, nor the way the sources are built, but its microkernel architecture.

We are going to switch the main source repository to version 3.3.0, in order to prepare for the next release. We are going to use this opportunity to adapt most - if not all - of the base system types to be defined as they are on a NetBSD system.

This will allow us to import more tools and have more packages from PKGSRC available on MINIX3. But this comes at a cost: an ABI break.

We are sorry not to be able to provide full forward, and backward, ABI compatibility this time.

Given our constrained ressources, we felt it was wiser to align as much as currently possible in one step, and ask our community to reinstall from the developper ISO, which will be provided, and spend the saved time on new MINIX hardware support as well as continuing to improve the system capabilities.

Of course we will provide the required steps to upgrade by compiling the new system, but that particular step will still require to generate an ISO, and install from it, over our usual source upgrade.


Regards,

Lionel Sambuc, on behalf of your MINIX team.

[r0ller]

Hi Lionel,

 

Do you happen to know roughly the release date of Minix3.3.0?

 

Regards,

r0ller

[Lazar Stricevic]

And could someone please come up with the list of the most important 3.3.0 changes, especially those which break ABI (I suspect there's more to it than just 'base system types'). A wiki-page would be nice if possible...

Thanks,

Lazar

[Lionel Sambuc]

Hi r0ller,


I don't expect to see the merge I announced published before mid/end of
september, but I wanted to give a heads up to the community for a future
disruptive change.

Now, to switch the mainline tree is a big step towards that. We have a
few other bullet points to achieve before that. So I can't really tell
when we will reach the release. As usual, we will release when we feel
it is ready.


Regards,

Lionel

[r0ller]

Hi Lionel,

 

Thanks, that's enough. I didn't expect an exact release date:)

 

Regards,

r0ller

[Alex B.]

Thanks Lionel,

I believe it's a wise choice to go this route at this time, it's about getting the kernel to a good place and get more developers in the door to work on this project with us. have no ABI compatibility at this time, in my own estimation is minor as compared to what has come forward
out of the hard work everyone has done in the past leading up to this point in time. At a later date, I am sure we can work on ABI and have no doubt we can work on it and have it ready for another stable release in the future. The way I look at it ABI doesn't make sense to implement now when we are constantly changing how the kernel and applications work together, it only hinders development at this time. Once we have all the drivers and the rest of the NetBSD applications and daemons coded for Minix 3 and we hammer out how we want to implement the application binary interface, that would be best.

Otherwise if we were to continue with ABI, we would have to create conversion utilities from one version to the next which in turn sooner rather than later, break the source code of some of the main applications, causing headaches and wasted time attempting to retrofit the old source of the applications to the new system.

[James T. Sprinkle]

snip snip rip

    Once we have all the drivers and the rest of the NetBSD applications and daemons coded for Minix 3 and we hammer out how we want to implement the application binary interface, that would be best.

Does this mean that writing a new driver for 3.2.1 would be a waste of time?

[Thomas Cort]

No, it wouldn't be a waste of time. The size of types will change, but
the API shouldn't change too much. You'll just have to recompile your
driver when the 3.3.0 snapshot comes out.

Thomas

[James T. Sprinkle]

Thanks Thomas!  I guess I misinterpreted the sentence I quoted.  Sounded like under 3.3.0 the drivers for NetBSD would be compilable on Minix.  Wishful thinking!

Jim

[Robert James]

Do we have any idea when the change over is happening as yet? I want to dive in and start developing but I want to wait till the next phase to really get my ears wet or what repo I can clone in to my system to start the process early.

Thanks.

[Lionel Sambuc]

Hi,


The 3.3.0 branch is progressing well, yet there is the main following tasks to be completed:
- Finish importing the /sys/sys headers and definitions.
- Finish PKGSRC bootstrap on 3.3.0.
- Switch time_t to 64bit.
- We also want to take the opportunity to cleanup the IPC protocols of a few
system protocol, as we have now more space available in the messages.

I can't give a deadline. What I can say is that this will not happen before february next year, given what has still to be done.

You can follow the progress on my personal git repository (http://git.minix3.org/?p=minix-lionel.git;a=summary) where you will find branches named 3.3.0[a-z].
The last letter is the latest rebase of the branch.


Regards,

Lionel

> --
> You received this message because you are subscribed to the Google Groups "minix3" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to minix3+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Robert James]

Thanks Lionel,

I just removed my src dir and resynced to your latest branch and now I'm rebuilding the system so get up to date. I like staying ahead of the curve so I'll stay in this branch and dive in where possible.  Keep up the great work and to all others involved, its a great project.

Rob.

[Lionel Sambuc]

Hi,


Just be careful when building, doing a 'make build' will fail, by definition.


Regards,

Lionel

[Robert James]

Yes, I just noticed this as I wasn't paying attention.  Damn stupid brain on Monday mornings. I'll run it via Archlinux and crosscompile.  The advantages of virtual machines.  I presume the steps are the same and its just the automated build that is screwed?

Rob

[Lionel Sambuc]

Hello,


The build is not screwed, it is just as we are breaking the ABI, you can't incrementally update the system, which is what make build does.

Using the two scripts (releasetools/arm_sdimage.sh, releasetools/x86_hdimage.sh) you can directly generate images which are bootable in qemu.

In my case I also add a second disk image which contains what I want to keep between two 3.3.0 changes. I just have to mount it somewhere, and I can work with it without having to transfer every time my work in progress.

This is the most convenient way I found to work with 3.3.0, until I can finish the pkgsrc bootstrap.


Regards,

Lionel

[Robert James]

noted, thanks for information.

Rob

On Wednesday, July 24, 2013 1:38:04 PM UTC+1, Lionel Sambuc wrote:

[Lionel Sambuc]

Hello Everyone, 

End of May has been reached, so I think an update on the state of the ABI break is due.

Before we can merge mainline, a few tasks are still required:

 1) The last task regarding the ABI break itself is being tackled by myself, I hope to see it through in a few days.

 2) PKGSRC has been bootstrapped on 3.3.0, but it is still missing two critical packages:

      - lang/gcc45 (being worked on by Ben right now)

      - minix/clang34

 3) Some minimal testing to check that our usual infrastructure (namely releasetools/release.sh) still works as expected, as well as our script to build the whole PKGSRC repository (so-called "pbulk").

As soon as this is cleared, we will be able to finally merge the 3.3.0 branch. We will then wait for a while, in order to get feedback from the community.

This grace period will also serve as a buffer to fix any bugs that may have crawled past our attention, despite our extensive testing.

After that, will we be able to make the new release ISO and official announcement.

 

Regards,

Lionel Sambuc, on behalf of the MINIX team.

[Patrick]

Thanks very much Lionel and the rest of the team !

If gcc 4.5 is not working from pkgsrc is 4.4 still working from pkgin ?

I would like to help with testing. If there are sections in particular you would like to have retested, please let me know.

-Patrick

--

You received this message because you are subscribed to the Google Groups "minix3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to minix3+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Lionel Sambuc]

Hello, 

pkgin is a front-end to PKGSRC. When I speak about gcc45, I am only speaking for the upcoming 3.3.0 release. On 3.2.1 versions and earlier, it is not planned to spend further effort (at least from the team) to add new packages / security fixes in PKGSRC for those releases.

At the present time we do not have the ressources to maintain multiple GCC versions. We switched to GCC45, as this is the default version (4.5.4) in the NetBSD sources we imported. In other word, GCC44 is being obsoleted, and in the new PKGSRC repository, no attempt will be made to backport our patches from 4.5.4 to 4.4.x, at least from me.

There is not much incentive either to do so, as the main compiler (for MINIX/x86) is LLVM/clang.

If anything, we would prefer to be able to switch to LLVM also for the ARM port, and for PKGSRC packages. The last point seems now possible, as most of the packages do compile with LLVM on NetBSD, at least according to the last quaterly report (http://mail-index.netbsd.org/pkgsrc-users/2014/04/04/msg019510.html) - 13841 compile out of a total of 14255 on NetBSD-current/x86_64.

While those number are not yet achieved on Minix, the ABI break *main* goal is to allow us to reach numbers as close as possible to those.

As far as testing goes, we will need help to test MINIX, and also the packages. There is just too many for us to be able to test them individually. My hope is that when 3.3.0 is released, we will be able to upstream most of the currently required patches we have for MINIX in PKGSRC, and if possible involve the PKGSRC regular maintainers into the process of making their packages work on MINIX. Meanwhile this will be up to our community to help us by trying out packages and reporting if they work or not, maybe even provide fixes.

Regards, 

Lionel 

[Ben Gras]

All,

Hey, big news, the 3.3.0 branch is merged with mainline now.

We need your help testing it for the coming 3.3.0 release.

A new dev iso is available on http://download.minix3.org/iso/snapshot/

This week we've just been doing a lot of final stuff like building packages to make the last pieces come together and it's all done now - all seems fine. So we've done it!

There are lots of changes compared to the old mainline; as you know the original motivation was to do a big NetBSD-aligning ABI break, but the scope has crept a bit so now there's more. See http://wiki.minix3.org/MinixReleases for a (growing.. we have to remember it all) list.

Worth mentioning explicitly: clang and the sources are currently not on the CD due to space limitations. You have to get the sources and clang and binutils after installing in order to rebuild the system. (In the future I think one or both will be there in compressed form.)

A few known limitations to be fixed later:

 - Clang is default now, also for crosscompiling but it doesn't support Minix/ARM yet. There's a workaround in the ARM script that selects GCC using BUILDVARS. So if you set BUILDVARS, include the GCC options already there.

 - The current 3.3.0 package set is quite minimal, a bit smaller than it was, but we'll rebuild a full set soon.

 - Xorg is missing for now.

Please help test the new state for the best Minix release ever!

Cheers,

Ben on behalf of the Minix team

[Thomas Cort]

> the 3.3.0 branch is merged with mainline now.

Way to go!

> There are lots of changes

Indeed, a lot of work has gone into 3.3.0... over 1000 commits[1] by
nearly 40 contributors[2]. I'll do some testing this weekend.

> Please help test the new state for the best Minix release ever!

Testing and updating the documentation is another place to help:
http://wiki.minix3.org/UsersGuide

TC

[1] git shortlog -n -s v3.2.1..master | cut -f 1 | awk '{s+=$1} END {print s}'
[2] git shortlog -n -s v3.2.1..master | wc -l

[JD Begin]

Congrats to everyone who worked on this. Minix 3.2 is dead, long live 3.3!

[r0ller]

Hi All,

First of all, congrats and thanks to everyone for this! I gave it a try and seems to be ok even though it was just a smoke test of a reinstall on a copy of my current 3.2.1 vmware system since my original "test plan" was to set up 3.3.0 and move my project to it. However, as some of the dependencies (zlib, readline, flex, bison, sqlite3) and dev tools (scmgit-base, gcc) are missing, I could not go far enough:( Therefore, the question arises: when do you think that more packages will be available?

Regards,

r0ller

[Lionel Sambuc]

Hi r0ller,


A few notes:
- scmgit-base as been renamed in pkgsrc to git-base (and is available right now)
- binutils is there, as well as all it's dependencies
- clang is there too.
- readline, bison are available as packages, we have zlib, flex and sqlite3 as part
of the base system system now.

- gcc is not available, gmp needs to be fixed when compiled with clang, at least.

I am currently going through a full build, which should take at least around 50 more
hours. I will announce on the mailing list as soon as the repository has been updated,
hopefully by the end of the week.

This will be only a first shot at the repository, we still have a few packages to fix
at least.

If I looked correctly, you should already be good to go, unless clang can't take care
of your sources.


Regards,

Lionel

[r0ller]

Hi Lionel,

Thanks for your answer! Poor me haven't checked if certain pkgs are part of the base system. I also suspected that scmgit-base has been renamed but was unsure. Anyway, then I can already report that when issuing make pkgsrc-create in /usr, it fails as scmgit-base is expected and therefore I get: "scmgit-base is not available on the repository" leading to "Error code 1".

Regards,

r0ller

[Roc Vallès]

Gave it a spin on some old hardware.

Improvements:

- Boot monitor can boot minix3. With 3.2.1, it would reboot immediately after it loaded the kernel (triple fault?) so I had to use grub2 to load it. 3.3.0 just works.

- Keyboard plugged on onboard OHCI USB just worked.

Issues (all with pkgsrc):

- make pkgsrc-create wants scmgit-base. The package is now named git-base.

- bmake not installed by default, but needed by pkgsrc

- trying to actually build a package will expose dependency on digest, and insist it's not there (doesn't like the binary package installed with pkgin)

- pkgsrc is setup to use gcc is used by default, but gcc isn't available via pkgin (only clang is).

At this point, I just gave up. I'll try again some other time. The intent was to build and try a few things, but that can't be done when pkgsrc doesn't work at all; pkgsrc is quite basic functionality and should just work.

[Antoine LECA]

On 2014-08-05 20:27Z, Lionel Sambuc wrote:
> - gcc is not available, gmp needs to be fixed when compiled with clang, at least.
>
> I am currently going through a full build, which should take at least around 50 more

[...]

What is the target plan for pkgsrc's compiler ${PKGSRC_CC}: llvm/clang
as rest of MINIX, gcc as mainstream NetBSD and most other pkgsrc ports
so far, or to have both available and to be able to interchange them
without hurt (like we interchange gnumake and bmake for most packages)?

I see advantages (and hurdles) to any of these proposals :-)

Antoine

[Lionel Sambuc]

Hello Antoine,


For C programs we have a solution which allows gcc-compiled packages to
work
on both a gcc-based system and a clang-based system. This is obviously very
experimental.

For C++ programs, we were not able to implement a solution for that.

This is why we decided to switch the main PKGSRC compiler to clang by
default.

This comes with a few limitations:
- As I have not yet completed a full pbulk with clang I don't know how
many packages
will be available.
- The current clang binary we have is about 3-4x slower to compile C++
programs
than GCC.

This means that to compile yourself LLVM/clang, on a good machine,
you have to
expect about 26 hours of build time. Works, just very slowly.

Oddly enough clang is not suffering from this while compiling C
programs. We have
not yet been able to explore this.

In summary, the answers to your questions are:
- We have switched to llvm/clang by default, for PKGSRC.
- Mixing and matching clang/gcc works for C only applications, but we
do not plan on
supporting this officially.


Regards,

Lionel

[Lionel Sambuc]

Hello,


Concerning PKGSRC, I have not yet changed /usr/Makefile to point to the new
pkgsrc repository, as I was waiting for the results of the full build before
creating the new official branch.

If you want to give it a spin before I create the official branch, and
update the
documentation, the development branch is available online there:

git.minix3.org/pkgsrc-ng/3.3.0-lionel.

Regards,


Lionel

> --
> You received this message because you are subscribed to the Google
> Groups "minix3" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to minix3+un...@googlegroups.com

> <mailto:minix3+un...@googlegroups.com>.

[r0ller]

Hi Lionel,

One more question from a noob like me: how can one tell which pkgs are part of the base system (besides checking /usr/pkg/bin and /usr/pkg/lib)? At least, pkgin ls does not show them (e.g. zlib, flex and sqlite3) as pkgs installed.

Thanks and regards,

r0ller

[Lionel Sambuc]

Hi r0ller,


Well, that's the trick, software which is part of the "base" system is
not found in packages.
There are no package descriptions for those. They might have a packaged
counterpart, but
that is not always the case.

Those are all the programs and libraries outside the /usr/pkg hierarchy.

Put in an other way, everything under /usr/pkg comes as a PKGSRC
package, or all packages
are installed under /usr/pkg.


Regards,

Lionel

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "minix3" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to minix3+un...@googlegroups.com

> <mailto:minix3+un...@googlegroups.com>.

[Antoine LECA]

On 2014-08-06 07:22Z, Lionel Sambuc wrote:
> In summary, the answers to your questions are:
> - We have switched to llvm/clang by default, for PKGSRC.
> - Mixing and matching clang/gcc works for C only applications, but we
> do not plan on supporting this officially.

Thanks for the detailed answer.

> This means that to compile yourself LLVM/clang, on a good machine, you
> have to expect about 26 hours of build time. Works, just very slowly.

Ugh. Not a good news. Particularly since I expect your definition of "a
good machine" to be your 4-core 3.7 GHz i7 G2 with SSD disks... pretty
much more efficient than the '08 Core available to me!

On the other hand, you spoke of 50 hours for a full pbulk, and now 26
just for llvm; do you mean the rest of pkgsrc takes 24 hours? 8-/

Antoine