Minix 3.4.0

[James T. Sprinkle]

Just curious.  When will Minix 3.4.0 be seeing a new RC or will it be released GA soon?

Also, does anyone know if AST is planning an updated OSDI in the future?

Thanks!

Jim

[James T. Sprinkle]

3 weeks and a lot of crickets later...

[Jean-Baptiste Boric]

On Sunday, September 24, 2017 at 9:40:14 PM UTC+2, James T. Sprinkle wrote:

3 weeks and a lot of crickets later...

It ain't the season of crickets anymore where I live :-P

MINIX 3.4.0 is blocked as long as https://github.com/Stichting-MINIX-Research-Foundation/minix/issues/104 is not solved. I don't know about a new RC, but development is kinda slow these days.

I haven't heard any rumors about a 4th edition OSDI either.

[r0ller]

Hi Jean-Baptiste,

Is there any VM where that issue can be reproduced? Provided that's possible and that VM has some trace tool we could make two traces for the last working and first non-working commit hash minix versions to compare them. Is such a thing feasible?

Best regards,

r0ller

[Jean-Baptiste Boric]

 

Is there any VM where that issue can be reproduced? Provided that's possible and that VM has some trace tool we could make two traces for the last working and first non-working commit hash minix versions to compare them. Is such a thing feasible?

It's been way too long, I don't remember the details. What I do recall is that this issue happens on hardware only (can't be reproduced with a VM), the commit responsible is 0a6a1f1 (a humongous NetBSD re-synchronization of the source code) and the fix is to ditch GCC and switch over to clang with pull request #195.

[James T. Sprinkle]

<SNIIP> the fix is to ditch GCC and switch over to clang with pull request #195.

Hi Jean-Baptiste!  How does one go about getting pull request #195 into their Minix source tree?  I would like to try this out, but could use some instruction on how to get there.

Thanks!

Jim

[Jean-Baptiste Boric]

 

Hi Jean-Baptiste!  How does one go about getting pull request #195 into their Minix source tree?  I would like to try this out, but could use some instruction on how to get there.

Something along these lines (I didn't actually test these commands) :

# register Lionel's repo

$ git remote add sambuc https://github.com/sambuc/minix.git

# fetch data from Lionel's repo

$ git fetch sambuc

# merge his pull request with your currently checked out branch

$ git merge sambuc/clang-arm

There will be merging conflicts (at least one according to GitHub) that will need to be taken care of. "git status" is your best friend.

[James T. Sprinkle]

Thank you!  I will give this a try when my current build finishes.

Jim

[James T. Sprinkle]

Resolved the merge conflicts and am building for both x86 and ARM now.

One question, does it also switch the installed compiler to clang on the ARM sdcard image or is it still GCC there?

Thanks again for the pointers!

Jim

[Jean-Baptiste Boric]

One question, does it also switch the installed compiler to clang on the ARM sdcard image or is it still GCC there?

It depends on what options were supplied to build.sh.

According to a commit message in the pull request:

ARM: Compile whole tree with clang.
Note: GCC is still the default compiler, to use clang do the following: $ BUILDVARS="" ./releasetools/arm_sdimage.sh

[rchmielarz]

Hi guys,

I've recently started to get interested in minix and it's a shame that there is no new version because of this issue.

But rather than removing gcc altogether I have a different plan. Thus I wanted to ask, would it be acceptable for You if I've done a rebase to the newer NetBSD version (and newer clang/gcc at the same time) and tried to solve the BeagleBone with it. Assuming this would work would You accept such a pull request? Or have You already decided to ditch gcc in favor of clang?

Cheers,

Radek

[James T. Sprinkle]

Hi Radek!  I can't speak for the core Minix development team, but since the changes Lionel has made to implement clang as the compiler haven't been merged in with the mainline due to the failed tests listed on the repo, I don't see that this wouldn't be a good thing to try in the meantime.  I actually spent some time over the weekend trying to track down the #104 bug.  If you are interested in setting up a community swat team/strike force/whatever to work on this, I would be more than happy to assist.  I am pretty new to the ARM architecture, but I have been working with this OS off and on since the mid 90s.

Jim

[rchmielarz]

Hi Jim,

Wow, that's quite a history You have with this system. Myself I'm brand fresh.

And sure. It would be nice to collaborate, especially when taking into account that even the core team didn't manage to fix it for those months that have passed. So the error is not trivial to put it mildly.

Getting back to the subject I had troubles even building the image and qemu-linaro on my system since the codebase is 2 years old. But after some tweaking (fixing perl scripts on qemu-linaro + some minor C changes and applying gcc patch [https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=ec1cc0263f156f70693a62cf17b254a0029f4852]) it worked. Unfortunately I don't own a beaglebone yet so my only option is qemu and here I've encountered a problem with loading the image:

[me@localhost minix]$ qemu-system-arm -M beaglexm -serial stdio -drive if=sd,cache=writeback,file=./minix_arm_sd.img

(process:20897): GLib-WARNING **: gmem.c:483: custom memory allocation vtable not supported
WARNING: Image format was not specified for '/home/radzio/experiments/minix/minix_arm_sd.img' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
qemu: fatal: Trying to execute code outside RAM or ROM at 0x402f0400

R00=40014044 R01=402f0400 R02=00000000 R03=00000000
R04=00000000 R05=00000000 R06=00000000 R07=00000000
R08=00000000 R09=00000000 R10=00000000 R11=00000000
R12=00000000 R13=4020fcb0 R14=00000000 R15=402f0400
PSR=400001df -Z-- A sys32
s00=00000000 s01=00000000 d00=0000000000000000
s02=00000000 s03=00000000 d01=0000000000000000
s04=00000000 s05=00000000 d02=0000000000000000
s06=00000000 s07=00000000 d03=0000000000000000
s08=00000000 s09=00000000 d04=0000000000000000
s10=00000000 s11=00000000 d05=0000000000000000
s12=00000000 s13=00000000 d06=0000000000000000
s14=00000000 s15=00000000 d07=0000000000000000
s16=00000000 s17=00000000 d08=0000000000000000
s18=00000000 s19=00000000 d09=0000000000000000
s20=00000000 s21=00000000 d10=0000000000000000
s22=00000000 s23=00000000 d11=0000000000000000
s24=00000000 s25=00000000 d12=0000000000000000
s26=00000000 s27=00000000 d13=0000000000000000
s28=00000000 s29=00000000 d14=0000000000000000
s30=00000000 s31=00000000 d15=0000000000000000
s32=00000000 s33=00000000 d16=0000000000000000
s34=00000000 s35=00000000 d17=0000000000000000
s36=00000000 s37=00000000 d18=0000000000000000
s38=00000000 s39=00000000 d19=0000000000000000
s40=00000000 s41=00000000 d20=0000000000000000
s42=00000000 s43=00000000 d21=0000000000000000
s44=00000000 s45=00000000 d22=0000000000000000
s46=00000000 s47=00000000 d23=0000000000000000
s48=00000000 s49=00000000 d24=0000000000000000
s50=00000000 s51=00000000 d25=0000000000000000
s52=00000000 s53=00000000 d26=0000000000000000
s54=00000000 s55=00000000 d27=0000000000000000
s56=00000000 s57=00000000 d28=0000000000000000
s58=00000000 s59=00000000 d29=0000000000000000
s60=00000000 s61=00000000 d30=0000000000000000
s62=00000000 s63=00000000 d31=0000000000000000
FPSCR: 00000000
Abortado (`core' generado)

There of course could be a variety of reasons why it doesn't work (wrong version of host gcc, error in qemu-linaro git, error in minix-git, wrong command, etc.), but maybe You've dealt with the problem in the past and can help out.

In the meantime I'll try to move back to the last minix commit that was working and build a new image to rule out problems with qemu.

Cheers,

Radek

[rchmielarz]

Hi,

I've just realized that I was building minix for beaglebone xm instead of black, but still this error should not be happening.

Cheers,

Radek

[Sambuc Lionel]

Hello,


On the test infrastructure, we use machines which run on :
* Ubuntu:
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

* Qemu:
qemu-linaro at commit e571600c4a260d3072abbe7ecf2013e48e82bea7

* host toolchain:
+ gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
+ GNU assembler (GNU Binutils for Ubuntu) 2.26.1
+ This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi
(with 58 registered patches, see perl -V for more detail)

This is known to work correctly, the last build with default
flags was done this night.

Some precision, the build above is still done using GCC (as
the branch is not yet merged).

The issue you see on qemu is most likely due to a .settings
being present in your build directory, configured for the
BBB.

When targeting the emulator, you have to target the BeagleXM,
which is the default.

Now, w.r.t to the bug with GCC, it works fine on qemu, the
reason is that qemu does not emulate properly unaligned
memory accesses exceptions. If I recall correctly, I could
not find how or why GCC would misalign the stack...


Kind regards,

Lionel

> --
> You received this message because you are subscribed to the Google Groups "minix3" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to minix3+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[r0ller]

Hi Lionel,

Does it have to do anything with this maybe? https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838

Best regards,

r0ller

[Sambuc Lionel]

Hello r0ller,


I can't say for sure, as it stems from a bug for intel, and here the
issue is only on ARM.

As it also involves some of their high-level optimisation passes, this
might be, but I am not sure to understand everything.

At the time I explored this issue I could not find a way to have GCC
spit out code which would ensure proper alignment in all cases, so at
some point in the debugging, I decided it was better to invest the time
in getting clang up and running for the following reasons:

1. It will be required for ARM if we want to be able to have liveupdate
running there as well.

2. Having only one compiler to care about will be enough work, given
the workforce we have.

At least my availabilities have clearly demonstrated I can't keep up
with all the things I am doing for minix right now.

3. Licenses, given llvm/clang is BSD, it is easier to work with in
MINIX. Being able to use only clang would mean the only thing GPL
licensed left would be binutils, and its requirements (gmake,
texinfo).

And provided a couple of fixes in i386 specific code, we might even
not need it anymore, and only use the tools from LLVM, and thus have
a fully BSD OS.


Kind regards,

Lionel

------------------------------------------------------------------------

PS: Provided unlimited time and budget, I would prefer to have GCC and
all possible compilers to work on / with / for minix, as code which
compiles on many compilers is more likely to be bug-free, and will
have to be written cleanly if one does not want to have a mess of
#ifdef or different files per compilers...

[r0ller]

Hi Lionel,

Do I get it right that we could have a 3.4.0/ARM release that can only be built with clang? Well, as far as I can judge, anyone who considers cross-compiling minix3 has already the knowledge that using clang instead of gcc would not prevent him/her from doing so. So, in my opinion it shouldn't be a release blocker. Though, I know that many won't like the example but afaik the linux kernel cannot be compiled with clang, only with gcc and still one release comes after the other:)

Best regards,

r0ller

[rchmielarz]

Hi,

I think something has got lost in the translation. I've managed to successfully run both i386 and beagleboard xm on qemu, but settings for beaglebone are producing an image which compiles but fails on qemu.

Hence I've got more fundamental questions:

1) Is qemu-linaro able to emulate beaglebone black?

2) Even if qemu-linaro is able to emulate it as Lionel is saying it won't be able to detect the problem that is blocking the release?

3) What are the major differences between beagleboard xm and beaglebone in terms of the build process? specifically if I'm able to pull the trick with rebasing to the newer gcc/clang and manage to run qemu with beagleboard xm and compile image for beaglebone black what is the level of certainty that it would work?

If the answer for the first question is positive than something is wrong with the image building process for beaglebone, if it is negative than the arm_sdimage.sh needs to be changed so it does not misinform about being able to run the image using qemu.

@r0ller: I believe the port to clang is still not merged as not everything is working (https://github.com/Stichting-MINIX-Research-Foundation/minix/pull/195 mentioned before in this thread).

Cheers,

Radek

[Sambuc Lionel]

Hello,

> 1) Is qemu-linaro able to emulate beaglebone black?

Not to my knowledge. qemu-linaro only support the BeagleBoard-XM.

> 2) Even if qemu-linaro is able to emulate it as Lionel is saying it won't be able to detect the problem that is blocking the release?

There was indeed a misunderstanding, I never meant to imply that one can emulate a BeagleBone (White or Black) with qemu-linaro.

> 3) What are the major differences between beagleboard xm and beaglebone in terms of the build process? specifically if I'm able to pull the trick with rebasing to the newer gcc/clang and manage to run qemu with beagleboard xm and compile image for beaglebone black what is the level of certainty that it would work?

If you rebase my branch it should work, minus the points presented in the readme file. A.k.a some regressions in several tests. Now the only changes are several constant in terms of where to find devices, the memory, etc… The code generated is actually the same, and the current bugs we are seeing are in generic ARM code.

What I mentionned about unaligned accesses exceptions not being triggered in the emulator used to be the root problem for GCC, which worked on qemu, but not on real HW. That was a nasty surprise as I had extensively tested things in emulation, and we have tests running on qemu.

> @r0ller: I believe the port to clang is still not merged as not everything is working (https://github.com/Stichting-MINIX-Research-Foundation/minix/pull/195 mentioned before in this thread).

Indeed the branch was not yet merged because it fixes several issues w.r.t to C++ exceptions, but it also generate the previously mentioned regressions.


Kind regards,

Lionel Sambuc

------------------------------------------------------------------------

[Sambuc Lionel]

Dear All,


I'll try to rebase my branch today, it does not seem like a big deal.

It will just take me some time to test things are working as expected.


Kind regards,

Lionel

------------------------------------------------------------------------

[Lionel Sambuc]

Dear All,


I have updated the pull request, I added two patches:

1. Make it so that by default it compiles using clang
2. I added on last change that I am not yet 100% sure is needed. If we get to fully support saving and restoring when needed the floating point register correctly, it would be for sure unneeded, as we could then switch to a full hardfloat build.


Kind regards,

Lionel
--
Sent from my phone

[rchmielarz]

Hi,

I should probably mention here that I haven't forgot about this issue. I'm still working on updating the gcc version. The reason it's taking so much time is that as I found out in order to update anything I first needed to learn about the netbsd build process. Then half-way through syncing minix with netbsd-CVS I've realized that there are just to many applications with changes specific to minix. I would never be able to tell if I haven't broken anything by accident. This has lead me in the direction of trying to update only the build tools and leave the rest as it is for now. But I didn't want to just copy gcc version from netbsd since this is an old version (5.3) and we are already on 7.3. So I have distilled the necessary changeset for the gcc package but although the tooldir builds fine the compiler that is used inside minix distibution still has compilation issues. And the proper way of updating the tooldir in netbsd (mknative) doesn't work for linux. And to my surprise minix arm can't built itself (meaning that running the releasetools scripts for arm inside emullated minix arm fails even without any changes). So I'm left with trying to guess what files are needed for the user facing compiler to build and function properly.

To summarize: the work is ongoing (https://github.com/rchmielarz/minix) but its taking more time than I could have imagined.

Cheers,

Radek 

[r0ller]

Hi Radek,

Thanks for the update, it seems there's hope:)

Best regards,

r0ller

[r0ller]

Hi Radek,

One more thing came into my mind concerning your sentence: "So I'm left with trying to guess what files are needed for the user facing compiler to build and function properly."

Well, if I get your problem right: you could get the list of files via a trace. Once I wanted to strip down the emscripten compiler to keep only the files used during compiling a project of mine so I managed to get the list of files by carrying out a trace for all "path name to inode translation". But as I'm not a guru in any fields I cannot tell you how it'd look like on linux but in case of NetBSD it was:

ktrace -id -tn ./<shell command to compile your stuff>

Strace may be an alternative in linux but I cannot tell which options may fit. I hope it helps:)

Best regards,

r0ller

2018. január 31., szerda 22:08:59 UTC+1 időpontban rchmielarz a következőt írta:

[r0ller]

Obviously, you'll need to do this trace on a successful build:)