Announcement: replacing the MINIX 3 network stack

[David van Moolenbroek]

Dear all,

As some of you may be aware, in the last year-and-then-some, I have been working on a major overhaul of the MINIX 3 network stack, for which I presented the plan at MINIXcon 2016 [1]. I am proud to announce that this networking project has now been finished and, barring any unexpected problems or serious objections, will be merged on short notice. The project effectively replaces the entire MINIX 3 network stack, all the way from the system call interface to the network driver API. Most notably, it provides a new implementation of the TCP/IP and UNIX Domain Sockets (UDS) system services.

The project had the following three main goals:

1) relieve the maintenance burden of the core TCP/IP implementation, of which the current incarnation has become unmaintainable;
2) establish compatibility with NetBSD networking userland, including low-level network administration utilities;
3) add support for IPv6.

All of these goals have been met. The first point is achieved by replacing the previous TCP/IP service (INET) with a completely new TCP/IP service (LWIP) which is based on the lwIP stack [2]: an actively maintained, BSD-licensed TCP/IP implementation. All factors considered, I believe the lwIP stack is currently the best option for us.

The second point is achieved with a considerable amount of code in the LWIP service that bridges the gap between the NetBSD socket and networking API on one side, and the core of the lwIP stack on the other side. As part of that, the networking project introduces a large number of BSD sockets and networking APIs and features and BSD-like behaviors that were previously absent. As a result, the project unlocks much more of the standard NetBSD libc/libutil/etc functionality, and replaces the set of traditional MINIX networking utilities with a set of (largely unchanged!) NetBSD networking utilities.

The lwIP stack also provides IPv6 support--the third point. As part of this project, I have invested substantial effort into improving lwIP's IPv6 support (among other things). I have been able to upstream all my changes to the lwIP project, of which we are now using an almost unchanged import. The plan is to continue updating to the latest lwIP source code on a regular basis, so that we get all the benefits from improvements made on their side.

Replacing the UNIX Domain Sockets (UDS) service was more a side effect of the project. While the previous UDS implementation mostly worked fine, the new version is much more POSIX compliant, is much closer to NetBSD in terms of behavior, and now also shares a fair amount of code with the LWIP service. As a practical outcome, we can also drop some previously necessary MINIX-only userland changes there (e.g. from tmux(1)).

As also listed in the corresponding pull request [3], the main overall highlights of the networking project include:

* first-class VFS support for socket-type file objects instead of a series of libc kludges;
* a socket driver library (libsockdriver) and a socket event dispatching library (libsockevent);
* a new, POSIX-compliant implementation of the UNIX Domain Sockets service (UDS);
* a new TCP/IP service (LWIP), built upon the core of the lwIP project;
* proper support for the BSD socket API and the BSD network management API;
* full-blown support for IPv6;
* replacement of old MINIX networking userland with modern NetBSD networking userland;
* network driver protocol extensions, from reporting link status to checksum offloading;
* auto-starting of plug-and-play network drivers;
* NetBSD-style rc-script based network initialization;
* about 18 KLoC worth of new tests (test90-94).

More detailed information about all the improvements can be found in the commit messages of various commits that are part of the pull request.

Practically speaking, the project comes with many user-visible changes. Not only are we switching to NetBSD networking utilities like ifconfig(8) and route(8) which work completely differently from before, but also the NetBSD networking system initialization (rc) scripts. In general, NetBSD documentation on networking (e.g., [4]) now applies to MINIX 3 too, although we do of course support only a subset of the full NetBSD functionality at this point.

On that point: in the next few days to weeks, I hope to compose a list of features that are expected and not expected to work. As an example of what works: MINIX 3 can be configured at run time to be an IPv4 router, and even a basic (functional but not strictly RFC-compliant) IPv6 router. As an example of what does not work: the functionality of the arp(8) and ndp(8) utilities is rather restricted because of current limitations in lwIP, and not all non-standard netstat(1) views are supported. In some cases, these limitations should be resolved in the future; in other cases, they are inherent to having a different TCP/IP stack implementation.

Perhaps most importantly right now, the way to configure networking has changed completely. As also mentioned in a docs/UPDATING entry, upgrading MINIX 3 users will have to reapply their network settings. In essence, network drivers are now automatically started whenever possible, and so netconf(8) is no longer concerned with selecting a network driver to start. Instead, it is now a front-end to NetBSD-style network configuration.

In any case, it is my strong belief that with this new network stack, one of the more important current obstacles to moving forward with MINIX 3 - in terms of both NetBSD userland compatibility and overall functionality - is out of the way, putting us on a healthy course when it comes to networking. We can now start looking into more advanced networking features such as support for a firewall, tunneling, wireless networking, etcetera.

As always, progress of our automatic testing process can be followed on the gerrit server [5]. Once the code has been merged, of course, please report any problems, bugs, questions/concerns (etc), either in the form of GitHub issues [6] (for concrete problems) or on this group (for questions and more open-ended issues). Given that this project involves a substantial amount of completely new code (a little over 50 KLoC in total), there are bound to be bugs, and I'd like to hear about them.

Finally, with this project formally coming to a close, I would like to express my gratitude to Andy Tanenbaum, in the role of the head of the MINIX Foundation, for sponsoring part of this project. Also, thanks go out to Lionel Sambuc for his testing and debugging, which among other things helped iron out a few major bugs.

Regards,
David

[1] http://www.minix3.org/conference/2016/program.html (first talk)
[2] https://savannah.nongnu.org/projects/lwip/
[3] https://github.com/Stichting-MINIX-Research-Foundation/minix/pull/213
[4] https://www.netbsd.org/docs/network/
[5] http://gerrit.minix3.org/
[6] https://github.com/Stichting-MINIX-Research-Foundation/minix/issues

[Greg Reagle]

Sounds great.  Congratulations, and thank you!

Will this new network code be a part of Minix 3.4.0?

[David van Moolenbroek]

Hi Greg,

Thank you!

On Sunday, March 26, 2017 at 8:08:22 PM UTC+2, Greg Reagle wrote:

Will this new network code be a part of Minix 3.4.0?

Good question. Yes it will. As you probably know, the 3.4.0 release is blocked on a fundamental problem with the ARM port [1], and despite recent significant progress [2], the resolution of this blocker is unlikely to be imminent. As such we have decided not to let that situation impede improvements in orthogonal areas, which is why there have already been several significant changes during the 3.4.0 release candidate trajectory (which, admittedly, is a bit of a misnomer at this point). Our best judgement is that the networking project is sufficiently ready that merging it now has more advantages than disadvantages.

Regards,
David

[1] https://github.com/Stichting-MINIX-Research-Foundation/minix/issues/104
[2] https://github.com/Stichting-MINIX-Research-Foundation/minix/pull/195

[Patrick McC^very]

Thanks very much for doing this David !

I am having some trouble with my network driver on 3.4 R5 so I will be first in line to try this out. Please let us know when we can try it out.

Thanks again

[David van Moolenbroek]

Hey,

On Sunday, March 26, 2017 at 10:38:43 PM UTC+2, Patrick McC^very wrote:

I am having some trouble with my network driver on 3.4 R5 so I will be first in line to try this out. Please let us know when we can try it out.

Will do! It should be a couple of days; I'm hoping/guessing that Lionel will build a new image after everything has been merged. Keep in mind that most network drivers will largely be left unchanged though, so there may be no difference there. If the problem persists, please open a GitHub issue so that we can look into it.

Regards,
David

[David van Moolenbroek]

All,

The network stack project has just been merged. Thanks once more to Lionel, for reviewing the patches and reviving our gerrit server a few times during the merge.. :)

The whole process has taken a while, and so there are a few more open pull requests now. It is probably best if we merge some of those before producing a new rc image, so stay tuned for that!

For those of you who are tracking minix-current [1] or cross-compiling from source [2], the new code is available right now through the normal procedure. If you are tracking current, keep in mind that this merge comes with a docs/UPDATING entry.

Regards,
David

[1] http://wiki.minix3.org/doku.php?id=developersguide:trackingcurrent
[2] http://wiki.minix3.org/doku.php?id=developersguide:crosscompiling

[David van Moolenbroek]

All,

On Sunday, March 26, 2017 at 10:38:43 PM UTC+2, Patrick McC^very wrote:

Please let us know when we can try it out.

Lionel has made a new snapshot image from minix-current, which as such also includes the new network stack. This new image, 3.4.0rc6, is available from:

  http://download.minix3.org/iso/snapshot/

Regards,
David

[PrasannaKumar Muralidharan]

Sounds nice. Great!!!

[Patrick]

Thanks David ! and thanks Lionel for making the image. Just in case,
anyone gets confused later, it's rc6.