New language based on Go

[Jamie Hall]

Hi,

We're starting a project to make a new programming language for writing operating system kernels, using Go as a starting point. It will be an open source project under the same BSD license, but re-branded, essentially forking from Go 1.5. We hope the Go team and community is comfortable with our approach and this seemed to be the best way to get in touch. This isn't an attempt to advertise the project, it just seemed wrong to publish the repository without letting you guys know first.

Best wishes

[andrewc...@gmail.com]

Do you have an official project page? I don't think there is a problem posting a link so long as you don't spam this list in the future. 

[Markus Zimmermann]

I am interested in what you are changing to justify a fork. Maybe you can just add new packages or even incorporate some changes back into the Go project so that you can manage your project without a fork. The whole Go community could benefit from this.

[hutton...@gmail.com]

The main constraints on using Go as a kernel development language, from our perspective, is the garbage collection and go-routines (which of course includes the scheduler and runtime). So removal of these parts will be the most significant changes between our language and Go.

I think the scale of these changes (particularly the loss of the garbage collector) definitely warrant the fork, as it affects every part of the language and standard library. 

Our main aim is a language with the simplicity and expressiveness of Go but with all the bits needed for kernel development. 

[David Chase]

I'm curious -- how certain are you that GC and goroutines are incompatible with kernel development?

I assume you have a better reason for this than conventional wisdom.

Years ago there was work done in Modula-3 for OS development: http://www-spin.cs.washington.edu/

(This was about 20 years ago.)

I'm asking this because it seems that people usually overestimate the need for twiddly bits in OS and

other 'low-level" code, and you might find that a modest extension (gc-independent goroutines?

safepoints with a guaranteed maximum polling interval?) or disciplined use of unsafe would serve

you better.  Modula-3 provides some useful examples here.

[Konstantin Khomoutov]

On Mon, 31 Aug 2015 08:55:03 -0700 (PDT)
David Chase <drc...@google.com> wrote:

> I'm curious -- how certain are you that GC and goroutines are
> incompatible with kernel development?
> I assume you have a better reason for this than conventional wisdom.
> Years ago there was work done in Modula-3 for OS
> development: http://www-spin.cs.washington.edu/
> (This was about 20 years ago.)
> I'm asking this because it seems that people usually overestimate the
> need for twiddly bits in OS and
> other 'low-level" code, and you might find that a modest extension
> (gc-independent goroutines?
> safepoints with a guaranteed maximum polling interval?) or
> disciplined use of unsafe would serve
> you better. Modula-3 provides some useful examples here.

[...]

The single interesting bit among all the usual blah-blah on the site
you've referred to is a document titled "When NEW breaks" apparently
discussing what happens if a memory allocation -- which is expected to
succeed -- fails. But that document is sadly unavailable, and I failed
to google an alternative source for it.

[Rob Pike]

I agree with David. There is a widespread assumption that garbage collection and kernels cannot coexist but history shows a number of examples that demonstrate otherwise.

Speaking as a one-time kernel writer, I would have loved to have garbage collection available inside the kernel. Hell, half the code in the kernel is tied to memory management.

-rob

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Jones]

Agree completely with the storied Mr. Pike on this point. The meaning of operating system has diffused enough that there are few if any set walls where most people have grown to expect them. 

When you look at hypervisors and descendants from IBM’s early VM/370 to Docker instances, you see that there is a natural tendancy to use the OS as a service library for a single program’s execution. I was part of an effort to build an application-level operating system; it worked and worked well across a range of computer systems. A few weeks ago I met with a group that has built provably secure microkernel / hypervisor stacks that run on systems as small as USB memory sticks(!) This area of work remains interesting and important. It seems increasingly valuable for the IoT world.

My instinct would be to focus on the GC topic from the other POV—how low can you go with a Go-like approach and how thin, tiny, and meta can the non-GC bit be below that. Likely the answer is to have an almost policy-free metakernel that turns to higher level general code for strategy and policy. The answer to how low can you go is related to the email here with the comment “what happens when NEW fails” because there is the controlling corollary question, “for which other programs, services, threads, goroutines, etc. should having NEW fail make this particular code terminate?”

If the answer is “all” then you have a single dedicated system without survivors; for “none” then you are executing in an isolated abstract machine. The trend is strongly for the decoupled none/few…

Michael

— 

Michael Jones, CEO  •  mic...@wearality.com  •  +1 650 656-6989 

Wearality Corporation  •  289 S. San Antonio Road  •  Los Altos, CA 94022

[Jamie Hall]

Our reasoning behind the lack of GC was that the resulting simplicity in the implementation would be worthwhile. We're very much open to suggestions and in fact plan not to remove the runtime until fairly late in the process, so we may have a chance to compare both approaches.

[Jesper Louis Andersen]

On Mon, Aug 31, 2015 at 9:13 PM, Jamie Hall <the.sl...@googlemail.com> wrote:

Our reasoning behind the lack of GC was that the resulting simplicity in the implementation would be worthwhile. We're very much open to suggestions and in fact plan not to remove the runtime until fairly late in the process, so we may have a chance to compare both approaches.

Take a look at the approach used by MirageOS, which is written in OCaml, with a GC. You can essentially push the layer which is not written in OCaml outward toward smaller and smaller pieces. For instance, the entirety of the TCP/IP stack can easily run under GC control. So can the VFS layer, policy managment and so on. For a 1995 precedent, look up papers on the Fox project which had a fairly complete environment in Standard ML at CMU, with the system running on the bare metal, with a GC.

--

J.

[Tim Hawkins]

I was periphialy involved in an attempt to build an os/distributed application stack in modular 3. This would have been about 1985,  while working for Linotype Paul in cheltenham, UK. I seem to remember it being a disaster, and it was abandoned in favor of the apple II os.  The requirment was to build a simple os that could boot a lot of diskless workstations for newspaper and large book publishing house data entry. In the end the plumetting cost of consumer PC's and comodity os's made it infeasable at thier volume levels. Apple lisa was around at the same time which was built using pascal, im not sure if that used GC.

--

[David Chase]

If it was before about 1989 it's not likely it was Modula-3; I don't recall the exact date the first implementation was released to the world, but it was around then, and certainly not before 1988 (because I did the backend, and we didn't even have a grammar for the front-end till sometime late in 1987).  It's possible you are thinking of Modula-2+ on some version of the BBC machine running CAMEL (the C And Modula Execution Library), though I am not entirely sure what state that was in in 1985.  I first saw one in late 1987, but got the impression that it was still under development.

As near as I understand it the main problems with using a mostly-managed language for an OS involve timing (response to interrupts) and the ability to reclaim resources "with extreme prejudice".  You can model an interrupt source as a channel, and the main issue is ensuring that the Goroutine waiting on it is activated very quickly -- thus, the idea that the compiler ensures very short intervals between safepoint polls.  A Goroutine is a fatter beast than an interrupt handler, so maybe not, or maybe a very slim one can be created.  Safepoint polling trades delays for having a much simpler and more well-defined thread state to save/restore.

Resource reclamation is complicated by programmers having the ability to write cleanup handlers that can contain any sort of code; as long as nothing prevents a malicious or creatively-buggy handler from resuming execution of normal code and creating cleanup contexts with even more handlers, you cannot guarantee that shutdown will always complete cleanly. Wearing a compiler-writer's hat, maybe you could work on this with something like a type system, if one could be devised to statically verify the sort of reasoning that humans already use to convince themselves that clean shutdown will occur.

[Sebastien Binet]

I don't know if the OP is part of the team behind `clive`, but it
seems somebody already forked the go toolchain (IIRC, to provide extra
semantics to channels) and wrote an OS in go:
http://lsub.org/ls/clive.html

-s

[naut...@googlemail.com]

Hello,

it is sort of funny, we are trying to do a very similar thing, but in a completely different way.

During the last years, we developed an object-based research processor that does real-time garbage collection in hardware. Now we want to leverage its benefits regarding object-based memory protection by designing a novel research operating system. In a nutshell, the main idea is to replace coarse-grained virtual memory protection by fine-grained object-based memory protection, and - as Rob wrote about in a reply to your post - to benefit from garbage collection in operating system design.

Our processor differs from a standard processor in that it knows about objects and pointers at the ISA level. Even in machine code, it is impossible to convert a pointer to a non-pointer or vice versa, to do pointer arithmetic, or to violate the bounds of an object. This is why we cannot use C to write an operating system. Regarding memory management, the processor's abstraction layer is just too high for C.

The first attempt was to develop a subset of Java (affectionately called java--), but the project was a complete failure. The main reason was the lack of orthogonality in Java. In our experience, taking away some features of the language made the rest of the language collapse.

Then we stumbled across Go. Thanks to its orthogonal design, it was possible to take away features without making the rest collapse. I guess you share that experience.

Like you, we took away goroutines (at least to start with). In contrast to you, however, we explicitly want to take advantage of the built-in hardware garbage collection feature of our processor. That processor, however, requires that there is a bijective relationship of pointer values and objects, a requirement that go violates here and there, for instance with slices and the address operator, both of which require internal pointers, i.e. pointers that point to somewhere inside an object. So we are currently designing a subset of go (affectionately called GoSUB [pun intended]) and a research prototype compiler, written in Java and started from scratch. 

At the moment, we work at the language and the compiler. Things look very promising. The compiler is 95% feature complete. But we did not actually start with the operating system design yet.

Because of the entirely different methodology used in our projects, the intersection of interests may be small, but maybe we can share our experiences, especially designing a subset of go (of course not on this list - we do not want to annoy genuine gophers).

Best wishes, Matthias. 

[David Walther]

On Mon, Aug 31, 2015 at 09:39:18AM -0700, Rob Pike wrote:
>I agree with David. There is a widespread assumption that garbage collection
>and kernels cannot coexist but history shows a number of examples that
>demonstrate otherwise.
>
>Speaking as a one-time kernel writer, I would have loved to have garbage
>collection available inside the kernel. Hell, half the code in the kernel is
>tied to memory management.

Did these kernels have a way to disable garbage collection for implementing the
top half of an interrupt handler?

I notice some superficial similarities in Go garbage collection, and the One
Reference Only GC in newLISP. With ORO I get consistent run times, no
unpredictable GC.

Since C.malloc and C.free work in Go, I suppose one could end run the GC in
that way. Is there an approved way of suspending GC in Go? I want to
implement my own garbage collected language. Nice to have GC most of the time,
but also nice to suspend it or get it out of the way at times.

David

[David Chase]

I don't want to make any promises about at all about what Go will do, but if you want to frame this as what sort of interface you'd like to have with your GC, I think you want to model interrupt handlers as special threads or goroutines, that don't touch managed memory at all.  If you can get to that point, then the GC can leave those threads alone no matter what.  HOWEVER, there's also the problem that these special threads must eventually hand their data off to some other thread that does interact with the GC, which means you need to be able to buffer as many interrupts as necessary to cover the maximum time T during which a thread might block a thread's progress.

Note that you can't really "disable" a garbage collection (in general) that is already underway.  It's not that big a deal to delay the start of a GC for the length of time it takes a (well-written) interrupt handler to do its business; the problem is what to do when the GC is already running.  It's also possible that an interrupt handler is compiled slightly differently; for example, write barriers always (and perhaps "better" write barriers or read barriers) that would allow it to run concurrently with GC even if it did manipulate managed pointers (I haven't figured this out, I'm just pointing out that if we focus on exactly where the problem is there might be solutions what wouldn't necessarily generalize to the entire program).

[minux]

On Thu, Sep 3, 2015 at 5:03 PM, <naut...@googlemail.com> wrote:

During the last years, we developed an object-based research processor that does real-time garbage collection in hardware. Now we want to leverage its benefits regarding object-based memory protection by designing a novel research operating system.

Do you have any pointer to that? Publications? Websites?

Thanks.

I'm also wondering what's the ultimate goal for such a processor targeting

high level languages? Secure/verifiable software perhaps?

Is executable code also managed as objects?

[Peter RK]

I think it should be based on C, but with features of Go, such as multiple return values, package and interface. We can try to translate Linux or FreeBSD source into a new form, like what has been done on the Go toolchain.

[andrewc...@gmail.com]

I started work on a language which is like this. C with go packages, syntax, multiple return + defer . I still have plans to continue, though first am finishing my own C compiler to be sure I understand all issues involved.

[naut...@googlemail.com]

The original motivation was real-time garbage collection for embedded systems. The processor does all the synchronisation stuff like read and write barrier checking and the (usually prohibitively expensive) synchronisation for fine-grained compaction (i.e. access an object while it is moved) in hardware. Now we are interested in secure and robust systems. The main idea is to enjoy the luxury and robustness of objects and garbage collection in operation system design.

The processor does not exactly target high level languages. Apart from support for objects and pointers, the processor is a standard RISC. Currently, we implement a superscalar out-of-order implementation of the architecture (FPGA prototype).

The original architecture does not manage code as objects. Version two of the architecture does (still unpublished) .

The specification of our subset of Go is almost complete, but still a bit too beta to be shown. Work on the operating system has not started yet.

Publications on the processor and hardware-supported garbage collector are here: http://www.ikr.uni-stuttgart.de/en/~meyer/XRDTabSelect=PUBS.

[Tim Hawkins]

i think the processor was from texas instruments, 9900,  you may be right about the 2+. You where spot on about the determinism problem. ISTR the project was plauged with timing and race condition problems in the lower levels of the stacks. But its going back a long time now, and as i said i was only periphialy involved.

[fatdo...@gmail.com]

The issue is getting it past those 10 core devs that have a strangle-hold on the project. I encourage this author to proceed, experiment. I encourage them to create a group, and occasionally return here and post, reminding golang programmers that your project exists and that there is another option similar to golang. I work with a small group, we are developing a tool set that based off the spec but with a few minor changes. Things are going great, we are just not as brave to show our work yet- its rough. One example is how we changed the syntax of maps and slices. However our work is done from scratch and the compiler only targets one platform- freebsd/amd64.

Perfect example is the table proposal. Or the many other proposals that are tagged unplanned. Seems the new way of dealing with new proposals is to appease those authors by apparently accepting the proposal and then tag it 'unplanned' to be forgotten as time passes, rather then the old way of just shooting them down immediately.

[Roger Pack]

On Monday, August 31, 2015 at 2:55:54 AM UTC-6, Jamie Hall wrote:

Hi,

We're starting a project to make a new programming language for writing operating system kernels, using Go as a starting point. It will be an open source project under the same BSD license, but re-branded, essentially forking from Go 1.5. We hope the Go team and community is comfortable with our approach and this seemed to be the best way to get in touch. This isn't an attempt to advertise the project, it just seemed wrong to publish the repository without letting you guys know first.

Interesting.  I wonder if there would be some utility to having an intermediary "go bytecode" format so multiple languages can target the go runtime more easily...just throwing it out there, I realize you could convert the code to go as well. :) 

[Sebastien Binet]

that *could* also be handy for a go interpreter :)

-s

[paulo....@gmail.com]

Have a look at all the OSes that were written in Oberon, Active Oberon at ETHZ.

Niklaus Wirth web site still has lots of information:

https://www.inf.ethz.ch/personal/wirth/

Specially relevant are the  Project Oberon books, the 2013 revision contains the whole stack
from the hardware and OS.

The AOS web site is still live and contains also relevant information

http://www.ocp.inf.ethz.ch/wiki/Documentation/Oberon

The screenshots show how it used to be like
http://www.ocp.inf.ethz.ch/wiki/Documentation/WindowManager

If you want to see screenshots for all applications, there is an article I wrote long time ago,

http://www.progtools.org/article.php?name=oberon&section=compilers&type=tutorial

As for Modula-3, check

http://www.modula3.org/threads/

http://www.hpl.hp.com/techreports/Compaq-DEC/SRC-RR-52.html

https://cseweb.ucsd.edu/~savage/papers/Sosp95.pdf

Going back to Xerox PARC, have a look at Mesa/Cedar

https://archive.org/details/bitsavers_xeroxparcteCedarProgrammingEnvironmentAMidtermRepo_13518000

--
Paulo

[you...@z505.com]

On Monday, August 31, 2015 at 2:55:54 AM UTC-6, Jamie Hall wrote:

Hi,

We're starting a project to make a new programming language for writing operating system kernels, using Go as a starting point. It will be an open source project under the same BSD license, but re-branded, essentially forking from Go 1.5. We hope the Go team and community is comfortable with our approach and this seemed to be the best way to get in touch. This isn't an attempt to advertise the project, it just seemed wrong to publish the repository without letting you guys know first.

Best wishes

Are you going to reintroduce the stack instead of heap based everything? One of the reasons I still use Delphi or C is because the stack is available.... to me the stack is like automatic garbage collection. Using a record in delphi requires no Free or Create or New or Dispose.... and speeds up programs that require such. Heap fragmentation? not an issue with the stack.  Also what about the runtime? It seems exe's and elfs in golang are increasing in size.... more and more run time stuff is thrown into the exe and elf .. this is a similar problem to Lazarus EXE's which grow huge. A hello world is 13MB with debug stuff included. Are you going to develop the language for embedded systems, like an Oberon style language, where the Exe is not 2MB or 13MB?  I realize all my concerns seem to be premature optimization, as hard drives are big these days. But that's what got java in trouble. Hard drives are big so who cares about bloatware. I do.

[you...@z505.com]

On Tuesday, September 29, 2015 at 12:36:22 PM UTC-6, paulo....@gmail.com wrote:

 

Have a look at all the OSes that were written in Oberon, Active Oberon at ETHZ.

Niklaus Wirth web site still has lots of information:

https://www.inf.ethz.ch/personal/wirth/

Yeah, oberon is a fascinating research project.  However one of the reasons that I started using Golang is that oberon is one of those fade away into academic obscurity projects, far disconnected from reality.  Niklaus Wirth is "out of site, out of mind". He doesn't participate on mailing lists, USENET, is hard to get a hold of... basically stuck in 1960's times when people wrote books about software and there was no internet communication... the Dijkstra way. Dijkstra wrote articles and started blogging back in the 1960's via postal mail, but was skeptical of the whole internet fad... Niklaus Wirth seems to be the same: he writes books, and expects you just to accept his big design up front without any conversation about it in public forums with him. Out of site, out of mind.... It's not like GoLang where you see people like Rob Pike participating in conversations, and you have a whole bunch of golang youtube videos, conferences, meetups, blogs.... Oberon is pretty much dead ever since 1995 or so when usenet posts were hot. But even back in 1995 when things were hot, Wirth was out of site, out of mind.. not participating in any discussions... AFAIK. I don't want to make too many judgments or criticisms because he's smart, I just don't think he communicates enough. Ivory tower fading away into obscurity, one step at a time...

[paulo....@gmail.com]

I have met him when he did an Oberon presentation at CERN around 2003.

He is a very interesting person.

However given Oberon's influence in Go, an interesting project would be something like "Goberon".

Rewriting Native Oberon System 3 in Go, including the nice Gadgets framework.

Maybe then more people would accept the work that many people have done in GC enabled systems programming languages.

However such project is an huge undertaking (bare metal Go runtime + rewrites + bare minimum drivers), probably worth of some CS OS thesis.

--
Paulo

[Henry Adi Sumarto]

If the main objection is GC, given what the author wanted to accomplish, it is more practical to just use C language and focus the energy in building more libraries and tool sets. At least I would have gone the C route if Go isn't an option.

[Steven Wiley]

Dave Chase said, " I think you want to model interrupt handlers as special threads or goroutines, that don't touch managed memory at all.  If you can get to that point, then the GC can leave those threads alone no matter what. "

I think that would be a great addition to Go; to have way to spawn threads (goroutines) that do not touch the GC, and cannot be interrupted by them.  Perhaps you could declare a bunch of standard, garbage collected structs, then tell the GC these are off limits for the duration of the code block, and defer their release until the goroutine completes?

Another great addition would be to be able to declare thread priorities, for instance declaring a minimal time interval a high priority thread (goroutine) can block before taking control over a lower thread.

IMHO, doing both of these things might Go competitive in embedded applications. 

Dave also mentions that the threads must pass off their data at some point. I think the streams can work here, and are Go idiom compatible.  Whatever the size is of the struct passed by the stream times the capacity of the stream is all you need as long as the (slow, garbage collecting) receiver understands it needs to copy the data. Perhaps the contract should be that if the high-priority guaranteed non-blocking sender exceeds the buffer while the receivers stall, some structs will be discarded and a dropped count reported to the receivers.

A classic example is user interface thread. How do you make sure that it never blocks even while the user might furiously click ahead of the background tasks? I don't think a good answer exists at the moment for Go or other less mature languages like nim, although I think nim has separate GC's for different threads. (I can't speak for rust.) Nothing, except maybe C, seems to be really able to get this right at the moment. Am I right in thinking that Chrome-browser and Firefox, all still stutter because no one can write an interrupt proof user interface thread?