Go and Linux Kernel Programming.

[Mohammad Nawazish Khan]

Hi,

Do we have built-in library in Go which could be used for doing low level stuffs like Linux Kernel programming. This is because, hitherto, most of the Linux Kernel programming was done in C and since Go is purported as a step beyond C, it would nice to see such low level library support in Go language.

Do let me know.

-Nawazish,

[Yongjian Xu]

I think like other general unix programming environment, Go relies on syscalls (not libc, it has it's own syscall lib), but no matter what syscall impls, it relies on kernel to provide fundamental resources, e.g. a simple Print will make syscalls. If you'd like to programming kernel, that means you can't rely on anything (there wouldn't be a syscall lib for you to use). Not saying it isn't possible, but it could be hard (you will need to write all fundamental utilities on your own). 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mohammad Nawazish Khan]

Thanks for your note.

Could you refer a good starting point for writing linux kernel in Go!

-Nawazish

[Mohammad Nawazish Khan]

Could you refer a good starting point for writing linux kernel in Go!

[branimir....@gmail.com]

Go is high level language. Writing kernel in go is exercise in  futility...

[Andy Balholm]

> On Oct 27, 2014, at 12:58 PM, branimir....@gmail.com wrote:
>
> Go is high level language. Writing kernel in go is exercise in futility…

At one time people would have probably said the same about C.

Go isn’t so high-level that it couldn’t be used to write a kernel. Of course you would have to do a few things in assembler, but that’s true with C as well.

The OP isn’t very clear what he wants to do: write modules for the Linux kernel in Go, or rewrite the whole kernel in Go. If he just wants to write modules, it seems like the first step would be to get dynamic loading of userspace code working. If he wants to rewrite the whole Linux kernel in Go, bringing the experimental bare-metal Go runtime up to date would be a trivial warmup exercise in comparison.

[branimir....@gmail.com]

Well, for a start how can I write library in Go that can get called from other language eg C... if that is possible next step would be writing module for kernel.
Also, without pointer arithmetic low level programming is pain. Also, one can't use GC in kernel.

[Andy Balholm]

> On Oct 27, 2014, at 2:43 PM, branimir....@gmail.com wrote:
> Well, for a start how can I write library in Go that can get called from other language eg C... if that is possible next step would be writing module for kernel.

Neither of these has been implemented yet. But that tells us more about the Go team’s priorities than about how high-level the language is. And neither is necessary if you write a complete kernel in Go.

> Also, without pointer arithmetic low level programming is pain.

The crypto/tls package contains some pretty low-level programming. But it doesn’t even import “unsafe”. And I would venture to guess that it’s easier to hack on than OpenSSL. :-P

Of course a kernel would need to contain some unsafe code. But if I needed to write a kernel, I would find C’s lack of strings, maps, channels and GC more of a pain than Go’s lack of (convenient) pointer arithmetic.

> Also, one can't use GC in kernel.

Why not?

Of course you need to be careful about latency, but kernels have been written in GC languages, such as Oberon. Microsoft’s Singularity is not just garbage-collected, but managed code (like C#).

[branimir....@gmail.com]

On Tuesday, October 28, 2014 12:48:25 AM UTC+1, Andy Balholm wrote:

> On Oct 27, 2014, at 2:43 PM, branimir....@gmail.com wrote:
> Also, without pointer arithmetic low level programming is pain.

The crypto/tls package contains some pretty low-level programming. But it doesn’t even import “unsafe”. And I would venture to guess that it’s easier to hack on than OpenSSL. :-P

Of course a kernel would need to contain some unsafe code. But if I needed to write a kernel, I would find C’s lack of strings, maps, channels and GC more of a pain than Go’s lack of (convenient) pointer arithmetic.

Well, I forgot that one can initialize slice from pointer and work normally. That is workaround for pointer arithmetic.

> Also, one can't use GC in kernel.

Why not?

Of course you need to be careful about latency, but kernels have been written in GC languages, such as Oberon. Microsoft’s Singularity is not just garbage-collected, but managed code (like C#).

I didn't thought about latency rather Linux kernel. If you write kernel from scratch  you can have GC, but in Linux kernel module you can't have GC (or default GC).

[Uli Kunitz]

C is the perfect language for writing a kernel, because it only requires an address space and a stack. The Linux assembler code starting the kernel maps the physical memory to virtual memory and sets the stack address and then switches to C. That's all.

Go requires a runtime supporting garbage collection, Go routines and support for maps, channels and slices. On top the kernel would need an adapted CGO version to support calls in both direction. I don't think it is impossible to do, but it's hard to see what the benefits should be rectifying the implementation and maintenance effort required. The code would have to live outside of the kernel for a long time requiring significant effort to adapt the code to new kernel versions, because there is no such thing as a stable kernel API.

I think it's much more simpler to stay with C in the kernel and write user-space components in Go. This can be done right now and doesn't require a major coding effort.

[Mohammad Nawazish Khan]

If we still have to rely on C as a language for writing kernels (entire kernel or modules/components thereof) then I feel Go would miss a vital armoury in its arsenal. On the other hand Go would justice to its inception if we are able to write a full stack kernel (*nix or otherwise) from the scratch in it. I am not against C - its an splendid language very close to the ISA layer - but those are just my views about Go. 

What do u think? 

-Nawazish

On Tuesday, October 28, 2014 12:03:10 AM UTC+5:30, Mohammad Nawazish Khan wrote:

[Dave Cheney]

On Tuesday, 28 October 2014 18:52:59 UTC+11, Mohammad Nawazish Khan wrote:

If we still have to rely on C as a language for writing kernels (entire kernel or modules/components thereof) then I feel Go would miss a vital armoury in its arsenal. On the other hand Go would justice to its inception if we are able to write a full stack kernel (*nix or otherwise) from the scratch in it. I am not against C - its an splendid language very close to the ISA layer - but those are just my views about Go. 

What do u think? 

I would be interesting to write a kernel in a language like Go. Microsoft research did something similar a few years ago with C# on raw tin with their Singularity project.

With that said, the number of kernels to be written vs the number of programs that run _on top_ of a kernel, is a very very small ratio. I think that reflects the priority of the Go team.

[Mohammad Nawazish Khan]

To be very honest, I am very new to low level programming, system programming, kernel programming and user-space components etc (although I have been programming for the web). But I am extremely interested for low level programming at the kernel and the user-space level. So I would earnestly request you to guide me (1) from where should I begin (2) what are the pre-requisites for kernel and user-space components programming (please elaborate) (3) What would be the easiest kernel module code I should try (4) what would the easiest user-space component code that I should try and finally (5) I have been writing in Java for all the web applications that I have worked on and so I have forgotten good part of C language; and I am currently learning Go, so do I need to refresh C one more time.

-Nawazish  

On Tuesday, October 28, 2014 12:03:10 AM UTC+5:30, Mohammad Nawazish Khan wrote:

[wkharold]

On Monday, October 27, 2014 4:17:28 PM UTC-5, Andy Balholm wrote:

> On Oct 27, 2014, at 12:58 PM, branimir....@gmail.com wrote:
>
> Go is high level language. Writing kernel in go is exercise in  futility…

At one time people would have probably said the same about C.

No, no one ever said that about C for the simple reason that parts of the Unix/Linux kernel were written in C from the beginning.

While I'm sure some clever person might be able to develop *an* operating system kernel primarily in Go it is currently difficult/impossible to write Go code that could run in the *Linux* kernel.
 

[Andy Balholm]

On Oct 28, 2014, at 7:26 AM, wkharold <wkha...@gmail.com> wrote:

No, no one ever said that about C for the simple reason that parts of the Unix/Linux kernel were written in C from the beginning.

The UNIX kernel was originally written in PDP-7 assembler, then ported to PDP-11 assembler. It was re-written in C mostly in 1972 and 1973. (See http://en.wikipedia.org/wiki/C_(programming_language)#Early_developments ) That article also states, “Unix was one of the first operating system kernels implemented in a language other than assembly.”

I don’t know if there was skepticism about the choice to re-write the kernel in C at the time, but using such a “high-level” language to write a kernel was at least somewhat nonstandard.

While I'm sure some clever person might be able to develop *an* operating system kernel primarily in Go it is currently difficult/impossible to write Go code that could run in the *Linux* kernel.

Yes, it is currently impossible to write Linux kernel modules in Go. But the problem is lack of compiler and runtime support, not the fact that Go is too high-level. (Go’s high-level features do make this compiler and runtime support more difficult, but a Go runtime that would work inside the Linux kernel would probably be no harder to develop than the current Go runtime was.)

[Yongjian Xu]

Agreed. I think it's mainly the lack of support. Go's runtime is a userspace application, everything related to resource management is relying on the kernel. In order to write the kernel itself, the runtime would have to support itself (doing all memory, IO, device, etc managements). Assume the runtime can run on bare metal, then it would be possible to implement kernel primitives (e.g. syscalls, interrupts) in pure Go...

--

[zj huang]

I am not good at golang so far, but I think that maybe you'd better use C instead of GO

[andrewc...@gmail.com]

Rust might be stable by the end of the year which might make slightly safer code overall if you can be bothered learning it.

[paulo....@gmail.com]

Am Montag, 27. Oktober 2014 20:58:23 UTC+1 schrieb branimir....@gmail.com:

Go is high level language. Writing kernel in go is exercise in  futility...

This is pure non-sense. Go only requires packages available that offer the required hardware access,
in addition to its unsafe package.

It has been done multiple times in system programming languages that even offer more abstractions than
Go, like Cedar, Oberon, Oberon-2, Active Oberon, Modula-3, Sing#.

The Swiss Federal Institute of Technology enjoyed quite a few Oberon workstations back in the mid-90's.

--
Paulo

[branimir....@gmail.com]

On Thursday, October 30, 2014 8:11:31 PM UTC+1, paulo....@gmail.com wrote:

Am Montag, 27. Oktober 2014 20:58:23 UTC+1 schrieb branimir....@gmail.com:

Go is high level language. Writing kernel in go is exercise in  futility...

This is pure non-sense. Go only requires packages available that offer the required hardware access,
in addition to its unsafe package.

Go uses userspace runtime . Besides packages that are useless on bare metal it needs
bare metal runtime.

It has been done multiple times in system programming languages that even offer more abstractions than
Go, like Cedar, Oberon, Oberon-2, Active Oberon, Modula-3, Sing#.

The Swiss Federal Institute of Technology enjoyed quite a few Oberon workstations back in the mid-90's.

For a start I really like Go, but I it is useless to me as I can't even write shared object library
and call it from C/C++. I see there is plugin for gccgo https://github.com/unbit/uwsgi/blob/master/plugins/gccgo/gccgo_plugin.c
but I try just to initialize runtime and failed :(
 

--
Paulo

[Gerard]

On Thursday, October 30, 2014 11:41:21 PM UTC+1, branimir....@gmail.com wrote:

Go uses userspace runtime . Besides packages that are useless on bare metal it needs

bare metal runtime.

The *current* implementation does, yes. The language itself however doesn't prevent writing kernels.
 

For a start I really like Go, but I it is useless to me as I can't even write shared object library
and call it from C/C++. I see there is plugin for gccgo https://github.com/unbit/uwsgi/blob/master/plugins/gccgo/gccgo_plugin.c
but I try just to initialize runtime and failed :(

 

Again, this is the *current* implementation. There are ideas however to implement shared libraries in Go. Just give it some time. The language is still young.

I think that for the Linux kernel Go isn't a good fit. Maybe Rust one day although that depends on Linus ;-)

Where I see Go is a microkernel. Well, not the microkernel itself, but the servers (drivers / filesystems / daemons etc.) where one server could be the runtime.

But this is just thinking out loud ;-)

[branimir....@gmail.com]

On Friday, October 31, 2014 1:04:10 AM UTC+1, Gerard wrote:

 

For a start I really like Go, but I it is useless to me as I can't even write shared object library
and call it from C/C++. I see there is plugin for gccgo https://github.com/unbit/uwsgi/blob/master/plugins/gccgo/gccgo_plugin.c
but I try just to initialize runtime and failed :(

 

Again, this is the *current* implementation. There are ideas however to implement shared libraries in Go. Just give it some time. The language is still young.

gccgo can produce shared libraries that can be called from C/C++ (whatever). Problem is that there is no exported Go function to initialize runtime from C and all
my attempts failed :(
 

[Gerard]

We are still talking about kernels?

[branimir....@gmail.com]

On Friday, October 31, 2014 1:15:27 AM UTC+1, Gerard wrote:

Again, this is the *current* implementation. There are ideas however to implement shared libraries in Go. Just give it some time. The language is still young.

gccgo can produce shared libraries that can be called from C/C++ (whatever). Problem is that there is no exported Go function to initialize runtime from C and all
my attempts failed :(

We are still talking about kernels?

No. For a start it would be really useful to me  to have library in Go.

[jonathan....@gmail.com]

People are writing unikernals in OCaml, which is a higher level language than Go, so I imagine the same thing could be done in Go. Raw Go on Xen would be pretty awesome.

[Gerard]

The discussion in golang-dev. In short: the Go team is aware and thinking about it.

Right now there is AFAIK not an easy compiler flag solution.

[wkoc...@gmail.com]

This guy tried with Rust: http://jvns.ca/blog/2014/09/18/you-can-be-a-kernel-hacker/ and http://jvns.ca/blog/2014/03/12/the-rust-os-story/
Maybe you could check his code or try talking to him. There's also a video of his talk somewhere on Mozilla Air.

[Konstantin Khomoutov]

On Thu, 30 Oct 2014 17:11:13 -0700 (PDT)
branimir....@gmail.com wrote:

[...]

> gccgo can produce shared libraries that can be called from C/C++
> (whatever). Problem is that there is no exported Go function to
> initialize runtime from C and all
> my attempts failed :(

In general this can't really work because the spec on the Go lanugage
guarantees certain things are available when Go code is running, which
are not possible in C environment. Discussed in detail at [1].
I hear you telling "initialize the runtime" but once you put more
thought into how that runtime initialization would have to be carried
out and how it would be supposed to play with the C runtime, you'll
understand that the matters are way more complicated that they seem on
the first sight. I'd say that currently I see no sensible way to make
Go runtime somehow "initializable" from a C program running on a
contemporary commodity OS.

1. https://groups.google.com/d/topic/golang-nuts/u9ZzPKNP4W0/discussion

[Fino]

my idea:

C itself need some improve.
1) drop the header file, use package;
2) have multi return value.
...

BR-fino

[Gerard]

3) Replace zero terminated strings/arrays with a pointer+length tuple.
4) Have some sort of "for each" in a for construct.
5) Use the package namespace.
6) Use some sort of "defer".
7) Standardize on POSIX so that the GNU Autotools are obsolete.
8) Have fixed length standard types. E.g. what is the size of int or long?
9) Strong types.
10) Improve the macro system. That of Rust looks OK (at first sight).
11) Have an option to use safe memory management (looks rather stupid btw).
12) Drastically improve the syntax.
13) ...
14 t/m 35) ...

So yes, C should have been improved, a long time ago. It also wouldn't and shouldn't be C anymore.

[Fino]

each hole has one proper language to sit,

C sits on the hole of 'hand memory management' + native + no class;
JAVA: complied, 'auto memory' + virtual opcode;
C++ : complied,  native + class + hand memory;
PY : dynamic + virtual opcode;
JS : inside browser + dynamic;
go: complied, native, auto memory management;
MATLAB: math + data visualization;
....

so what is missing?  can we put golang inside browser, +JIT, to replace JS(missing strong type)? 

在 2014年11月2日星期日UTC+8下午4时29分22秒，Gerard写道：

[Gerard]

On Sunday, November 2, 2014 11:07:38 AM UTC+1, Fino wrote:

 

so what is missing?  can we put golang inside browser, +JIT, to replace JS(missing strong type)? 

I don't understand the question.