Porting Go to New Platforms

[distributed]

I have read on the web about the talk Aram Hăvărneanu gave at FOSDEM'14:

https://fosdem.org/2014/schedule/event/porting_go_to_new_platforms/

Assuming there were slides, are they available somewhere? Will there be a video of the talk at some point? :)

[Aram Hăvărneanu]

Hi, thanks for your interest, my slides are here:
https://bitbucket.org/4ad/gofosdem2014.

They are just text files, but I can give you the acme scripts I
used, if you care about these things. The numbered files are the
actual slides, porting.talk are my speaker notes.

Videos will be available here, at some point:
http://video.fosdem.org/2014/K4601, but the presentation wasn't
very good. I underestimated the time it would take me to talk about
things so I only talked about one third or maybe half the things
I hoped to talk about, and I didn't show any demos.

I might write in depth about all this at some point though.

--
Aram Hăvărneanu

[Mathieu Lonjaret]

Yes, we filmed everything, but we're waiting for the fosdem peeps to
do the editing/encoding/etc and put them online.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[distributed]

Thanks for the slides!

 

They are just text files, but I can give you the acme scripts I
used, if you care about these things. The numbered files are the
actual slides, porting.talk are my speaker notes.

Yes, please. This would be a good opportunity for me to try out acme.

I might write in depth about all this at some point though.

I'd be very happy to read that kind of thing.

I'm currently, at least in part, working with QNX systems. Out of curiousity, I'd like to gauge how much work a port would be, though I suspect that the answer is "way above my pay grade" ;)

[Aram Hăvărneanu]

>> used, if you care about these things. The numbered files are the
>> actual slides, porting.talk are my speaker notes.
>
> Yes, please. This would be a good opportunity for me to try out
> acme.

mkdir /tmp/1
hg clone https://code.google.com/p/aram-dotfiles/ /tmp/1

Copy /tmp/1/bin/Slide{,-,+} somewhere in your path.

In acme, open 000.start from my presentation repo, copy Slide-
Slide+ in the tag and middle click on them to move between slides.

> I'm currently, at least in part, working with QNX systems. Out of
> curiousity, I'd like to gauge how much work a port would be, though I
> suspect that the answer is "way above my pay grade" ;)

Go on QNX would definitely be something I'd like to see. For such
a port to really shine, you'd need to rewrite the scheduler and the
garbage collector, which is a huge task, but that's not required
for an inital port.

QNX uses ELF, but I believe the ELF layout it uses is quite different
compared to other operating systems. The Go linker generates ELF,
but you need to modify it to produce what QNX expects. It won't be
hard adapting it to QNX, the hard bit is that you will (probably)
have to break lots of abstractions in the linker, and you need to
do that without breaking ELF on other target systems.

This job won't be big (in number of lines changed), but it will
probably be the most difficult part of the port. You might find
that QNX actually accepts more traditional ELF layouts, even though
it prefers a special layout for performance. In that case your job
will be easier, although reading QNX documentation I don't think
that's the case. The bright side is that in principle it's easier
to generate the kind of ELF binaries QNX expects that what ELF
binaries we generate now.

I believe statically linked binaries are supported on QNX, so you
don't need to use a shared library to call system calls. Reading
the loader documentation is seems if you do want to use a shared
library (like Solaris does), it's easier than on more traditional
systems.

Other than what I mentioned above, QNX is a Unix system and Go
should be straightforward to port.

--
Aram Hăvărneanu

[distributed]

Nice explanation! 

Go on QNX would definitely be something I'd like to see. For such
a port to really shine, you'd need to rewrite the scheduler and the
garbage collector, which is a huge task, but that's not required
for an inital port.

What would need to be changed in the scheduler and garbage collector? Off the top of my head, I guess that the Go scheduler might need get a notion of thread priorities to interact nicely with other QNX processes. The garbage collector is not very friendly towards real time uses, but that's an incredibly tough problem to solve.

 

I believe statically linked binaries are supported on QNX, so you
don't need to use a shared library to call system calls. Reading
the loader documentation is seems if you do want to use a shared
library (like Solaris does), it's easier than on more traditional
systems.

I'll investigate that. Would Go programs need to link against libc? IIRC, things that are "taditionally" system calls, such as open(), are implemented in libc which uses actual system calls such as MsgSend() and companions.

[Brad Fitzpatrick]

Available now at: http://video.fosdem.org/2014/K4601/Sunday/Porting_Go_to_New_Platforms.webm

[Aram Hăvărneanu]

> What would need to be changed in the scheduler and garbage collector?

If you want to use the real time properties of QNX, you need 1:1
threading, new runtime/debug knobs and a pauseless GC.

But Go can be useful on QNX even without these properties. I have many
small tools written in Go. Even if I couln't write a real time program in
Go, they could help be in developing and debugging the real time program.

> I'll investigate that. Would Go programs need to link against libc? IIRC,
> things that are "taditionally" system calls, such as open(), are
> implemented in libc which uses actual system calls such as MsgSend()
> and companions.

They don't need to link against libc (provided QNX supports statically
linked binaries, which seems to be the case), but it would be much
simpler to use the Unix APIs from libc than re-writting everything to
use QNX primitives.

--
Aram Hăvărneanu

[distributed]

The QNX libc is statically linkable.

I have had some success executing Go programs compiled for linux/386 under QNX. The QNX loader loads and executes these binaries happily. They run up to the first int $0x80, which seems natural. Hacking QNX syscalls via int $0x28 directly into runtime.rt0_linux_386, I have been able to write to stdout and exit the program gracefully.

I have applied for a non commercial developer license, providing the right to run the OS, the C library source, the kernel source and possibly even more. Contrary to the information on the QNX homepage, I have been assured this license does not exist any longer. The QNX ecosystem is changing. For everybody who isn't buying a lot, it seems to be drying up.

[Aram Hăvărneanu]

A shame for something that used to be open source (albeit for a short time).

--
Aram Hăvărneanu

[Aram Hăvărneanu]

> I have applied for a non commercial developer license, providing the right
> to run the OS, the C library source, the kernel source and possibly even
> more. Contrary to the information on the QNX homepage, I have been assured
> this license does not exist any longer. The QNX ecosystem is changing. For
> everybody who isn't buying a lot, it seems to be drying up.

I tried to download QNX today and couldn't find any links. All I could
find are evaluations versions of the SDK (so not even QNX itself).
Such a shame, I was excited when it became open source and when the
DTrace port was announced (which has also been scrapped).

--
Aram Hăvărneanu

[Efraim Sealman]

Hello,

Can you please share some info about how you made the go binary to run on QNX? 

Aram,

Can you please share what have you done for solaris to support go compiler? Watched the video that you posted and couldn't get much from it. Couldn't find the slides though. We're looking to make the same thing for QNX and need your help. Any info is appreciated .

Thanks,

Efraim Sealman 

[distributed]

Hi Efraim

You can take a binary built for linux/386, copy it to a QNX system and run it. The QNX loader will load and execute these binaries. Of course the program will not get very far during its execution, I think it crashes at the first "int $0x80" instruction, which is to be expected. This what I mean by "some success executing": instructions from the program get run until the process hits an instruction that does not make sense for a QNX process.

I also wrote

Hacking QNX syscalls via int $0x28 directly into runtime.rt0_linux_386, I have been able to write to stdout and exit the program gracefully.

That means I edited rt0_linux_386.s, specifially the function _rt0_386_linux, which begins with the first instruction run by a Go program. I wrote code, in this assembler function, to synthesize an _IO_WRITE message and call MsgSendv through the syscall interface, which works by executing "int $0x28".

I never had anything close to a QNX port. The only thing I showed is that the output of the current linux/386 toolchain can be used to run code on a QNX system.

Are your port efforts public?

Cheers,

Michael