PNaCl and LLVM IR portability

[Richard Franks]

Hi, I'm new to all this, and while the confusion seems to be lifting a
tiny bit, I'm sure I've got more than a few things horribly wrong -
any clarification would be awesome. This is what I think I know:

1) LLVM IR is not inherently target-independent. Or rather - it isn't
designed to be target-independent, but it can be if the front-end
plays nice.
2) Because of this, PNaCl uses a modified ARM toolchain - a .pexe
contains bitcode compiled for this custom/virtual ARM architecture.
The "translate" stage then converts/optimizes/compiles this custom ARM
architecture for the targetted platform .nexe's.
3) Presumably this is more practical than translating x86_64 to ARM.
4) This wouldn't be necessary if LLVM already had some sort of high-
level target-independent IR, but this may never happen since they're
not entirely convinced of its utility.

Richard

[Samuel Crow]

Hi Richard,

There are some people working on an improved "Wordcode" based on LLVM Bitcode.  Here's a slideshow and video presentation from the European LLVM DevMeeting:

http://www.llvm.org/devmtg/2011-09-16/EuroLLVM2011-MoreTargetIndependentLLVMBitcode.pdf

http://www.llvm.org/devmtg/2011-09-16/videos/Kang-TargetIndependentBitcode-desktop.mp4

I'm not sure if it will make the cut before PNaCl makes its showing.  The more I hear about PNaCl's growing pains, the more I'm having my doubts about it.  Maybe this Wordcode file format will gain better traction.

--Sam

[Derek Schuff]

Hi Richard,

see responses inline

On Sat, Nov 19, 2011 at 6:27 PM, Richard Franks <spont...@gmail.com> wrote:

Hi, I'm new to all this, and while the confusion seems to be lifting a
tiny bit, I'm sure I've got more than a few things horribly wrong -
any clarification would be awesome.  This is what I think I know:

1) LLVM IR is not inherently target-independent.  Or rather - it isn't
designed to be target-independent, but it can be if the front-end
plays nice.

True. This causes problems in several cases (and not just for PNaCl) e.g. the frontend must do some of the calling convention lowering itself on x86-64.

 

2) Because of this, PNaCl uses a modified ARM toolchain - a .pexe
contains bitcode compiled for this custom/virtual ARM architecture.
The "translate" stage then converts/optimizes/compiles this custom ARM
architecture for the targetted platform .nexe's.

This is the current state, but it will be changing, and hopefully soon. We are introducing a generic 32-bit little-endian target (It's already in clang as "le32") into LLVM. Fixing the byte order and pointer size (and a few other things like some alignments, long double, etc) gets us most of the way toward the level of portability that we need, but we also are doing a few other modifications. For example in our current code, structures which are passed by value must always be passed in memory (if you look at the byzantine x86-64 ABI specification on how structures are passed by value, or look at what clang has to do in this case, you'll see why). In the backend this generic target is translated to native instructions, calling conventions, etc in a fairly straightforward way.

 

3) Presumably this is more practical than translating x86_64 to ARM.
4) This wouldn't be necessary if LLVM already had some sort of high-
level target-independent IR, but this may never happen since they're
not entirely convinced of its utility.

right; adding higher-level type information to the IR could allow, for example, fully native-compatible calling convention lowering in the backend; but of course there would be a lot of complications (which passes will work on which IR? how much of C/C++/$LANGUAGE type system do we really want to have in the IR? etc).

Our approach of using a generic target but with fixed endianness and pointer size is a problem specific enough to be tractable (no need to create a grand utopian portable universal IR), but still general enough to be useful (le32 is useful for us, emscripten, etc and it would be straightforward for someone to create e.g. be64 if they wanted to).

It's true that the discussions online have gotten off the rails several times about the portability and stability questions, and it hasn't always been clear about, e.g. the scope of the various problems and solutions. There was actually a great discussion just this past week at the LLVM developers meeting, and there was general support for this approach I've described here, at least as a start. It's may not be perfect for all situations (some have expressed a desire for endian-agnostic bitcode for example, that's a much harder problem), but it's good enough for us because it's much closer to the current state of things, and it will allow us to run almost all the optimizations on the pexe before it downloads to the browser.

[Richard Franks]

Hi Sam,
since the KIST Ocean Virtual Machine (OVM) development process is
closed we just don't know what they have working and what problems
they face. But it looks like they've been at it about the same amount
of time - oceanvm.org was registered to KIST in August 2009. It's
possible that they're further ahead than what was demonstrated.

Funnily enough - the OVM approach seems similar to PNaCl - it's not
a true high-level/target-independent IR, but a lowest-common-
denominator LLVM architecture from which you can then derive target-
dependent bitcode. And the OVM architecture, as defined in the target
datalayout section, looks like 32bit ARM to me *cough* ;-)

Richard

[Samuel Crow]

>   since the KIST Ocean Virtual Machine (OVM) development process is

> closed we just don't know what they have working and what problems
> they face.  But it looks like they've been at it about the same amount
> of time - oceanvm.org was registered to KIST in August 2009.  It's
> possible that they're further ahead than what was demonstrated.

Wow.  I wasn't able to Google anything about that name.  Their website must still be empty.

>   Funnily enough - the OVM approach seems similar to PNaCl - it's not
> a true high-level/target-independent IR, but a lowest-common-
> denominator LLVM architecture from which you can then derive target-
> dependent bitcode.  And the OVM architecture, as defined in the target
> datalayout section, looks like 32bit ARM to me *cough* ;-)

It doesn't look as ARM specific to me as it does to you.  It might only be geared toward x86 and ARM at the moment though.

[Samuel Crow]

Hi Derek,

>2) Because of this, PNaCl uses a modified ARM toolchain - a .pexe
>>contains bitcode compiled for this custom/virtual ARM architecture.
>>The "translate" stage then converts/optimizes/compiles this custom ARM
>>architecture for the targetted platform .nexe's.
>>
>
>This is the current state, but it will be changing, and hopefully soon. We are introducing a generic 32-bit little-endian target (It's already in clang as "le32") into LLVM. Fixing the byte order and pointer size (and a few other things like some alignments, long double, etc) gets us most of the way toward the level of portability that we need, but we also are doing a few other modifications. For example in our current code, structures which are passed by value must always be passed in memory (if you look at the byzantine x86-64 ABI specification on how structures are passed by value, or look at what clang has to do in this case, you'll see why). In the backend this generic target is translated to native instructions, calling conventions, etc in a fairly straightforward way.

I'm interested in using PNaCl for a static-compiled VM for the AROS operating system.  I wondered what similarities might arise.  Since AROS supports ARM, x86, AMD64, PPC, and 68k, I wondered what difficulties might arise in having similar generic platforms for big-endian 32-bit systems as well.  The main ones we support are the x86, 68k, PPC and ARM in about that order (with ARM set to gain soon because Hosted AROS is being ported to Android).  It would be nice to only have to support big-endian and little-endian machines as two targets instead of a bunch of individual targets.

--Sam

[Derek Schuff]

I'm interested in using PNaCl for a static-compiled VM for the AROS operating system.  I wondered what similarities might arise.  Since AROS supports ARM, x86, AMD64, PPC, and 68k, I wondered what difficulties might arise in having similar generic platforms for big-endian 32-bit systems as well.  The main ones we support are the x86, 68k, PPC and ARM in about that order (with ARM set to gain soon because Hosted AROS is being ported to Android).  It would be nice to only have to support big-endian and little-endian machines as two targets instead of a bunch of individual targets.

Don't forget pointer size. NaCl is a little bit unique in that our x86-64 ABI has 32-bit pointers and a structure layout which resembles x86 and ARM (but with -malign-double as the default). This means that le32 works for us on AMD64, but it might not work for you. (Unless you want to use something like the x32 abi e.g. https://sites.google.com/site/x32abi/ ).

As for endianness, I would imagine that defining a similar big-endian target would be pretty straightforward, but haven't thought through the details. Note however that both ARM and PPC are bi-endian, so if you care less about 68k, you could probably punt on that for now  (although PPC at least requires some OS support since it defaults to big).

[Samuel Crow]

Hi Derek,

>Don't forget pointer size. NaCl is a little bit unique in that our x86-64 ABI has 32-bit pointers and a structure layout which resembles x86 and ARM (but with -malign-double as the default). This means that le32 works for us on AMD64, but it might not work for you. (Unless you want to use something like the x32 abi e.g. https://sites.google.com/site/x32abi/ ).
>As for endianness, I would imagine that defining a similar big-endian target would be pretty straightforward, but haven't thought through the details. Note however that both ARM and PPC are bi-endian, so if you care less about 68k, you could probably punt on that for now  (although PPC at least requires some OS support since it defaults to big).

The only 64-bit version we have now is AMD64 (aka x86_64).  It's purely in the experimental stages.  It doesn't even support full 64-bit addressing yet.  It can have its own native code format unlike the others.

As for the 68k, we'd rate that as being the MOST important version of all of them after the x86.  The 68k backend isn't done yet but since the AROS operating system is based on the old AmigaOS version 3.1, it will likely be the most supported version since that's what most of our software is written for.

Also, I'd rank PPC big-endian higher than that of ARM for AROS.  PPC is what most people went to to emulate the old 68k code on.  We've got 2 competing operating systems trying to carry on the Amiga heritage that we'll want to be developing hosted versions of AROS for so that they can all run the same code.

Thanks for your interest in our project though,

--Sam