[RFC] New 3.1/trunk port and new backend for x86

[Gianluca Guida]

Hello libcpu people,

After quite sometime I decided to get back at libcpu and in my cute NetBSD box I ported libcpu to current LLVM (3.1/trunk).

When looking at github.com/libcpu page, I found a request pull from Ryan Govostes (rgov) with quite a bunch of patch. I need to apologize to Ryan and Garrison for not responding quickly to this request, I will find some magic switch hopefully in github to send email notices.

The change in the pull request were:

1 Merge of Garrison Venn's patch to LLVM 3.0
2 Enabling clang compiling (again by Garrison Venn)
3 a new, MC-based, UCPL-using i386 backend

I think that getting in trunk #2 and #3 would be a good idea.

What I would like to discuss is how to deal with LLVM inter-version incompabilities. Right now we still don't have any real client or use for libcpu. IMHO until libcpu is useful and actually *used* we could happily keep libcpu in line with trunk.
But, for example, Pekka has in a previous thread in this group raised a point about keeping compatibility with LLVM versions that are used in common distributions.

If we want to support multiple LLVM version in the same code, we might need to #ifdef the code heavily, or create a llvm-version-independent layer (details overlooked by purpose), or something else.

What do you people think?

Right now my libcpu on 3.1/trunk with the new x86 backend is at http://github.com/glguida/libcpu. Feel free to comment. If there are no strong objections and I receive enough ACKs here, I might push it to the main repository (http://github.com/libcpu/libcpu).


Gianluca

[Pekka Enberg]

Hi Gianluca,

On Sat, Jun 2, 2012 at 3:22 PM, Gianluca Guida <glg...@tlbflush.org> wrote:
> What I would like to discuss is how to deal with LLVM inter-version incompabilities. Right now we still don't have any real client or use for libcpu. IMHO until libcpu is useful and actually *used* we could happily keep libcpu in line with trunk.
> But, for example, Pekka has in a previous thread in this group raised a point about keeping compatibility with LLVM versions that are used in common distributions.

I don't have any strong objections to using trunk if you think it's helpful.

On Sat, Jun 2, 2012 at 3:22 PM, Gianluca Guida <glg...@tlbflush.org> wrote:
> If we want to support multiple LLVM version in the same code, we might need to #ifdef the code heavily, or create a llvm-version-independent layer (details overlooked by purpose), or something else.

The LLVM API seems to change quite a bit over time so we probably only
want to try to support few releases at a time.

> Right now my libcpu on 3.1/trunk with the new x86 backend is at http://github.com/glguida/libcpu. Feel free to comment. If there are no strong objections and I receive enough ACKs here, I might push it to the main repository (http://github.com/libcpu/libcpu).

Looks good to me!

[Stefan Schmidt-Bilkenroth]

Hi,

indeed we at the company are busy with implementing something similar to libcpu. In our case it is about (partial) code transformation from native code (x86) to a special VM. It's about moving client code to an external device for license enforcement purposes.

So far we currently face more issues with the backend of LLVM and the special VM.

So maybe we can more share knowledge about this...

Is anyone of you at the WWDC this year? We could meet then in person...

Cheers,

Stefan Schmidt-Bilkenroth

Von meinem iPad gesendet

[Gianluca Guida]

On Sun, Jun 03, 2012 at 10:01:31AM +0300, Pekka Enberg wrote:
> On Sat, Jun 2, 2012 at 3:22 PM, Gianluca Guida <glg...@tlbflush.org> wrote:
> > If we want to support multiple LLVM version in the same code, we might need to #ifdef the code heavily, or create a llvm-version-independent layer (details overlooked by purpose), or something else.
>
> The LLVM API seems to change quite a bit over time so we probably only
> want to try to support few releases at a time.

This, or we could attempt at forking master to a llvm-version-specific branch
every time they do a release, keeping this way development on LLVM-trunk and
mantaining only a couple of branches for bug fixing.

Any opinion?

G.

[Pekka Enberg]

Hi Gianluca,

On Sun, Jun 03, 2012 at 10:01:31AM +0300, Pekka Enberg wrote:
>> The LLVM API seems to change quite a bit over time so we probably only
>> want to try to support few releases at a time.

On Sun, Jun 3, 2012 at 8:17 AM, Gianluca Guida <glg...@tlbflush.org> wrote:
> This, or we could attempt at forking master to a llvm-version-specific branch
> every time they do a release, keeping this way development on LLVM-trunk and
> mantaining only a couple of branches for bug fixing.
>
> Any opinion?

Who would benefit from the LLVM-version-specific branch? If there are
users out there that depend on libcpu, we should do proper releases
that are, of course, tied to a specific LLVM version.

So I don't think it's worth it. The only thing I hope is that we don't
track LLVM master too closely and force libcpu developers to upgrade
LLVM every time they pull from libcpu git, that's all.

Pekka

[Pekka Enberg]

On Sun, Jun 3, 2012 at 3:02 PM, Stefan Schmidt-Bilkenroth <s...@mac.com> wrote:
> indeed we at the company are busy with implementing something similar to libcpu. In our case it is about (partial) code transformation from native code (x86) to a special VM. It's about moving client code to an external device for license enforcement purposes.

Cool. Do you have any plans to push that code to libcpu in the future?

On Sun, Jun 3, 2012 at 3:02 PM, Stefan Schmidt-Bilkenroth <s...@mac.com> wrote:
> So far we currently face more issues with the backend of LLVM and the special VM.
>
> So maybe we can more share knowledge about this...

What kind of problems are you having with LLVM?

[Stefan Schmidt-Bilkenroth]

Hi,

see comments inline...
No one of you at WWDC?

Cheers, ssb

Von meinem iPhone gesendet

Am 03.06.2012 um 19:30 schrieb Pekka Enberg <pen...@kernel.org>:

> On Sun, Jun 3, 2012 at 3:02 PM, Stefan Schmidt-Bilkenroth <s...@mac.com> wrote:
>> indeed we at the company are busy with implementing something similar to libcpu. In our case it is about (partial) code transformation from native code (x86) to a special VM. It's about moving client code to an external device for license enforcement purposes.
>
> Cool. Do you have any plans to push that code to libcpu in the future?

I am not sure if we can really push that code back, depends on the companies opinion. It is anyway similar, but no line of libcpu code is used. Also it is by intention limited by several constraints, like the device can not access process memory, so many instructions can't be supported at all.

> On Sun, Jun 3, 2012 at 3:02 PM, Stefan Schmidt-Bilkenroth <s...@mac.com> wrote:
>> So far we currently face more issues with the backend of LLVM and the special VM.
>>
>> So maybe we can more share knowledge about this...
>
> What kind of problems are you having with LLVM?

The biggest problem right now is the LLVM backend vs. the VM implemented in the license device. It is about moving from SSA with unlimited registers of LLVM to a pure stack based VM without registers, similar to a Java VM. So far for research we use LLJVM plus a modified Jasmin as workaround, but this wont be suitable for the final product.

[Stefan Schmidt-Bilkenroth]

From my point of view, itis by far more important, to keep up with LLVM changes, while keeping backwards compatibility is less important. It is easier to upgrade LLVM, than to port the related projects from older LLVM to the current one. We those problems with a few LLVM Subprojects. Sometimes it is sad, that some interesting projects are poorly maintained and it takes some time to get them running on the current LLVM. Several hot projects really look dead...

I am just glad, libcpu is alive. When i get spare time, I can invest a bit more on integrating my PPC disassembler with LLVM...

Von meinem iPhone gesendet

[Gianluca Guida]

On Sun, Jun 03, 2012 at 08:28:39PM +0300, Pekka Enberg wrote:
> On Sun, Jun 03, 2012 at 10:01:31AM +0300, Pekka Enberg wrote:
> >> The LLVM API seems to change quite a bit over time so we probably only
> >> want to try to support few releases at a time.
>
> On Sun, Jun 3, 2012 at 8:17 AM, Gianluca Guida <glg...@tlbflush.org> wrote:
> > This, or we could attempt at forking master to a llvm-version-specific branch
> > every time they do a release, keeping this way development on LLVM-trunk and
> > mantaining only a couple of branches for bug fixing.
> >
> > Any opinion?
>
> Who would benefit from the LLVM-version-specific branch? If there are
> users out there that depend on libcpu, we should do proper releases
> that are, of course, tied to a specific LLVM version.

Well, Ok. I skipped some part. I would really like to keep the architectures
frontend independent from LLVM. Right now we mostly have it -- not sure if
completely or not, anyway it's doable -- and I would like to enforce this.

It would mean that the only component that will be affected by LLVM changes
will be the libcpu backend itself.

Keeping around the backend working with older versions of LLVM will make
users happy, developers of frontends won't be affected and the core part
of libcpu could be rewritt^H^H^H^H^H^H^Himproved using new features available
on trunk LLVM.

Furthermore, we would get a product always in line with LLVM, so when we get
something useful we could even get it listed as a LLVM project.

> So I don't think it's worth it. The only thing I hope is that we don't
> track LLVM master too closely and force libcpu developers to upgrade
> LLVM every time they pull from libcpu git, that's all.

In my scheme, is up to what you mean by libcpu developer. If you want to
develop arch frontends, a stable interface will help us all and you won't
be affected.
If you want to change the inner architecture of libcpu, then yes, you'd be
affected, but not at the extreme to update llvm everytime you pull libcpu.

I might have over-simplified things here, but I hope this makes my point clearer.

Gianluca

[Pekka Enberg]

On Sun, 3 Jun 2012, Gianluca Guida wrote:
> Well, Ok. I skipped some part. I would really like to keep the architectures
> frontend independent from LLVM. Right now we mostly have it -- not sure if
> completely or not, anyway it's doable -- and I would like to enforce this.
>
> It would mean that the only component that will be affected by LLVM changes
> will be the libcpu backend itself.
>
> Keeping around the backend working with older versions of LLVM will make
> users happy, developers of frontends won't be affected and the core part
> of libcpu could be rewritt^H^H^H^H^H^H^Himproved using new features available
> on trunk LLVM.
>
> Furthermore, we would get a product always in line with LLVM, so when we get
> something useful we could even get it listed as a LLVM project.

No objections from me for trying out this scheme.

Pekka