RISC-V LLVM status update

[Alex Bradbury]

As you will have seen from previous postings, I've been working on upstream
LLVM support for the RISC-V instruction set architecture. The initial RFC
<http://lists.llvm.org/pipermail/llvm-dev/2016-August/103748.html>
provides a good overview of my approach. Thanks to funding from a third party,
I've recently been able to return to this effort as my main focus. Now feels
like a good time to give an update on where the RISCV backend is at, and how
you can help.

Note: a version of this message has also been sent to the llvm-dev
mailing list <http://lists.llvm.org/pipermail/llvm-dev/2017-August/116709.html>.

## Current status
* A full, regularly rebased patchset can be found here
<https://github.com/lowRISC/riscv-llvm>.
* 16 of these patches have been put up for review so far. 7 have been
committed, and 8 are awaiting review.
* The vast majority of the GCC torture suite compiles and runs at O0,
targeting RV32I. 1315 out of 1352 compile and run (32 compile-time failures, 5
run-time failures).
* I intend to keep <http://www.lowrisc.org/llvm/status> updated with status,
test results etc.

## Next steps and getting involved
The plan has always been to work from the MC-layer upwards towards reliable
RV32I codegen. This then provides a stable 'core' of the backend where it's
easy for further development work to be parallelised, and for others to make
contributions. I think we're now at that point.

I would really like to avoid setting up a new 'downstream', and to use this
opportunity to pull in new people to upstream LLVM development. However
collaboration is made rather difficult for now due to the large gap between
the patches that have been reviewed and committed upstream vs the complete
patchset. If you would like to help, reviewing the remaining patches
<https://reviews.llvm.org/differential/?authors=asb&order=updated> is an
incredibly valuable way to do so. If you would like to be listed as a reviewer
for future patches, just let me know (and feel free to add yourself for
existing patches). I'll be doing some refactoring of RISCVInstrInfo.td as
discussed here
<http://lists.llvm.org/pipermail/llvm-dev/2017-August/116635.html>, but this
is a straight-forward non-functional change. Other than that, there are no
planned changes to the patches currently up for review.

My current plans look like the below, though of course will accelerate
significantly with more contributors:
* Next 2-3 weeks: resolve remaining torture suite issues, ABI compliance
testing, documentation on MC layer backend implementation
* End of Oct: Clang toolchain driver, initial MAFD MC layer + codegen +
calling conventions, initial benchmarking vs RISC-V GCC and generated code
quality improvements
* End of Dec: finalise MAFD support, improve MAFD compliance testing and
larger-scale codegen testing, further expanded backend docs, generated code
quality improvements

I've mapped out a number of TODO items here
<https://github.com/lowRISC/riscv-llvm/issues>, which I hope can help to
co-ordinate efforts. Where possible, this indicates the current preferred
approach (e.g. we plan to provide RV64 support building upon Krzysztof's
variable-sized register class work). I've submitted a RISC-V BoF session
proposal for the upcoming LLVM Dev Meeting. If accepted, this should provide a
great opportunity to further discuss and co-ordinate work between the multiple
parties who have expressed an interest.

## Credits and support
Thank you to everyone who has contributed review comments, suggestions, or code
to this patchset and related support patches: Chandler Carruth, Chih-Mao Chen,
David Chisnall, Simon Cook, David Craven, Hal Finkel, James Y Knight, David
Majnemer, Ed Maste, Dylan McKay, Krzysztof Parzyszek, Jordy Portman, Philip
Reames, Tim Northover, Eugene Zalenko, Florian Zeitz. Apologies to anybody who
I've missed.

This work has been performed through lowRISC CIC <http://www.lowrisc.org>, a
not-for-profit company. If your company would like to see my work on RISC-V
LLVM to be sustained or to accelerate, contributing sponsorship and/or
development time is the best way to do that. Please contact a...@lowrisc.org if
you would like to discuss sponsorship, or have questions about code
contributions that you can't discuss on-list.

[Angelo Bulfone]

Wow. Great work!

Any chance on some information about what upstream version of LLVM to use and how to install it? I try running the patches and patch seems to just hang on me.

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CA%2BwH294u5VtyHfryWcN5s0ku74j%2Br5N-MVJq8yHKG64AeqDcmw%40mail.gmail.com.

[Alex Bradbury]

On 22 August 2017 at 16:14, Angelo Bulfone <mbul...@gmail.com> wrote:
> Wow. Great work!
> Any chance on some information about what upstream version of LLVM to use
> and how to install it? I try running the patches and patch seems to just
> hang on me.

Hi Angelo, I've added some guidance in the README which should clarify
things <https://github.com/lowrisc/riscv-llvm#how-to-build>. As I
indicate in the README, things should be less awkward once more of the
patchset is committed upstream. As I hoped, since publicising this
current milestone there seems to have been an uptick in reviewer
activity on the patches submitted upstream, so I'm hopeful this
awkwardness will soon be in the past.

I'll repeat the two "notes" from these build instructions here:

1. This repository currently serves a dual purpose - providing a clean
and constantly rebased "perfect" history of the full RISC-V patchset,
as well as being a way to grab all patches in one go, including those
that haven't yet been committed upstream. As more and more patches are
committed upstream, this repository will become mainly interesting as
a reference on how to implement an LLVM backend, and aid in
understanding the RISC-V backend. In the near future, the answer to
"how can I test out the latest RISC-V LLVM support?" will be "check
out the latest upstream LLVM repository and build it".

2. This LLVM and Clang work is not quite at a stage where end-users
are likely to find it useful. However, building and trying it out is a
great first step towards contributing to the development effort.

I hope that helps. Any issues, just let me know. There are some great
starter tasks on the issues page if you're interested in getting start
with LLVM development (e.g.
https://github.com/lowRISC/riscv-llvm/issues/39)

Best,

Alex

[John Leidel]

Alex, as always, this is awesome work.  I perused through your outstanding issue list and I believe there are definitely places whereby the community can help contribute at this stage in the development.  Regarding community involvement, how would you like to proceed?  Would you prefer to funnel the patches through lowRISC such that we maintain continuity in the upstream patch process (at least until this is fully merged into the upstream tree)?  David Chisnall, thoughts? 

cheers

john 

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+unsubscribe@groups.riscv.org.

[Alex Bradbury]

On 22 August 2017 at 20:26, John Leidel <john....@gmail.com> wrote:
> Alex, as always, this is awesome work. I perused through your outstanding
> issue list and I believe there are definitely places whereby the community
> can help contribute at this stage in the development. Regarding community
> involvement, how would you like to proceed? Would you prefer to funnel the
> patches through lowRISC such that we maintain continuity in the upstream
> patch process (at least until this is fully merged into the upstream tree)?

My thoughts are:
1) Having contributors work directly on upstream LLVM infrastructure
is where we want to end up. It benefits both the RISC-V community and
the LLVM community to do as soon as possible
2) Perfect shouldn't be the enemy of good. If there's no reasonable
chance of the majority of the patchset being reviewed and committed in
the near future, we should bite the bullet and collect work downstream
on a temporary basis. This does require ensuring contributors agree to
the LLVM license and developer policy. Doable, but I'd rather skip
this
3) There's no shortage of people interested in this work, and
indicating that interest either on or off list. This status update and
recent milestones seems to be having the desired effect of stoking
that interest, and there does seem to be an uptick in review activity
on the remaining patches. As such, getting the remaining patches
committed in the near future isn't infeasible

Taking in all of the above, I was thinking to see where things are
with the submitted patches by the beginning of next week. If it looks
like that's going to remain a bottleneck, then pooling work downstream
is the pragmatic way to go despite the downsides. If reviewing the
outstanding patches is no longer problem, then of course we can stick
with plan A.

How does that sound?

Alex

[Angelo Bulfone]

I built the latest patches last night and was able to get a rust implementation of factorial working! I was a little surprised when I saw that it was doing actual overflow checking for the subtraction and multiplication. Given that it's incomplete, I was very surprised by how ugly the resulting assembly turned out and trying to optimize it at all said that insertBranch is unimplemented. It looks like a clear feature that can be worked on for someone not concerned with more pressing features.

All in all, great work so far!

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.

To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CA%2BwH296Nfyaw2YmoqECm4pCgzRbW7cccaSGgspCR3vrN5J2L_g%40mail.gmail.com.

[David Craven]

Hi Angelo!

Nice to hear... If you want to get libcore compiling you can try this
branch (latest patches with inline asm enabled), will currently only
build with opt-level=1

I'm guessing you want to run rust on your hifive board? :)

[0] https://github.com/dvc94ch/llvm/commits/rust-llvm-riscv-2017-08-23

[Angelo Bulfone]

Hi David. I'll check out that branch. Would you mind giving a brief summery of the differences between that fork and Alex's patches?

I'm personally big on OS development, so while I'm not big on embedded development or have a hifive board, I am interested in writing a kernel from scratch to support RISC-V from the start. As for Rust, it's basically a dream come true for me to have memory safety and a freestanding runtime. If I can find another language with those, I might try them out. The fact that LLVM is being worked on for RISC-V means that many languages now become available for development for RISC-V.

[kr...@berkeley.edu]

Great to see this happening.
Thanks for all the hard work Alex!

Krste

| --
| You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
| To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
| To post to this group, send email to sw-...@groups.riscv.org.
| Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.

| To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CA%2BwH294u5VtyHfryWcN5s0ku74j%2Br5N-MVJq8yHKG64AeqDcmw%40mail.gmail.com.

[Michael Clark]

Yes. It’s pretty neat. I remember spending a few days looking at the LLVM code base and realising it would be no small commitment to get up-to-speed and productive. It is a relatively daunting task to familiarise oneself with the LLVM codebase, in addition to this port essentially being started from a clean slate. Hopefully I can help when the port is further along.

>> email to sw-dev+un...@groups.riscv.org.

[David Craven]

Hi Angelo.

It's the rust-llvm-2017-07-20 branch, which is llvm-5 with a few
patches. We mainly need it for the ValueTracking patch and the
MCSubtargetInfo patches. Then asb's patches are applied on top.

Then there's three TMP: patches related to backporting the patchset to
llvm-5 and rust-llvm (reverting a build related patch).

And then a couple of trivial patches mostly hacks (see the commit
messages), the only interesting one is the one that implements branch
analysis - as you already noted (insertBranch).

[ValueTracking] 7bfe090f9dc92b9c732d0d35bb4a2a53acb6b44e
[SubtargetInfo] 8dde6702fc292b571d8808843c1db43f909ccf53

[Alex Bradbury]

Sadly, the lack of review activity remains a major barrier to getting
the patchset merged upstream. I've therefore set up a temporary
'riscv-llvm-integration' repository to allow downstream collaboration
using pull requests. See the README at
<https://github.com/lowRISC/riscv-llvm-integration> for more
information. Do note that this is intended for integrating
upstream-ready code, so temporary hacks or code without matching tests
will not be merged.

Would people be happy for this repo to be moved to
github.com/riscv/riscv-llvm-integration?

For those who are seeing the challenges I'm having with code review
and are put off upstream RISC-V LLVM contributions, I wanted to make a
clarification. Although my experience here is frustrating, it's a
bootstrapping problem that you won't have to face if and when you
contribute RISC-V patches upstream. LLVM policy
<https://llvm.org/docs/DeveloperPolicy.html#code-reviews> requires
major changes to be reviewed prior to commit. I'm clearly having
trouble getting reviewers, but as the RISC-V target code owner, I'm
here to review anything you might submit with as rapid a turnaround
time as possible. I'm very grateful that my current funding source is
keen to support this, and recognises the long-term benefits of getting
RISC-V merged upstream in LLVM.

Best,

Alex

[David Chisnall]

On 6 Sep 2017, at 15:21, Alex Bradbury <a...@asbradbury.org> wrote:
>
> Sadly, the lack of review activity remains a major barrier to getting
> the patchset merged upstream.

Note that LLVM permits post-commit review at the discretion of the code owner. I’d suggest that you request permission to do this from the designated code owner for the RISC-V back end. He’s some guy called Alex Bradbury, maybe you know him?

David

[Alex Bradbury]

On 6 September 2017 at 16:02, David Chisnall

It's possible I misunderstood the review policy here, but I understood
that it is considered inappropriate to switch to post-commit review
after the code review process has been started. I agree, many future
patches might be best off going in with post-commit review. This would
be a shame, as when reviewers have waded in I've always had useful
feedback which has improved code quality.

Alex

[David Chisnall]

The relevant part is:

> • Code can be reviewed either before it is committed or after. We expect major changes to be reviewed before being committed, but smaller changes (or changes where the developer owns the component) can be reviewed after commit.

Specifically, note the ‘or where the developer owns the component’ part.

I haven’t given feedback on the most recent patches because they look fine (at least, nothing structurally wrong with them), and minor cleanups are easier to do after it’s all committed.

David

[Alex Bradbury]

On 6 September 2017 at 16:10, David Chisnall

<David.C...@cl.cam.ac.uk> wrote:
>
>> On 6 Sep 2017, at 16:08, Alex Bradbury <a...@asbradbury.org> wrote:
>>
>> On 6 September 2017 at 16:02, David Chisnall
>> <David.C...@cl.cam.ac.uk> wrote:
>>> On 6 Sep 2017, at 15:21, Alex Bradbury <a...@asbradbury.org> wrote:
>>>>
>>>> Sadly, the lack of review activity remains a major barrier to getting
>>>> the patchset merged upstream.
>>>
>>> Note that LLVM permits post-commit review at the discretion of the code owner. I’d suggest that you request permission to do this from the designated code owner for the RISC-V back end. He’s some guy called Alex Bradbury, maybe you know him?
>>
>> It's possible I misunderstood the review policy here, but I understood
>> that it is considered inappropriate to switch to post-commit review
>> after the code review process has been started. I agree, many future
>> patches might be best off going in with post-commit review. This would
>> be a shame, as when reviewers have waded in I've always had useful
>> feedback which has improved code quality.
>
> The relevant part is:
>
>> • Code can be reviewed either before it is committed or after. We expect major changes to be reviewed before being committed, but smaller changes (or changes where the developer owns the component) can be reviewed after commit.
>
> Specifically, note the ‘or where the developer owns the component’ part.

Sure, though I was put off by the subsequent "Specifically, once a
patch is sent out for review, it needs an explicit “looks good” before
it is submitted. Do not assume silent approval, or request active
objections to the patch with a deadline." part. I'll pursue this issue
on llvm-dev.

Thanks,

Alex

[Shiva Chen]

Hi Alex,

This is Shiva. I work with Kito as one of the main Andes developers for LLVM.

Thanks for your great effort for riscv backend which makes it easy for
every interested party to follow and extend.
Following your code base, we started our porting.
For accelerating llvm riscv backend development, we think it's a good
time to integrate our code base.

We would like to start from the implementation of RV32 C extension
mentioned in the issue list:
https://github.com/lowRISC/riscv-llvm/issues
We'll prepare patch set as the upstreaming standard requires and
definitely need your review to make the code quality better.

We plan to contribute our patchset to riscv-llvm-integration repo.
But it seems that the riscv-llvm-integration repo is not the latest
code base available in
https://github.com/lowRISC/riscv-llvm, where our porting started.
Is there any reason why they are different? If not,
would you mind updating riscv-llvm-integration repo ?

Thanks,
Shiva

> --
> You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
> To post to this group, send email to sw-...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CA%2BwH296KCE1CbSztycbv1%2B8eHBkd9-1SAeMMyUf9yUY1sgs0nw%40mail.gmail.com.

[Alex Bradbury]

On 19 September 2017 at 10:49, Shiva Chen <shiv...@gmail.com> wrote:
> Hi Alex,
>
> This is Shiva. I work with Kito as one of the main Andes developers for LLVM.
>
> Thanks for your great effort for riscv backend which makes it easy for
> every interested party to follow and extend.
> Following your code base, we started our porting.
> For accelerating llvm riscv backend development, we think it's a good
> time to integrate our code base.
>
> We would like to start from the implementation of RV32 C extension
> mentioned in the issue list:
> https://github.com/lowRISC/riscv-llvm/issues
> We'll prepare patch set as the upstreaming standard requires and
> definitely need your review to make the code quality better.

Fantastic! I'm really glad Andes are starting to join the upstream
RISC-V LLVM effort.

> We plan to contribute our patchset to riscv-llvm-integration repo.
> But it seems that the riscv-llvm-integration repo is not the latest
> code base available in
> https://github.com/lowRISC/riscv-llvm, where our porting started.
> Is there any reason why they are different? If not,
> would you mind updating riscv-llvm-integration repo ?

I was actually going to make a slight change in approach for the
llvm-integration repo, motivated by the desire from you (Andes) to
work on top of an LLVM 5.0 base. There are advantages and
disadvantages to this, but I'd really like to minimise the number of
different trees we all have to contend with. Therefore I'm going to
set up a riscv-llvm-integration branch based on 5.0 and make that the
new default. I'll push it later today - it will take a little while to
compile-test each commit and I've got a meeting soon which is going to
take up some of my time. In the meantime, you should be able to work
against a tree with the lowrisc/riscv-llvm patches applied, and your
work can be trivially transplanted as soon as I push the new branch.

Even though we hope this work on the compressed instruction set should
be merged relatively quickly, I'd really recommend trying to keep your
commit history clean and making use of git rebase as you develop - it
will make it much easier to update the code if there are any other
changes in lib/Target/RISCV.

Best,

Alex

[David Chisnall]

On 19 Sep 2017, at 11:43, Alex Bradbury <a...@asbradbury.org> wrote:
>
> I was actually going to make a slight change in approach for the
> llvm-integration repo, motivated by the desire from you (Andes) to
> work on top of an LLVM 5.0 base. There are advantages and
> disadvantages to this

Having tried both approaches, I’d say that the disadvantages massively outweigh the advantages. There’s usually a lot of churn in the back-end APIs in LLVM (and as GlobalISel becomes the default, this is not going to slow down any time soon). Trying to upgrade an entire release at once is a lot of effort and a lot more effort than doing regular merges from upstream. Worse, you don’t catch regressions in other parts of the system that only affect your back end until a release is out, at which point it’s hard to find the exact commit that broke things for you and it’s no longer fresh in the mind of the person that wrote it.

David

[Alex Bradbury]

On 19 September 2017 at 11:54, David Chisnall

I do agree with that, but I think the situation is little different vs
a typical out-of-tree backend here. The goal is to bridge the gap
between now and the code being merged upstream. The hope is that the
5.0 tree can effectively be discarded by the time 6.0 is branched. I'm
already committed to maintaining a continuously rebased patchset on
the latest LLVM master (github.com/lowrisc/riscv-llvm) of course. I'm
trying to strike a balance here by having 'just' the following three
trees:
* Upstream LLVM
* Continuously rebased patch queue
* Patches applied on 5.0

Rather than four cases:
* Upstream LLVM
* Continuously rebased patch queue
* Patches applied on 5.0
* Patches applied on upstream, never rebased

I too prefer merging in upstream changes regularly, but it's difficult
to fight the perception that this would be more "unstable" vs
something based on 5.0.

Best,

Alex

[Alex Bradbury]

Hi Shiva, I've set this up at https://github.com/lowRISC/riscv-llvm-integration.

Best,

Alex

[Kito Cheng]

Hi David:

I think we just have different concern, for Andes, we need a stable version for production and release to customer, so base on stable version is best choice for us, but we also intended to put resource on upstream.

Hi Alex:

It great to hear that you consider to maintain a 5.0 branch :) BTW, would you mind to create a repo for clang too?

Best,

Alex

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CA%2BwH296wXG-iYZ20FLXpHtVP6Lz%2BxCx6F%2BEhQENzw4HjbKf9ow%40mail.gmail.com.

[David Chisnall]

On 20 Sep 2017, at 05:37, Kito Cheng <kito....@gmail.com> wrote:
>
> I think we just have different concern, for Andes, we need a stable version for production and release to customer, so base on stable version is best choice for us, but we also intended to put resource on upstream.

The two constraints (a shipping product vs a staging area for upstreaming) are difficult to reconcile. It’s worth noting that neither Apple nor Google bases their releases on LLVM releases - they both branch LLVM trunk at points that are convenient for their release cycles and then run their own QA on top. Other vendors have different strategies, but this is a fairly common approach. The only advantage of using the release branches is that a subset of non-ABI-breaking bug fixes are merged into these branches for the point releases. The vast majority of fixes are ABI breaking and are not merged, so this is of questionable value.

In either case, you’ll have to either forward-port changes from whatever snapshot you pick before you can upstream them, or you’ll have to backport them from trunk to your product. The real question is which of the two is easier:

- Following trunk with regular merges, back port changes to 5.x
- Working on 5.x, forward port improvements to trunk

In almost all cases, I have found that the former is easier. If you stick with 5.x, then in six months 6.0 will be out and there will be a large divergence in internal APIs. At this point, you can either do a big upgrade to 6.0 and handle all of the QA (late, because it may be 4 months since the thing that broke your code was committed upstream), or you can stick with 5.x. If you stick with 5.x, then you’re now in a state of even larger divergence with trunk and the effort of forward porting your changes for upstreaming grows.

Alternatively, if you’re following trunk (ARM, for example, does nightly merges from upstream, as I recall Qualcomm does them weekly) and regularly running your tests then you’ll find regressions immediately and be able to work with the upstream committer to fix them. When 6.0 is branched, your trunk version will exactly match it and so you can easily create a 6.0 branch. You will need to back-port features from your trunk to your 5.x branch, but this is generally easier to do individually to a non-moving target.

I’d strongly recommend that you watch the videos from the various working downstream BoFs at the LLVM DevMeetings for opinions of others who have tried both approaches. The TL;DR version is that working from releases costs a lot in developer time and everyone who has tried it long term eventually gives up.

David