How to add RV32E support in riscv-llvm?

[Dong Tong]

Hi guys.

Now I’m doing something about rve. it seems like rve is not supported in llvm upstream.
How can I add the rve to the LLVM backend? I have try to learn the source code,but it seems really hard to find the specific position to add rve.
Did there have some useful examples or some specific patches which were in the repositories that I can learn or imitate? thanks
If the llvm could support the rve someday is great! :-)

Any suggestion will be appreciate.

Dong

[Alex Bradbury]

Hi Dong,

Firstly, I think the RVE ABI is under-documented. I see that you've
started to clarify some aspects in
<https://github.com/riscv/riscv-elf-psabi-doc/issues/77>, and I'd
encourage you to continue in that direction. I'm slightly confused as
the status of the current RV32E ABI given the stated plans to unveil a
set of new "embedded" ABIs. One of the motivations for upstreaming and
thus standardising the RV32E ABI implemented in GCC was that it was a
counterpart to the ilp32/ilp32f/ilp32d/lp64/lp64f/lp64d "Linux ABIs"
and thus orthogonal to whatever the is going to be proposed for the
new "embedded" ABIs
<https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CAFyWVaZsAWt8j4D%2B9ZvUJwNRMOiabwxeb45uRCAMvFwAaUFiwA%40mail.gmail.com>.
But the linked issue indicates there's no plan to support PLT.

As for implementing it, I'd separate the tasks somewhat. I've written
some notes here, which should get you started though I warn you that
it's unlikely I successfully enumerate every piece of code you may
need to change. You'll want to support both -march=rv32i -mabi=lp32e
and -march=rv32e -mabi=ilp32e.

* MC layer (assembler) support for -march-rv32e. Ensure that using
registers other than x0-x15 in assembly results in an error.
* RV32E calling convention support:
* LLVM-side support: Look specifically at CC_RISCV in
RISCVISelLowering.cpp as well as the callee-saved register sets in
RISCVCallingConv.td. Stack alignment is different for RV32E, so you'd
want to grep 'stackalign' in RISCVFrameLowering.cpp. I imagine you'll
want to modify RISCVRegisterInfo::getCalleeSavedRegs too, but if not
supporting -march=rv32i -mabi=ilp32e, it might not be necessary to
modify the calee-saved register set logic.
* Clang-side support: Due to the design of LLVM and its IR,
target-specific code is needed to do some amount of ABI lowering in
the frontend. For the integer RISC-V ABIs, this is basically limited
to counting the number of GPRs that are used used so that parameters
can be marked as signext/zeroext as appropriate if they are passed in
registers (they are anyext if passed on the stack). Look at NumArgGPRs
in RISCVABIInfo in lib/CodeGen/TargetInfo.cpp.
* RV32E codegen support: Need to ensure that when targeting
-march=rv32e that x16-x31 will never be selected.

In order to provide the MC layer and codegen support, we want to avoid
duplicating every instruction and instruction pattern in
RISCVInstrInfo.td with an RV32E instruction variant that uses a new
'EGPR' register class. I should think you'd be able to mark x16-x31 as
reserved in RISCVRegisterInfo if the target is rv32e. You'll want to
add a little extra logic in the AsmParser and Disassembler to reject
out-of-range registers for RV32E. I'd recommend looking at how the ARM
backend (lib/Target/ARM) handles the "+d16" target feature which
disables the top 16 double precision registers. Grep for FeatureD16
and HasD16.

Best,

Alex

[Alex Bradbury]

> need to change. You'll want to support both -march=rv32i -mabi=ilp32e
> and -march=rv32e -mabi=ilp32e.

Actually it seems the ABI docs currently state
<https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md#rv32e-calling-convention>
that -march=rv32i -mabi=ilp32e is an illegal combination. RISC-V GCC
accepts that combination and just uses the ilp32 ABI though. I've
filed a bug <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87391>

Best,

Alex

[xudali...@gmail.com]

Hi  Alex,

     I'm a graduate student of  dong.   I have done something on llvm-rve.

     I have reserved the x16-x31 reg in  RISCVRegisterInfo and I can use llc command to make a source code  from IR to binery code.  such as 

         

          llc -march=rv32e -filetype=obj sum.bc -o sum.o 

     I wonder if that means I have successfully reserved the x16-x31 reg ? if we don't consider the ABI's problem such as callee-save etc. I may want to kown how to verify that thing.

    Moreover, I'm now working wtih how to add ilp32e, It seems only the -march=rv32e -mabi=ilp32e is legal as you have said.

Actually it seems the ABI docs currently state
<https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv-elf.md#rv32e-calling-convention>
that -march=rv32i -mabi=ilp32e is an illegal combination. RISC-V GCC
accepts that combination and just uses the ilp32 ABI though. I've
filed a bug <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87391>

   So , it means I only need to add -march=rv32e -mabi=ilp32e .  I will follow your notes above which tells How to make llvm-rve in both clang and llvm.  

> As for implementing it, I'd separate the tasks somewhat. I've written
> some notes here, which should get you started though I warn you that
> it's unlikely I successfully enumerate every piece of code you may
> need to change. You'll want to support both -march=rv32i -mabi=ilp32e
> and -march=rv32e -mabi=ilp32e.

  Thanks  a lot .

  Daliang

[Jim Wilson]

On Sat, Sep 22, 2018 at 7:30 AM Alex Bradbury <a...@lowrisc.org> wrote:
> Firstly, I think the RVE ABI is under-documented. I see that you've
> started to clarify some aspects in
> <https://github.com/riscv/riscv-elf-psabi-doc/issues/77>, and I'd
> encourage you to continue in that direction. I'm slightly confused as
> the status of the current RV32E ABI given the stated plans to unveil a
> set of new "embedded" ABIs. One of the motivations for upstreaming and
> thus standardising the RV32E ABI implemented in GCC was that it was a
> counterpart to the ilp32/ilp32f/ilp32d/lp64/lp64f/lp64d "Linux ABIs"
> and thus orthogonal to whatever the is going to be proposed for the
> new "embedded" ABIs
> <https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CAFyWVaZsAWt8j4D%2B9ZvUJwNRMOiabwxeb45uRCAMvFwAaUFiwA%40mail.gmail.com>.
> But the linked issue indicates there's no plan to support PLT.

A new "embedded" ABI will be added at some point in the future.
However, meanwhile, there are people building and using RVE parts
today, so we need an ABI today, and the only way to do that is with a
trivial modification to the existing "Linux" ilp32 ABI, even though
there is no intent to ever support Linux running on ilp32e parts. So
there are two separate problems here, one is the need for an RVE ABI,
and one is the need for an embedded ABI. The ilp32e ABI fixes the
first one. The second one will be fixed at some point in the future.

Jim

[Michael Clark]

Because the PLTs are essentially private to a shlib, it might be possible to rejig then to use t0-t2 without breaking compatibility.

The current _dl_runtime_resolve trampoline appears to use t0-t3 but i’m not sure if the code will fit into 16 bytes while using only 3 temporaries (t0-t2) and preserving the current arguments to _dl_runtime_resolve. It needs some analysis.

That said, it makes sense not to special case the change for RV32E if _dl_runtime_resolve ends up to be not using using t2 (so we can link through t2) and the rejigged trampoline still fits in 32 bytes, then the PLT change could be uniform across ABIs while maintaining backwards compatibility.

The proviso here is that _dl_runtime_resolve is only using t0 and t1 and that the trampoline is not already optimal i.e. can be rewritten to avoid t3 in the same space. The only public interface here is the interface to _dl_runtime_resolve. The PLT registers are free to change as long as the input registers to the exported symbols remains the same.

Note there may be bugs in my transcription of the current PLT trampoline code in riscv-elf-psabi-docs

> --
> You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
> To post to this group, send email to sw-...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/CAFyWVab-KHw0zAYgsYve_xFFY2NLGG1%2Bt2z_yF_U4scoHM1iQQ%40mail.gmail.com.

[Andrew Waterman]

On Mon, Sep 24, 2018 at 6:35 PM 'Michael Clark' via RISC-V SW Dev
<sw-...@groups.riscv.org> wrote:
>
> Because the PLTs are essentially private to a shlib, it might be possible to rejig then to use t0-t2 without breaking compatibility.
>
> The current _dl_runtime_resolve trampoline appears to use t0-t3 but i’m not sure if the code will fit into 16 bytes while using only 3 temporaries (t0-t2) and preserving the current arguments to _dl_runtime_resolve. It needs some analysis.

It can be done in the same instruction count with only 3 registers,
but with lower performance. (Not that performance matters in this
case.)

Nevertheless, I don't support changing anything here until someone
demonstrates a need for PLT-based dynamic linking with the ILP32E ABI.

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/79F6D294-2ADE-40AD-B401-9B5425E16F06%40mac.com.

[xudali...@gmail.com]

 Hi, Alex

Recently I have completed the LLVM-RVE developing work  following your guides. I'm really appreciate your help.

Now I wonder how I can push the patch on the llvm.org or llvm-riscv.git or some other ways to share them.

Best,

Daliang

[Alex Bradbury]

Hi Daliang,

Great. Please post a patch to LLVM's phabricator (reviews.llvm.org)
following the guidelines at
https://llvm.org/docs/Contributing.html#how-to-submit-a-patch

Best,

Alex

[xudali...@gmail.com]

 Hi Alex,

Firstly, Thank you for your reply about how to post a patch to LLVM . I will lately post it following the guidelines.

And,  There is another problem.  When I review my work about rve yesterday,  I found something interesting.

When I use compile cmd like this：

      clang -march=rv64i  -S sum.c -o sum.S

I got a  print "  target 21 "   by  lib/Target/RISCV/RISCVRegisterInfo.cpp  

I have add some print  code like this in that cpp

BitVector RISCVRegisterInfo::getReservedRegs(const MachineFunction &MF) const {

    BitVector Reserved(getNumRegs());

    // Use markSuperRegs to ensure any register aliases are also reserved

    markSuperRegs(Reserved, RISCV::X0); // zero

    markSuperRegs(Reserved, RISCV::X1); // ra

    markSuperRegs(Reserved, RISCV::X2); // sp

    markSuperRegs(Reserved, RISCV::X3); // gp

    markSuperRegs(Reserved, RISCV::X4); // tp

    markSuperRegs(Reserved, RISCV::X8); // fp

    // this is the code I add for print
    // print current target 

    Triple triple = MF.getTarget().getTargetTriple();

    llvm::dbgs() << "target " << triple.getArch() << "\n";

    assert(checkAllSuperRegsMarked(Reserved));

    return Reserved;

}

I have lookup the Triple.h which 21 means riscv32 not riscv64. I was confused about that. I think I have removed what I have changed all except the print.

And when I use clang -march=rv32i  -S sum.c -o sum.S   I have got the same print 21 again.It seems right.

It seems the Target only changed when using --target  options，then the target will change to riscv64  (22 for the number). 

I'm not sure ,but  the Target seems to be determined when cmake ,by using -DLLVM_DEFAULT_TARGET_TRIPLE="riscv32-unknown-elf"

I don't know is that right?

And according to what I said above , it seems have two method to build rve program.

first one, clang -march=rv32e  -S sum.c -o sum.S  ( which target is riscv32 in default.)

the other one, clang  --target=riscve -S sum.c -o sum.S (riscve corresponding to riscv32 which is not formal style.)

which one is better? and  Need we support both? is that different?

Best,

Daliang.

[xudali...@gmail.com]

Hi Alex,

Recently, I have worked on RVE with your advice on https://reviews.llvm.org/D53291.

But I found something confused about how can I write a  Regression test for RVE CodeGen support.That's  mean How can I write a test to verify -march=rv32e only use the x0-x15 regs.

I know I can write a rv32e-valid.s and rv32e-invaild.s for MC layer. But I realy don't know how to do it in CodeGen.

need help!

Best,

Daliang.

[Alex Bradbury]

Hi Daliang,

There are at least a couple of things you might do to test codegen:
* Write a .ll input where the register allocator will use at least one
register from x16-x31 when targeting for RV32I. Ensure that correct
output is produced for RV32E, without using any register x16-x31. This
provides a simple sanity check
* Tests for the ilp32e calling convention

Best,

Alex

[xudali...@gmail.com]

在 2018年10月22日星期一 UTC+8下午9:45:03，Alex Bradbury写道：

Hi Alex,

Thanks for your help,Now I know How to test codegen.

But there is still a little problem,as you have said,testing codegen should be with test ilp32e calling convention.

when I add the rve to the llvm, I first worked the codegen which reserved the x16-x31 regs.

And then I add the -mabi=ilp32e calling conventing.  So,tests for the ilp32e calling convention may be passed until I have added the calling convention.

Is that true?

Tests for the ilp32e calling convention may Imitate calling-conv-*.ll in test/CodeGen/RISCV. Is That right ?

Best,

Daliang

[Alex Bradbury]

On Mon, 22 Oct 2018 at 15:56, <xudali...@gmail.com> wrote:
> Thanks for your help,Now I know How to test codegen.
>
> But there is still a little problem,as you have said,testing codegen should be with test ilp32e calling convention.
> when I add the rve to the llvm, I first worked the codegen which reserved the x16-x31 regs.
> And then I add the -mabi=ilp32e calling conventing. So,tests for the ilp32e calling convention may be passed until I have added the calling convention.
> Is that true?

I think implementing and testing the ilp32e calling convention is the
best first step, then test that x16-x31 aren't being selected in
function bodies. As -march=rv32e doesn't make sense without support
for the calling convention.

> Tests for the ilp32e calling convention may Imitate calling-conv-*.ll in test/CodeGen/RISCV. Is That right ?

Yes, something like calling-conv-ilp32e.ll but with tests chosen to
exercise the "interesting" cases given that ilp32e has fewer registers
used for argument passing.

Best,

Alex

[xudali...@gmail.com]

在 2018年10月23日星期二 UTC+8下午5:33:25，Alex Bradbury写道：

Hi Alex,

Ok, I have known what I should do.Thanks again !

Best,

Daliang