Large Virtual Addresses

[Graham Gobieski]

When running spike, does anyone know what is loaded at 0xFFFFFFFF800190B0 or addresses near that range. Seems like it is a syscall thing, but it isn't clear. Is there a linker script file I am missing or some definitions for where memory is loaded? Thanks.

[Michael Clark]

> On 22/08/2018, at 4:02 AM, Graham Gobieski <graham....@gmail.com> wrote:
>
> When running spike, does anyone know what is loaded at 0xFFFFFFFF800190B0 or addresses near that range. Seems like it is a syscall thing, but it isn't clear. Is there a linker script file I am missing or some definitions for where memory is loaded? Thanks.

I think you need to look at the symbol table of the binary you have loaded. The simulator is agnostic to magic addresses like the one you have given above.

[Graham Gobieski]

Okay so I think I have determined the symbol. Thanks. I just needed to note that addresses beginning with 0xFFFFFFFFxxxxxxxx are equivalent to  0xxxxxxxxx.

[Michael Clark]

> On 22/08/2018, at 7:37 AM, Graham Gobieski <graham....@gmail.com> wrote:
>
> Okay so I think I have determined the symbol. Thanks. I just needed to note that addresses beginning with 0xFFFFFFFFxxxxxxxx are equivalent to 0xxxxxxxxx.

They are not equivalent.

They differ in sign. Virtual addresses on RISC-V must be sign extended e.g. for sv39, bits[63:38] must all be the same, likewise for sv48, bits[63:47] must all be the same.

If you are running a 32-bit simulation, you may see this if a 32-bit signed address is cast to 64-bit during formatting, where bit 31 is extended to bit 63. Are you using rv32?

If they were equivalent, then this implies the upper address bits could be ignored by the memory system, but they are indeed significant.

There should be some tests in riscv-tests that check this constraint...

[Graham Gobieski]

I am using rv32e. 

Looking into pk further I find that dummy_payload is loaded at 0xFFFFFFFF80000000 (-0x80000000), which is what I was looking for. I don't really understand why this is (why a sign-extended address). 

Also the linker script for pk and bbl, seem to suggest that they are loaded at 0x8000000, which does make sense. But when I load an arbitrary binary that is loaded at some physical address less than 0x8000000. However, you can probe the addresses at 0x8000000 + offset from symbols in code and see the same code (i.e. symbols can be viewed at the virtual/physical address specified by the elf file, but also at 0x8000000 + offset). Once again I think this makes sense because the DRAM starts at 0x8000000. I guess though my question with this is how is Spike managing symbols so they don't conflict between pk source, bbl source, dummy_payload and an arbitrary binary.

[Graham Gobieski]

I guess my larger question is how I can divide up the memory. I want a section to represent volatile memory and another to represent non-volatile. This means elf sections need to load (e.g. .bss in RAM, .text in NVRAM) into either RAM or NVRAM. I can do this on the executable by relinking with a custom linker script. However, I am having difficulties doing the same with pk. It seems that there is no code to copy .data section from the NVRAM to RAM (do you know if this is correct?), so I will probably need to add such code (this should be standard bootloader stuff I assume). I further imagine that I would add this in minit.c init_first_hart, which seems to be the first thing called after do_reset. Yeah this is very technical...sorry about that, just want another person's opinion who has a better understanding of Spike and Pk.

On Tuesday, August 21, 2018 at 12:02:31 PM UTC-4, Graham Gobieski wrote:

When running spike, does anyone know what is loaded at 0xFFFFFFFF800190B0 or addresses near that range. Seems like it is a syscall thing, but it isn't clear. Is there a linker script file I am missing or some definitions for where memory is loaded? Thanks.

[Michael Clark]

> On 23/08/2018, at 2:24 AM, Graham Gobieski <graham....@gmail.com> wrote:
>
> I am using rv32e.
>
> Looking into pk further I find that dummy_payload is loaded at 0xFFFFFFFF80000000 (-0x80000000), which is what I was looking for. I don't really understand why this is (why a sign-extended address).
>
> Also the linker script for pk and bbl, seem to suggest that they are loaded at 0x8000000, which does make sense. But when I load an arbitrary binary that is loaded at some physical address less than 0x8000000. However, you can probe the addresses at 0x8000000 + offset from symbols in code and see the same code (i.e. symbols can be viewed at the virtual/physical address specified by the elf file, but also at 0x8000000 + offset). Once again I think this makes sense because the DRAM starts at 0x8000000. I guess though my question with this is how is Spike managing symbols so they don't conflict between pk source, bbl source, dummy_payload and an arbitrary binary.

That’s a good question. It doesn’t.

We made some changes to QEMU to separate the firmware/monitor (bbl in this case) and the kernel (vmlinux in this case). By doing so we can get symbolicated trace for the monitor and early boot in Linux before the page tables have been mapped. This helps with debugging early boot. Typically for virtual addresses I have a console open with riscv-objdump -d vmlinux with a big console history setting and then use cmd-F (on Mac) to search for the symbol. This is a little slow.

With a more sophisticated simulator, (extensions to spike) we could allow “symbol resolution expressions” e.g. in the case of rv64 Linux we know that the load addresses for the kernel, perhaps 0x80200000 will be mapped to 0xffffffff80000000 and thus we could add a static translation “expression” to the symbolicator to allow one to debug the kernel with symbols.

Of course once the kernel starts loading it’s own ELF files, we won’t be able to track the symbols for each process (unique satp address from the perspective of the simulator).

It would be perceivable possible to associate a user program ELF symbol table with a satp value, and use those symbols when (satp == <value>). This would be useful for debugging persistent processes in a full system emulator.

Something to think about. These are all relatively easy to do if one has a symbolicating simulator with a user defined symbol resolution function and multiple ELF symbol tables loaded into memory...

I know of other symbolicating simulators, but they share the same problem as spike in that they load only the symbol tables of one ELF image.

Symbolicated trace is nice. It makes debugging some issues much much easier.

Thanks for the thought provoking email.

[Michael Clark]

> On 23/08/2018, at 9:26 AM, Graham Gobieski <graham....@gmail.com> wrote:
>
> I guess my larger question is how I can divide up the memory. I want a section to represent volatile memory and another to represent non-volatile. This means elf sections need to load (e.g. .bss in RAM, .text in NVRAM) into either RAM or NVRAM. I can do this on the executable by relinking with a custom linker script. However, I am having difficulties doing the same with pk. It seems that there is no code to copy .data section from the NVRAM to RAM (do you know if this is correct?), so I will probably need to add such code (this should be standard bootloader stuff I assume). I further imagine that I would add this in minit.c init_first_hart, which seems to be the first thing called after do_reset. Yeah this is very technical...sorry about that, just want another person's opinion who has a better understanding of Spike and Pk.

I would have to see your exact use case with pk/bbl, but I am sure the loader could be improved. Firstly it can’t be used as an M-mode only loader that copies sections from ROM to RAM (presumably using load command metadata stored in ROM). riscv-pk is designed to start a Supervisor.

I can already see a use case based on what you described for an M-mode only loader to copy .data to RAM, zero .bss, etc. I would like a tiny M-mode ELF loader to initialise code from ROM instead of using a debugger to load the image to the desired locations. Good idea.

While there are other loaders that are being ported to RISC-V it would be nice to be able to solve some common uses using riscv-pk. It already exists. It’s not like it’s going to die or anything.

Thanks again for your thought provoking use cases.

bbl lives!

[Michael Clark]

Sorry for top posting. I’m still thinking about this...

If we store ELF in ROM on RV32E systems, we can solve these problems with a very tiny loader. Perhaps we could define a subset of ELF called TELF that has the bare minimum needed information  e.g. .bss has no data. Header is not page aligned (like Linux/binutils which is designed for big system).

sizeof(ELF32Header) + sizeof(ELF32ProgramHeader) * 4 +

sizeof(.text) +

sizeof(.rodata) +

sizeof(.data) +

0 /* .bss has size and NO_BITS */

We can make some assumption like .sdata is at the beginning of .data and so forth to reduce the number of PT_LOAD entries. We also don’t need any section entries or symbols for ROM, but they should be saved for debugging with an external debugger.

The loader can be smart i.e. if the text address equals the address of the address of the text in the ELF, then it’s XIP (Execute in place), so we just interpret the PT_LOAD commands necessary to copy .data to RAM and initialise .bss to zero. Of course we can just omit redundant PT_LOAD commands.

binutils executables are quite fat. We could use binutils and a regular linker script but then run them through a telf-util to slim them down and pack them into the TELF in-ROM format.

There are some problems with doing this post link as without PIC/PIE, we’d need the text addresses to be relative to a start address after the ELF headers.

XIP is one use case. Relocation of text into RAM is another use case (scratchpad may run faster on some chips). I think PIE is very important.

We actually need relocs so that we can re-write pointers to functions and data in the case that, the text and data addresses are not known at compile time. This is needed for static PIE as well as dynamic modules. Fortunately this is all relatively easy.

BTW I would really like shared to be re-enabled on the ELF toolchain. We just need to remove all of the .gnu and Glibc hacks from the resulting ELFs. ELF shared and runtime relocs for PIE is a very well understood and relatively simple problem.

The code to perform RELATIVE relocs at runtime is tiny and would not take too much ROM or nvram space. I’ll measure...

http://www.openwall.com/lists/musl/2012/05/24/1

It looks like it would just be several bytes which could run out of presumably slow XIP, assuming an architecture with slow XIP, no instruction or data caches and some fast (zero-wait state) SRAM attached to the main memory port.

Here static PIE makes a lot of sense and assuming XIP size is not too strict (e.g. more than 128KiB), then having the relocation code run slowly out of XIP at reset would not be too costly. Assuming XIP non-volatile memories are cheaper than the SRAMs (this seems to be the case based on data on sizes and pricing).

I will measure musl’s static_pie init code. 

musl has some nice properties. It doesn’t strictly need to be Linux specific because it has a very modular design with low coupling and uses weak linkage strategically. If I was making a femto-libc (vs newlib-nano) I would start with musl and do the following:

- remove POSIX and Linux syscalls

- keep the C stdlib stuff like <string.h>

- keep stdio - musl’s files have callbacks which don’t need to be POSIX. They could be linked to uart read write functions

- keep all the libm stuff (optional)

- make printf lighter e.g. make float support optional. one could even do a runtime check on a weak symbol to see if any math is linked in, so that linking printf doesn’t link in all of the math support. 

- make it work on RV32E in M mode with a tiny XIP loader that relocates an ELF payload, optionally .text (if XIP is slow and there is no cache)

--
You received this message because you are subscribed to the Google Groups "RISC-V SW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sw-dev+un...@groups.riscv.org.
To post to this group, send email to sw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/sw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/3FD6B013-2B1A-4F27-9B74-A20CB19F7F39%40mac.com.

[Graham Gobieski]

These are good things to think about. I wonder what a minimal example would look like. I tried messing with the bare metal tests, but unfortunately they are just too bare metal and I don't fully understand how it is emulating any system calls at all (there is a sycall.c file, but printf obviously doesn't work, etc.). Maybe a bare metal example with a new small bootloader and musl with the modifications you suggested (maybe I will spend some more time looking into this). The scope of this is sort of growing though... I wonder for testing purposes if there is a super easy hack in Spike, like can I get access to sections/symbols and then just manually move/modify them). 

[Michael Clark]

There is probably a relatively easy hack to spike. Essentially you want to trace the init path and remove the device tree parser, machine setup and enter_supervisor.

The utility to pack an ELF would be relatively fast for someone to write who had already written such a utility for ELF and mach-O several times.

If I have some time in the weekend i might have a look at this.

I have a derived version of some of bbl’s code here:

https://github.com/michaeljclark/riscv-probe

It has the minimum needed to get interrupts and printf working. We could copy add some code to embed an ELF, but instead of flat, included the entire ELF and iterate through the load segments. For me it would only take a few hours. More than half an hour. Perhaps a half day on the weekend...

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/acf9598c-1fbe-41ca-b1d2-223dcd7effcf%40groups.riscv.org.

[Michael Clark]

I actually wrote an ELF re-packer but it was trying to resize the text segment by rewriting code. This is somewhat achievable with symbols and PIC/PIE code, but really needs relocs (similar to PIE executables). If I repurpose it to just move the text after the header and trim out anything that we can’t control in binutils like hardcoded minimum alignment rules (for Linux), then we could perhaps make a telf-util that minimises the size of the ELF.

It’s a weekends worth of work.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/acf9598c-1fbe-41ca-b1d2-223dcd7effcf%40groups.riscv.org.

[Graham Gobieski]

Wow you made some great progress on this. I have been messing around with a stripped version of musl (I took out the floating point support and linux support). I have gotten it to compile and link to a program, but haven't yet got it to load properly. This is probably because none of the syscalls or traps are actually handled. Maybe I will use your repo as a jumping off point and link in standard musl stuff.

[Michael Clark]

> On 28/08/2018, at 9:27 AM, Graham Gobieski <graham....@gmail.com> wrote:
>
> Wow you made some great progress on this. I have been messing around with a stripped version of musl (I took out the floating point support and linux support). I have gotten it to compile and link to a program, but haven't yet got it to load properly. This is probably because none of the syscalls or traps are actually handled. Maybe I will use your repo as a jumping off point and link in standard musl stuff.

Yeah I spent a bit of time in the weekend factoring the riscv-probe generic code into a library, and updated to the more modern style linker scripts with MEMORY sections. The code originally came from bbl which has the old style linker scripts. I actually haven’t pushed it all yet. I was working on an experiment to run PIE executables.

One of the issues with running tests is that each executable needs to be tailored to each environment due to the different memory layouts and io devices, which necessitates linking separate binaries for each environment.

I have been thinking about a semi-hosting use case where we could link PIE executables that uses the documented SBI ecall interface for getchar, putchar and poweroff, but unlike bbl, reuse this ecall ABI in M-mode on no-mmu systems. An environment specific loader (that has io and memory layout) could decide where to copy the executable to e.g DRAM. This would allow the creation of universal binaries. Useful for small tests such as launching probes.

One could even include code in a monitor that receives executables via the serial console using xmoden, runs them and then returns to the monitor. xmodem is quite often used as a method to load firmware to devices with a serial console.

The issue with portability is that we have to fit the executable and it’s data (and the monitors .data) into a 64KiB scratchpad like the HiFive1, due to the current ABIs hardcoding the gap between .text/.rodata and .data/.bss. I think it’s possible for small programs.

When looking at PIE/PIC and the existing linker scripts, it made me realise that FDPIC would be very useful. With medlow, medany, no-pic or pic, with the current ABI, accesses to the .data/.bss sections are linked as absolute or PC-relative offsets to the .text section. i.e. the gap between non volatile read-only memory and RAM must be fixed and cannot differ.

We’d actually need FDPIC to be able to run the same binary with text/rodata and data/bss relocated to separate addresses with a variable difference between the read-only and read-write sections. That’s probably quite a lot of work as it requires a new FDPIC ABI. FDPIC is necessary to have relocatable objects on no-mmu systems. The static and executable case without PLT is much easier than dynamic as function calls are still just PC-Relative calls and they don’t need to change; it’s primarily GOT and other data accesses that can’t be PC-Relative, if the gap differs, rather .data and .bss would need to be accessed using gp. The problem is gp relative accesses have limited reach so .data and .bss address synthesis would necessarily be larger. Assuming gp is pointing at data (and read-only data is accesses PC-Relative), then data accesses would look something like this:

LUI t0, %gprel_hi(sym)
LW t1, t0, %gprel_lo(sym)(t0)

It would be nice to have universal binaries for no-mmu systems. The code sizes for static data would be larger. Accesses to .text and .rodata would still use AUIPC. The code size for globals in some cases could be smaller than PIC if the GOT was reachable by gp and could be relaxed e.g

LW t0, %gprel12(sym)(gp)

I have often wondered that FDPIC would be useful for security, as ASLR could apply different entropy to .text/.rodata vs .data/.bss. This would prevent a subset of pointer leaks such as predictable heap addresses due to program break (brk) from revealing the offset to .text

Michael

[Graham Gobieski]

So a quick update. I took your code and made it work with rv32e, then I added a _start function that copied ROM to RAM for .data sections and zeroed the .bss sections(s) (as part of a crt0.S file). This simulates and yields the expected results for my small test programs.

I agree universal binaries would be an interesting addition, but it sounds like it would take a decent-amount of work to support FDPIC. 

Also, I have begun to realize the musl is probably overkill for the embedded domain. Although it says it is positioned for embedded development, I think it is mainly meant for embedded Linux devices. Maybe a port of klibc or something similar would be better. On bare-metal embedded devices I don't really need dirent, pthreads, ipc, etc. or even complex.h or math.h (I would probably roll my own, especially since I wouldn't support floats).

[Tommy Murphy]

On Wednesday, 29 August 2018 15:07:09 UTC+1, Graham Gobieski wrote:

Also, I have begun to realize the musl is probably overkill for the embedded domain. Although it says it is positioned for embedded development, I think it is mainly meant for embedded Linux devices. Maybe a port of klibc or something similar would be better. 

Why not simply use newlib?

[Michael Clark]

> On 30/08/2018, at 2:07 AM, Graham Gobieski <graham....@gmail.com> wrote:
>
> So a quick update. I took your code and made it work with rv32e, then I added a _start function that copied ROM to RAM for .data sections and zeroed the .bss sections(s) (as part of a crt0.S file). This simulates and yields the expected results for my small test programs.

That makes sense.

I had actually simplified the linker scripts to the bare minimum as there are lots of redundant legacy aliases. Just adding __bss_start and __bss_end symbols would allow the CRT to zero .bss. That doesn’t require any additional ELF headers or metadata.

I am curious how you got the linker script and code to handle copying .data without needing any special hacks. The code needs to be linked against the RAM addresses for .data, but if you make a flat binary how do you know the offset of .data within the image at runtime? i.e. do you have a general solution? or do you have some code that is specific to your particular app? (Note: Linux gets around this using pagetables as it runs in S mode so can make PTEs match the flat section of the ELF file).

I did some work on the Makefile last night as I wanted to eliminate Makefile hacking when adding new tests. It was pretty ghetto before and only supported building one app “probe”. It’s now magic and somewhat Android NDK like but simpler.

I have used freedom-u-sdk a bit and didn’t like how it overwrites executables when I build for another target as I often ended up loading the wrong binary so it includes target architecture (rv32,rv64) for libs in the paths as well as target environment for linked binaries. I should add ABI as it will need to compile separate objects for each ABI.

I found a way for scan directories at runtime (musl has some nice make magic too) and use a single template for each target ISA:environment:io-library tuple. It expands the product of environments and examples and generates all rules on the fly. Subdirectories just define “<myapp>_objs = myapp.o”

I could probably add “<myapp>_targets = spike virt” so that the make rules just build the app for the specified target environment instead of all environments.

I added the ability to override the entry point (currently called reset_vector) in an app. There is an example call “tiny” and the stripped ELF is about 652 bytes. This is similar to the asm_main example in freedom-e-sdk. I probably should rename the entry point to _start as that is much more standard.

I’m not trying to duplicate work (as there are some similar efforts) as I actually have a branch of probe in freedom-e-sdk. The binary is much larger linked against newlib, the compiler flags are much more complex and I could not get it working, so I wanted to strip away newlib and get to the absolute bare minimum bare metal code needed to run some simple C code, without have any other bundled dependencies (the code compiles and runs with either the ELF/Newlib or Linux/Glibc toolchains).

This is for a QEMU vs hardware testing use case... freedom-e-sdk is useful to take inspiration from, but in this case most of the code came from riscv-pk and hence is under the Berkeley license.

I wanted a simple no nonsense scaffold for bare metal C tests. I especially didn’t want to bundle any other dependencies (this lowers the effect of entropy to the toolchain). It works with any riscv toolchain. I also hate autoconf. I didn’t want to have to configure it for a specific environment rather have able to build for multiple environments like the android NDK.

[Michael Clark]

On 30/08/2018, at 2:07 AM, Graham Gobieski <graham....@gmail.com> wrote:

I agree universal binaries would be an interesting addition, but it sounds like it would take a decent-amount of work to support FDPIC. 

Yes, that’s a long term thing. I’m kind of done with this for the moment anyway as I need to move back on to actually using it for some tests i’m working on. I only had an environment for asm tests. For QEMU, we need some simple open source verification (or perhaps sanity) checks so that upstream QEMU can check our changes for fixes and regressions and I need to compare QEMU to hardware.

Also, I have begun to realize the musl is probably overkill for the embedded domain. Although it says it is positioned for embedded development, I think it is mainly meant for embedded Linux devices. Maybe a port of klibc or something similar would be better. On bare-metal embedded devices I don't really need dirent, pthreads, ipc, etc. or even complex.h or math.h (I would probably roll my own, especially since I wouldn't support floats).

That’s true. Anyway thanks for the update.

riscv-probe is one of my evolving tests for QEMU vs spike vs hardware. What I did last night is make it trivial to add extra tests. No autoconf, no Makefile hacking beyond adding a directory and a rules.mk

For everyone else, here’s the link (the repo has been around for probably six months, and it evolves in spurts when I don’t have any high pressure deadlines).

https://github.com/michaeljclark/riscv-probe

[Michael Clark]

Size.

While I know there is newlib-nano, I wanted something smaller. Slightly larger than a minimal assembly program, but not much. So I called the library libfemto.

For example we can’t optimize memcpy like musl does, instead we use the minimal loop, etc.

Verification tests tend to want to have control from the first instruction so they will avoid library dependencies like newlib or any other large dependencies.

I have another repo for assembly only tests:

https://github.com/michaeljclark/qemu-riscv-tests

Ideally these can be merged together at some point. I don’t know. Independent repos are sometimes a good idea.

One of the repos just focuses on  running some assembly tests which was the focus at that point in time. Some of the asm tests are in the order of 200 or so bytes. The riscv-probe repo predates this, and was a bare metal M mode environment derived from riscv-pk/bbl with all of the Supervisor mode stuff stripped out, so I could compare control and status registers between spike and QEMU. It now supports SiFive’s E20 and E21 cores (apparently one third the area and power of our competitors).

libfemto seemed appropriate for a tiny lib that is leaner and simpler than newlib-nano.

[Graham Gobieski]

This sounds good.

To your question about linker scripts. I could only think about two solutions: first put the .data section into it's own NVRAM section (i.e RAM > AT > DATA_NVRAM), then you could just use the fixed origin and size of that memory region to copy; second and the one I ended up going with, is to export symbols from the linker script into my code (suggested by some libc reference I was looking at) and assume that the .rodata (export _end_ro_data)section comes right before the .data section in NVRAM. The first is more general than the second, but I like not fragmenting my memory any more.