TLB Flush

[Florian Zaruba]

Hi,

I am starting to wonder whether I understood the synchronisation points (e.g.: where TLBs are flushed) correctly. In particular Spike seems to flush the TLBs quite often:

1. On setting the privilege level (https://github.com/riscv/riscv-isa-sim/blob/master/riscv/processor.cc#L234)
2. On setting SSTATUS/MSTATUS (https://github.com/riscv/riscv-isa-sim/blob/master/riscv/processor.cc#L377)
3. On setting SATP (https://github.com/riscv/riscv-isa-sim/blob/master/riscv/processor.cc#L463)

I couldn’t find anything in the privileged specification that 1 or 2 is necessary. According to the specification 3 should not be necessary:

Note that writing satp does not imply any ordering constraints between page-table updates and subsequent address translations. If the new address space’s page tables have been modified, it may be necessary to execute an SFENCE.VMA instruction (see Section 4.2.1) prior to writing satp.

I am asking because I am seeing strange behaviour regarding address translation when booting Linux to user-space on my core. Of-course the most likely is that I am having a bug somewhere but just want to make sure I understood the specification correctly and that flush_tlb in Spike actually has some other side-effect as well (like a pipeline flush for example).

Thanks,

Florian

--
Florian Zaruba
ETH Zurich | Integrated Systems Laboratory | Digital Circuits and Systems Group
ETZ J 89, Gloriastrasse 35, CH-8092 Zurich, Switzerland
m: zar...@iis.ee.ethz.ch | skype: florianzaruba

[Hesham Almatary]

Hi Florian,

The sfence.vma instruction is generally used to flush TLBs. From the riscv-spec:

"The SFENCE.VMA is used to flush any local hardware caches related to
address translation.
It is specified as a fence rather than a TLB flush to provide cleaner
semantics with respect to
which instructions are affected by the flush operation and to support
a wider variety of dynamic
caching structures and memory-management schemes. SFENCE.VMA is also
used by higher
privilege levels to synchronize page table writes and the address
translation hardware."

In Spike it's here [4].

Normally the OS executes sfence.vma tp update/flush TLB entries after
each page table update.

[4] https://github.com/riscv/riscv-isa-sim/blob/master/riscv/insns/sfence_vma.h#L2

Hope that helps.

Cheers,
Hesham

> --
> You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
> To post to this group, send email to hw-...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/27F4558A-EBEA-4E31-A43D-DA19A772C834%40iis.ee.ethz.ch.



--
Hesham

[Andrew Waterman]

Spike's TLB is meant to accelerate the simulator itself; it's not a model of a realistic HW TLB.  It's designed to minimize the number of host instructions in the TLB-hit case, mainly by eliding the permissions checks.  To accomplish this, it needs to flush the TLB every time the interpretation of the permission bits might change.

Privilege-mode changes change the interpretation of the permission bits for obvious reasons, so that requires a TLB flush.  (Privilege-mode changes can also disable virtual memory.  Since Spike uses the TLB even when paging is disabled, the change in meaning of virtual addresses also requires a TLB flush.)

Writing xstatus/mstatus changes the interpretation of the permission bits (MXR, MPRV+MPP) or virtual addresses (MPRV+MPP), so Spike needs to flush its TLB in these cases.

Writing satp can change the interpretation of virtual addresses (e.g. toggling MODE from Sv39 to Bare), so Spike needs to flush the TLB in this case, again because it uses the TLB even when paging is disabled.

If you look at the Rocket RTL, for example, you'll see it doesn't flush the TLB in any of these cases.

--

[Florian Zaruba]

Thanks Andrew and Hesham for your replies those clear up my doubts.