[Gemmini] flush command purpose

[Nathan Kaplan]

Hello,

I noticed that gemmini's spike simulation doesn't perform any operations when the flush command is issued. I was wondering what the purpose of the flush command is.

Is it just for consistent benchmarking results? Or could there be a situation where Gemmini's local TLB loses coherence with main memory's page table and the TLB should be flushed? Or is there another purpose that I'm not thinking about?

Would appreciate any help.  Thanks

[Hasan Genc]

If you're running just a single workload, then the initial flush isn't necessary. However, when running Gemmini workloads on FireSim, we would often run multiple binaries one-after-another on the same simulation. The OS assigns each binary its own process ID and its own virtual memory space.

If you've already run one binary on Gemmini, then the TLB will now be full of entries which no longer map to valid addresses. When you run the next binary, the TLB therefore returns physical addresses for the last process instead of the current process (which invariably leads to segfaults). Flushing at the beginning of the process avoids this possibility.

A better solution would probably have been to just check the pid of each memory request and flush the TLB silently if the pid changes. I don't remember why we didn't do that to be honest. Adding an explicit flush instruction was probably just easier to implement at the time.

Hope that helps,

Hasan

--
You received this message because you are subscribed to the Google Groups "Chipyard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/fbd6f85f-68d4-4a40-92d3-d866bbe6f9afn%40googlegroups.com.

[Albert Ou]

On Sat, Jul 20, 2024 at 12:35 PM 'Hasan Genc' via Chipyard <chip...@googlegroups.com> wrote:

If you're running just a single workload, then the initial flush isn't necessary. However, when running Gemmini workloads on FireSim, we would often run multiple binaries one-after-another on the same simulation. The OS assigns each binary its own process ID and its own virtual memory space.

If you've already run one binary on Gemmini, then the TLB will now be full of entries which no longer map to valid addresses. When you run the next binary, the TLB therefore returns physical addresses for the last process instead of the current process (which invariably leads to segfaults). Flushing at the beginning of the process avoids this possibility.

A better solution would probably have been to just check the pid of each memory request and flush the TLB silently if the pid changes. I don't remember why we didn't do that to be honest. Adding an explicit flush instruction was probably just easier to implement at the time.

Executing sfence.vma during a context switch only flushes the local core's TLB and not Gemmini's TLB, which I suspect is why the flush command was added.

In practice, however, the way that Gemmini appears to currently implement the flush command isn't much different than simply tying both TLBs to the same synchronous flush signal.  In both cases, there is a risk of invalidating a translation before Gemmini has completed executing all older commands that may depend on it, so a fence must be inserted in the context switch code to wait for the accelerator to drain.

The OS can support ASIDs to reduce the number of unnecessary invalidations, but that doesn't solve all TLB coherence issues.  There still needs to be a mechanism to flush the TLB if an ASID is reused.  Also note there is a similarly problematic race condition if the OS swaps the page table pointer (if satp is shared between the core and Gemmini) or modifies the page table in memory while Gemmini still needs it for a pending page table walk.

 

Hope that helps,

Hasan

On Fri, Jul 19, 2024 at 3:26 PM Nathan Kaplan <nska...@gmail.com> wrote:

Hello,

I noticed that gemmini's spike simulation doesn't perform any operations when the flush command is issued. I was wondering what the purpose of the flush command is.

Is it just for consistent benchmarking results? Or could there be a situation where Gemmini's local TLB loses coherence with main memory's page table and the TLB should be flushed? Or is there another purpose that I'm not thinking about?

Would appreciate any help.  Thanks

--
You received this message because you are subscribed to the Google Groups "Chipyard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/fbd6f85f-68d4-4a40-92d3-d866bbe6f9afn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Chipyard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/CAMPrpMDU3wsJ9nWCtwHuKG9pa9-7Vpg7Yp%2BHUzs2AO2qoLc70Q%40mail.gmail.com.

--

Albert Ou