PLIC and CLIC

[Michael Clark]

Hi,

I have been thinking a bit more about the PLIC (Platform Level Interrupt Controller) and what I am now referring to as the CLIC (Core Level Interrupt Controller).

I have made some outline level documentation of the virtual hardware for the template based interpreter I am working on. What is remaining to complete are SBI stubs or alternatively Linux drivers that interact directly with well-documented MMIO regions. I have a MIPI aperture for interprocessor interrupts (this is not really specified so I guessed), a TIME aperture for mtime and mtimecmp as per the specification. The model is functioning with test cases for timer, the PLIC and now the UART. I have an M-mode console echo demo for the UART. Next up is trying to bring up an OS. The layout is an artefact of looking at what Spike and QEMU were doing. Here are the docs on the layout (although I propose something a little more different with the CLIC proposal below and of course the MMIO regions should be movable with the config string):

- https://github.com/michaeljclark/riscv-meta/blob/master/doc/src/mmio.md

The thoughts that came to mind during the process:

- the MIPI hart bit field or integer MMIO register for sending IPIs, mtime and mtimecmp could be coalesced into a CLIC (Core Level Interrupt Controller) device

- there is a question as to whether MIPI would be better placed in the PLIC as a bit field, given it has wires to cores in the cluster or if there are IPI fields in each CLIC

- preferably the {M,H,S,U}IPI is an integer per CLIC that can indicate a XLEN-bit message communicated during the IPI (what I have sketched below 1 = TLB SHOOT-DOWN, etc)

- the CLIC is per hart, and each hart requires their own timer compare registers for each priority/privilege level (same as a typical linux SMP setup).

PLIC (Platform Level Interrupt Controller) MMIO Region (per node)

- pending bitfield (per IRQ), whether an interrupt has been triggered, clear to acknowledge.

- priority bitfield (per IRQ), 4 priority levels that map to masks for the 4 privilege levels. i.e. 2-bits per IRQ

- enabled bitfield (per NODE, per HART, per IRQ) that designates whether the destination hart is masked for each IRQ.

- things I have forgotten or am unaware of

u64 pending[irq_words]

u64 pritority0[irq_words]

u64 pritority1[irq_words]

u64 enabled[enabled_words]

CLIC (Core Level Interrupt Controller) MMIO Region (per core)

- mipi,hipi,sipi,uipi, 4-words (M mandatory) each corresponding to raising software interrupts at M,H,S,U levels. e.g. msip, hsip, ssip, usip

- mtime, mtimecmp, htimecmp, stimecmp, utimecmp

- mtie, htie, stie, utie would control which timers are active for the “core” raising timer interrupts at M,H,S,U levels e.g. mtip, htip, stip, utip

- the mipi region would be accessible from other cores to raise interprocessor interrupts.

- if mipi,hipi,sipi,uipi are words, then the word can store a message signal (0 means no interrupt pending)

- the IPI word is communicated along with the level m_software, h_software, s_software, u_software

- e.g. a hart receiving s_software can read sipi to get message dispatch info (e.g. TLB shoot-down vs queue wakeup) 

- things I have forgotten or am unaware of

u64 mtime

u64 mtimecmp

u64 htimecmp <reserved>

u64 stimecmp <reserved>

u64 utimecmp <reserved>

u64 mipi

u64 hipi <reserved>

u64 sipi <reserved>

u64 uipi <reserved>

It seems logical that per core timers and IPI be combined together into a CLIC (Core Level Interrupt Controller). I’m not suggesting to follow these memory layouts as is, rather this is how I am thinking about the PLIC and the CLICs and this email is a tangible sketch. This email contains a little more details on specific layout than the current specification. Of course I understand the privilege spec is a work in progress…  so this is just feedback. I think the work-to-date is excellent and is becoming easier to follow. My link above is not using this layout sketch, and is instead closer to what is visible in the current version of the privilege spec, which I believe needs more consideration of SMP timers, and (optional) S-mode autonomy for timers (for those who implement S-mode) and of course H-mode and U-mode timers in a full scale implementation.

UART

Suggest the platform spec mentions a de-facto standard such as an 8550 or 16550 style UART for console. The UART was present in the config string example in one of the earlier versions of the privileged spec. I also see 16550 UART in PowerPC Linux and MIPS Linux ports as an early boot console devices. I am not sure whether there are IP issues with the use of 16550 2x5 headers and register layout (essentially a struct). A UART makes early boot logging much much easier for bring up. Ethernet, Framebuffer and VirtIO are clearly more complicated for early bring up stage.

Interrupt Routing.

I have not thought about interrupt routing. Routing can be static or dynamic. I’ve wired the UART to IRQ 3 on the PLIC. I eventually intend to make the device configuration dynamic. Physical Interrupt wires on various devices could be statically or dynamically assigned to IRQs. It may be such that the wiring is indicated in the config so that the interrupt trap entry knows which device driver bottom half ISR routines to call. I think static mapping is sufficient for many purposes, but once PCI is involved there may need to be dynamic bus interrupt pin to IRQ routing.

BTW Good luck to everyone at the workshop!

Now the question is SBI… or direct MMIO device drivers in Linux… 

 

🤔

Regards,

Michael

[Michael Clark]

Essentially one cause vector means exceptions and interrupts can use the same jump table, and reduce cycles in the monitor. It also means one delegation register. I am not sure whether it would simplify hardware… The other is the interrupt controller asserting priority/mode to allow direct trap entry to S-mode. Some hardware can be removed by collapsing delegation mechanism into one vector where the lower-32 bits allows the lower 32 causes to be delegated. The XLEN width MIPI register per core would allow injection of message signalled interrupts (for virtualisation) or real message signalled interrupted (for physical).

For the H extension, interrupt sources may requires both a protection domain and privilege level to avoid exiting VMs. i.e. direct interrupt if the processor is running in the protection domain of the protection domain of the interrupt source in the IOMMU.

I don’t know what happens to edge interrupts for a particular protection domain while the Supervisor for that domain is not running. I guess it has to trap to the Hypervisor.

Interrupts and Virtualisation are interesting. I think we are going to need to model/prototype this before we know exactly what we need…