Small System MMU config settings

[Robert Finch]

I have a small systems MMU that supports base/bound addressing and a page mapping table. I am wondering how this could be fit into the RISCV standard memory config. The satp and other register setting to use? This MMU is meant to support only a few MB at most. The MMU is always active for all operating modes. The test system is 512kB for instance. A four or five-bit ASID is in use. The page mapping table maps 1kB pages. There are two custom instructions for loading the base/bounds and mapping table ram. A more complete description is here: Cores/RISCV SSMMU Proposal.md at master · robfinch/Cores · GitHub

 

[Kelvin Goveas]

Hi Robert,

This seems like a good idea for a smaller system where the hardware and associated complexity of the data/instruction TLBs and associated table walk engine(s) can be eliminated. Some thoughts/comments:

1. What OS will you run the SMMU with - RV Linux?  RV has support for 2 virtual paging modes, Sv39 with 39-bits of physical addressing, and Sv48 with 48-bits of physical addressing. I noticed the SMMU base physical address (PA) can be at most 28-bits + 10-bits = 38-bits which is one bit less than Sv39.
2. The 'satp' register contains 3 pieces of information for virtual memory management: (1) mode - bare, Sv39 or Sv48 (2) asid and (3) the PA of the root or base of the page table for the asid specified. With the SMMU, it appears that you wouldn't need to use the satp base as all the page tables are stored locally in the processor.
3. RV also has sfence instructions which are used to order implicit accesses to the page tables in memory with respect to older stores that modify the page tables. With SMMU on a processor with out-of-order loads and stores, the sfence could instead apply to the mvmap instruction?

Kelvin

[Robert Finch]

I have been working on an OS called Femtiki (originally FMTK for Finitron Multi-Tasking Kernel). Which is a simple multi-tasker based on time-slice interrupts. I think Linux would be too large. For the device in use xc7a15T there is only about 100kB of block RAM/ROM which the OS and other functionality is pre-loaded into. So, the core of LINUX would probably need to fit into 64kB. Possible? It may be possible to load some of the OS into external RAM from a serial prom. I anticipate that the PA where the SSMMU is used would likely be very much less than 38 bits. While the base/bounds registers allow a 38-bit address range the mapping table does not support the same range. In the SSMMU of the test system only 9 bits are allowed for the physical page number (PPN) in the mapping table. There is only 512kB ram in the system. So physical memory is limited to 19 bits max. For other systems memory may be a single 16MB cell ram chip for instance. I have used the SSMMU for a system with 256MB ram but it does not work very well since a very large page size is required (64kB+) to keep the mapping table small.

 

I think the ASID needs to be stored in the satp register. From RISC-V Privileged Architectures V1.12 draft: “For RV32, the satp encodings corresponding to MODE=Bare and ASID[8:7]=3 are designated for custom use,” So, I think that means I can use the satp for custom use by selecting mode=bare and ASID[8:7]=3. With custom use selected I assume its up to the implementation to decide how to use the remaining bits. There is no base address for the page table since its internal to the core. This field may be used to indicate the size of the mapping table.

 

I had not thought about the sfence instruction as the core is in-order. But it makes sense that it would be applied for the mvmap instruction. mvmap does a swap operation. I wonder if it has the same semantics as an AMO swap? Having mvmap do an automatic fence operation is another thought. All memory and IO operation before mvmap need to be complete before mvmap can be done.

[Kelvin Goveas]

Hi Robert,

My knowledge about Linux is limited as I am a hardware designer :-), but a search on the Internet reveals that the standard Ubuntu version requires at least 2GB of memory. There may be Linux distributions targeted towards smaller (embedded) systems that may be smaller. One benefit of using Linux is that a lot of work has already been done by the RV community to port it to the RV architecture. However, with it, comes the RV virtual memory management paging architecture which requires a table walk.

Yes, satp.asid[8:7]=0x3 is reserved which allows the satp.ppn to be used for custom purposes. What processor core will a system with an SMMU be using and will it run existing software that has been compiled for RV? I assume the Femtiki OS will have its own access fault and page fault handlers, which then are free to use the custom mvmap instruction to update the page tables. RV arch defines the physical memory protection (PMP) registers for controlling RWX permissions for up to 64 memory regions. The physical memory attribute (PMA) register spec is largely left to the platform to define. The SMMU seems to be combining both PMP and PMA in one set of registers which is a more concise implementation. The RV ePMP extension also adds support for protection between privilege levels but may not be relevant if there is only 1 privilege level. The AMO extension can be implemented through hardware coherency whereas coherency of the page tables in RV is via software coherency for which the sfence is provided. 

Kelvin

[Palmer Dabbelt]

On Fri, 20 Aug 2021 08:21:45 PDT (-0700), kgo...@ventanamicro.com wrote:
> Hi Robert,
>
> My knowledge about Linux is limited as I am a hardware designer :-), but a
> search on the Internet reveals that the standard Ubuntu version requires at
> least 2GB of memory. There may be Linux distributions targeted towards
> smaller (embedded) systems that may be smaller. One benefit of using Linux
> is that a lot of work has already been done by the RV community to port it
> to the RV architecture. However, with it, comes the RV virtual memory
> management paging architecture which requires a table walk.

A normal-smelling Linux port can run in systems with significantly less
than 2GiB of memory. Getting down to 128MiB of memory can be done in a
fairly straight-forward fashion (ie, portable kernel and fully featured
userspace). I still have some systems I use with 32MiB of memory, but
that's abound as low as I bother going -- 16MiB is probably still
feasible as a demo, but I don't have any workloads I can about that will
fit.

Additionally, the RISC-V Linux port supports configurations without
page-based virtual memory (or even S-mode at all). We support a handful
of K210-based boards upstream, which have 8MiB of useable memory. This
is obviously an extremely limited port of Linux (specifically it
requires a specially crafted userspace), but it can be used to run real
programs.

Porting the !MMU configurations to other systems is a bit more
complicated than porting the more mainstream configurations, as we have
explicitly punted on portable kernels in this space (there's no way to
probe in M-mode, and users probably don't want binary compatibility any
way). If you have hardware then I'm happy to talk futher, the kernel
mailing list is probably the most efficient place to do that.

>>> 1. What OS will you run the SMMU with - RV Linux? RV has support for

>>> 2 virtual paging modes, Sv39 with 39-bits of physical addressing, and Sv48
>>> with 48-bits of physical addressing. I noticed the SMMU base physical
>>> address (PA) can be at most 28-bits + 10-bits = 38-bits which is one bit
>>> less than Sv39.

>>> 2. The 'satp' register contains 3 pieces of information for virtual

>>> memory management: (1) mode - bare, Sv39 or Sv48 (2) asid and (3) the PA of
>>> the root or base of the page table for the asid specified. With the SMMU,
>>> it appears that you wouldn't need to use the satp base as all the page
>>> tables are stored locally in the processor.

>>> 3. RV also has sfence instructions which are used to order implicit

>>> accesses to the page tables in memory with respect to older stores that
>>> modify the page tables. With SMMU on a processor with out-of-order loads
>>> and stores, the sfence could instead apply to the mvmap instruction?
>>>
>>> Kelvin
>>>
>>> On Monday, August 16, 2021 at 5:40:13 AM UTC-5 robf...@gmail.com wrote:
>>>
>>>> I have a small systems MMU that supports base/bound addressing and a
>>>> page mapping table. I am wondering how this could be fit into the RISCV
>>>> standard memory config. The satp and other register setting to use? This
>>>> MMU is meant to support only a few MB at most. The MMU is always active for
>>>> all operating modes. The test system is 512kB for instance. A four or
>>>> five-bit ASID is in use. The page mapping table maps 1kB pages. There are
>>>> two custom instructions for loading the base/bounds and mapping table ram.
>>>> A more complete description is here: Cores/RISCV SSMMU Proposal.md at
>>>> master · robfinch/Cores · GitHub

>>>> <https://github.com/robfinch/Cores/blob/master/Proposals/RISCV%20SSMMU%20Proposal.md>
>>>>
>>>>
>>>>
>>>