Coyotos/Rust Update: Tooling Hell

[Jonathan S. Shapiro]

I'm trying to send out occasional updates. This one is kind of embarrassing.

The "Good" News

I got caught up in host tool hell, and I really should have read my own code first, because it had a delightfully disgusting and undocumented solution for almost everything I was worried about.

There are a bunch of low-level dsta structures (cap representation, GPT, Process) that involve bitfield manipulations. Rust doesn't natively have bitfields.

Because we have (or at least will eventually have) a durable object store, and Processes are durable objects in Coyotos, it's tempting to imagine that mkimage needs to know things about how to initialize the target register set. It needs to know about all of the other data structures, but (up to alignment and padding concerns) those do not have any target-dependent fields. And in the on-disk process data structure, by design, all of the target-specific fields (i.e. the register set) follow the target-independent fields. The only target register that mkimage needs to set is the initial PC value.

Now that's obviously wrong. On x86, segment registers and EFLAGS obviously need to get initialized, but mkimage doesn't do that, so how the hell does that work? Answer: it cheats. It turns out that every process introduced in mkimage is initially running, because the application has to initialize itself before it can usefully do anything (including receive messages). It turns out that the per-process faultCode and faultInfo are part of the target-independent process state. And the faultInfo field has to be wide enough to hold a pointer. In fact, it has to be able to hold a coyaddr_t (always 64 bits).

So rather than try to initialize highly machine-dependent code in mkimage, I invented a new fault code PROCESS_FC_STARTUP, and initialize all processes with this fault code with their initial PC stuck in faultInfo. The moment the kernel tries to run the process, it immediately takes this distinguished trap, whereupon the kernel - which already knows everything there is to know about the target process structure - sets the rest of the process register set to clean initial values.

Which explains why looking for that initialization logic in mkimage took a long time without a useful outcome.

In the current code (which will still be relevant on FPGA), mkimage doesn't even bother to write out the register set portion of the process structure. It truncates them. When the kernel loads the process in cache_preload_image(), it zeros the object structure on general principles and then loads the leading (non-register) part from the mkimage. The register set will get initialized later when the PROCESS_FC_STARTUP fault hits.

All of that would still work for writing to an actual store, except that the tool that "pours" the image onto the disk will need to know the concrete size of the process structure.

But it means that I did a whole bunch of work building what may be the wrong tool.

The Bad News

This trick isn't quite going to work going forward, for three reasons:

1. Explicit padding for offsets and alignments doesn't work well enough across different word sizes. Something like the asm-offsets trick can be used to extract that information for use by mkimage. This means that mkimage will have to treat the structures as fixed-length byte arrays, but that's merely a nuisance.
2. Rust doesn't have bitfields natively, so there's no clean way to extract information about bitfields from the source code. Manifest constants for shift, width, and mask are probably the answer here. That will also cover most of the unions in the capability structure.
3. There is an unsafe union at the end of the capability structure. In C, it mixes pointers and integers and is discriminated by the capability "type" field. Unfortunately, "type" is a 6 bit bitfield.

Within the capability structure, one of those pointers can probably be turned into an index, though that will come at a cost. It is possible because the OTEntry table is built as a contiguous array.

Unfortunately, the same is not true of the ObjectHeader pointer. Which is the topic of my next email.