> On 13/11/2017, at 9:43 AM, Allen J. Baum <allen...@esperantotech.com
> I am under the impression, mistaken as it may be, that this is a platform specific definition, just as the reset vector is.
> I am charged with defining it for our implementations, as it turns out.
> The two obvious options are that every HART starts executing at reset (and they can find out their own ID and diverge), or that only HART#0 starts, and it is responsible for kickstarting the others.
> This slightly begs the questions of
> - how does it even know there are other HARTs, how many, and what their HART IDs are? Obvious choices:
> baked into the boot ROM
> baked into the config string
> - even if it knows, how does it start them running? Obvious choices:
> the others reset into a WFI state, so an IPI wakes them
> There is a magic MMIO locations that starts them running
> (not that IPIs are also writes to a platform defined MMIO locations
> - where do they all start running? Obvious choices:
> reset vector
> some other fixed location
> configurable by HART 0
> at the SW interrupt vector location (in the case of WFI and IPI)
> All of these will work; which is best depends on the application and specific platform features, of course.
> When you ask about stopping a HART - do you mean how a HART can stop itself, or how some other HART can cause it to suddenly stop? A HART can stop itself with a WFI.
> Reaching and and stopping it externally would be a custom platform feature (possibly as simple as an IPI to a handler that performs a WFI, or a drastic as an MMIO write that causes it clock to stop or something).
> Defining how to work out if a HART is free to execute code: are you asking how you know which HARTs are implemented, or how a master process can figure out which HARTs that it knows about are available to assign to a task?
> The former is discussed above, and the latter sounds like a standard OS feature,
I agree with your assessment however it would be beneficial that this part of the platform is standardized rather than remaining implementation defined.
I actually like the idea of a single boot hart coming up with the remaining harts sitting in WFI, waiting for an IPI, however what the early boot firmware does is a different fromhow the firmware hands off to the boot loader and OS. i.e. the early boot firmware might have all harts executing the reset vector, but select one of them as a boot hart, putting the other harts to sleep in a WFI loop, waiting for an IPI, before calling a secondary vector from a known wakeup address. i.e. a race using atomics and a timeout.
These were questions that I was drafting in response to Liviu’s email.
1) What is the upper limit on the time bound between the first and last harts executing the reset vector. This should be (implementation/specification) defined? A time limit is required to find out all harts that are present, as they might come out of reset at different times.
2) What are the interrupt enable flags defined to be coming out of reset? and does an implementation need a mechanism to change the interrupt enable flags and trap vector of a remote hart?
3). How do we reset a particular remote hart? Some implementations may define a PRCI (Power, Reset, Clock, Interrupt) MMIO region to allow a given hart to be reset. Should this be standardized? I believe it should be.
If you control the code executed at reset, and have some known area of RAM, atomic instructions, fences and time limits can be used to assign and record the harts present at boot. Some implementations memory map CSR space for all harts so hart 0 could alter a remote hart s interrupt enable flags (mstatus.MIE and mie.MSIE) and machine trap vector (mtvec) then send the hart an IPI, however this would currently be implementation defined. The problem may be deciding which hart gets assigned hart id 0, as shared memory contents may not be known to be zero.
LI t3, counter_address
AMOADD t1, t2, t3
ANDI t1, t1, npot(max_harts)-1 # nearest power of two
SW time vec(t1)
LOOP RDTIME 1ms # wait 1 millisecond
LOOP over time vec to find ones position
# sort by time+vector position, winner is earliest time within last 1 ms, vector position tie breaker
# assign hart ids starting from 0
CSRRW mhartid # write my hart id
This approach depends on AMOs, a shared address with unknown initial contents, a shared clock, and writable hart ids.
Palmer has blogged about early boot and the hart lottery used by Linux, which expects only one boot processor to be enabled at boot. The hart id problem I guess is already solved at this point. i.e. they are all unique.
I guess it is more complex if some harts don’t come up and the topology of harts and sockets is unknown by the very early boot code. i.e. all that is known is that there is shared memory access and shared access to the clock.
> At 12:52 PM +0200 11/12/17, Liviu Ionescu wrote:
>>> On 12 Nov 2017, at 12:48, Frohic Food <fro...@gmail.com
>>> Hart 0 starts at reset, but how do you start other harts executing, stop them, work out if you have a hart free to execute code on, etc.? I can't find where this is documented.
>> good question.
>> I'm also interested in such details.
>> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org
>> To post to this group, send email to isa...@groups.riscv.org
>> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/
>> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/B3AA8382-2464-4221-B44F-11C79D72859A%40livius.net
> * Allen Baum tel. (908)BIT-BAUM *
> * 248-2286 *
> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org
> To post to this group, send email to isa...@groups.riscv.org
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/p0624084ad62e5e7d0174%40%5B192.168.1.50%5D