How do you start/stop harts?

[Frohic Food]

Hart 0 starts at reset, but how do you start other harts executing, stop them, work out if you have a hart free to execute code on, etc.?  I can't find where this is documented.

Thanks in advance :-)

[Liviu Ionescu]

> On 12 Nov 2017, at 12:48, Frohic Food <fro...@gmail.com> wrote:
>
> Hart 0 starts at reset, but how do you start other harts executing, stop them, work out if you have a hart free to execute code on, etc.? I can't find where this is documented.

good question.

I'm also interested in such details.


Liviu

[Allen J. Baum]

I am under the impression, mistaken as it may be, that this is a platform specific definition, just as the reset vector is.

I am charged with defining it for our implementations, as it turns out.
The two obvious options are that every HART starts executing at reset (and they can find out their own ID and diverge), or that only HART#0 starts, and it is responsible for kickstarting the others.

This slightly begs the questions of
- how does it even know there are other HARTs, how many, and what their HART IDs are? Obvious choices:
baked into the boot ROM
baked into the config string
- even if it knows, how does it start them running? Obvious choices:
the others reset into a WFI state, so an IPI wakes them
There is a magic MMIO locations that starts them running
(not that IPIs are also writes to a platform defined MMIO locations
- where do they all start running? Obvious choices:
reset vector
some other fixed location
configurable by HART 0
at the SW interrupt vector location (in the case of WFI and IPI)

All of these will work; which is best depends on the application and specific platform features, of course.

When you ask about stopping a HART - do you mean how a HART can stop itself, or how some other HART can cause it to suddenly stop? A HART can stop itself with a WFI.
Reaching and and stopping it externally would be a custom platform feature (possibly as simple as an IPI to a handler that performs a WFI, or a drastic as an MMIO write that causes it clock to stop or something).

Defining how to work out if a HART is free to execute code: are you asking how you know which HARTs are implemented, or how a master process can figure out which HARTs that it knows about are available to assign to a task?
The former is discussed above, and the latter sounds like a standard OS feature,

>--
>You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
>To post to this group, send email to isa...@groups.riscv.org.
>Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/B3AA8382-2464-4221-B44F-11C79D72859A%40livius.net.


--
**************************************************
* Allen Baum tel. (908)BIT-BAUM *
* 248-2286 *
**************************************************

[Frohic Food]

HART#0 kickstarts others: that's what I'd expect.

How does it know what other HARTs are available?

I guess that's part of my question.  

On a core I have worked on any hart could request another hart as a resource which either succeeded if there was one available or failed if they were all in use and the code execution could be delayed or transferred to another core.  If a hart was available the PC for that hart is set and the hart started.

Stopping a Hart: Harts should be able to stop themselves to release the resource for other threads of execution; but also the hart that requested and started the hart should be able to kill that hart.

WFI: well this will relieve the core from having to execute this hart, which will improve the performance of the other harts executing on the core, but since you need to maintain the WFI hart's state you're not releasing the hart.  I assume there must be a way of killing the thread of execution on the hart, but I can't find it.  

Perhaps there's a RISC-V workgroup looking at this?  Or maybe it's a free for all and we can all implement what suits our own needs.

On Sun, Nov 12, 2017 at 8:43 PM, Allen J. Baum <allen...@esperantotech.com> wrote:

I am under the impression, mistaken as it may be, that this is a platform specific definition, just as the reset vector is.

I am charged with defining it for our implementations, as it turns out.
The two obvious options are that every HART starts executing at reset (and they can find out their own ID and diverge), or that only HART#0 starts, and it is responsible for kickstarting the others.

This slightly begs the questions of
 - how does it even know there are other HARTs, how many, and what their HART IDs are? Obvious choices:
        baked into the boot ROM
        baked into the config string
- even if it knows, how does it start them running? Obvious choices:
        the others reset into a WFI state, so an IPI wakes them
        There is a magic MMIO locations that starts them running
        (not that IPIs are also writes to a platform defined MMIO locations
- where do they all start running? Obvious choices:
        reset vector
        some other fixed location
        configurable by HART 0
        at the SW interrupt vector location (in the case of WFI and IPI)

All of these will work; which is best depends on the application and specific platform features, of course.

When you ask about stopping a HART - do you mean how a HART can stop itself, or how some other HART can cause it to suddenly stop? A HART can stop itself with a WFI.
Reaching and and stopping it externally would be a custom platform feature (possibly as simple as an IPI to a handler that performs a WFI, or a drastic as an MMIO write that causes it clock to stop or something).

Defining how to work out if a HART is free to execute code: are you asking how you know which HARTs are implemented, or how a master process can figure out which HARTs that it knows about are available to assign to a task?
The former is discussed above, and the latter sounds like a standard OS feature,

At 12:52 PM +0200 11/12/17, Liviu Ionescu wrote:

>--
>You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

>To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.

>To post to this group, send email to isa...@groups.riscv.org.
>Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

[Samuel Falvo II]

On Sun, Nov 12, 2017 at 12:43 PM, Allen J. Baum
<allen...@esperantotech.com> wrote:
> The two obvious options are that every HART starts executing at reset (and they can find out their own ID and diverge), or that only HART#0 starts, and it is responsible for kickstarting the others.

Third option maybe: every HART starts with a unique reset address,
executes its own bootstrap code, terminates in a WFI loop until
orchestrated by HART 0.

> - even if it knows, how does it start them running? Obvious choices:
> the others reset into a WFI state, so an IPI wakes them
> There is a magic MMIO locations that starts them running
> (not that IPIs are also writes to a platform defined MMIO locations

This is the approach that a GA144 (144-core Forth CPU fabric) chip takes.

--
Samuel A. Falvo II

[Michael Clark]

> On 13/11/2017, at 9:43 AM, Allen J. Baum <allen...@esperantotech.com> wrote:
>
> I am under the impression, mistaken as it may be, that this is a platform specific definition, just as the reset vector is.
>
> I am charged with defining it for our implementations, as it turns out.
> The two obvious options are that every HART starts executing at reset (and they can find out their own ID and diverge), or that only HART#0 starts, and it is responsible for kickstarting the others.
>
> This slightly begs the questions of
> - how does it even know there are other HARTs, how many, and what their HART IDs are? Obvious choices:
> baked into the boot ROM
> baked into the config string
> - even if it knows, how does it start them running? Obvious choices:
> the others reset into a WFI state, so an IPI wakes them
> There is a magic MMIO locations that starts them running
> (not that IPIs are also writes to a platform defined MMIO locations
> - where do they all start running? Obvious choices:
> reset vector
> some other fixed location
> configurable by HART 0
> at the SW interrupt vector location (in the case of WFI and IPI)
>
> All of these will work; which is best depends on the application and specific platform features, of course.
>
> When you ask about stopping a HART - do you mean how a HART can stop itself, or how some other HART can cause it to suddenly stop? A HART can stop itself with a WFI.
> Reaching and and stopping it externally would be a custom platform feature (possibly as simple as an IPI to a handler that performs a WFI, or a drastic as an MMIO write that causes it clock to stop or something).
>
> Defining how to work out if a HART is free to execute code: are you asking how you know which HARTs are implemented, or how a master process can figure out which HARTs that it knows about are available to assign to a task?
> The former is discussed above, and the latter sounds like a standard OS feature,

It’s interesting.

I agree with your assessment however it would be beneficial that this part of the platform is standardized rather than remaining implementation defined.

I actually like the idea of a single boot hart coming up with the remaining harts sitting in WFI, waiting for an IPI, however what the early boot firmware does is a different fromhow the firmware hands off to the boot loader and OS. i.e. the early boot firmware might have all harts executing the reset vector, but select one of them as a boot hart, putting the other harts to sleep in a WFI loop, waiting for an IPI, before calling a secondary vector from a known wakeup address. i.e. a race using atomics and a timeout.

These were questions that I was drafting in response to Liviu’s email.

1) What is the upper limit on the time bound between the first and last harts executing the reset vector. This should be (implementation/specification) defined? A time limit is required to find out all harts that are present, as they might come out of reset at different times.

2) What are the interrupt enable flags defined to be coming out of reset? and does an implementation need a mechanism to change the interrupt enable flags and trap vector of a remote hart?

3). How do we reset a particular remote hart? Some implementations may define a PRCI (Power, Reset, Clock, Interrupt) MMIO region to allow a given hart to be reset. Should this be standardized? I believe it should be.

If you control the code executed at reset, and have some known area of RAM, atomic instructions, fences and time limits can be used to assign and record the harts present at boot. Some implementations memory map CSR space for all harts so hart 0 could alter a remote hart s interrupt enable flags (mstatus.MIE and mie.MSIE) and machine trap vector (mtvec) then send the hart an IPI, however this would currently be implementation defined. The problem may be deciding which hart gets assigned hart id 0, as shared memory contents may not be known to be zero.

RDTIME t2
LI t3, counter_address
AMOADD t1, t2, t3
ANDI t1, t1, npot(max_harts)-1 # nearest power of two
SW time vec(t1)
LOOP RDTIME 1ms # wait 1 millisecond
LOOP over time vec to find ones position
# sort by time+vector position, winner is earliest time within last 1 ms, vector position tie breaker
# assign hart ids starting from 0
CSRRW mhartid # write my hart id
BEQZ boot_hart
WFI
J wakeup_vec

This approach depends on AMOs, a shared address with unknown initial contents, a shared clock, and writable hart ids.

Palmer has blogged about early boot and the hart lottery used by Linux, which expects only one boot processor to be enabled at boot. The hart id problem I guess is already solved at this point. i.e. they are all unique.

- https://www.sifive.com/blog/2017/10/09/all-aboard-part-6-booting-a-risc-v-linux-kernel/

I guess it is more complex if some harts don’t come up and the topology of harts and sockets is unknown by the very early boot code. i.e. all that is known is that there is shared memory access and shared access to the clock.

> At 12:52 PM +0200 11/12/17, Liviu Ionescu wrote:
>>> On 12 Nov 2017, at 12:48, Frohic Food <fro...@gmail.com> wrote:
>>>
>>> Hart 0 starts at reset, but how do you start other harts executing, stop them, work out if you have a hart free to execute code on, etc.? I can't find where this is documented.
>>
>> good question.
>>
>> I'm also interested in such details.
>>
>>
>> Liviu
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
>> To post to this group, send email to isa...@groups.riscv.org.
>> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/B3AA8382-2464-4221-B44F-11C79D72859A%40livius.net.
>
>
> --
> **************************************************
> * Allen Baum tel. (908)BIT-BAUM *
> * 248-2286 *
> **************************************************
>
> --
> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
> To post to this group, send email to isa...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/p0624084ad62e5e7d0174%40%5B192.168.1.50%5D.

[Liviu Ionescu]

> On 12 Nov 2017, at 22:43, Allen J. Baum <allen...@esperantotech.com> wrote:
>
> The two obvious options are that every HART starts executing at reset (and they can find out their own ID and diverge), or that only HART#0 starts, and it is responsible for kickstarting the others.

according to the actual specs, it looks like every hart starts executing at reset.

the reset address is device specific, and it is not required for the addresses to be different, so for a smp device I would expect all harts to start the same code and one of them (probably hart 0) to take the lead, all the other enter in a form of sleep.

regards,

Liviu

[Andrew Waterman]

It's unbounded, as other harts may remain unpowered/unclocked until some action is taken by another hart.

The devicetree should be used to discover the other harts, rather than shared memory and a timeout.

2) What are the interrupt enable flags defined to be coming out of reset? and does an implementation need a mechanism to change the interrupt enable flags and trap vector of a remote hart?

mstatus.MIE is 0 out of reset; the rest is unspecified.

 

3). How do we reset a particular remote hart? Some implementations may define a PRCI (Power, Reset, Clock, Interrupt) MMIO region to allow a given hart to be reset. Should this be standardized? I believe it should be.

The act of putting a core into reset can require a litany of platform-specific actions, which can be complicated enough they require some degree of software sequencing.

This should be hidden behind an SBI call so that it's standardized at the OS level, but it's unrealistic to expect the SBI call implementation to be the same across implementations.

>> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.

>> To post to this group, send email to isa...@groups.riscv.org.
>> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/B3AA8382-2464-4221-B44F-11C79D72859A%40livius.net.
>
>
> --
> **************************************************
> * Allen Baum              tel. (908)BIT-BAUM     *
> *                                   248-2286     *
> **************************************************
>
> --
> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.

> To post to this group, send email to isa...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/p0624084ad62e5e7d0174%40%5B192.168.1.50%5D.

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.

To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/7FCF6BDF-33A5-488B-831B-6D0C551A040B%40mac.com.

[Liviu Ionescu]

On 13 Nov 2017, at 00:40, Michael Clark <michae...@mac.com> wrote:

1) What is the upper limit on the time bound between the first and last harts executing the reset vector. This should be (implementation/specification) defined? A time limit is required to find out all harts that are present, as they might come out of reset at different times.

isn't there any other mechanism to discover the number of harts?

between adding such a timeout to the specs and adding a reliable mechanism to get the number (or the individual IDs?) of all harts, I would choose the second.

2) What are the interrupt enable flags defined to be coming out of reset? and does an implementation need a mechanism to change the interrupt enable flags and trap vector of a remote hart?

if I understood the specs right, only very few registers need to be cleared by reset, most of them should be cleared by software.

3). How do we reset a particular remote hart? Some implementations may define a PRCI (Power, Reset, Clock, Interrupt) MMIO region to allow a given hart to be reset. Should this be standardized? I believe it should be.

this seems to imply a certain hierarchy for the harts, or at least a master hart that orchestrates the others, similar to the U54-MC.

regards,

Liviu

[ron minnich]

On Sun, Nov 12, 2017 at 2:55 PM Liviu Ionescu <i...@livius.net> wrote:

according to the actual specs, it looks like every hart starts executing at reset.

kind of like the k7.

 

the reset address is device specific, and it is not required for the addresses to be different, so for a smp device I would expect all harts to start the same code and one of them (probably hart 0) to take the lead, all the other enter in a form of sleep.

In coreboot for the k7 we had to elect the leader and let the others sleep, via a software process involving a lock. You resumed the APs once the platform was set up by releasing the lock. For k8 and intel, there was a guarantee that only the BSP was running and you had to awake the APs via an IPI mechanism. 

Neither approach  is unambiguously better than the other in our experience.

Hope this helps.

[Allen J. Baum]

I'm pretty much agreeing with Andrew here:

The configuration should be determined by reading the config string ( at an implementation defined address).

You can't set a timeout because some HARTs maybe be powered off or clock disabled (which contradicts Liviu's assertion that

    "according to the actual specs, it looks like every hart starts executing at reset."

I think the spec says that only HART ID0 is required to run at reset:

     "In certain cases, we must ensure exactly one hart runs some code (e.g., at reset), and so require one hart to have a known hart ID of zero."

The spec says only that, at reset:

 priv_mode      <-- M

 mstatus.MIE    <-- 0

 mstatus.MPRV   <-- 0

 PC             <-- (implementation defined value)

 mcause <-- (0 or implementation defined values indicating type of reset

and all other state is undefined (though I think the wording should be "not specified", as "undefined" has a pretty specific meaning when you're taping out a chip).

I actually don't like the idea for all HARTs to start executing all at the same time, primarily because that will cause possibly unpleasant current inrush.

 True story: the Apple Newton had an issue that, when the battery was low, waking it up would cause enough current inrush to drop the supply voltage, which signalled a low-voltage battery condition that would put it to sleep - at which point voltage recovered and it would try to wake up again.... until it totally drained the battery and couldn't.

The software fix was to start it in slow clock mode and gradually ramp it up.

Having multiple HARTs multiplies that issue.

Better to either stagger wakeup, or just leave them powered off/unclocked, or both.

Putting my RAS hat on - if you only wake up one HART - how do you select which one to wake? If it is always HART zero, you now have a single point of failure- especially since HART 0 is required to exist.

There are solutions, but it seems to me that HART ids should not be fixed in HW if you want to avoid that problem. Since The HART ID CSR is in a read_only region, I would infer that some backdoor access is required to change it with HW sequencer support.

>> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

>> To post to this group, send email to isa...@groups.riscv.org.
>> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/B3AA8382-2464-4221-B44F-11C79D72859A%40livius.net.
>
>
> --
> **************************************************
> * Allen Baum              tel. (908)BIT-BAUM     *
> *                                   248-2286     *
> **************************************************
>
> --
> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

> To post to this group, send email to isa...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/p0624084ad62e5e7d0174%40%5B192.168.1.50%5D.

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/7FCF6BDF-33A5-488B-831B-6D0C551A040B%40mac.com.

[Andrew Waterman]

On Sun, Nov 12, 2017 at 4:08 PM, Allen J. Baum <allen...@esperantotech.com> wrote:

I'm pretty much agreeing with Andrew here:

The configuration should be determined by reading the config string ( at an implementation defined address).

You can't set a timeout because some HARTs maybe be powered off or clock disabled (which contradicts Liviu's assertion that

    "according to the actual specs, it looks like every hart starts executing at reset."

I think the spec says that only HART ID0 is required to run at reset:

     "In certain cases, we must ensure exactly one hart runs some code (e.g., at reset), and so require one hart to have a known hart ID of zero."

The spec says only that, at reset:

 priv_mode      <-- M

 mstatus.MIE    <-- 0

 mstatus.MPRV   <-- 0

 PC             <-- (implementation defined value)

 mcause <-- (0 or implementation defined values indicating type of reset

and all other state is undefined (though I think the wording should be "not specified", as "undefined" has a pretty specific meaning when you're taping out a chip).

I actually don't like the idea for all HARTs to start executing all at the same time, primarily because that will cause possibly unpleasant current inrush.

 True story: the Apple Newton had an issue that, when the battery was low, waking it up would cause enough current inrush to drop the supply voltage, which signalled a low-voltage battery condition that would put it to sleep - at which point voltage recovered and it would try to wake up again.... until it totally drained the battery and couldn't.

The software fix was to start it in slow clock mode and gradually ramp it up.

Having multiple HARTs multiplies that issue.

Better to either stagger wakeup, or just leave them powered off/unclocked, or both.

Putting my RAS hat on - if you only wake up one HART - how do you select which one to wake? If it is always HART zero, you now have a single point of failure- especially since HART 0 is required to exist.

There are solutions, but it seems to me that HART ids should not be fixed in HW if you want to avoid that problem. Since The HART ID CSR is in a read_only region, I would infer that some backdoor access is required to change it with HW sequencer support.

Indeed.

Hardwiring the hart ID inside the core can be problematic for other reasons (e.g. if you want to perform physical implementation once then tile multiple cores).

At 2:58 PM -0800 11/12/17, Andrew Waterman wrote:

[Michael Clark]

Device tree is fine and there needs to be an abstraction layer that is appropriate to hand off to the OS.

The case of an SOC where the core topology is known in advance is one particular case where static device tree is most likely fine.

There may however be cases where early firmware wants to create the device tree by dynamic hardware introspection, and it may be using MMIO apertures exposed by devices that the firmware is intimate with. Imagine ROM on a board with multiple CPU sockets and non uniform latency between sockets which may contain heterogeneous cores which come of out reset at different times. Static device tree won’t work.

> 2) What are the interrupt enable flags defined to be coming out of reset? and does an implementation need a mechanism to change the interrupt enable flags and trap vector of a remote hart?
>
> mstatus.MIE is 0 out of reset; the rest is unspecified.
>
>
> 3). How do we reset a particular remote hart? Some implementations may define a PRCI (Power, Reset, Clock, Interrupt) MMIO region to allow a given hart to be reset. Should this be standardized? I believe it should be.
>
> The act of putting a core into reset can require a litany of platform-specific actions, which can be complicated enough they require some degree of software sequencing.

Sure. Early boot likely has to do clock and PLL programming, set voltages, download RAM timings, configure the memory controller and caches before it can talk to DRAM. And in the case where RAM is socketed, the firmware may perform power on tests and decide whether or not it wants to use particular channels, etc. A myriad of issues depending on the complexity of the particular system and its components…

> This should be hidden behind an SBI call so that it’s standardized at the OS level, but it's unrealistic to expect the SBI call implementation to be the same across implementations.

Agree, however whatever is hidden behind the SBI needs to be specified well enough that firmware writers have some degree of control. Many folk won’t want binary blobs at this level, however given RISC-V doesn’t preclude proprietary implementations there may certainly be implementations that do have binary blobs. That said, the market might prefer open systems, given the open nature of RISC-V is one of its main selling points.

> >> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

> >> To post to this group, send email to isa...@groups.riscv.org.
> >> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
> >> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/B3AA8382-2464-4221-B44F-11C79D72859A%40livius.net.
> >
> >
> > --
> > **************************************************
> > * Allen Baum tel. (908)BIT-BAUM *
> > * 248-2286 *
> > **************************************************
> >
> > --
> > You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

> > To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

> > To post to this group, send email to isa...@groups.riscv.org.
> > Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
> > To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/p0624084ad62e5e7d0174%40%5B192.168.1.50%5D.
>
> --
> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

[Liviu Ionescu]

On 13 Nov 2017, at 02:08, Allen J. Baum <allen...@esperantotech.com> wrote:

I'm pretty much agreeing with Andrew here:

The configuration should be determined by reading the config string ( at an implementation defined address).

a config string is ok, but not having it at a fixed address requires another complication to discover it.

You can't set a timeout because some HARTs maybe be powered off or clock disabled (which contradicts Liviu's assertion that

    "according to the actual specs, it looks like every hart starts executing at reset."

I think the spec says that only HART ID0 is required to run at reset:

     "In certain cases, we must ensure exactly one hart runs some code (e.g., at reset), and so require one hart to have a known hart ID of zero."

The spec says only that, at reset:

 priv_mode      <-- M

 mstatus.MIE    <-- 0

 mstatus.MPRV   <-- 0

 PC             <-- (implementation defined value)

 mcause <-- (0 or implementation defined values indicating type of reset

and all other state is undefined (though I think the wording should be "not specified", as "undefined" has a pretty specific meaning when you're taping out a chip).

if so, then in 3.3, the statement should read  'The pc of HART ID 0 is set to an implementation-defined reset vector.' and all other HARTs should be kept in RESET until HART 0  programatically releases the RESET, probably via a separate register with bits for each HART.

for regular applications I think that starting HART ID0 is enough. however for high availability devices, which must account for some redundancy, it might be necessary to start multiple HARTs. (if I remember right, some TI Cortex-R devices have two cores, placed at 90 degrees on the chip, such that if some high energy particles strike the chip, to reduce impact).

probably the generic way would be to add to the specs the register with the RESET bits (if not already there), and to specify how to programatically set/release the reset line for a given HART, but also alow the implementation to define which HARTs start automatically at reset, since most applications will probably be happy to start a single HART.

I actually don't like the idea for all HARTs to start executing all at the same time, primarily because that will cause possibly unpleasant current inrush.

a very good reason.

... it seems to me that HART ids should not be fixed in HW if you want to avoid that problem.

there might be solutions for this, but with even more increased complexity.

the problem I faced, related to HART IDs was the definitions required for the PLIC driver. I know that the PLIC is device specific, but the SiFive devices implement it as a MMIO, with some arrays of structures, sized by the number of HARTs.

once we do this, we also restrict the HART IDs to the subset 0, 1, ..., (n-1), since the ID is used as index in these arrays.

to make things worse, for U54-MC this was not even an array, it is a separate element for HART0, and an array for HART1-4, since the definitions for HART0 differ from the definitions for the other HARTs. this makes access to these not-uniform registers complicated and slow.

making the IDs configurable in software will probably further complicate these cases.

as long as these devices still need some software to run, it would be good not to make the lives of the software guys more difficult than necessary.

regards,

Liviu

[Mauro Olivieri]

Hello,

sorry for re-starting this discussion back from the initial question, but I have doubts about assuming that only "Hart 0 starts at reset" by specification. While it may be a convenient implementation for many reasons discussed in the previous comments, it seems the specs do not require such solution and actually there are representative implementations in which all the harts start at reset (possibly stopping immediately by a software mechanism, if required), e.g. in the PULP architecture by ETH.

In summary, one issue is to consider the most convenient implementation, another thing is to consider what implementation is legal according to RISC-V specs.

thanks

Mauro

2017-11-12 11:48 GMT+01:00 Frohic Food <fro...@gmail.com>:

Hart 0 starts at reset, but how do you start other harts executing, stop them, work out if you have a hart free to execute code on, etc.?  I can't find where this is documented.

Thanks in advance :-)

--

You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/58c862c4-99f0-4ed5-890e-91b12cdd602f%40groups.riscv.org.

[Liviu Ionescu]

> On 13 Nov 2017, at 12:00, Mauro Olivieri <olivieri...@gmail.com> wrote:
>
> ... I have doubts about assuming that only "Hart 0 starts at reset" by specification.

agree

> ... one issue is to consider the most convenient implementation, another thing is to consider what implementation is legal according to RISC-V specs.

Andrew might help us to better understand the specs, but as long as '3.3 Reset', states 'The pc is set to an implementation-defined reset vector.' and does not restrict this behaviour to a limited set of harts, a 1.10 compliant implementation should probably start all harts at reset.


regards,

Liviu

[Andrew Waterman]

My reading of the spec is that it is permitted but not required to start all harts. So, it is left up to the platform to decide.

regards,

Liviu

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/2FA586F5-E9FE-497A-B15C-60C69CFE53EB%40livius.net.

[Liviu Ionescu]

> On 13 Nov 2017, at 20:00, Andrew Waterman <wate...@eecs.berkeley.edu> wrote:
>
> My reading of the spec is that it is permitted but not required to start all harts. So, it is left up to the platform to decide.

if so, to avoid threads like this, don't you think that the specs should explicitly mention this?


Liviu

[Allen Baum]

-Allen

> On Nov 12, 2017, at 6:12 PM, Andrew Waterman <wate...@eecs.berkeley.edu> wrote:
>
> Hardwiring the hart ID inside the core can be problematic for other reasons (e.g. if you want to perform physical implementation once then tile multiple cores).

I don’t believe that’s much of a problem. The (an?) industry solution is to make the HART ID be a set of strap pins, which are tied to power and ground outside the tile.

[Rishiyur Nikhil]

This is not the same question, of course, as production, free-running
harts, but just wanted to mention, in case people were not aware, that
the "External Debug Module" spec has facilities to reset/start/stop
individual harts, either individually or in groups.

Nikhil

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.
To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CC5F2F6A-BFF6-41A3-9E21-C9056699A678%40esperantotech.com.

[Allen J. Baum]

At 2:58 PM -0800 11/12/17, Andrew Waterman wrote:

mstatus.MIE is 0 out of reset; the rest is unspecified.

 

What I would like to do is have a core come out of a full reset in WFI state, and have some other core wake it up with an IPI

(question about IPI: your slides say:

  - One hart can write to different hart's MSIP register

  - Mechanism for inter-hart interrupts.

But MSIP is a bit in the MIP CSR - are you saying that from an MMIO point of view,

        -there is a address corresponding to just that bit, as opposed to writing the entire CSR?

        - Or would an AMOOR to the MIP MMIO view what is expected

        - Or is it simply completely implementation dependent?

 

 - is there anything in the spec that prohibits waking up from reset in a WFI state?

 - if is woken up, what is the next instruction it should execute?

 Reset vector? Reset vector+4 (since execution will resume at pc + 4, and the PC is initialized to the reset vector)?

 - MIE is clear at reset. But WFI wakeup ignores xIE, ignores xDELEG, and only pays attention to  xSIP/xTIP/xEIP iff the corresponding xSIE/xTIE/xEIE bits are set- or at least that is how I am interpreting the spec.

   This implies that MSIE most also be set to one in the WFI core at reset.

   I believe setting the initial state of MSIE=1 is legal to implement (i.e. not prohibited, left up to implementation)

[Andrew Waterman]

On Tue, Nov 14, 2017 at 9:31 AM, Allen J. Baum <allen...@esperantotech.com> wrote:

At 2:58 PM -0800 11/12/17, Andrew Waterman wrote:

mstatus.MIE is 0 out of reset; the rest is unspecified.

 

What I would like to do is have a core come out of a full reset in WFI state, and have some other core wake it up with an IPI

(question about IPI: your slides say:

  - One hart can write to different hart's MSIP register

  - Mechanism for inter-hart interrupts.

But MSIP is a bit in the MIP CSR - are you saying that from an MMIO point of view,

        -there is a address corresponding to just that bit, as opposed to writing the entire CSR?

        - Or would an AMOOR to the MIP MMIO view what is expected

        - Or is it simply completely implementation dependent?

The MSIP bit is writable through a memory-mapped address.  The bit is read-only through the MIP CSR.

The details of the memory-mapped MSIP registers are up to the platform.

 

 - is there anything in the spec that prohibits waking up from reset in a WFI state?

I think this is a valid decision for a hardware platform to make.

 - if is woken up, what is the next instruction it should execute?

 Reset vector? Reset vector+4 (since execution will resume at pc + 4, and the PC is initialized to the reset vector)?

Reset vector seems like the sensible way to define it (i.e., WFI-state was entered before attempting to execute the instruction at the reset vector).

 - MIE is clear at reset. But WFI wakeup ignores xIE, ignores xDELEG, and only pays attention to  xSIP/xTIP/xEIP iff the corresponding xSIE/xTIE/xEIE bits are set- or at least that is how I am interpreting the spec.

   This implies that MSIE most also be set to one in the WFI core at reset.

   I believe setting the initial state of MSIE=1 is legal to implement (i.e. not prohibited, left up to implementation)

Right - setting MSIE=1 is legal, and is necessary to make this strategy work.

[Mauro Olivieri]

Hi all, concerning harts in wfi state, when they are waken up by an inter-hart interrupt through the MIP bit, I think they should start at the MTVEC address according to the specs, like for any interrupt Then the software should identify the cause through MCAUSE and jump to a platform defined address.
Am I going wrong?
M

> --
> You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

> To post to this group, send email to isa...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CA%2B%2B6G0ADWEAmtySgNQR8kZ1VZCTCyW2VK_Ki0H6gw15Nr3Z%2B2g%40mail.gmail.com.
>