Questions about MTIMECMP and timer interrupts

[Rishiyur Nikhil]

In Priv Spec 1.10, Sec 3.1.15 "Machine Timer Registers" it says:

    "The interrupt remains posted until it clears by writing the MTIMECMP register".

Does "clears" mean resetting the MIP.MTIP bit, or just in an interrupt controller?

Must it be immediate, i.e., precondition to retiring the STORE instruction that writes MTIMECMP,

or can there be a delay?

Thanks,

Nikhil

[Alex Solomatnikov]

On Tue, Jul 17, 2018 at 9:45 AM, Rishiyur Nikhil <nik...@bluespec.com> wrote:

In Priv Spec 1.10, Sec 3.1.15 "Machine Timer Registers" it says:

    "The interrupt remains posted until it clears by writing the MTIMECMP register".

Does "clears" mean resetting the MIP.MTIP bit, or just in an interrupt controller?

This is implementation dependent, e.g. in https://github.com/sifive/clic-spec/blob/master/clic.adoc MIP.MTIP is not used at all.

 

Must it be immediate, i.e., precondition to retiring the STORE instruction that writes MTIMECMP,

or can there be a delay?

Definitely not a precondition to retiring the STORE instruction, STORE reaches MTIMECMP some time after it is retired. This latency is implementation dependent.

Alex

Thanks,

Nikhil

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+unsubscribe@groups.riscv.org.
To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPmv9kznik8aCvi-SvORfeE%3DLNGBu_9R78HwiBMn8BNBbg%40mail.gmail.com.

[Rishiyur Nikhil]

Thanks Alex. What you say makes sense, given how timers are likely to

be implmented somewhere else on the chip.

But I'm still puzzled by this: how does the CPU _predictably_ resume,

after one timer interrupt, ready for the next timer interrupt, without

the danger of re-triggering on the first timer interrupt?

Detail:

- Section 3.1.14 of Priv Spec 1.10 says:

   "The MTIP bit is READ-ONLY [emphasis mine] and is cleared by

    writing to the memory-mapped machine-mode timer compare register."

- You said: "... the store reaches MTIMECMP some time after it is

    retired. This latency is implementation dependent."

- So, consider this scenario:

    1. MTIME crosses MTIMECMP, generating a timer interrupt,

        eventually setting MIP.MTIP ("first timer interrupt"), after

        some implementation-dependent latency.

    2. The CPU responds to MIP.MTIP and takes the timer interrupt.  On

        taking the interrupt, MSTATUS.MIE is automatically reset to 0,

        so the CPU is immediately protected from interrupts. It can

        further programmatically protect from timer interrupts by

        setting MIE.MTIE to 0.

The question is, when can the CPU _safely_ re-enable timer interrupts

(via MIE.MTIE and MSTATUS.MIE)??  Just doing a store to MTIMECMP is

not enough, because of the implementation-dependent latency.  If it

does not wait long enough, it might re-trigger on the MIP.MTIP setting

from the first timer interrupt.

Thanks for any insight on this.

Nikhil

On Tue, Jul 17, 2018 at 1:58 PM, Alex Solomatnikov <so...@sifive.com> wrote:

On Tue, Jul 17, 2018 at 9:45 AM, Rishiyur Nikhil <nik...@bluespec.com> wrote:

In Priv Spec 1.10, Sec 3.1.15 "Machine Timer Registers" it says:

    "The interrupt remains posted until it clears by writing the MTIMECMP register".

Does "clears" mean resetting the MIP.MTIP bit, or just in an interrupt controller?

This is implementation dependent, e.g. in https://github.com/sifive/clic-spec/blob/master/clic.adoc MIP.MTIP is not used at all.

 

Must it be immediate, i.e., precondition to retiring the STORE instruction that writes MTIMECMP,

or can there be a delay?

Definitely not a precondition to retiring the STORE instruction, STORE reaches MTIMECMP some time after it is retired. This latency is implementation dependent.

Alex

Thanks,

Nikhil

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.

[Andrew Waterman]

You could do something like "while (mip.mtip && mtime < mtimecmp) ;".

> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPkONJLYYBjM7Zi0Y%2B%3DT9wvJ2zTBab9VenKrXK7ZkqOqzg%40mail.gmail.com.

[Rishiyur Nikhil]

Thanks, Andrew.

Busy-waiting like this is clearly a solution; I was wondering if there was some "simpler" way.

Ok, thanks.

Nikhil

>>> email to isa-dev+unsubscribe@groups.riscv.org.

>>> To post to this group, send email to isa...@groups.riscv.org.
>>> Visit this group at
>>> https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPmv9kznik8aCvi-SvORfeE%3DLNGBu_9R78HwiBMn8BNBbg%40mail.gmail.com.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "RISC-V ISA Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to isa-dev+unsubscribe@groups.riscv.org.

> To post to this group, send email to isa...@groups.riscv.org.
> Visit this group at
> https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
> To view this discussion on the web visit

> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPkONJLYYBjM7Zi0Y%2B%3DT9wvJ2zTBab9VenKrXK7ZkqOqzg%40mail.gmail.com.

[Samuel Falvo II]

On Tue, Jul 17, 2018 at 1:11 PM, Rishiyur Nikhil <nik...@bluespec.com> wrote:

> But I'm still puzzled by this: how does the CPU _predictably_ resume,
> after one timer interrupt, ready for the next timer interrupt, without
> the danger of re-triggering on the first timer interrupt?

This might not be a possibility; but given ideal circumstances, I
think I might design a system like this:

When mtime == mtimecmp, put a copy of mtime into a queue and trigger
an interrupt *pulse*. The interrupt pulse is a unidirectional
message: it could be expressed as a single pulse for a CPU IRQ line,
or expressed as a MSI transaction on an interconnect somehow. That
this is a unidirectional transaction cleaves the dependency between
the CPU and the peripheral.

When it finally gets around to executing the ISR, the CPU can then
check the timer's queue, and if non-empty, it knows that an interrupt
occurred *and when*. The CPU can then update mtimecmp and return from
the ISR without fear of retriggering the same interrupt, since the
interrupt itself was just a pulse. You could even have multiple
mtimecmp registers to allow for high precision notifications (and a
queue depth to match, of course).

This implies that MTIP can be set from external logic, but not
cleared. Use a CSRRW instruction to clear the relevant flags. This,
of course, requires altering the interrupt-pending flags as read/write
(or, at least, introducing an "interrupt clear" register local to the
hart).


--
Samuel A. Falvo II

[Samuel Falvo II]

On Tue, Jul 17, 2018 at 1:19 PM, Andrew Waterman <and...@sifive.com> wrote:
> You could do something like "while (mip.mtip && mtime < mtimecmp) ;".

Hmmmm. I'm probably over-thinking this, but I've been trained over
the years to have a knee-jerk reaction to any kind of busy-waiting in
a multitasking environment, much less a multi-processing environment.
To me, even though this seems like it ought not be any more expensive
than any other kind of spin-lock, this seems inefficient, and
potentially ill-advised especially for battery-operated equipment.

If mip.mtip is clear, then we break the loop for a known good reason
and can safely return from the ISR. I agree with this logic, even if
I'm unhappy with a busy-wait loop.

The other case, however, remains a bigger source of my concern. If
mip.mtip remains set, there are two possibilities:

1) that mtimecmp updates and mtime >= mtimecmp in the same cycle (or
in a cycle between when last check and the current check), which will
set mtip in a way that the loop never sees it cleared.

2) that mtimecmp has not yet updated since the last loop iteration.

It seems to me that the ISR should be wrapped in an infinite loop and
explicitly checking for valid exit-points, instead of busy-waiting on
a specific condition that, under rare conditions, might deadlock the
hart.

Your ISR might have to be structured like:

// As long as mtime >= mtimecmp, we know we have expired the timer.
// If we're here and this isn't the case, we have either a spurious interrupt,
// or a hopefully harmless kernel bug, in which case it should be safe to
// return from the ISR without further processing.

while(mtime >= mtimecmp) {

// If the MTIP flag isn't set, we can safely return from the ISR.
// Alternatively, this might be a possible hardware issue to look out for.

if(!mip.mtip) break;

// Otherwise, handle the interrupt, calculate the new mtimecmp value, and
// wait for it to update. It's possible that the mtimecmp can cut things
// so close that we might generate an interrupt while returning
from this ISR.
// This can be seen as MTIP never clearing, but mtimecmp updating.

mtimecmp = calculate_new_mtime_value();
while(mip.mtip && mtimecmp < mtime);

// To avoid unnecessary register save/restore overhead, loop until we're
// confident that the next timer event is sufficiently in the future that
// ISR dispatch overhead is reasonably well amortized.

[Guy Lemieux]

Normally, I would say you must execute a FENCE to ensure the write is
complete before proceeding.

The problem is things are still not fully specified... it's not clear
which minimal options must be passed to the fence. Is mtimecmp an I/O
location, or a R/W location? According to the ISA Spec:

"The execution environment will define what I/O operations are
possible, and in particular,
which load and store instructions might be treated and ordered as
device input and device output
operations respectively rather than memory reads and writes. For
example, memory-mapped I/O
devices will typically be accessed with uncached loads and stores that
are ordered using the I and O
bits rather than the R and W bits."

so, would it be the minimal
FENCE PO,SI
because the MMIO locations are uncached?

or would it be the more inclusive:
FENCE PO,SIO

or would it use R/W instead:
FENCE PW,SR
or
FENCE PW,SRW

or the fully (over)specified:
FENCE *,*

Further, executing the FENCE with R/W options may have the side effect
of writing back & invalidating the entire data cache in a noncoherent
system.

I've long advocated to clean this up:
FENCE should be specified to not flush the cache or force writebacks
of a cache, only to ensure store buffers are flushed out to cache
CACHE.FLUSH and CACHE.WRITEBACK instructions should be added for
noncoherent systems; software on noncoherent systems would need to
execute these in addition to the FENCE if they are worried about data
stuck in the cache or ensuring external visibility and ordering.

Guy

On Tue, Jul 17, 2018 at 1:11 PM, Rishiyur Nikhil <nik...@bluespec.com> wrote:

> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPkONJLYYBjM7Zi0Y%2B%3DT9wvJ2zTBab9VenKrXK7ZkqOqzg%40mail.gmail.com.

[Rishiyur Nikhil]

Guy,

In this particular case, even a FENCE is not enough. It addresses only one half of the round trip communication from CPU to MTIMECMP and back.

A FENCE just says something about when the store may reach the MTIMECMP location. There is still the question of how long that location takes to communicate back to the CPU to reset its MTIME.MTIP bit.

I don't think there is any safe solution other than the busy-wait suggested by Andrew.

Nikhil

[Rishiyur Nikhil]

MIP.MTIP, I meant, not MTIME, apologies.

Nikhil

[Prof. Michael Taylor]

This is a very interesting discussion. I’m not overly familiar with where things are with RISC-V’s interrupt design, but I have struggled with similar issues in previous systems.

I think fundamentally the interrupt line level-sensitive model, as opposed to an event model is not well suited to modern day “where the heck is this address in my mazeway of a memory hierarchy” and where wire delay has been mitigated by OCN and pipelining, making atomicity more challenging.

A cleaner approach is to say that the timer widget is sitting way out in memory space, and will send a one time message the first time the event occurs to an interrupt block that is sitting close to the core. The timer widget will send no more messages until it receives a new command; i.e. MTIMECMP is written. The local interrupt block is responsible for asserting a level sensitive line locally to the processor. When the processor services the interrupt, it can atomically clear the local interrupt line from the local block, and then a message to the MTIMECMP with a new value and cause it to once again be capable of sending a single event message to the interrupt block.

M

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPny0A-s2YcTBdkcE7whdRSENy1p6ZwChyPxaFsBhYU%2BmA%40mail.gmail.com.

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

[Guy Lemieux]

Software can’t do this because it creates a race condition where a new interrupt arrives just as it is being cleared, and the software loop livelocks.

After a fence, all future reads of mtimecmp must reflect the updated write. To me, that implies all types of reads, not just load-data instructions on mtimecmp, but since it is a special mmio it must include all derived copies as well. This must occur because, in the event that an interrupt bit was being cleared, but a new one is generated in the same cycle, software must not have a race condition.

So it includes the full round trip, since that would be a future read.

As a system/chip designer, if there was any latency propagating the clear back to MTIP, then I would not release the FENCE.

If any other MMIOs must behave similarly, these should all be all documented together for clarity.

Guy

[Vikram Sethi]

+1 on Guy’s suggestion of a fence. Since the timer register is a MMIO register (device type) I don’t see why it would be mapped cacheable, so I’m not envisioning the caching problems for the store to timer registers.

However, the fence in this case would not be the typical load/store memory barrier but could be a flavor that guarantees context synchronization (such as Guy hints in his last email), such that the *effects* of the timer store also have to be visible when the fence completes.

 

As an aside, having implementation specific behavior creep into how the timer appears to operate to software would be problematic. The timer must architecturally function correctly the same way across all implementations and the specification must define the architecturally correct sequence to set, read, and clear the timer (sure different frequencies, latencies etc across implementations, but not programmatically different).

 

Thanks,

Vikram

 

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CALo5CZxqMJR6BaPpC-J0U1_vNo_NkiMo5U7pEtZOdOq8TZmFyA%40mail.gmail.com.

[Alex Solomatnikov]

I don't think this is actually necessary.

As with other level-triggered interrupts the general approach is to be able to deal with spurious interrupts - if timer interrupt happens again spuriously, timer interrupt handler can check MTIME and MTIMECMP and ignore/return immediately without incrementing MTIMECMP again.

>>> email to isa-dev+unsubscribe@groups.riscv.org.

>>> To post to this group, send email to isa...@groups.riscv.org.
>>> Visit this group at
>>> https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPmv9kznik8aCvi-SvORfeE%3DLNGBu_9R78HwiBMn8BNBbg%40mail.gmail.com.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "RISC-V ISA Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to isa-dev+unsubscribe@groups.riscv.org.

> To post to this group, send email to isa...@groups.riscv.org.
> Visit this group at
> https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
> To view this discussion on the web visit

> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPkONJLYYBjM7Zi0Y%2B%3DT9wvJ2zTBab9VenKrXK7ZkqOqzg%40mail.gmail.com.

[Andrew Waterman]

On Wed, Jul 18, 2018 at 11:23 AM, Alex Solomatnikov <so...@sifive.com> wrote:
> I don't think this is actually necessary.
>
> As with other level-triggered interrupts the general approach is to be able
> to deal with spurious interrupts - if timer interrupt happens again
> spuriously, timer interrupt handler can check MTIME and MTIMECMP and
> ignore/return immediately without incrementing MTIMECMP again.

Agreed - this is a non-problem. Following this logic, the Linux
kernel doesn't do anything like I suggested.

>> >>> email to isa-dev+u...@groups.riscv.org.

>> >>> To post to this group, send email to isa...@groups.riscv.org.
>> >>> Visit this group at
>> >>> https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
>> >>> To view this discussion on the web visit
>> >>>
>> >>> https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/CAAVo%2BPmv9kznik8aCvi-SvORfeE%3DLNGBu_9R78HwiBMn8BNBbg%40mail.gmail.com.
>> >>
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "RISC-V ISA Dev" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to isa-dev+u...@groups.riscv.org.