Tilelink - ProbeBlock toN while inflight AcquireBlock BtoT

[Charles Papon]

Hi,

One case which doesn't seems to be specified in the tilelink spec is : 

- What' happen when a master get a "ProbeBlock toT" while it had a "AcquireBlock BtoT" inflight for the same block.

I could see 2 potential "sane" outcomes : 

1) Either the master execute the probe, and will receive later  a "GrantData toT"

2) Either the master execute the probe, and ignore the following GrantData, then start all over with a new fresh  "AcquireBlock NtoT"

Not sure what is the Tilelink intends about it. I would guess it is option 1)

Regards
Charles

[Asutosh Mishra]

Hi Charles,

I am not authorized to answer your concern, but as we have also worked on similar implementation I can share my understanding.

I assume current permission status is at Branch. In that case it will be option 1,  Because-

    1. Probe will be executed prior to Acquire as "messages from higher priority channels must be able to bypass lower priority messages in the network." 

    2. "ProbeBlock toT" will result in Report category "ProbeAck/ProbeAckData BtoB." (reason: Probing Master Agents to cap their permissions at a more permissive level than they currently have is allowed, and does not result in a permissions change)

    3. When master sends Acquire BtoT, Slave can accept it.

Story would have changed if Current Permission status is at Branch, a master get a "ProbeBlock toN" while it had a "AcquireBlock BtoT" inflight for the same block. In that case

   1. Permission would have changed to BtoN.

   2. Then the Acquire request (BtoT) is not valid one, So Slave could have ignored with Deny.

Please let me know if you have further concern or different opinion.

Regards,

Asutosh

[Charles Papon]

Hi Asutosh,

Thanks for the answer !

Sorry, i didn't meant "ProbeBlock toT", but  "ProbeBlock toN" (I made the title right, but the message content wrong XD)

> Story would have changed if Current Permission status is at Branch, a master get a "ProbeBlock toN

It was exactly what i meant :D

> 1. Permission would have changed to BtoN.

Same opinon as you.

> 2. Then the Acquire request (BtoT) is not valid one, So Slave could have ignored with Deny.

Here i'm not sure. 

My understanding so far of the d_denied pin, is that it isn't a "please retry later", but more a "you can't do that" (for example : that memory access isn't legal,  software exception expected)

From the tilelink spec : 

"""

How errors that have been signaled via either corrupt or denied fields are reported to the broader
system is beyond the scope of this document. For example, interrupts, traps, or exceptions may be
used to notify software that an error has occurred.
"""

I looked into some opensource tilelink interconnect to see their usage of the denied pin.
There seems to be no "denied" pin usage on https://github.com/chipsalliance/rocket-chip/blob/master/src/main/scala/tilelink/Broadcast.scala.
Also on the sifive L2 cache, denied only seems propagated from downstream to upstream ? 
https://github.com/sifive/block-inclusivecache-sifive/blob/a260c905acc3645a49583f1c304f1a83025a860b/design/craft/inclusivecache/src/SourceD.scala#L226

https://github.com/sifive/block-inclusivecache-sifive/blob/a260c905acc3645a49583f1c304f1a83025a860b/design/craft/inclusivecache/src/MSHR.scala#L485

Not exactly sure.

I think a system would work using the denied pin, but i don't think it is the intend of the tilelink spec.
So far, my guess is that : 

1) The master execute the probe (goes toN), 

2) Then the interconnect receive the AcquireBlock BtoT

3) The interconnect figure out that the master losed the B permition, and so will handle it as if it was a AcquireBlock NtoT

4) The master will receive a GrantData toT

Can't find any spec / info to back it up.

Let's me know what you think :)

Regards,
Charles

[Asutosh Mishra]

As per my understanding if wrong permission access is requested from Master (e.g. BtoT when Current Permission is at N) then Slave can not process the Requested access, so it can deny it. Now it is Master's choice whether to retry it with NtoT permission or not.

=================================================================================================

Snapshot from Chapter 4.5 Errors:

"Access denied is a single bit control signal that indicates whether an attempted access or permissions transfer operation was processed by the recipient or not. When an operation is denied, it must have no effect on the permissions of the data block, nor change its contents, nor trigger any side effects related to accessing the data.

This control signal is only present on the D channel, which is the only channel for responses to A channel requests, which are the only type of requests which require increasing permissions. All other requests are guaranteed sufficient permissions and so may not be denied."

=================================================================================================

Regards,

Asutosh

[john.i...@sifive.com]

The TileLink spec says: "Slaves must include an up-to-date copy of the data in Grants responding to Acquires upgrading permissions, unless they are certain that that master has not been probed since the Acquire was issued."

Therefore, when a master gets a ProbeBlock.toN while it had a AcquireBlock.BtoT inflight for the same block, then the master must still process the probe, and the slave will reply GrantData.toT with an up-to-date copy of the data.

I can confirm that Charles is correct:

1) The master execute the probe (goes toN), 

2) Then the interconnect receive the AcquireBlock.BtoT

3) The interconnect figure out that the master lost the B permission, and so will handle it as if it was a AcquireBlock.NtoT

4) The master will receive a GrantData.toT

The `denied` signal is separate from this.

[Charles Papon]

Ahhh nice, thanks :D

[Asutosh Mishra]

Hi John,

As you have correctly explained  "Slaves must include an up-to-date copy of the data in Grants responding to Acquires upgrading permissions, unless they are certain that that master has not been probed since the Acquire was issued."

Here it is being probed, so should not be Aquire denied or atleast permission not changed toT?

Though looking at the statement it struck to my mind that there can 2 cases-

Case 1- Where Acquire is sent with BtoT permission request while current permission is at None.

            - In this case Slave can allow change in permission toT.

Case 2- As explained in this Mail thread, Acquire is sent to BtoT permission request while current permission is probed to None.

           - In this case Slave can not allow to changing permission to toT.

Please let me know your opinion on it and correct me if I am interpreting it wrong.

Regrads,

Asutosh

[John Ingalls]

In this case, the slave knows that the master was probed toN since the Acquire BtoT was issued, else the Acquire would have been NtoT (and the slave would return data then anyway).  So, the slave grants the Acquire with data, and grants the permission toT.

[Asutosh Mishra]

Yes, got the point now. I think the responses for cases I mentioned above will alter. i.e

Case 1- Where Acquire is sent with BtoT permission request while current permission is at None.

            - In this case Slave can not allow to changing permission to toT, as Slave is not aware of Probing of Master.

Case 2- As explained in this Mail thread, Acquire is sent to BtoT permission request while current permission is probed to None.

           - In this case Slave can allow change in permission toT, as it knows that Master is probed.

Is the response from Slave for case 1 correct?

Regards,

Asutosh

[Charles Papon]

I think case 1 (Acquire BtoT sent, despite the master know it has None permition, if it is what  you mean) is just a violation of the tilelink spec.

Still, from a interconnect perspective, i think such case 1 should be handled the same way than case 2, as the interconnect do not know exactly from how long ago that Acquire BtoT was asked (may have been in a legal time)

Or in case 1, you meant that the Acquire BtoT is sent (master think he is in B state), while from a interconnect perspective, that master permition is None ?
(i think it isn't an allowed state)

Regards
Charles

[Asutosh Mishra]

I meant this->  Acquire BtoT sent, despite the master know it has None permission.

Yes, we also believe it as spec violation.

That is why in our implementation slave denies these kind of request. Only concern is whether it is correct behavior from Slave? Or should Slave would allow permission update.

Regards,

Asutosh

[Charles Papon]

Ahhh ok ^^

> That is why in our implementation slave denies these kind of request. 

Hmm i think that the issue with that design is that the slave has no way to accuratly identify that case (Acquire BtoT sent, despite the master know it has None permission)

Here are two scenario :
scenario case 2 :

- Master is in B state(and the slave knows about it)

- Master send Acquire BtoT

- Slave send probe toT

- Master execute probe

- Slave now consider master is in N state

- ... Some time passes, that  Acquire BtoT got stuck a loooong time somewere in the interconnect

- Slave receive the master Acquire BtoT (and mutate it to a Acquire NtoT)

scenario case 1 :

- Master is in N state (and the slave knows about it)

- Master send Acquire BtoT (spec violation)

- Slave receive the master Acquire BtoT (what should it do now ?)

Question is, in scenario case 1, what should the slave do ? the only information it has, is that it receive a Acquire BtoT from a "N" master. That's "exactly" the same amount of information than in scenario case 2.

So for that reason, i think scenario case 1 and scenario case 2 can't be properly distinguished by the slave, and so, should always be managed the same (mutate  Acquire BtoT into a Acquire NtoT)

Unless I miss something  :)