UNIX-Class Platform Specification Working Group

[Palmer Dabbelt]

Atish and I have approval from the RISC-V technical committee and board to
start a group to manage the UNIX-class platform specification. This working
group will start by defining a subset of this platform specification that both
allows compatibility with existing implementations and extensibility for the
future needs. The contents of this first platform specification will include:

* A versioning scheme for future versions of the UNIX-class platform
specification.
* A specification for a platform-level interrupt controller that is compatible
with existing implementations of SiFive's PLIC.
* A specification for a standard SBI that is compatible with the current
implementation in Linux as well being extensible for future needs.
* An SBI extension for CPU power management.
* A specification of the interface between a bootloader and supervisor-mode
software.

Since this working group mostly consists of standardizing existing
implementations, I hope it can progress quickly. As such, I'd like to set a
6-month deadline between when the working group is started (ie, today) and when
a specification enters the 45-day review period.

[Luke Kenneth Casson Leighton]

this sounds great, palmer, it will help resolve some of the
uncertainty surrounding the Unix platform, allowing implementors to
commit to silicon with confidence.

as you are no doubt aware, the commercial and mass-volume libre risc-v
SoC is a hybrid CPU / VPU / GPU [1], where for example the TLB /
Virtual Memory handling and other aspects *may* need to deviate from
the standard UNIX-class platform specification, or suffer significant
(commercially unacceptable) performance degradation, jeapordising its
chances of commercial success if its hybrid requirements are not met
by the UNIX-class platform.

in addition, as it is not intended as a proprietary (secretive,
closed doors) design, those (potential) modifications *will* end up as
publicly-released common-place mass-volume and high profile
modifications to the associated software and toolchains that will
ultimately require (potentially forcibly - not by our team! - due to
user demand) upstream long-term distribution and support.

in addition to *that*, there are funding applications being
considered and under review that *require* full disclosure and full
transparency, as part of "Enhanced Privacy and Trust" for end-users
[2]. secretive closed-doors discussions of innovations and potential
enhancements to, and ratification of, the UNIX-class platform are *not
possible*, because engaging in secretive discussions would be viewed
by investors, customers and end-users alike as "something to hide",
and thus irrevocably cause harm to the reputation of the project by
undermining its business case as a "trusted independently auditable
product at every level of its development and manufacture".

question: is the discussion of the UNIX-class platform specification
to take place in a fashion that is inclusive of the *business* needs
of this commercial *and* libre project to remain entirely public and
transparent at all times?

difficult as this is to point out: we have asked about this in the
past, many times, and have not received a response. investigations
into Trademark Law show that the point at which financial and other
damages occur due to manufacturing and design delays caused by
persistently and systematically not receiving a response is the point
at which a Trademark may be irrevocably invalidated.

whilst we continue to develop this product in good faith (an
important factor to note under Trademark Law), and will make our best
efforts to ensure interoperability (another extremely important factor
to note under Trademark Law), our coninued exclusion from access to
ongoing innovations and the Standards Development Process *will* at
some point require us to make decisions or suffer substantial
financial loss as a result.

l.

[1] unlike a discrete GPU, which is a completely separate isolated
processor where IPC is deployed to shuttle data between the CPU and
GPU, and thus any augmentations or non-compliance with Standards *does
not matter*, a hybrid CPU / GPU / VPU has CPU workloads *and* GPU
workloads *and* VPU workloads operating on the exact same SMP cores.
the number of *general-purpose* registers has therefore been increased
to a whopping 256 (128 INT and 128 FP), the ISA extended to cope (as
hinted at as a possibility right at the top of the priv-spec), the TLB
will almost certainly require pre-load "hints" due to the
significantly different nature of GPU parallel workloads, and many
other aspects besides.

[2] https://nlnet.nl/PET/

[Bert]

I am confused, SiFive is commercial, right?

Why would one commercial company drive everything and use its own implementation for it?

It would be horrible if it is about "standardizing existing implementations", especially

(and it could be outdated), the list of cores and SoC's mentions none have been

evaluated as compliant (compliance in development), more importantly,

without knowing which embedded OS's will be used and for what application.

I doubt SiFive covers all applications from AI extreme performance to a million

IoT variants.

Best regards,

Bert

Veteran SoC designer and blogger (>1M views on Quora, topwriter 2018)

Linkedin showcase page (HW accelerators for AI)

Portugal Silver Coast Holiday Rentals

My 3D printed objects

Life is too short, act now and enjoy!

Op di 2 apr. 2019 om 17:50 schreef Palmer Dabbelt <pal...@sifive.com>:

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/mhng-99966160-5466-4cfc-b3d8-48ad48d3dc74%40palmer-si-x1c4.

[Gurjeet Singh]

Not that I'm interested in participating, but just curious as to how these officially approved groups work. Is there a mailing list for this group? Is it open to public? How does one request to participate in these lists? How does one follow along in these conversations, that is, where are the mailing list archives for this group?

The list [1] seems to be for this purpose, but it is not hosted @groups.riscv.org so I'm not so sure.

[1]: linux...@lists.infradead.org

Best regards,

--
You received this message because you are subscribed to the Google Groups "RISC-V ISA Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isa-dev+u...@groups.riscv.org.
To post to this group, send email to isa...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/isa-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/isa-dev/mhng-99966160-5466-4cfc-b3d8-48ad48d3dc74%40palmer-si-x1c4.

--

Gurjeet Singh http://gurjeet.singh.im/

[Palmer Dabbelt]

On Tue, 02 Apr 2019 10:57:38 PDT (-0700), deli...@gmail.com wrote:
> I am confused, SiFive is commercial, right?
>
> Why would one commercial company drive everything and use its own
> implementation for it?

The goal here is not to standardize exactly the existing SiFive implementation,
but we do want to maintain compatibility with pre-standard implementations
where the cost to do so is low. Note that this isn't really driven by the
SiFive hardware implementation, it's more driven by the existing software stack
-- for example, we're defining the SBI to be compatible with existing
implementations while still being extensible in the future.

The only SiFive hardware standard that will be proposed as a RISC-V standard
here is the SiFive PLIC, which as far as I can tell has been implemented by
both Andes and Kentryte (can't tell for sure, though as I don't see proper
documentation for either of these on a quick search) and should therefor really
be a RISC-V standard as opposed to one owned by SiFive. The PLIC as it
currently stands is sufficient for some but not all systems, so it will be an
optional part of the platform to allow for a different interrupt controller to
be added in the future.

> It would be horrible if it is about "standardizing existing
> implementations", especially
> (and it could be outdated), the list of cores and SoC's mentions none have
> been
> evaluated as compliant (compliance in development), more importantly,
> without knowing which embedded OS's will be used and for what application.
> I doubt SiFive covers all applications from AI extreme performance to a
> million
> IoT variants.

The platform spec will simply refer to the other RISC-V ISA specifications for
ISA compliance, so if it turns out that existing implementations are not
compliant then that's tough for them. There's a lot of work going on WRT the
compliance suite, but I'm confident that we will have one so I don't see any
reason to block defining a platform on finishing the compliance specification
-- if anything, having a platform specification will make the mechanics of
running the compliance tests easier because there will be more infrastructure
to share.

As far as OSes go: this is aimed at UNIX-class systems, where Linux is the main
contender. RISC-V has had a FreeBSD port for a long time, and one of the goals
of this effort is to make sure that the various UNIX-style OS implementations
can all run on hardware from various vendors without too much duplication of
effort.

>
> Best regards,
> Bert
> Veteran SoC designer and blogger <https://delisvhdl.com/blog> (>1M views on
> Quora, topwriter 2018)
> Linkedin showcase page <https://www.linkedin.com/showcase/5396093> (HW
> accelerators for AI)
> Portugal Silver Coast Holiday Rentals <https://www.vilaoasisverde.com>
> My 3D printed objects
> <https://nl.pinterest.com/bertdelis/cool-3d-printed-objects/>

[Palmer Dabbelt]

On Tue, 02 Apr 2019 11:59:04 PDT (-0700), lk...@lkcl.net wrote:

> On Wednesday, April 3, 2019, Bert <deli...@gmail.com> wrote:
>
> It would be horrible if it is about "standardizing existing
>> implementations",
>>
>>
>>

> The PLIC was one of the first key strategic parts of hardware that needed
> heavy interdependence on software, particularly the linux kernel.
>
> Without an interrupt controller, you don't have a modern processor. So the
> PLIC was designed out of necessity, and SiFive should be respected for
> taking the initiative.
>
> Since its creation the PLIC has wide de-facto adoption, with several
> independent implementations, many of them in actual silicon.
>
> For example, Richard Herveille, an independent Open Hardware designer, has
> made available a libre licenced PLIC.
>
> For example, the Shakti Design by IIT Madras University, which had working
> silicon last year, and they have several more designs in development which
> would be extremely costly to start changing.
>
> Whilst the PLIC (a global interrupt controller) is pretty much de-facto
> already a standard, the CLIC (a per-core-local interrupt controller) is
> nowhere near the same level of adoption, so its inclusion in a first
> iteration would not be wise.
>
> That having been said, you are right in that backwards retrofitting of
> standards to preexisting (incompatible) implementations is a rubber stamp
> recipe for interoperability disaster. Look at the quotes specialisation
> quotes in IEEE754 NaN arithmetic as a good example.
>
> Biting the bullet and saying "sorry guys, you jumped the gun, you should
> have discussed the required changes publicly, that you didn't do so, you're
> on your own" is better than having permanent incompatibility that requires
> indefinite support in upstream source code for multiple implementations.
>
> That having been said I do not believe that any such incompatibilities
> actually exist, so the issue is moot.

That's a big part of the reason we want the PLIC as a RISC-V standard and not a
SiFive standard -- if there's an incompatibility it's probably due to an
ambiguity in the spec, and having a RISC-V standard means we can pick the
answer that makes the most sense rather than just picking whatever SiFive has
done.

[Palmer Dabbelt]

On Tue, 02 Apr 2019 16:42:30 PDT (-0700), gur...@singh.im wrote:
> Not that I'm interested in participating, but just curious as to how these
> officially approved groups work. Is there a mailing list for this group? Is
> it open to public? How does one request to participate in these lists? How
> does one follow along in these conversations, that is, where are the
> mailing list archives for this group?

There is (or will soon be) an internal RISC-V foundation mailing list. You
sign up for it via your RISC-V foundation account -- I've never quite figured
out how to use that, mostly because Arun handles it for me.

The tool is called workspace, which lives here:
http://workspace.riscv.org/higherlogic/ws/public . If you're a RISC-V member
then you should be able to log in. I can't find the platform spec group,
though, so I may be screwing something up...

> The list [1] seems to be for this purpose, but it is not hosted @
> groups.riscv.org so I'm not so sure.
>
> [1]: linux...@lists.infradead.org

That's the Linux kernel mailing list for RISC-V. It's CC'd here because users
on it may be interested in the platform specification, but it's not the
platform specification mailing list.

[Palmer Dabbelt]

On Tue, 02 Apr 2019 17:31:23 PDT (-0700), lk...@lkcl.net wrote:

> On Wed, Apr 3, 2019 at 12:42 AM Gurjeet Singh <gur...@singh.im> wrote:
>>
>> Not that I'm interested in participating, but just curious
>> as to how these officially approved groups work.
>> Is there a mailing list for this group?
>

> that's the question that i asked. i had to ask it in the way that i
> did, using the words that i did, due to issues related to Trademark
> Law.

>
>> Is it open to public?
>

> it has not been made clear, hence my question.
>
> (as an aside: none of the "official" RISC-V Working Group mmailing
> lists are open access.)

It's a RISC-V working group and therefor follows the same policies as the rest
of them. IIRC we're in a bit of a transition from workspace to groups.io, but
either way the list (and associated meetings) will only be available to RISC-V
members. There's a free membership for open source projects, as well as a
cheap membership for individuals.

>
>> How does one request to participate in these lists?
>

> again: it wasn't made clear. if however it has been expected that
> the lists be "RISC-V Working Group Mailing lists", then it is
> mandatory to sign the "RISC-V Membership Agreement", which prevents
> and prohibits all and any public discussion or release of information
> outside of the group without the expressed and explicit consent and
> approval of the RISC-V Foundation (Section 5).
>
> this being what our team absolutely cannot do [enter into secret
> agreements that interfere with the BUSINESS objectives], hence why i
> requested clarification.

OK, well, I'm sorry you'll be unable to contribute. We considered different
contribution models as part of starting the group, but decided the standard
RISC-V arrangement was the best fit for this group as well.

>> How does one follow along in these conversations, that is, where are the mailing list archives for this group?
>

> again it was not stated (apologies, to palmer), so we do not know.
>
>> The list [1] seems to be for this purpose, but it is not hosted @groups.riscv.org so I'm not so sure.
>> [1]: linux...@lists.infradead.org
>
> exactly: it's not clear. so, it needs to be made clear exactly what
> the Standards Development Process is to be, where discussion is to
> take place, and whether there are any terms or conditions required for
> participation.
>
> at that point, once it has been made clear, people will know what
> they are getting into.

That's the RISC-V Linux kernel mailing list, where is where RISC-V kernel
development happens. That is an open list, but does not directly discuss the
development of RISC-V standards like the platform specification. Platform spec
discussions will happen the same place as all the other RISC-V discussions --
my other groups are on workspace.riscv.org, but I don't see a platform spec
group there and with the LF transition IIRC we're deprecating workspace in
favor of something else. Maybe we'll be the first group to figure out how it
works :)

I pinged some of the LF people to try and figure out how to get at least myself
added to the list.

[Luke Kenneth Casson Leighton]

On Wed, Apr 3, 2019 at 3:43 AM Palmer Dabbelt <pal...@sifive.com> wrote:
>
> On Tue, 02 Apr 2019 17:31:23 PDT (-0700), lk...@lkcl.net wrote:
> > On Wed, Apr 3, 2019 at 12:42 AM Gurjeet Singh <gur...@singh.im> wrote:
> >>
> >> Not that I'm interested in participating, but just curious
> >> as to how these officially approved groups work.
> >> Is there a mailing list for this group?
> >
> > that's the question that i asked. i had to ask it in the way that i
> > did, using the words that i did, due to issues related to Trademark
> > Law.
> >
> >> Is it open to public?
> >
> > it has not been made clear, hence my question.
> >
> > (as an aside: none of the "official" RISC-V Working Group mmailing
> > lists are open access.)
>
> It's a RISC-V working group and therefor follows the same policies as the rest
> of them. IIRC we're in a bit of a transition from workspace to groups.io, but
> either way the list (and associated meetings) will only be available to RISC-V
> members. There's a free membership for open source projects, as well as a
> cheap membership for individuals.

monetarily zero cost... as long as they have no conflict of interest
that prevents and prohibits joining. and we have a clear business
case (full transparency and independent third party audit
requirements).

to be clear: it would actually be considered to be *fraud* to have
put in a Grant Application under false pretenses (namely: to state
that we intended to develop a commercial chip in a fully transparent
fashion, then immediately on doing so join a *CLOSED* secretive
group).

> >> How does one request to participate in these lists?
> >
> > again: it wasn't made clear. if however it has been expected that
> > the lists be "RISC-V Working Group Mailing lists", then it is
> > mandatory to sign the "RISC-V Membership Agreement", which prevents
> > and prohibits all and any public discussion or release of information
> > outside of the group without the expressed and explicit consent and
> > approval of the RISC-V Foundation (Section 5).
> >
> > this being what our team absolutely cannot do [enter into secret
> > agreements that interfere with the BUSINESS objectives], hence why i
> > requested clarification.
>
> OK, well, I'm sorry you'll be unable to contribute.

you may be misunderstanding: we have a clear business case for
involvement in the standards development process, and have been
consistently requesting access to resources such that we may
participate in innovation.

we have yet to receive a response.

failure of the RISC-V Foundation to accept this and take it into
account is not an option for the RISC-V Foundation if the RISC-V
Foundation wishes to continue to enjoy the benefits afforded by
Trademark Law.

so we are being excluded from the standards development process,
despite having a clear-cut business case for being included. given
the failure of the RISC-V Foundation to take our business case into
consideration, we are not going to just "give up", and will proceed as
best we can under the circumstances with our business objectives.

acting in good faith as we are, we will endeavour to ensure that our
business objectives are met with the minimum disruption to the wider
RISC-V community, however given that we are clearly being
discriminated against, we cannot and will not be held responsible for
any incompatibilities or non-interoperability: that is clearly down to
the failure of the RISC-V Foundation to take our business objectives
into consideration.

if anyone has a problem with that, particularly that the libre/open
community depends critically on good-will, PLEASE SPEAK UP. it is
very important that you do so, as it will strengthen the case that
substantial damages have arisen to our business objectives as direct
consequence of continued and persistent silence and abdication of
responsibility regarding Trademark Law by the RISC-V Foundation.
(Executive Director once again cc'd so that there is no possibility
that the RISC-V Foundation may claim "It Did Not Know")

> e considered different
> contribution models as part of starting the group, but decided the standard
> RISC-V arrangement was the best fit for this group as well.

this constitutes discrimination. Trademark Law is unfortunately quite
clear: discrimination in this fashion is not permitted. any Tradmark
owner that engages in discriminatory practices irrevocably forfeits
the Trademark.

it is also very clear that persistent failure by a Trademark holder
to respond to communications to resolve such matters also results in
permanent forfeiture, and a consequent loss of right of enforcement.

we're *going* to go ahead, palmer. we *will* meet the business
objective of creating an entirely and wholly libre-licensed RISC-V
hybrid GPU/CPU/VPU that is fully transparent in its development and
manufacture, right to the bedrock, with or without the RISC-V
Foundation's cooperation.

l.

[Luke Kenneth Casson Leighton]

On Wed, Apr 3, 2019 at 3:43 AM Palmer Dabbelt <pal...@sifive.com> wrote:

> We considered different
> contribution models as part of starting the group, but decided the standard
> RISC-V arrangement was the best fit for this group as well.

to make it clear as to why this process constitutes "discrimination":
why was there no invitattion made public for actively interested
parties aka "stakeholders" to participate in the decision-making
process? why was there no wider consultation? why was the decision
made in secret? why are we being told *after* the decision has been
made, and why do you believe that our business objectives do not
matter?

executive director of the RISC-V Foundation once again cc'd so that
there is no way that the RISC-V Foundation may claim that they are
unaware.

l.

[Luke Kenneth Casson Leighton]

https://entertainment.slashdot.org/story/19/04/02/218226/justice-department-warns-academy-about-changing-oscar-rules-to-exclude-streaming#comments

that's interesting. i was not previously aware of "Section 1 of the
Sherman Act".

Delrahim cited Section 1 of the Sherman Act that "prohibits
anticompetitive agreements among competitors." "Accordingly,
agreements among competitors to exclude new competitors can violate
the antitrust laws when their purpose or effect is to impede
competition by goods or services that consumers purchase and enjoy but
which threaten the profits of incumbent firms," Delrahim wrote.

i wonder therefore if "deciding the standard RISC-V arrangement was
the best fit for this group as well" - excluding us from that
decision-making process constitutes a violation of the Sherman Act,
given that "profits of incumbent firms are threatened", and
"competition by goods that consumers purchase is impeded"?

l.

[Dr Jonathan Kimmitt]

Dear Palmer,

I appreciate your pragmatic decision to adopt the specification of the
SiFive PLIC as a working model of how things should be done. However at
Cambridge and ETHZ we recently observed a problem with the PLIC claim
mechanism if the idempotency PMA is not implemented. Since I assume the
latter is an optional and perhaps rather complicated feature, we were
wondering whether it would be possible to support an optional
specification to the PLIC to use a write operation to claim an interrupt
instead of a read. The bug arises because, to maximise performance,
speculative load instructions are initiated at the beginning of the
pipeline, this will have the effect of claiming the interrupt whether or
not that instruction reaches the commit stage. As you know in Linux this
claim routine happens with interrupts off but there is always the
possibility of a machine mode interrupt or other event causing a
pipeline flush. Another option would be to force the PLIC to support
atomic reads. I appreciate backward compatibility is an issue and there
could be sound architectural reasons why your group chose to do it this way.

Regards,

Jonathan Kimmitt

[Palmer Dabbelt]

Sorry for the delay, but it appears we now have a mailing list set up!  I managed to announce the group right in the middle of the workspace -> groups.io transition, which means we're the first group to try out the new infrastructure.  As a result there may be some wrinkles, but at least a handful of us can get into the group so with any luck it'll work for anyone else.

You should be able to join here: https://lists.risc-v.org/g/tech-unixplatformspec/topics .  Just like the other RISC-V specs, you'll need to be a member of the RISC-V foundation to contribute to the platform specification -- sorry if there was some confusion in my original email.

[Christopher Celio]

However at 
Cambridge and ETHZ we recently observed a problem with the PLIC claim 
mechanism if the idempotency PMA is not implemented. Since I assume the 
latter is an optional and perhaps rather complicated feature,   

The idempotency check can be very simple; and holding loads till commit (or till the relaxed cache is ordered) is only slightly annoying.  Unless I'm misunderstanding the issue, I would think a Unix-platform processor would be required to understand the difference between device IO and cacheable regions, no? If not, the PLIC would not be the only problematic device in the system. 

-Chris

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/e93dee44-11ac-860f-47bd-204b2b4b7479%40cam.ac.uk.

[Andrew Waterman]

On Fri, Apr 5, 2019 at 6:45 PM Christopher Celio <ce...@berkeley.edu> wrote:

However at 
Cambridge and ETHZ we recently observed a problem with the PLIC claim 
mechanism if the idempotency PMA is not implemented. Since I assume the 
latter is an optional and perhaps rather complicated feature,   

The idempotency check can be very simple; and holding loads till commit (or till the relaxed cache is ordered) is only slightly annoying.  Unless I'm misunderstanding the issue, I would think a Unix-platform processor would be required to understand the difference between device IO and cacheable regions, no? If not, the PLIC would not be the only problematic device in the system. 

This matches my thinking.  For better or worse, the PLIC is not unique in having memory-mapped registers with non-idempotent read behavior.  And it's well understood how to handle this microarchitecturally.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiFiZCO6Jjg5qW2WRUd4EUjz9kJm2XxEyXrRFdOO2O9C1w%40mail.gmail.com.

[Prof. Michael Taylor]

Seems like we have established that microarchitecture can handle this. But I did not understand the argument for architecting a system deliberately where reads have side effects?

M

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CA%2B%2B6G0DY0Rim9MYA5Cja3N_ptU8NXMbu3%2B6_fuka7Kmb1cuM0w%40mail.gmail.com.

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

[Paul Walmsley]

In a multi-hart system, using a single read ensures that the reading hart atomically claims the next pending interrupt ID without risking a race with other harts.

This is a common pattern for memory-mapped I/O devices - for example, consider FIFO registers.

- Paul

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx20eE%2BDf0HzbuUcWa%3DU0v8uCL-Tpx8FRGSY9E4vQdVbGA%40mail.gmail.com.

[Andrew Waterman]

On Fri, Apr 5, 2019 at 8:34 PM Prof. Michael Taylor <prof....@gmail.com> wrote:

Seems like we have established that microarchitecture can handle this. But I did not understand the argument for architecting a system deliberately where reads have side effects?

The alternatives are using a store and a load, or using one AMO, to accomplish the same thing as the one non-idempotent load. Those are defensible options, to be sure, but they, too, have obvious downsides. We opted for the solution that only requires one transaction but does not mandate the A extension. (The PLIC design needs to work efficiently for less sophisticated systems.)

[Prof. Michael Taylor]

Thanks Paul and Andrew. That’s a neat trick. Violating the standard invariants always has a price, but beyond making sure you do not speculatively do loads to I/O regions, I can’t say I have figured out what it is in this case. :-)

M

[Dr Jonathan Kimmitt]

So the conclusion is: nuclear non-proliferation. To avoid everybody needing to have atomics in our arsenal, we need to mandate physical memory attributes for UNIX capable platforms.

Sent from my iPhone

[Florian Zaruba]

Definitely, but it missing a character from a UART and causing a hang-up on a System are two different things in user experience (although both are wrong) ;-) 

The differentiation between NC and memory Region can be efficiently handled in the cache, while waiting for a load to be (and remain non-speculative) is, imho, another story. So I need to drain the remaining pipeline and mask-off all interrupt sources before I can commit the load.. 

Maybe I am missing something but can you please share your experience for your Boom implementation? 

Florian

On Sat, 6 Apr 2019 at 03:45, Christopher Celio <ce...@berkeley.edu> wrote:

However at 
Cambridge and ETHZ we recently observed a problem with the PLIC claim 
mechanism if the idempotency PMA is not implemented. Since I assume the 
latter is an optional and perhaps rather complicated feature,   

The idempotency check can be very simple; and holding loads till commit (or till the relaxed cache is ordered) is only slightly annoying.  Unless I'm misunderstanding the issue, I would think a Unix-platform processor would be required to understand the difference between device IO and cacheable regions, no? If not, the PLIC would not be the only problematic device in the system. 

-Chris

On Fri, Apr 5, 2019 at 8:16 AM Dr Jonathan Kimmitt <jr...@cam.ac.uk> wrote:

Dear Palmer,

I appreciate your pragmatic decision to adopt the specification of the
SiFive PLIC as a working model of how things should be done. However at
Cambridge and ETHZ we recently observed a problem with the PLIC claim
mechanism if the idempotency PMA is not implemented. Since I assume the
latter is an optional and perhaps rather complicated feature, we were
wondering whether it would be possible to support an optional
specification to the PLIC to use a write operation to claim an interrupt
instead of a read. The bug arises because, to maximise performance,
speculative load instructions are initiated at the beginning of the
pipeline, this will have the effect of claiming the interrupt whether or
not that instruction reaches the commit stage. As you know in Linux this
claim routine happens with interrupts off but there is always the
possibility of a machine mode interrupt or other event causing a
pipeline flush. Another option would be to force the PLIC to support
atomic reads. I appreciate backward compatibility is an issue and there

could be sound architectural reasons why your group chose to do it this way..

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiFiZCO6Jjg5qW2WRUd4EUjz9kJm2XxEyXrRFdOO2O9C1w%40mail.gmail.com..

--

Florian Zaruba

PhD Student

Integrated Systems Laboratory, ETH Zurich

Mail:    zar...@iis.ee.ethz.ch

Skype: florianzaruba

[lk...@lkcl.net]

On Saturday, April 6, 2019 at 7:51:09 AM UTC+1, Florian Zaruba wrote:

Definitely, but it missing a character from a UART and causing a hang-up on a System are two different things in user experience (although both are wrong) ;-) 

The differentiation between NC and memory Region can be efficiently handled in the cache, while waiting for a load to be (and remain non-speculative) is, imho, another story. So I need to drain the remaining pipeline and mask-off all interrupt sources before I can commit the load.. 

out-of-order systems have a slightly easier time of it, as there's a pre-existing pattern for preventing commits until certain conditions (other dependencies) are met and cleared, first.

the "opportunity for throwing exceptions" in particular hold a write hazard (dependency) preventing and prohibiting all "sequentially identified as future" instructions from committing (doing any kind of irreversible write-damaging operation), until such time as the condition where the exception *could* occur is known (absolutely, guaranteed) *not* to occur... the exception-write-hazard is then dropped, and two things happen:

(1) the current instruction no longer has any "damage on future instructions" conditions, so can (assuming all other write hazards are clear) commit.

(2) future instructions, now one write-hazard less, may also potentially commit.

and it turns out that interrrupts are... just a type of exception.

everything in a scoreboard / OoO system holds these really really simple single-wire "write hazard" lines, they're ORed together (from all the different sources), when all clear, that means "ok you can commit now".

unfortunately, florian... deep breath... without that infrastructure in place, it's a pig.  as in: the effect of introducing the kind of infrastructure needed basically turns an in-order system *into* an out-of-order one... just a degenerate one that can only issue a single instruction at a time, down a single pipeline.

end result: precisely the stalling / pipeline-flushing you describe.  that will occur in an OoO system *as well*... it's just that it doesn't matter as much.

the reason: the advantage of going "full OoO" is that it's effectively a parallel processor (multiple pipelines) with an instruction-order-preservation dependency system [and yes, even the instruction-order-preservation system is... a write-hazard chain!  Tomasulo uses a Reorder Buffer for that (just a rotating buffer to create a FIFO), and the 6600-style augmented algorithm is an NxN bit-matrix to create a kind of linked-list].

so in an OoO design, because of the multiple (parallel) pipelines, the one totally-stalled-and-flushed pipeline you describe doesn't matter: the other pipelines will still be executing.

you can probably tell, i'm not a huge fan of in-order systems any more :)

l.

[lk...@lkcl.net]

On Saturday, April 6, 2019 at 10:42:17 AM UTC+1, lk...@lkcl.net wrote:

 

the "opportunity for throwing exceptions" in particular hold a write hazard (dependency) preventing and prohibiting all "sequentially identified as future" instructions from committing (doing any kind of irreversible write-damaging operation), until such time as the condition where the exception *could* occur is known (absolutely, guaranteed) *not* to occur... the exception-write-hazard is then dropped, and two things happen:

p.s. many people believe that 6600-style scoreboards are incapable of precise exceptions, due to a mis-reading of the original patent.  this is horseshit. the above algorithm, which is an adaptation of existing write-hazard logic, shows why.

[Samuel Falvo II]

It's not clear to me why would you need to do this?

If you're speculating loads, then you already have the effective address computed, and thus, can use PMAs to determine if it's to an I/O region or not.  If it is to an I/O region, your speculation fails out of hand -- the bus transaction never appears outside the core, and is therefore required to wait until the memory stage of the pipeline.  This doesn't flush pipeline (unless a branch happens), but it will stall it until the read can be properly executed.

This guarantees that a buffer-constrained UART will not lose characters unless your software is physically incapable of servicing the UART before another character comes in.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CADZArczNm8bc6Bftuntd%2BE19O1MQ8aZ%3DztYTDS9S_BTHM1PSHw%40mail.gmail.com.

--

Samuel A. Falvo II

[Florian Zaruba]

in your response, you are implying that I have a way to simply not do loads speculatively, which I don't (otherwise it would, of course, be easy). Even in the simpler pipelines, a load can potentially be in the shadow of an interrupt which effectively makes the load speculative. So either you "take" all the interrupts after the speculative load (in a decode stage) or you mask them until you've committed the load. In any case, it adds another semantic (and path) to loads. 

I was simply interested in how Chris solved this for his OoO machine.  

  

Funny that we are discussing this in the UNIX-class specification which makes atomics mandatory and that apparently the CLINT requires atomic support (https://github.com/riscv/riscv-pk/blob/16476bd8219f58417a401ea0a720d9588d1d8ebc/machine/mtrap.c#L61). I am not advocating using atomics on peripherals though.

   

could be sound architectural reasons why your group chose to do it this way...

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiFiZCO6Jjg5qW2WRUd4EUjz9kJm2XxEyXrRFdOO2O9C1w%40mail.gmail.com...

--

Florian Zaruba

PhD Student

Integrated Systems Laboratory, ETH Zurich

Mail:    zar...@iis.ee.ethz.ch

Skype: florianzaruba

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CADZArczNm8bc6Bftuntd%2BE19O1MQ8aZ%3DztYTDS9S_BTHM1PSHw%40mail.gmail..com.

--

Samuel A. Falvo II

[Prof. Michael Taylor]

Hi Florian,

Don’t you have a store buffer? I wonder if this might map onto the case of an uncached store in terms of commit logic. 

M

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CADZArcxtjzXdHz88_pOBMR21E4iP8Eh2qW7qAiXQJW_naaHp3g%40mail.gmail.com.

[Florian Zaruba]

Hi Michael,

the idea is good! Unfortunately, the storebuffer has no direct way of altering the core's architectural state (e.g. returning something to a register).  I think the closest to which a non-idempotent load would map (in terms of my pipeline at least) is an atomic memory operation.

Florian

could be sound architectural reasons why your group chose to do it this way....

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiFiZCO6Jjg5qW2WRUd4EUjz9kJm2XxEyXrRFdOO2O9C1w%40mail.gmail.com....

--

Florian Zaruba

PhD Student

Integrated Systems Laboratory, ETH Zurich

Mail:    zar...@iis.ee.ethz.ch

Skype: florianzaruba

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CADZArczNm8bc6Bftuntd%2BE19O1MQ8aZ%3DztYTDS9S_BTHM1PSHw%40mail.gmail...com.

--

Samuel A. Falvo II

--

Florian Zaruba

PhD Student

Integrated Systems Laboratory, ETH Zurich

Mail:    zar...@iis.ee.ethz.ch

Skype: florianzaruba

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CADZArcxtjzXdHz88_pOBMR21E4iP8Eh2qW7qAiXQJW_naaHp3g%40mail.gmail.com..

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

[Christopher Celio]

Hi Florian!

As Samuel mentioned, once the load's address is computed, you know if the address is non-idempotent (at the end of EXE). All loads are sent to the D$ on the next cycle (MEM), but I have the full cycle to kill my memory request for things like page faults, access violations, store/load order violations, etc. So a load can be recognized for many reasons that it is not allowed to execute at this time and is put to sleep in the Load Address Queue. For loads marked as "non-speculative", they sit in the load-address queue until the ROB notifies the LAQ that the oldest instruction in the machine is a load, and then our non-speculative load is re-sent to the D$ to execute for real this time. 

The non-blocking D$ likewise has a separate MSHR for IO; I don't recall the details, but I believe it waits for the rest of the cache's inflight ops to finish, then it handles the non-idempotent load, and then it blocks further cache ops until the load data returns.

For simplicity, BOOM (and I think rocket?) treats all uncacheable memory as non-idempotent, but naturally more performance can be gained by separating out those two orthogonal attributes. 

To comment on your specific experiences regarding interrupts and speculative loads, I would need to have a better picture of your specific pipeline. For a standard 5-stage pipeline, the commit point is at the end of MEM, so the load is "committed" well before data has ever returned. Then I would handle any state changes due to interrupts in MEM or WB, which is fine, as the core will eventually block if it tries to read the register waiting for its load-data to return.

-Chris

[Samuel Falvo II]

On Sat, Apr 6, 2019 at 2:28 PM Luke Kenneth Casson Leighton <lk...@lkcl.net> wrote:

On Sat, Apr 6, 2019 at 4:59 PM Florian Zaruba <zar...@iis.ee.ethz.ch> wrote:
>
> in your response, you are implying that I have a way to simply not do loads speculatively,
> which I don't (otherwise it would, of course, be easy).

 i'm not: i'm saying that the method that is needed is identical to

I think he's replying to my question.

[Prof. Michael Taylor]

Getting back to Florian’s question, I have wondered a little bit about the general question of the PLIC. Pardon my ignorance, but to what extent does the Linux RISC-V implementation rely upon a particular Interrupt controller design? Is the dependence deeply embedded across many lines of Linux code, or something that is fairly well localized, to the point that it might make sense for Florian to provide a driver for an interrupt controller that meets his own constraints?

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiGgCDd6XWU2Lm_MkRcoPfRiOSz6btZ0vChg7DvDLfmmyQ%40mail.gmail.com.

[Dr Jonathan Kimmitt]

Dear Michael,

  The interrupt controller specification was provided by SiFive and as such is already enshrined in Silicon. This thread started as a discussion as to what extent the SiFive PLIC specification could be enshrined by the RISCV foundation as an exemplar of how to achieve safe SMP modern Unix compliant semantics. If successfully adopted it would facilitate one Linux kernel for everybody encompassing existing silicon and new designs.

I have no doubt there is a valid solution to the technical problem for many kinds of micro-architecture, the primary question is one of requiring or desiring software compatibility. Once two rival designs are produced, there is nothing to stop a dozen.

Regards,

Jonathan 

Sent from my iPhone

[kr...@berkeley.edu]

While one of the goals of the RISC-V project has always been to
encourage experimentation, there is also a great desire for
standardization to allow reuse of software effort. Oftentimes, a
software-friendly portable standard will require hardware implementers
to do more than the bare minimum needed for certain core styles in
restricted use cases.

Originally, we left the interrupt controller unspecified in the
privileged spec as different systems have different needs, but there
was considerable demand for a standard interrupt controller for Linux
to save software effort. The Linux source code is modular enough to
support many different styles of interrupt controller, but supporting
multiple different Linux binary distributions is a non-trivial effort,
and we did not want to repeat the mistakes of earlier ISAs in this
regard. So, we developed the PLIC spec as a simple standard
centralized interrupt controller for SMP Unix-like systems. Our
constraints on initial PLIC design were that it be purely memory-mapped,
instead of having some special connection to the core pipeline logic
beyond the external asynchronous interrupt input, and that it not
require the A extension (although Linux now does require A exists).

Of course there are many possible improvements to make to the PLIC
spec, but first we want to standardize the existing PLIC v1 before
working on a PLIC v2. Small improvements to the PLIC are not worth
any resulting incompatibilities. Goals of a PLIC v2 are a large jump
in functionality including, e.g., handling virtualization well, but
that will require experience with implementations of hypervisors.

Krste

| https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiFiZCO6Jjg5qW2WRUd4EUjz9kJm2XxEyXrRFdOO2O9C1w%40mail.gmail.com
| ...

| --
| Florian Zaruba
| PhD Student
| Integrated Systems Laboratory, ETH Zurich
| Mail: zar...@iis.ee.ethz.ch
| Skype: florianzaruba

| --
| You received this message because you are subscribed to
| the Google Groups "RISC-V HW Dev" group.
| To unsubscribe from this group and stop receiving emails
| from it, send an email to
| hw-dev+un...@groups.riscv.org.
| To post to this group, send email to
| hw-...@groups.riscv.org.
| Visit this group at
| https://groups.google.com/a/groups.riscv.org/group/hw-dev/
| .

| To view this discussion on the web visit

| https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CADZArczNm8bc6Bftuntd%2BE19O1MQ8aZ%3DztYTDS9S_BTHM1PSHw%40mail.gmail..com
| .

| --
| Samuel A. Falvo II

| --
| Florian Zaruba
| PhD Student
| Integrated Systems Laboratory, ETH Zurich
| Mail: zar...@iis.ee.ethz.ch
| Skype: florianzaruba

| --
| You received this message because you are subscribed to the Google
| Groups "RISC-V HW Dev" group.
| To unsubscribe from this group and stop receiving emails from it, send
| an email to hw-dev+un...@groups.riscv.org.
| To post to this group, send email to hw-...@groups.riscv.org.
| Visit this group at
| https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
| To view this discussion on the web visit
| https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAJ9VFiGgCDd6XWU2Lm_MkRcoPfRiOSz6btZ0vChg7DvDLfmmyQ%40mail.gmail.com
| .

| --
| Apologies for all typos. Message sent by combination of an approximate
| neural network and a smartphone.

| --
| You received this message because you are subscribed to the Google Groups
| "RISC-V HW Dev" group.
| To unsubscribe from this group and stop receiving emails from it, send an
| email to hw-dev+un...@groups.riscv.org.
| To post to this group, send email to hw-...@groups.riscv.org.
| Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/
| .
| To view this discussion on the web visit

| https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/028B372A-331A-490A-930B-74CB94CADE2E%40cam.ac.uk
| .

[高野茂幸]

Is it difficult to decouple interrupt from core definition, and defines an interrupt interface?

Then his her own interrupt functionality can wrap or connect to the interface although interrupt should have shortest overhead.

Best,

S.Takano

2019年4月8日(月) 8:15 <kr...@berkeley.edu>:

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/23722.33906.576602.874520%40KAMacBookPro2016.hsd1.ca.comcast.net.

[Krste Asanovic]

Some core-interrupt interfaces can be quite intrusive, if seeking the lowest interrupt latency and occupancy.

For example:

- an interrupt controller might be aware of which cores are waiting to service an interrupt versus busy doing other work.

- a core might tell the interrupt controller directly that it wants to claim and cleardown an edge-triggered interrupt, and there needs to be an atomic handshake that the core has committed to take the interrupt and the IC has committed to giving this core that interrupt

The simple PLIC interface only requires a memory-mapped I/O interface and a single external asynchronous interrupt input to the core (really, to a hart on a core).

The hart responds to the interrupt notification by eventually explicitly reading the PLIC state.  Because the PLIC state read has to be from a committed instruction, there is no danger of the hart not committing to taking the interrupt handler path.  Because the PLIC handles races centrally, it is straightfoward to ensure only one hart will get each interrupt.  In the worst case, there can be a spurious interrupt, in which case handler simply returns.  The system software can arrange to what degree spurious interrupts are possible by routing interrupts to target harts appropriately.  This model copes with any degree of communication latency between endpoints and does not require any new interfaces to the core hardware, beyond what is already generally required (memory mapped I/O and basic interrupt inputs).

A per-core interrupt controller, like the CLIC, can have lower overhead, both because it can be physically close to the core and does not have to worry about multiple harts contending for an interrupt, but requires a wider interface between the IC and the core.

Krste

[高野茂幸]

Krste-san,

Thank you very much for your explanation.

I think minimum definition is ideal similar to RISC policy.

Best,

S.Takano

2019年4月8日(月) 9:29 Krste Asanovic <kr...@berkeley.edu>:

[Dr Jonathan Kimmitt]

If it’s not possible to implement any small improvements due to incompatibility with existing silicon we should at least document potential race hazards and recommend that non-idempotent interfaces use atomic transactions in future releases. This will help to get the message across to potential implementors.

Sent from my iPhone

[Paul Walmsley]

The problem with such a recommendation is that it would imply that there is something race-prone or incorrect about reads from MMIO device registers with read side-effects.

- Paul

On 4/8/19 12:09 AM, Dr Jonathan Kimmitt wrote:

If it’s not possible to implement any small improvements due to incompatibility with existing silicon we should at least document potential race hazards and recommend that non-idempotent interfaces use atomic transactions in future releases. This will help to get the message across to potential implementors.

Sent from my iPhone

On 8 Apr 2019, at 00:14, kr...@berkeley.edu

 wrote:


While one of the goals of the RISC-V project has always been to
encourage experimentation, there is also a great desire for
standardization to allow reuse of software effort.  Oftentimes, a
software-friendly portable standard will require hardware implementers
to do more than the bare minimum needed for certain core styles in
restricted use cases.

Originally, we left the interrupt controller unspecified in theg
privileged spec as different systems have different needs, but there
was considerable demand for a standard interrupt controller for Linux
to save software effort.  The Linux source code is modular enough to
support many different styles of interrupt controller, but supporting
multiple different Linux binary distributions is a non-trivial effort,
and we did not want to repeat the mistakes of earlier ISAs in this
regard.  So, we developed the PLIC spec as a simple standard
centralized interrupt controller for SMP Unix-like systems.  Our
constraints on initial PLIC design were that it be purely memory-mapped,
instead of having some special connection to the core pipeline logic
beyond the external asynchronous interrupt input, and that it not
require the A extension (although Linux now does require A exists).

Of course there are many possible improvements to make to the PLIC
spec, but first we want to standardize the existing PLIC v1 before
working on a PLIC v2.  Small improvements to the PLIC are not worth
any resulting incompatibilities.  Goals of a PLIC v2 are a large jump
in functionality including, e.g., handling virtualization well, but
that will require experience with implementations of hypervisors.

Krste

[Prof. Michael Taylor]

I see Krste’s arguments for standardization but also feel Florian’s pain, and wonder how we have constrained the space of valid OOO implementations.

It seems to be that idempotency and atomicity can be attained relatively easily by what I will call “gumball machine” acquisition:

- each core has two unique addresses for the interrupt controller, the coin slot and the gumball slot

- to acquire an interrupt from the controller, a core will write to coin slot (“sticking the quarter in the gumball machine”). This will grab the interrupt off the queue.

- then the core will read from the gumball slot, to find out which interrupt it acquired, if any

M

[Krste Asanovic]

Yes, you can support cores that don’t support non-idempotency in this way.  The interrupt controller will need additional state per hart per privmode to store the captured interrupt ID.

You will also need to ensure ordering between the store and the read, either with a FENCE (which will hurt performance on simple cores) or with a strongly ordered PMA.

You don’t need two addresses, just define reads and writes to same address to have the desired effect.  This PLIC variant is certainly bigger and lower performance than the current design.

The standardization discussion was about the PLIC used in Unix platform spec.

If you’re running Unix, you’ll have data caches.  If you have data caches, you’ll need to support uncacheable MM I/O, which for most simple implementations ends up also supporting non-idempotent I/O.

The question is how many cores that want to run standard Unix distros won’t need to support uncacheable/non-idempotent I/O anyway?

Can someone explain the core they’re trying to build to run Unix that would have difficulty supporting non-idempotent loads to MMIO regions?

Most core designs only ever issue non-speculative loads to any MMIO region, so would already have necessary support.

More sophisticated designs support PMAs that allow speculative loads to idempotent regions of MMIO space (e.g., memory scratchpads on device controllers).

Krste

[Prof. Michael Taylor]

Hi Krste — thanks for reply — see Florian’s earlier posts in the thread for his use case. 

Essentially he wants to do loads early in the pipe on his OOO ish core.

With RISC-V and the open source movement (for example the Pulp ecosystem), I think it’s quite conceivable that entire SoCs can be built where no I/O component has side effects on reads, but clearly if the RISC-V interrupt mechanism requires it, that will put an end to that possibility.

Regarding overheads, I think we both know that it’s not the high order bit.

I don’t think anybody is saying we shouldn’t support the SiFive design, which seems good to me. But there is an interesting question about the universe of designs we want to support.

M

[Krste Asanovic]

On Apr 8, 2019, at 7:29 PM, Prof. Michael Taylor <prof....@gmail.com> wrote:
>
> Hi Krste — thanks for reply — see Florian’s earlier posts in the thread for his use case.
> Essentially he wants to do loads early in the pipe on his OOO ish core.

The load can still issue speculatively to memory regions or to a known-idempotent MM I/O region.
Implementing idempotency PMAs just requires statically checking a few physical address bits.
As Chris mentioned, there are many reasons a load has to wait in an OoO core, and waiting for commit can occur for other reasons.

(As an orthogonal but related issue, I am hoping Florian's core does not speculatively issue MMIO before TLB/PMP checks have passed)

> With RISC-V and the open source movement (for example the Pulp ecosystem), I think it’s quite conceivable that entire SoCs can be built where no I/O component has side effects on reads,

I understand the purist desire to avoid non-idempotent loads everywhere, but the performance difference is important enough that there is a lot of existing, and newly designed, IP out there that has side-effects on loads.
Building a core that requires every peripheral has idempotent MMIO is self limiting, and there is negligible cost to supporting non-idempotent loads in most OoO pipelines.

> but clearly if the RISC-V interrupt mechanism requires it, that will put an end to that possibility.

Adding an optional second memory-mapped aperture to the PLIC to support cores that can’t implement non-idempotent loads is not a big deal at all in specification-land or hardware-land, but just more hassle in software-land and compliance-land.

> Regarding overheads, I think we both know that it’s not the high order bit.

I’d agree overhead of issuing two memory operations instead of one for Linux on single-socket SoCs is not that bad overhead, but enough people complain already about the PLIC overheads.
(That assumes you have a strongly ordered PMA on that region, otherwise you’ll see two full I/O bus round trip latencies due to a FENCE in the middle - which means adding this option really means adding two options from software/compliance perspective, ones with and without strongly ordered PMAs).

> I don’t think anybody is saying we shouldn’t support the SiFive design, which seems good to me. But there is an interesting question about the universe of designs we want to support.

The “we” in “want to support” is not you or I, but the people supporting Unix distributions and writing compliance suites. They would have to agree to do the work to support all three options in the PLIC v1 and make available in standard distributions.

I’d rather spend my “annoy Unix maintainers” credit on working on a better PLIC v2.

Krste

> M
>

[Prof. Michael Taylor]

The “we” in “want to support” is not you or I, but the people supporting Unix distributions and writing compliance suites.  They would have to agree to do the work to support all three options in the PLIC v1 and make available in standard distributions.

I’d rather spend my “annoy Unix maintainers” credit on working on a better PLIC v2.

Krste

Just curious, will this “let’s not annoy the software people” argument be made everytime SiFive has developed something and hacked in preliminary support in the Linux kernel?

Aren’t we talking about basically one or two lines of code in this file? 

https://github.com/torvalds/linux/blob/master/drivers/irqchip/irq-sifive-plic.c

I am sure that the changes to Florian’s design (and all others who travel down this path and then realize too late) will be much greater.

> M
>

[Alex Solomatnikov]

The problem is not just PLIC.

PCIe device config can indicate non-prefetchable BAR, which means the BAR has read side effects:

Prefetchable vs. Non-prefetchable Memory Space

Figure 4‐1 shows two different types of MMIO being claimed by PCIe devices: Prefetchable MMIO (P‐MMIO) and Non‐Prefetchable MMIO (NP‐MMIO). It’s important to describe the distinction between prefetchable and non‐prefetch‐ able memory space. Prefetchable space has two very well defined attributes:

• Reads do not have side effects

• Write merging is allowed

Are you proposing that RISC-V UNIX-Class Platform does not need to be compatible with PCIe?

Alex

--

You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3P37hBvyrfjSK_XqbFs40jKUByWwfrmrPn8LuY6p-EhA%40mail.gmail.com.

[Dr Jonathan Kimmitt]

See inline comments.

Sent from my iPhone

> On 9 Apr 2019, at 04:52, Krste Asanovic <kr...@berkeley.edu> wrote:
>
>> On Apr 8, 2019, at 7:29 PM, Prof. Michael Taylor <prof....@gmail.com> wrote:
>>
>> Hi Krste — thanks for reply — see Florian’s earlier posts in the thread for his use case.
>> Essentially he wants to do loads early in the pipe on his OOO ish core.
>
> The load can still issue speculatively to memory regions or to a known-idempotent MM I/O region.
> Implementing idempotency PMAs just requires statically checking a few physical address bits.
> As Chris mentioned, there are many reasons a load has to wait in an OoO core, and waiting for commit can occur for other reasons.
>
> (As an orthogonal but related issue, I am hoping Florian's core does not speculatively issue MMIO before TLB/PMP checks have passed)
>
>> With RISC-V and the open source movement (for example the Pulp ecosystem), I think it’s quite conceivable that entire SoCs can be built where no I/O component has side effects on reads,
>
> I understand the purist desire to avoid non-idempotent loads everywhere, but the performance difference is important enough that there is a lot of existing, and newly designed, IP out there that has side-effects on loads.
> Building a core that requires every peripheral has idempotent MMIO is self limiting, and there is negligible cost to supporting non-idempotent loads in most OoO pipelines.
>
>> but clearly if the RISC-V interrupt mechanism requires it, that will put an end to that possibility.
>
>
> Adding an optional second memory-mapped aperture to the PLIC to support cores that can’t implement non-idempotent loads is not a big deal at all in specification-land or hardware-land, but just more hassle in software-land and compliance-land.
>
>> Regarding overheads, I think we both know that it’s not the high order bit.
>
> I’d agree overhead of issuing two memory operations instead of one for Linux on single-socket SoCs is not that bad overhead, but enough people complain already about the PLIC overheads.
> (That assumes you have a strongly ordered PMA on that region, otherwise you’ll see two full I/O bus round trip latencies due to a FENCE in the middle -

I have looked at a disassembly of the PLIC support code in stock RISCV Linux, and lo and behold there is a “fence i, r” right after the interrupt claim. This can be used in a micro architecture specific way to confirm to other harts and the PLIC that a critical synchronisation point has been reached. Unfortunately that instruction does not mention the I/O address that is being fenced. Perhaps that could be a future enhancement to the RISCV user specification, if it’s not frozen already. I’m not personally a fan of non idempotent I/O but it’s a bit late to complain now. The true key to better interrupt performance in PLICv2 is vectored interrupt control and multiple register banks, not tinkering with a few loads here and there.

[Krste Asanovic]

On Apr 8, 2019, at 9:17 PM, Prof. Michael Taylor <prof....@gmail.com> wrote:

The “we” in “want to support” is not you or I, but the people supporting Unix distributions and writing compliance suites.  They would have to agree to do the work to support all three options in the PLIC v1 and make available in standard distributions.

I’d rather spend my “annoy Unix maintainers” credit on working on a better PLIC v2.

Krste

Just curious, will this “let’s not annoy the software people” argument be made everytime SiFive has developed something and hacked in preliminary support in the Linux kernel?

“Donning SiFive hat”:  SiFive donated the PLIC design to fill a vacuum in Unix land.  In other areas, SiFive has donated an initial version of a spec that was then changed in Foundation Task Group after SiFive put early version into silicon (debug, CLIC).    We didn’t force our implementation down people’s throats, but donated the substantial initial effort and then changed our existing in-silicon designs based on community feedback to support others' use cases.   The vector spec has gone through many drastic revisions driven by community feedback, before hopefully settling on an agreed form.  We’re not the only ones doing this - I’d highlight Andes spirit of cooperation here too, along with other members. 

“Donning Foundation hat”: For this particular case of the PLIC interface, first note that for a commercial Unix SoC project, you generally can’t sell a commercial core or use a free core, that speculatively executes all MMIO accesses.  So, there’s incentive to retain the current PLIC interface as it is faster and supported by cores you would use in a Unix platform.  Asking the community to support an optional second slower idempotent PLIC interface for other cores does not seem like a good use of Foundation resources right now.

Aren’t we talking about basically one or two lines of code in this file? 

https://github.com/torvalds/linux/blob/master/drivers/irqchip/irq-sifive-plic.c

Of course, any group can port Linux to their variants. As you well know, it’s not how many lines are changed to support a variant, it’s the multiplicative effect of how many simultaneous variants there are to support.

(It’d be easy to highlight the single bit change needed to accommodate an alternate encoding of an ADD instruction, for example.

The support ramifications would be massive, despite the tiny difference of the alternate encoding.)

I am sure that the changes to Florian’s design (and all others who travel down this path and then realize too late) will be much greater.

The PLIC spec has been out for years, and has been implemented by several developers outside of SiFive.

Most commercial SoCs have to deal with devices that have side-effects on reads.

This should not be a surprise to anyone building a core intended to be widely used.

Again, there is a plan to develop a PLIC v2 which is a more substantial upgrade over current PLIC. This could try to avoid non-idempotent accesses for example, but I can guess there will be significant push back if that implies lower performance. While Unix is less concerned with low-level interrupt handler overheads, AMP systems can mix Unix on some cores and RTOS on others, and PLIC interrupt latency will be more exposed.

Krste

> M
>

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3P37hBvyrfjSK_XqbFs40jKUByWwfrmrPn8LuY6p-EhA%40mail.gmail.com.

[Prof. Michael Taylor]

Thank you for your email, Alex. It is helpful to have some concrete examples of things that will not work if you do not support read side effects in your core.

PCIE is not a requirement of the POSIX UNIX standard, and many chips that run Linux do not have PCIE blocks, so it would not make sense that PCIE support should be a requirement for Linux RISC-V chips (or more specifically,  non-prefetchable PCIE devices). PCIE had to support legacy HW devices from ancient x86 days, so it’s not surprising it has support for this.

I think there is a subtle but important distinction between whether we insist that a processor cannot do a purely speculative load operation on a potentially wrong address (which could create chaos in an I/O device), and whether we allow the processor to retry an otherwise good load operation for whatever reason. For example, suppose we have a transient parity error on the value read from the PLIC. The PLIC’s current semantics prevent a simple retry mechanism in the processor from reissuing the read to correct the error. 

[Krste Asanovic]

On Apr 8, 2019, at 10:57 PM, Dr Jonathan Kimmitt <jr...@cam.ac.uk> wrote:

See inline comments.

Sent from my iPhone

On 9 Apr 2019, at 04:52, Krste Asanovic <kr...@berkeley.edu> wrote:

I’d agree overhead of issuing two memory operations instead of one for Linux on single-socket SoCs is not that bad overhead, but enough people complain already about the PLIC overheads.
(That assumes you have a strongly ordered PMA on that region, otherwise you’ll see two full I/O bus round trip latencies due to a FENCE in the middle -

I have looked at a disassembly of the PLIC support code in stock RISCV Linux, and lo and behold there is a “fence i, r” right after the interrupt claim. This can be used in a micro architecture specific way to confirm to other harts and the PLIC that a critical synchronisation point has been reached. Unfortunately that instruction does not mention the I/O address that is being fenced.

I believe that FENCE is needed for the separate problem of ensuring that a following memory read only happens after the claim is returned by the PLIC and not before in a speculative processor. I.e., this is to order what is done after the claim is received not part of the claim process itself.

Perhaps that could be a future enhancement to the RISCV user specification, if it’s not frozen already. I’m not personally a fan of non idempotent I/O but it’s a bit late to complain now.  

I don’t think anyone is a fan of non-idempotent I/O, but it is a fact of life.

The true key to better interrupt performance in PLICv2 is vectored interrupt control and multiple register banks, not tinkering with a few loads here and there.

Even modern micro controllers avoid multiple register banks for interrupts.  Saving and restoring necessary registers to cache is quite fast compared to a few serialized MMIOs, and supports nesting better.

Hardware vectoring is not a good match to regular linux, which wants to treat "interrupts as data” not “interrupts as control". (See my 4th workshop talk).

Using real driver interrupt overhead measurements to drive improvements would be ideal.

Krste

[Bert]

Hi,

As a potential implementer, I am worried about this discussion.

Blame it on being naieve, but I consider RISC-V an alternative to the proprietary locked systems from unnamed other
CPU cores.

Just searching for more information: (source: https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.1-RISC-V-Maturing)

"""

Moving forward though with the RISC-V kernel support maturing, RISC-V architecture maintainer Palmer Dabbelt is wanting to ensure the support doesn't regress and as such will begin instituting that all kernel changes are tested on this board as opposed to just the RISC-V emulator. 

Palmer commented in the 5.1 pull request, "I've used my standard testing flow (QEMU in Fedora), but now that we're starting to get the kernel in better shape I think it's time to impose some more testing here -- specifically I'm going to require that patches boot on the HiFive Unleashed because we're getting to the point where we can actually expect that to work. I haven't done that for this tag, but I'm going to do it for future ones." 

"""

As an implementer, am I allowed to repeat my initial question, is one implementation being used to

define the standard for the whole open source community and not based on the requirements and

arguments of the whole open source community? Do all mailing list members agree this is

the open source standard way forward?

I need to know. Because as an implementer, I enjoy the fact that the ISA can be used for a slow IoT device
or for high performance. Because the ISA allows various implementations fast and slow. And that is
great, this is what I want. If later on, I cannot validate my design (as certified RV compliant) because my
design runs a linux or unix kernel (the terms are used both, but I think that is not correct) with my own

choice of -for example- interrupt controller, because it does not adhere to the IRQ standard,

that could be a showstopper to many. I expected the IRQ requirements and not an implementation

as the standard. That allows many implementations or applications whatever RAM or comm protocol

requirements they have.

How does the hardware development (*) community feel about the potential limitations

of defining a standard for all?

BR,

B.

(*) people that do ASIC design for a living for a significant period of time. It is a pure hardware concern

that I have related to the standard. I feel that sw things are mixed with hw which is never a good idea.

Op di 9 apr. 2019 om 08:09 schreef Krste Asanovic <kr...@berkeley.edu>:

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/DEE2ECC5-3B38-48BB-A428-1EB92C6F9023%40berkeley.edu.

[Dr Jonathan Kimmitt]

There are not many examples of Linux capable RISCV silicon out there. Until there are, it makes sense to use the HiFive-unleashed as a reference. Also there are many software bugs that only show up on multicore. Other FPGA emulation platforms such as the OpenPiton (http://parallel.princeton.edu/openpiton/) support RISCV but not sufficiently maturely to be tape-out ready (yet).

The goal of a single Linux image is a laudable one, but it does mean loads of drivers being compiled in that may not be loaded. So this is not a block on incompatible interrupt controllers as such. Nor do we expect every peripheral and every micro-architecture to be open-source. And of course it would not mean that a particular device driver patch is correct, if that peripheral is not present on the HiFive.

[Greg Wright]

> On Apr 9, 2019, at 2:15 AM, Prof. Michael Taylor <prof....@gmail.com> wrote:
>
> Thank you for your email, Alex. It is helpful to have some concrete examples of things that will not work if you do not support read side effects in your core.
>
> PCIE is not a requirement of the POSIX UNIX standard, and many chips that run Linux do not have PCIE blocks, so it would not make sense that PCIE support should be a requirement for Linux RISC-V chips (or more specifically, non-prefetchable PCIE devices). PCIE had to support legacy HW devices from ancient x86 days, so it’s not surprising it has support for this.
>
> I think there is a subtle but important distinction between whether we insist that a processor cannot do a purely speculative load operation on a potentially wrong address (which could create chaos in an I/O device), and whether we allow the processor to retry an otherwise good load operation for whatever reason. For example, suppose we have a transient parity error on the value read from the PLIC. The PLIC’s current semantics prevent a simple retry mechanism in the processor from reissuing the read to correct the error.
>

Yes, there’s a difference between mathematical idempotence (i.e. a load executing twice produces the same result as executing once) and side-effect freedom (where a speculative load may be undone with no long term damage). For example, an idempotent load may have the side-effect of launching all the missiles; trying to launch them twice results in the same end state. Similarly at the microarch level, we can distinguish speculation that may result in the I/O load never committing (address speculation, or control - branch, exception, etc.) vs. cases where the load will definitely occur but may be replayed e.g. for ordering purposes. The PMP spec seems to use idempotence and side-effect freedom as equivalent, but doesn’t locally define ‘idempotence’; in any case, it seems it’s not the usual mathematical meaning.

To come back to the main thread, I believe RISC-V in general will need to support the masses of devices out there with load side-effects. It’s one thing giving a new ISA to the world, it’s another to insist that the rest of the system around the core also change, or that drivers be rewritten. Many peripherals, and their drivers, will make use of hardware FIFOs (for example).

That said, there are some options if you’re starting with a pipeline that always speculatively issues loads:
1. Only work with I/O devices that have no side-effects, such as a custom non-standard PLIC.
2. Push the domain of speculation out to include the device. For something as tightly-coupled as a PLIC, this might work. Can the core signal commit/flush on that load out to the PLIC? I have not looked at what this does to the PLIC implementation…

- Greg.

[Prof. Michael Taylor]

On Mon, Apr 8, 2019 at 11:12 PM Krste Asanovic <kr...@berkeley.edu> wrote:

On Apr 8, 2019, at 9:17 PM, Prof. Michael Taylor <prof....@gmail.com> wrote:

The “we” in “want to support” is not you or I, but the people supporting Unix distributions and writing compliance suites.  They would have to agree to do the work to support all three options in the PLIC v1 and make available in standard distributions.

I’d rather spend my “annoy Unix maintainers” credit on working on a better PLIC v2.

Krste

Just curious, will this “let’s not annoy the software people” argument be made everytime SiFive has developed something and hacked in preliminary support in the Linux kernel?

“Donning SiFive hat”:  SiFive donated the PLIC design to fill a vacuum in Unix land.  In other areas, SiFive has donated an initial version of a spec that was then changed in Foundation Task Group after SiFive put early version into silicon (debug, CLIC).    We didn’t force our implementation down people’s throats, but donated the substantial initial effort and then changed our existing in-silicon designs based on community feedback to support others' use cases.   

Seems like things are unfolding differently here.

“Donning Foundation hat”: For this particular case of the PLIC interface, first note that for a commercial Unix SoC project, you generally

Sorry, I did not realize that commercial SoC projects are the only intended stakeholders of RISC-V.

can’t sell a commercial core or use a free core, that speculatively executes all MMIO accesses.  

I think a better way to say it “occasionally retries”.

So, there’s incentive to retain the current PLIC interface as it is faster and supported by cores you would use in a Unix platform.  Asking the community to support an optional second slower idempotent PLIC interface for other cores does not seem like a good use of Foundation resources right now.

Aren’t we talking about basically one or two lines of code in this file? 

https://github.com/torvalds/linux/blob/master/drivers/irqchip/irq-sifive-plic.c

Of course, any group can port Linux to their variants. As you well know, it’s not how many lines are changed to support a variant, it’s the multiplicative effect of how many simultaneous variants there are to support.

(It’d be easy to highlight the single bit change needed to accommodate an alternate encoding of an ADD instruction, for example.

The support ramifications would be massive, despite the tiny difference of the alternate encoding.)

The IRQ code is segregated into a single file in Linux, and almost certainly is the same for any portable OS. Does not seem like an equal comparison to me.

I am sure that the changes to Florian’s design (and all others who travel down this path and then realize too late) will be much greater.

The PLIC spec has been out for years, and has been implemented by several developers outside of SiFive.

Most commercial SoCs have to deal with devices that have side-effects on reads.

This should not be a surprise to anyone building a core intended to be widely used.

Seems like it was, that’s why we have this thread?

Again, there is a plan to develop a PLIC v2 which is a more substantial upgrade over current PLIC. This could try to avoid non-idempotent accesses for example, but I can guess there will be significant push back if that implies lower performance. While Unix is less concerned with low-level interrupt handler overheads, AMP systems can mix Unix on some

cores and RTOS on others, and PLIC interrupt latency will be more exposed.

I don’t think anybody will waste their time trying to give feedback on v2 after reading this thread. 

I don’t really care about the PLIC, our designs will work either way, but the process seems pretty broken. 

The takeaway I got from this is that two of the long-time major open source stake holders in RISC-V, ETHZ and Cambridge, raised a concern and the response was super dismissive.

Krste

> M
>

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3P37hBvyrfjSK_XqbFs40jKUByWwfrmrPn8LuY6p-EhA%40mail.gmail.com.

[Alex Solomatnikov]

Clearly, you are not very familiar with PCIe devices :)

If you look at a typical modern server (a snippet of lspci -vvvv output is below), you find that there are many modern PCIe devices that use non-prefetchable BARs, e.g. a Mellanox Ethernet controller from 2015. So, it is not just for legacy PCIe devices.

I agree that not every UNIX/Linux system has PCIe devices, however, PCIe is extremely common, especially in data center systems but not only.

Alex

00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)

        Subsystem: Super Micro Computer Inc Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 64 bytes

        Interrupt: pin A routed to IRQ 83

        Region 0: Memory at c752c000 (64-bit, non-prefetchable) [size=16K]

        Capabilities: <access denied>

        Kernel driver in use: ioatdma

        Kernel modules: ioatdma

...

03:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

        Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

        Physical Slot: 15

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 32 bytes

        Interrupt: pin A routed to IRQ 36

        Region 0: Memory at c7200000 (64-bit, non-prefetchable) [size=1M]

        Region 2: Memory at c5800000 (64-bit, prefetchable) [size=8M]

        Expansion ROM at c7100000 [disabled] [size=1M]

        Capabilities: <access denied>

        Kernel driver in use: mlx4_core

        Kernel modules: mlx4_core

...

00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])

        Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode]

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Interrupt: pin A routed to IRQ 35

        Region 0: I/O ports at 7110 [size=8]

        Region 1: I/O ports at 7100 [size=4]

        Region 2: I/O ports at 70f0 [size=8]

        Region 3: I/O ports at 70e0 [size=4]

        Region 4: I/O ports at 7020 [size=32]

        Region 5: Memory at c7538000 (32-bit, non-prefetchable) [size=2K]

        Capabilities: <access denied>

        Kernel driver in use: ahci

        Kernel modules: ahci

...

06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) (prog-if 00 [VGA controller])

        DeviceName: ASPEED Video AST2400

        Subsystem: Super Micro Computer Inc ASPEED Graphics Family

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0

        Interrupt: pin A routed to IRQ 16

        Region 0: Memory at c6000000 (32-bit, non-prefetchable) [size=16M]

        Region 1: Memory at c7000000 (32-bit, non-prefetchable) [size=128K]

        Region 2: I/O ports at 6000 [size=128]

        Expansion ROM at <unassigned> [disabled]

        Capabilities: <access denied>

        Kernel driver in use: ast

        Kernel modules: ast

07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

        DeviceName: Intel Ethernet i210 #1

        Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection

        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 64 bytes

        Interrupt: pin A routed to IRQ 18

        Region 0: Memory at c7400000 (32-bit, non-prefetchable) [size=512K]

        Region 2: I/O ports at 5000 [size=32]

        Region 3: Memory at c7480000 (32-bit, non-prefetchable) [size=16K]

        Capabilities: <access denied>

        Kernel driver in use: igb

        Kernel modules: igb

[Prof. Michael Taylor]

I didn’t say that PCIE devices do not have it, I said that the support came from the need to support legacy X86 (I.e. ISA, PCI, ...) , and now that the support is there, it will get used and then be necessary for all future generations of hardware, just as will happen with RISC-V as a result of the PLIC.

M

[Josh Scheid]

On Mon, Apr 8, 2019 at 10:34 PM Alex Solomatnikov <so...@sifive.com> wrote:

The problem is not just PLIC.

PCIe device config can indicate non-prefetchable BAR, which means the BAR has read side effects:

UART programming models also require non-speculative read behavior for the receive buffer access.

-Josh

 

[Samuel Falvo II]

On Tue, Apr 9, 2019 at 7:44 AM Prof. Michael Taylor <prof....@gmail.com> wrote:

I don’t think anybody will waste their time trying to give feedback on v2 after reading this thread. 

Years ago now, I can't even remember when, back when the RISC-V foundation was first forming and starting to make its first choices about processes and the like, I specifically mentioned the two opposing methods of standards setting: what I call the ITU-T approach and the IETF approach.

Neither is without their respective problems, but I do recall raising some concerns about ITU-T's method seeming baroque, unapproachable, and even exclusionary (much less leading to suboptimal standards).  This whole thread pains me, because it seems to vindicate those concerns.

What's the core difference between the two?  If I may over-generalize:

ITU-T is a standards-by-committee approach.  IETF is a "standardize already known-working solutions" approach.  To get an RFC through the process, you need to demonstrate (not just speculate) some minimum number of independently developed, yet compatible, implementations of your software or hardware protocol.

Right now, the RISC-V Foundation follows a process not unlike ITU-T.  They've been significantly more open in some areas than ITU-T, but at its core, it's a committee-driven, consensus-seeking approach which, despite having the best of intentions, has generated a bit of heat in the past from outside contributors who raise valid technical concerns, but who feel they have no say in the matter otherwise.  (In fact, this is explicitly by design; voting rights only comes with Foundation membership.)

However, if it's one thing we've learned from history, it's that the IETF-style of standards setting largely won out in the end.  ITU-T specifications dominate protocols which seem to have government mandate (e.g., some countries still require the use of X.25 networks, for example); however, for everything else, you'll find Internet standards taking over or already firmly established.  It took quite a while, but it did eventually happen.

I raise this point not to thread-jack the topic, but to attempt to put to ease both sides of the issues discussed.  On the one hand, when PLIC was first published, it was fairly understood to be a recommendation, and comments about it were solicited, and a lot of feedback and counter-proposals were exchanged (including from myself; but I'm not a foundation member and I'm largely inexperienced with large-scale systems design).  Today, it's enshrined in one of the best known examples of RISC-V silicon (even if unfairly by virtue of SiFive's visibility in the technical press), and that popularity now confers unto PLIC some semblance of an official standard.  Especially with working versions of Linux built for it.

But, by way of allegory again, people found ways of using Intel 8255 PIOs with their Motorola 6809 processors, despite the wildly different bus architectures.  I may be waxing optimistic here; but, I predict in the future that IETF-inspired, community-driven agreements will become more prevalent as people start to notice other people's work.  Perhaps the Shakti[1] project has a certain way of doing things for X, and Cambridge has a way of doing Y that is quite interesting.  Provided these aren't overly burdened by IP limitations, we will see standardization (eventually) on certain sets of features.  Maybe, the lifespan of the PLIC standard will be relatively short if Cambridge, ETH Zurich, Shakti, et. al. all settle on an alternative design.  Maybe, Linux will need to support two or three interrupt controller standards, as it currently does for x86-based systems.  Having a choice between implementations is OK, I think, as long as you don't go hog-wild and end up with 52 different interrupt controller designs on equal footing.

I want to point out that, in practice, while it appears that SiFive has a perhaps disproportionate amount of influence on the standards setting process today, they have so far done nothing to dissuade me or anyone else on these lists from not pursuing their own unique designs or contributions.  In fact, just the opposite!  I've received encouragement from Andrew Waterman, Chris Celio, Krste, and many others.  I haven't heard from him in a long while, but I remember that even Yunsup Lee was always eager to hear my unique interpretation of RISC-V.  They don't need to agree with me, but having received support and understanding was very encouraging nonetheless.

Case in point, I just completed the implementation of my next processor's CSR Unit.  It's a total chungus, weighing in at 1157 iCE40HX8K logic cells.  That's enormous!  It's mostly due, I think, to the WPRI logic, and perhaps a not so efficient way of exposing the registers to the rest of the processor logic.  Also, 64-bit event counters, even when the high bits are almost never going to be used.  I can save a lot of space on my design if I turn all WPRI fields to WIRI or WARL fields instead, and shave half the number of bits off the event counters.  Ideally, I'd like to not have to do this; but, I will if I can't fit my design onto the FPGA.  I will no longer be compliant with the 1.10 privilege specifications; but I'm willing to bet I'll be compatible "enough" that most software will work on it.  Absolutely nobody has yet told me not to do this.

So, if PLIC isn't to your tastes for some reason, and you're willing to invest the effort to make your part work with Linux (or whatever), I do not foresee any retributive action coming from SiFive at all.  Given all this round-about discussion, I suppose what I want to drive home is that I don't think emotions here need to run particularly hot over this issue.  That said, keep on debating!  It's how progress will be made, one way or another.

________

1.  Forgive any poor spelling; I'm writing from memory.

 

I don’t really care about the PLIC, our designs will work either way, but the process seems pretty broken. 

The takeaway I got from this is that two of the long-time major open source stake holders in RISC-V, ETHZ and Cambridge, raised a concern and the response was super dismissive.

Krste

> M
>

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3P37hBvyrfjSK_XqbFs40jKUByWwfrmrPn8LuY6p-EhA%40mail.gmail.com.

--

Apologies for all typos. Message sent by combination of an approximate neural network and a smartphone.

[Samuel Falvo II]

My Kestrel-2 project (the original Kestrel-2, not the 2DX) used a processor which did not support interrupts.  To make the keyboard interface work well, I put the keyboard queue in hardware as a FIFO.  I originally wanted to use an effecting read to pop the queue, but realized that my software was smaller and simpler if I instead used the "gumball machine" approach.  The read port would always return the same character, until you poked a value (any value; didn't matter) back at the FIFO.  The write caused the explicit pop.  I absolutely *love* this interface; so much so that my Keyboard Interface Adapter has become a standard fixture on Kestrels made since then (Kestrel-2DX, and soon, Kestrel-3).

The disadvantage for gumball-style FIFOs is, as can be seen in the above example, it doubles the traffic to the I/O device for a given amount of data transfer.  For a keyboard controller or perhaps a "slow enough" serial connection, this won't be a problem.  But, if you're writing for a system with constrained-resources and/or for a dedicated I/O processor, and you need to keep transfer rates high, the gumball approach might not work for you.

Remember that a lot of UARTs back in the day were intended to be used with CPUs with 1MHz to 4MHz clock speeds, and which didn't often have DMA dedicated to the purpose.  So, transferring large amounts of data often involved tight loops to maintain data throughput.  But, even with today's architectures, writing to I/O space can incur significant bus overheads, precisely because I/O cannot be cached.  Even if idempotency can be guaranteed, even with an effect-free read to a FIFO or mailbox register, the need for a secondary acknowledgement write will incur measurable performance degradation by virtue of the requirement for non-cached accesses.  On many buses, back-to-back read/write cycles incurs additional overheads to support bus turn-around (e.g., PCI).  It seems the economics of high-performance systems still prefers single-access effecting-reads or effecting-writes over gumball approaches, at least for channels which demand high performance.

[Prof. Michael Taylor]

Thanks, Samuel, this was very thought provoking. And I echo all of your praise for Chris, Krste, Yunsup and Andrew.

There were a few questions that were brought up by your email, and mostly I am participating in this thread so I understand the state of RISC-V better.

1. Does SiFive reserve the right to railroad through their HW into the RISC-V specifications?

I think the practical answer is yes.

I'm actually fine either way, but let's not pretend.

2. Will SiFive work to prevent other interrupt controller designs (or more generally, alternative peripheral architectures) from being committed to Linux?

I think the answer is ultimately yes, since it will be argued that they are not "compliant with the standard", and SiFive will always have the most hands on the Linux kernel source.

3. Should independent groups develop an alternative way of doing things than SiFive in RISC-V like the IETF process mentioned by Samuel (as opposed to iterating a new standard), is it likely to be considered for adoption?

I think the answer is generally no, based on this thread and prior history, this would be time wasted.

If I'm wrong on this, maybe Krste can clarify.

[Christopher Celio]

I don’t think anybody will waste their time trying to give feedback on v2 after reading this thread. 

I don’t really care about the PLIC, our designs will work either way, but the process seems pretty broken. 

Please don't mistake an email thread on riscv-hw as the place where standards are hashed out; there are a lot of RISC-V engineers who avoid these lists like the plague and who don't feel "locked out" from providing feedback. The PLIC is not a surprise; it's been around for a few years now and SiFive made it very clear their intentions and goals behind it being a future RISC-V standard and their desire for feedback on it. 

-Chris

 

[Prof. Michael Taylor]

Thanks, I will update the list of what I think I have learned in the thread:

4. Everything we post in this forum is considered to be irrelevant.

[Paul Walmsley]

There's probably a miscommunication or a misunderstanding about what the "UNIX-class Platform Specification" proposal is, and isn't, intended to be.  This is perhaps an artifact of this list being oriented towards hardware designers and engineers, rather than software developers.

It doesn't help that the name of the proposed specification is confusing.  Despite being called "UNIX-class", there is nothing preventing anyone from porting, running, and distributing the Linux kernel, or any other operating system, to systems that don't align with what ends up in the "UNIX-class Platform Specification" (beyond the license terms of the software itself, of course).  This is almost always the right approach for new devices.  Every ARM Linux-based smartphone SoC vendor - Samsung, Qualcomm, TI, many others - has taken this approach, maintaining their own "forks" of open-source software and providing that to their users.

Similarly, there is nothing preventing anyone from sending code changes for their specialized hardware to the open-source projects to ask that their changes be included in "mainline" project code.  Of course, the maintainers may reject those changes, or require changes to be made to the proposed code, or may take a long time to include the changes.  That's simply the nature of the process.  Maintainer responsibilities include ensuring a codebase can be easily developed going forward, and that it doesn't spiral out of control due to complexity.

For example, suppose someone creates a hardware device that breaks key assumptions in decades of microprocessor or SoC design practice, that requires code changes all over an operating system to function.  Open source maintainers may be reluctant to merge code changes for that device.  That reluctance still wouldn't block that operating system from running on the device, since whoever creates those devices can just distribute their own "board support package", until the rest of the world comes around to appreciating what they've done.   This is what all the major SoC providers do today.  Linux-class hardware designs don't exist in a vacuum.  Those hardware designs rely on software to run on them to be useful, and the process is a way of focusing maintainers' limited resources on the parts of the codebase that need those resources the most. 

Similarly, just as open-source software developers have limited resources, so too do the major operating system distribution maintainers, like Red Hat, the Debian Project, Canonical (Ubuntu), Microsoft, etc.  Even for the ARM architecture - a change-restricted, proprietary architecture, backed by billions of dollars of capital - it is only barely becoming practical to download a single binary image that boots and runs on devices from different ARM vendors.  For ARM, many distribution providers don't even try.  That single bootable system image is much easier for Intel devices, for many reasons.  One major reason is that Intel has enforced a consistent platform interface and specification across their ecosystem, in cooperation with the OS distribution providers.  If OS distribution vendors target those specifications, they can feel reasonably confident that their software will boot and at least limp along on future systems.  The RISC-V community doesn't have the same options, since Intel maintains iron-clad corporate control of their instruction set and platform architecture.  The RISC-V community has to take a more collaborative, open approach.

Thus a more useful way to think of the RISC-V "UNIX-class" proposal might be as a "Major OS Distribution Platform Specification" proposal.  One of the goals is to provide a document where OS distribution vendors, hardware vendors, and other interested members of the RISC-V community can come together to define what the boot and operating system interface should look like, and define expectations for hardware, firmware, and software binaries that should "just work" across different device vendors.

Will the "UNIX-class" platform specification restrict which devices can run "UNIX" or Linux?  No.  Just as there are millions of Linux-class ARM devices in the world that will never run Red Hat Enterprise Linux for ARM, in devices like smartphones, thermostats, smartwatches, and cellphone base stations; the same will likely be true for RISC-V.    However, there will be a subset of hardware devices whose designers do target these traditional desktop or server use-cases, and for them, the "UNIX-class Platform Specification" will be important.

- Paul

[Prof. Michael Taylor]

Thank you Paul.

Let’s say that 1) ETHZ, Cambridge, and UW get together and define an idempotent Linux irqchip, PLIC-alternative spec which their open source Linux capable RISC-V designs — Ariane, Cambridge’s design, and BlackParrot, respectively — support and release open source code for, and 2) we submit the code modifications compliant with Linux coding guidelines, will SiFive and the RISC-V foundation folks support its acceptance into the Linux kernel base?

M

[Palmer Dabbelt]

On Tue, 09 Apr 2019 09:59:55 PDT (-0700), prof....@gmail.com wrote:
> Thanks, Samuel, this was very thought provoking. And I echo all of your
> praise for Chris, Krste, Yunsup and Andrew.
>
> There were a few questions that were brought up by your email, and mostly I
> am participating in this thread so I understand the state of RISC-V better.
>
> 1. Does SiFive reserve the right to railroad through their HW into the
> RISC-V specifications?
>
> I think the practical answer is yes.
>
> I'm actually fine either way, but let's not pretend.

In this case we really don't: every member of the RISC-V foundation gets to
vote on the specifications for approval. The only SiFive thing suggested in
the platform specification is the PLIC, but as far as I can tell that has been
implemented by multiple RISC-V vendors and is frequently confused to be a
RISC-V standard. All we're really trying to do here is make sure all the
RISC-V vendors have an equal say in the specification.

As a concrete example: if the PLIC was to remain a SiFive standard and we found
a mismatch between the implementation and the specification that manifested in
chips that had already been shipped, then a reasonable option would be to
change the specification (assuming it's just a mismatch that's not too much of
a problem). This would be bad for the other RISC-V vendors because they would
then have implementations that don't meet the specifications.

Avoiding this sort of vendor control over the specifications is the whole point
of RISC-V, which is why we want the PLIC spec to be a RISC-V spec not a SiFive
spec.

> 2. Will SiFive work to prevent other interrupt controller designs (or more
> generally, alternative peripheral architectures) from being committed to
> Linux?
>
> I think the answer is ultimately yes, since it will be argued that they are
> not "compliant with the standard", and SiFive will always have the most
> hands on the Linux kernel source.

That's far from true. By my metrics there have been 1434 contributors to
drivers/irqchip, 3 of which have SiFive email addresses. 0 of the 3
maintainers of the interrupt subsystem in Linux are employed by SiFive, so even
if we wanted to there would be no way for us to prevent you from getting your
interrupt controller driver merged.

The intent here is simply to document the PLIC, which was originally designed
by SiFive but as far is I know it has been implemented by other RISC-V vendors.
The PLIC has been confused for a RISC-V standard a handful of times, so all
we're really trying to do here is make sure that this standard is owned by all
RISC-V vendors and not just SiFive.

I don't intend the platform specification to mandate the presence of a PLIC,
but simply to provide it as an option for an interrupt controller that is
expected to be supported by standard software. In general my philosophy behind
this platform spec is to avoid trying to force one way of doing things on
people, but to instead provide a standard way of doing things in places where
there are fairly well understood ways of doing something. The general idea
here is to reduce the amount of unnecessary deviation between different
hardware platforms, while still allowing vendors to do whatever they want. I'm
treating interrupt controllers this way: the PLIC seems to be good enough that
it's been implemented a few times, so having it in the platform spec will
reduce unnecessary deviations for use cases when the PLIC can meet the
requirements.

If a PLIC doesn't meet your requirements then you're free to go do some other
interrupt controller, it's just now your problem to make sure that software can
run on your implementation. That may mean convincing another iteration of the
platform spec to adopt your interrupt controller as one of the standard
interrupt controller interfaces, or it may mean going around to the software
vendors you care about and convincing them to support your interrupt
controller. The second option is what you'd have to do without a platform spec
to rely on, so it's not like we're taking away any options here. We're just
giving vendors the option to rely on a RISC-V standard (and the associated
implementations) for the parts of the system where the existing standards meet
their requirements.

This is really the same as the rest of the RISC-V idea. For example: we have
an extension for multiplication, which is supported by standard software and
meets many use cases. There's nothing that prevents you from doing your own
ISA extension that performs multiplication, and there's nothing that stops you
from building a software ecosystem that supports that extension. Getting all
the upstream maintainers to support your custom extension might be a hard sell,
but ultimately it's up to you to convince whomever owns the code to support
your custom extension.

While the employees of SiFive do own the RISC-V subsystems in a handful of
projects, we try really hard to ensure that we put the needs of the RISC-V
community first in our upstream maintenance roles and also aim to have an
active non-SiFive maintainer on every project in order to act as a check
against us having too much control. Most of SiFive's upstream maintainers have
a long history of working for companies while maintaining open source software,
and so far I think we have a decent track record of avoiding too much
SiFive-focused influence over the upstream ports.

> 3. Should independent groups develop an alternative way of doing things
> than SiFive in RISC-V like the IETF process mentioned by Samuel (as opposed
> to iterating a new standard), is it likely to be considered for adoption?
>
> I think the answer is generally no, based on this thread and prior history,
> this would be time wasted.
>
> If I'm wrong on this, maybe Krste can clarify.

I'm not sure what your concern is here. When you say a IETF-stlye process, do
you mean:

* Lack of formal membership agreements? That decision is far above my pay
grade, but FWIW I think the membership agreement of the RISC-V foundation is
the best way to get the broadest set of contributors to the RISC-V
specifications.
* Voluntary nature of the specifications? Every RISC-V specification is
voluntary, it's just that if you don't adhere to a standard then you might
have to do more work to produce a viable system.
* "Rough consensus and running code"? How a specification is designed is
really up to the members of the working group, and while many RISC-V
specifications are run in the "build a spec from the group up" way that's not
how the platform spec is being run. Here we're focused on maintaining
compatibility with existing implementations that arose on their own.

Either way, the best way to have your opinion heard is to join the development
of the specification.

>> comments about it *were* solicited, and a lot of feedback and
>> counter-proposals *were* exchanged (including from myself; but I'm not a
>> foundation member *and* I'm largely inexperienced with large-scale

>> systems design). Today, it's enshrined in one of the best known examples
>> of RISC-V silicon (even if unfairly by virtue of SiFive's visibility in the
>> technical press), and that popularity now confers unto PLIC some semblance
>> of an official standard. Especially with working versions of Linux built
>> for it.
>>
>> But, by way of allegory again, people found ways of using Intel 8255 PIOs
>> with their Motorola 6809 processors, despite the wildly different bus
>> architectures. I may be waxing optimistic here; but, I predict in the
>> future that IETF-inspired, community-driven agreements will become more
>> prevalent as people start to notice other people's work. Perhaps the
>> Shakti[1] project has a certain way of doing things for X, and Cambridge
>> has a way of doing Y that is quite interesting. Provided these aren't
>> overly burdened by IP limitations, we will see standardization (eventually)

>> on certain sets of features. *Maybe*, the lifespan of the PLIC standard

>> will be relatively short if Cambridge, ETH Zurich, Shakti, et. al. all

>> settle on an alternative design. *Maybe*, Linux will need to support two

>> or three interrupt controller standards, as it currently does for x86-based
>> systems. Having a choice between implementations is OK, I think, as long
>> as you don't go hog-wild and end up with 52 different interrupt controller
>> designs on equal footing.
>>
>> I want to point out that, in practice, while it appears that SiFive has a
>> perhaps disproportionate amount of influence on the standards setting
>> process today, they have so far done nothing to dissuade me or anyone else

>> on these lists from *not* pursuing their own unique designs or

>> contributions. In fact, just the opposite! I've received encouragement
>> from Andrew Waterman, Chris Celio, Krste, and many others. I haven't heard
>> from him in a long while, but I remember that even Yunsup Lee was always
>> eager to hear my unique interpretation of RISC-V. They don't need to agree
>> with me, but having received support and understanding was very encouraging
>> nonetheless.
>>
>> Case in point, I just completed the implementation of my next processor's
>> CSR Unit. It's a total chungus, weighing in at 1157 iCE40HX8K logic
>> cells. That's enormous! It's mostly due, I think, to the WPRI logic, and
>> perhaps a not so efficient way of exposing the registers to the rest of the
>> processor logic. Also, 64-bit event counters, even when the high bits are
>> almost never going to be used. I can save a lot of space on my design if I
>> turn all WPRI fields to WIRI or WARL fields instead, and shave half the
>> number of bits off the event counters. Ideally, I'd like to not have to do
>> this; but, I will if I can't fit my design onto the FPGA. I will no longer
>> be compliant with the 1.10 privilege specifications; but I'm willing to bet
>> I'll be compatible "enough" that most software will work on it. Absolutely
>> nobody has yet told me not to do this.
>>
>> So, if PLIC isn't to your tastes for some reason, and you're willing to
>> invest the effort to make your part work with Linux (or whatever), I do not
>> foresee any retributive action coming from SiFive at all. Given all this
>> round-about discussion, I suppose what I want to drive home is that I don't

>> think emotions here need to run particularly hot over this issue. *That
>> said,* keep on debating! It's how progress will be made, one way or

>> another.
>>
>> ________
>> 1. Forgive any poor spelling; I'm writing from memory.
>>
>>
>>
>>> I don’t really care about the PLIC, our designs will work either way, but
>>> the process seems pretty broken.
>>>
>>> The takeaway I got from this is that two of the long-time major open
>>> source stake holders in RISC-V, ETHZ and Cambridge, raised a concern and
>>> the response was super dismissive.
>>>
>>>
>>>> Krste
>>>>
>>>>
>>>>
>>>> > M
>>>>> >
>>>>>
>>>>> --
>>>> Apologies for all typos. Message sent by combination of an approximate
>>>> neural network and a smartphone.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "RISC-V HW Dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to hw-dev+un...@groups.riscv.org.
>>>> To post to this group, send email to hw-...@groups.riscv.org.
>>>> Visit this group at
>>>> https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3P37hBvyrfjSK_XqbFs40jKUByWwfrmrPn8LuY6p-EhA%40mail.gmail.com

>>>> <https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3P37hBvyrfjSK_XqbFs40jKUByWwfrmrPn8LuY6p-EhA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .

>>>>
>>>>
>>>> --
>>> Apologies for all typos. Message sent by combination of an approximate
>>> neural network and a smartphone.
>>>
>>
>>
>> --
>> Samuel A. Falvo II
>>
>

> --
> You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
> To post to this group, send email to hw-...@groups.riscv.org.
> Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/CAKoatx3ZfuZKCP-h3ysy%2ByqiSerxyXtdGX8Ea_jLN6wVXhV6Jg%40mail.gmail.com.

[Chang, Abner (HPS SW/FW Technologist)]

Stand at the viewpoint of RISC-V PC/Server system, I am happy to see PLIC is moved to platform spec from RISC-V privilege spec, and UNIX-class platform WG is working on standardizing it. To standardize PLIC in platform spec helps lot to have RISC-V based PC/Server systems and boot to single (unified) OS image. I am one of those who need these standardizing works happen in Unix-Platform WG because I am working on standardizing whatever we need for RISC-V PC/server system. It will be hard to standardize something like PLIC in PC/Server spec for every H/W vendors and each of PLIC has different architecture. This is not saying SiFive PLIC is the only one choice for UNIX-class platform, however, that SiFive PLIC is what I can see on the table so far which is flexible and mature to be part of PC/Server spec, says ACPI spec. Instead of saying to standardize SiFive PLIC, I think it is more like to standardize the operation parameters of platform interrupt controller in RISC-V platform spec and SiFive PLIC is a reference for building up operation parameters in spec.
Not just only PLIC, we also need to standardize CLIC (or CLINT), RTC, Reset mechanism, etc. in UNIX-class platform spec (maybe a specific section for PC/Server platform).

Abner

>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_groups.riscv.org_group_hw-2Ddev_&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4vmYa9iv_hGZ5pxSd-V4eRH5JW3vWlM_UhGSGvsqo&s=U-6RyGZNy_OZagtJ5kCyWVgd5xfhcLbcxGmmLbGuoyY&e=.

>>>> To view this discussion on the web visit

>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.
>>>> com_a_groups.riscv.org_d_msgid_hw-2Ddev_CAKoatx3P37hBvyrfjSK-5FXqbF
>>>> s40jKUByWwfrmrPn8LuY6p-2DEhA-2540mail.gmail.com&d=DwIFaQ&c=C5b8zRQO
>>>> 1miGmBeVZ2LFWg&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4v
>>>> mYa9iv_hGZ5pxSd-V4eRH5JW3vWlM_UhGSGvsqo&s=WpbPF1ov7dPQmpyat3TrxAzqb
>>>> rio_o9eSVhVgHs9PrQ&e=
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google
>>>> .com_a_groups.riscv.org_d_msgid_hw-2Ddev_CAKoatx3P37hBvyrfjSK-5FXqb
>>>> Fs40jKUByWwfrmrPn8LuY6p-2DEhA-2540mail.gmail.com-3Futm-5Fmedium-3De
>>>> mail-26utm-5Fsource-3Dfooter&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=_S
>>>> N6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4vmYa9iv_hGZ5pxSd-V4e
>>>> RH5JW3vWlM_UhGSGvsqo&s=3ZY1D8zZj17VyIHa8UpLqcv37oNeGNPvfbxK6s7o8yQ&
>>>> e=>

>>>> .
>>>>
>>>>
>>>> --
>>> Apologies for all typos. Message sent by combination of an
>>> approximate neural network and a smartphone.
>>>
>>
>>
>> --
>> Samuel A. Falvo II
>>
>
> --
> You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
> To post to this group, send email to hw-...@groups.riscv.org.

> Visit this group at https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_groups.riscv.org_group_hw-2Ddev_&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4vmYa9iv_hGZ5pxSd-V4eRH5JW3vWlM_UhGSGvsqo&s=U-6RyGZNy_OZagtJ5kCyWVgd5xfhcLbcxGmmLbGuoyY&e=.
> To view this discussion on the web visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_groups.riscv.org_d_msgid_hw-2Ddev_CAKoatx3ZfuZKCP-2Dh3ysy-252ByqiSerxyXtdGX8Ea-5FjLN6wVXhV6Jg-2540mail.gmail.com&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4vmYa9iv_hGZ5pxSd-V4eRH5JW3vWlM_UhGSGvsqo&s=NSVT5vNWL2Fbg65fdUSeMZFqH25qOsE6fOU7PekGFus&e=.

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.

Visit this group at https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_groups.riscv.org_group_hw-2Ddev_&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4vmYa9iv_hGZ5pxSd-V4eRH5JW3vWlM_UhGSGvsqo&s=U-6RyGZNy_OZagtJ5kCyWVgd5xfhcLbcxGmmLbGuoyY&e=.
To view this discussion on the web visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_groups.riscv.org_d_msgid_hw-2Ddev_mhng-2Dd057b339-2D135e-2D4692-2Db71a-2D8690467c3831-2540palmer-2Dsi-2Dx1c4&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=6X4vmYa9iv_hGZ5pxSd-V4eRH5JW3vWlM_UhGSGvsqo&s=Qyqgzb0mtQWbFzEVKnQdx813RAwD0__a9SFbkBBu2hA&e=.

[Dom Rizzo]

Voting power isn’t the same as reasonable consensus among all interested parties.

So long as you’re relying on the labor of the open source community, in all its various domains, it’s worth considering their input.

I don’t have a dog in this fight, but there’s some clear contention here. Maybe it would be good to split off a working session in Zurich to talk through both the concerns about spec consensus and the PLIC in particular. I suspect a lot of the interests parties will be in attendance.

-d

> Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.
> To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/TU4PR8401MB0958FDF482CC270E3BD23FB9FF2E0%40TU4PR8401MB0958.NAMPRD84.PROD.OUTLOOK.COM.

[David Chisnall]

On 09/04/2019 16:35, Prof. Michael Taylor wrote:
> I didn’t say that PCIE devices do not have it, I said that the support
> came from the need to support legacy X86 (I.e. ISA, PCI, ...) , and now
> that the support is there, it will get used and then be necessary for
> all future generations of hardware, just as will happen with RISC-V as a
> result of the PLIC.

The complexity has to go somewhere. If you want to have a memory-mapped
FIFO, then you end up with memory regions where a read pulls a value
from the FIFO. If you don't do this, then you have one of two options:

1. Move the complexity from the CPU into software and have a separate
read and advance operation for the FIFO, so every read is now a load and
a store to advance to the next value.

2. Move the complexity from the CPU to the device and provide a
memory-mapped ring buffer (or a ring buffer in host memory as a DMA
target) instead of a single FIFO, with explicit stores to notify the
device that it can overwrite parts of the buffer.

All three options require complexity somewhere. Generally, you have
more devices and more drivers than you have CPUs, so requiring the
complexity to be in the CPU makes sense.

A simple UART and its driver is trivial to implement if your CPU
guarantees that reads to uncached regions are not speculatively
executed. This is very important for OS bringup, where you want to get
a console UART working very early in the boot for logging error
messages, typically before you've initialised the virtual memory
subsystem, let alone bus enumeration.

For more complex devices, if you're doing DMA anyway then having a
control ring that is DMA'd and a producer and consumer counter as
memory-mapped registers doesn't add much because you're already
implementing most of the complexity.

You could define a CPU that can speculatively execute uncached loads and
provide a shim device that lets you read from a side-effecting register
and inserts a caching layer so that you need to explicitly advance to
the next read, but that would add complexity to both the device driver
and to your chip.

Most operating systems use the same drivers for peripherals across
architectures. If you require devices to be used via a shim that
changes their memory-mapped control interfaces, then this will add a
significant barrier to porting.

This is even worse for things like GPUs or anything that supports SRIOV,
where the device can be exposed directly to userspace, so arbitrary bits
of userspace code can end up depending on specific layouts.

David

[高野茂幸]

Hi,

I want to understand about user ISA and supervisor ISA on RISC-V.

Is user ISA needed interrupt standard? Or is It needed only for supervisor ISA?

Or, there is no user and supervisor ISAs? as my misunderstanding.

And can we make a matrix table, not only dimension of modules of ISA, but also dimension of a class for such the implementation matters, if this does not introduce complexity and not damage to community not only now but also future?

Best,

S.Takano

2019年4月10日(水) 17:41 David Chisnall <David.C...@cl.cam.ac.uk>:

--
You received this message because you are subscribed to the Google Groups "RISC-V HW Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hw-dev+un...@groups.riscv.org.
To post to this group, send email to hw-...@groups.riscv.org.
Visit this group at https://groups.google.com/a/groups.riscv.org/group/hw-dev/.

To view this discussion on the web visit https://groups.google.com/a/groups.riscv.org/d/msgid/hw-dev/41adef0b-d91b-75d5-8ec1-bbef68c40b1e%40cl.cam.ac.uk.

[高野茂幸]

My worry for such classification is that after introducing classes, someone begin to say “I want subclass of the class!”, continues this routine and finally makes a divergence of community.

Best,

S.Takano

2019年4月10日(水) 19:21 高野茂幸 <adaptive...@gmail.com>: