howto use ioremap_wc?
Brice Goglin
We're looking at using ioremap_wc() in myri10ge. No drivers seem to be
using it yet, so I'd like to get some clarification regarding ioremap_wc
failures, MTRR and so on.
What we currently do is mtrr_add() and then ioremap. Depending on the
mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to
submit requests to the NIC. How are we supposed to switch this to
ioremap_wc?
Are we sure that if the arch supports _wc, it does not return success
when the underlying plain ioremap worked but setting up _wc() failed? If
so, why does it revert to ioremap_nocache when PAT isn't enabled?
Can we keep the mtrr_add() in the regular path? Or are we supposed to
drop it when the arch provides ioremap_wc and it did not fail?
thanks,
Brice
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Arjan van de Ven
Brice Goglin <br...@myri.com> wrote:
> Hello,
>
> We're looking at using ioremap_wc() in myri10ge. No drivers seem to be
> using it yet, so I'd like to get some clarification regarding
> ioremap_wc failures, MTRR and so on.
>
> What we currently do is mtrr_add() and then ioremap. Depending on the
> mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to
> submit requests to the NIC.
Ok this leads to a question: since write combining is effectively an
extension (eg relaxation) to uncached, how much do you care if you
actually get uncached? Eg can you just use the "WC" function even for
the case where you get an uncached mapping ?
> How are we supposed to switch this to
> ioremap_wc?
>
> Are we sure that if the arch supports _wc, it does not return success
> when the underlying plain ioremap worked but setting up _wc() failed?
why would it not return success? You asked for a mapping with a set of
guarantees, and you got something that adheres to those guarantees (and
then some ;(...
Would you expect ioremap_cache() to fail if you couldn't get a cachable
mapping? That would suck (hint: on PC's it will then fail 99.9% of the
time)....
> If so, why does it revert to ioremap_nocache when PAT isn't enabled?
Actually it would make sense for the ioremap_wc() implementation to try
to add an mtrr I suppose... it's better to be done there than trying to
do it in some driver....
--
If you want to reach me at my work email, use ar...@linux.intel.com
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
Brice Goglin
> Brice Goglin <br...@myri.com> wrote
>> We're looking at using ioremap_wc() in myri10ge. No drivers seem to be
>> using it yet, so I'd like to get some clarification regarding
>> ioremap_wc failures, MTRR and so on.
>>
>> What we currently do is mtrr_add() and then ioremap. Depending on the
>> mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to
>> submit requests to the NIC.
>>
>
> Ok this leads to a question: since write combining is effectively an
> extension (eg relaxation) to uncached, how much do you care if you
> actually get uncached? Eg can you just use the "WC" function even for
> the case where you get an uncached mapping ?
>
WC is strictly required for our "wcfifo" path, but this path is actually
not so important nowadays. It is disabled by default and might even be
removed in the future. So, no, myri10ge itself does not really need to
know whether the mapping is actually WC.
The only case I see where it would be helpful is to avoid calling
mtrr_add when WC got enabled through PAT. See below.
> Actually it would make sense for the ioremap_wc() implementation to try
> to add an mtrr I suppose... it's better to be done there than trying to
> do it in some driver....
>
Agreed, that would be nice!
From what we discussed here after your reply, our plan is now to just
replace
ioremap+mtrr_add
with
ioremap_wc+mtrr_add
If mtrr_add() is moved into ioremap_wc(), we'll remove it from myri10ge.
For now, when PAT is enabled, we may have PAT + MTRR both doing WC, but
I don't think it can break anything, right?
thanks,
Brice
Loic Prylli
> Arjan van de Ven wrote:
>
>> Ok this leads to a question: since write combining is effectively an
>> extension (eg relaxation) to uncached, how much do you care if you
>> actually get uncached? Eg can you just use the "WC" function even for
>> the case where you get an uncached mapping ?
>>
>>
>
> WC is strictly required for our "wcfifo" path,
To be more accurate about the above statement, that codepath will still
work correctly even if the mapping ends-up being uncached, but that
could lead to a 16X slowdown on some machines, since that path was
really designed for the WC case.
Anyway that codepath does not really matter as Brice mentioned
afterwards :-)
> but this path is actually
> not so important nowadays. It is disabled by default and might even be
> removed in the future. So, no, myri10ge itself does not really need to
> know whether the mapping is actually WC.
>
Loic
Arjan van de Ven
Brice Goglin <br...@myri.com> wrote:
>
> Agreed, that would be nice!
>
> >From what we discussed here after your reply, our plan is now to just
> replace
> ioremap+mtrr_add
which doesn't work as ioremap() may/will force uncached ;)
> with
> ioremap_wc+mtrr_add
>
> If mtrr_add() is moved into ioremap_wc(), we'll remove it from
> myri10ge. For now, when PAT is enabled, we may have PAT + MTRR both
> doing WC, but I don't think it can break anything, right?
Both having WC is nicely consistent and the right thing happens...
--
If you want to reach me at my work email, use ar...@linux.intel.com
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
Roland Dreier
PAT. Are there any plans for making pgprot_writecombine() a generic
interface that portable drivers can use to map PCI memory into userspace
with WC turned on? (Falling back to uncached when WC is not available
for whatever reason is fine for the case I have in mind)
Maybe it's better to ask at a higher level: what's the right thing for a
driver to do in an mmap method when it wants to expose PCI memory to
userspace with WC turned on if possible? I see that the PCI sysfs code
on x86 does it using non-exported (hence not available to a driver
buildable as a module) interfaces.
Thanks,
Roland