Question: Cached vs. Non-cached sharedMemory in bus-master NDIS miniport.
Calvin Guan
DDK says
Cache Considerations for Receiving Data
1) If the NIC managed by the miniport driver receives consecutive packets
into contiguous memory, the end of one packet can occupy the same cache
block as the start of the next packet. Therefore, the miniport driver must
allocate noncached memory because reading the first packet can cause stale
cache data for the start of the second one.
2) If releasing the buffers involves writing to them. For example, the
receive buffer could be contiguous with a header containing an ownership bit
that indicates whether the buffer belongs to the adapter or to the miniport
driver. In this case, the adapter may write to that bit before releasing the
buffer. A miniport driver that uses this technique must allocate these
buffers from noncached memory.
The first statement seems to avoid unwanted "cache read hit" I can
understand it.
I don't understand the second statement. My NIC falls into this category.
Why I need noncached memory in this case? What is it trying to avoid?A cache
write hit? If I add padding between header/buffer and it's guaranteed they
are not in a cache line, can I use cached memory? I'm sure that the
onwership bit is written to the sharedmem not the processor cache. It
doesn't seem to break anything in my NIC. NDIStest is pretty happy with it.
Thanks,
Calvin
Stephan Wolf
unwanted cache read or write hits.
Please keep in mind that there are many system architectures on which
Windows could run. I know there are none left except x86 for NT but
for instance consider looking at Windows CE.
Anyway. On x86, there is no need to take care of any caching issues at
all because x86 always guarantees cache coherency. That is, the
software does not have to explicitly flush the cache of any memory
areas before passing "ownership" of a memory area to a DMA busmaster.
That is probably the reason why everthing works ok in your case.
But! If your driver would run on some other system like MIPS, ALPHA,
PPC, etc., then you would have to take care of caching issues because
there is no built-in hardware cache coherency on these systems.
Trying to shed some light on what you found in the DDK docs:
First, please note that you need to call NdisFlushBuffer() for any
NDIS_BUFFER *before* you pass its "ownership" to your adapter's DMA
busmaster engine. That is true for *both* send and receive buffers.
1) If the adapter writes receive data to a buffer and you read that
data from software, i.e. the CPU, the cache is filled with that data.
Thus, any *further* writes by the adapter to the memory immediately
behind the buffer will not be available to the software if that memory
shares the same cache line.
1. flush cache
2. adapter writes memory[0..100]
3. CPU reads memory[0..100] => cache lines for memory[0..100 + X]
filled!
4. adapter writes memory[101..200]
5. CPU reads memory[101..200] => memory[101..n] *not* read from memory
but served from cache!
Note: "X" depends on cache line size and buffer alignment
2) If you need to poll the "own" bit of a buffer (usually in some
descriptor) by software, then the actual memory is only read once by
the CPU because all consecutive reads will be served from cache.
That of course means that when the adapter writes the "own" bit, the
software will never see that change because it only "sees" the cache
and not the real memory.
Again, there is no problem in x86 because any write to real memory by
some busmaster *automatically* flushes the according cache lines.
AFAIK, NdisFlushBuffer() and NdisMUpdateSharedMemory() are actually
no-ops on x86.
Stephan
---
Stephan Wolf
of "flush cache".
Stephan
Calvin Guan
Hi Stephan
Thanks for your detailed explanation.
Can I assume that, on any platforms not just i386, it's always safe to
allocate non-cached memory for descriptors and cached memory for traffic
buffer(buffers to hold the rx ethernet frame)? This assumption is based on
the following facts:
1)NIC polls the descriptor to determine if a descriptor and its associated
traffic buffer are available to receive frame. Also, NIC writes
something(owner bit, numOfBytes received, RxFrameStatus) to the descriptor
*after* an incoming frame was loaded into the associated traffic buffer.
2)Upon reception, CPU checks the descriptor's owner bit and RxSize to see if
an RxFrame is ready to indicate and how big is the frame.
3)Upon ReturnPacket, CPU writes to descriptor to clear the RxStatus and
returns the ownership to NIC.
DDK provides NdisFlushBuffer and NdisMUpdateSharedMemory to deal with the
cache coherency in NDIS environment. It seems to me NdisFlushBuffer can make
sure the cache coherency for *traffic* buffer only. As for
NdisMUpdateSharedMemory, here's what DDK said:
NdisMUpdateSharedMemory ensures that data, just transferred from a
bus-master NIC, to be *read* from driver-allocated shared memory is current.
My understanding is that it ensures the CPU-bound data is cache-coherent. I
would assume it does NOT write the modified cache line to the shared memory
if a cache write-back policy is in force. That means, if CPU writes to a
cached descriptor for whatever purposes, the dirty cache line for the
descriptor will not go to the shared memory until the cache line was
evicted, explicitly invalidated, or retired, therefore, NIC does NOT see the
changes in descriptor at the right time.
Similarly, all reads from the cached descriptor are served from the cache
until a read miss/cache line fill happens. But NdisMUpdateSharedMemory seems
to help in this case(CPU-bound).
As for traffic buffers, it's always safe to use cached memory if
NdisFlushBuffer is called at the right place and the right time.
Am I in the right path?
I'm just trying to fully understand the cache coherency issue and make my
Tx/Rx logic bullet-proof on any platforms NT might run.
> First, please note that you need to call NdisFlushBuffer() for any
> NDIS_BUFFER *before* you pass its "ownership" to your adapter's DMA
> busmaster engine. That is true for *both* send and receive buffers.
Can you shed me more light on this? why call NdisFlushBuffer() before
passing the ownership bit? In my case, NDIS_BUFFER points to the Rx traffic
Buffer which is not sharing the same cache line with it descriptor.
Again, thank you very much for your great helps since I developed my first
NDIS protocol driver.
cheers,
Calvin
Stephan Wolf
<ndisnew_antisp...@yahoo.com> wrote:
>Can I assume that, on any platforms not just i386, it's always safe to
>allocate non-cached memory for descriptors and cached memory for traffic
>buffer(buffers to hold the rx ethernet frame)?
Yes. Descriptors need to be in non-cached memory *if* cache coherency
is not guaranteed by the hardware (which *is* the case on x86).
Frame buffers can (and should) be in cached memory.
>DDK provides NdisFlushBuffer and NdisMUpdateSharedMemory to deal with the
>cache coherency in NDIS environment. It seems to me NdisFlushBuffer can make
>sure the cache coherency for *traffic* buffer only. As for
>NdisMUpdateSharedMemory, here's what DDK said:
>
>NdisMUpdateSharedMemory ensures that data, just transferred from a
>bus-master NIC, to be *read* from driver-allocated shared memory is current.
Umm, I just read the description of NdisFlushBuffer() and
NdisMUpdateSharedMemory() in the XP DDK docs and must say that the
description there is completely contradictory.
For instance, as you quoted from the docs, it is my understanding that
NdisMUpdateSharedMemory() is to be used on *receive* buffers only. But
the docs at several places say NdisMUpdateSharedMemory() should also
be called for *transmit* buffers.
Also, I am under the impression that the person who wrote the
description of when and how to use NdisFlushBuffer() and
NdisMUpdateSharedMemory() apparently didn't fully understand the
matter.
Ali: Can you please have someone review the whole caching/flushing
stuff in the docs.
All I can say is what I think is right:
NdisFlushBuffer()
=================
The 'WriteToDevice' parameter passed to NdisFlushBuffer() determines
what happens to the cache lines that map to the 'Buffer':
WriteToDevice = TRUE
The software (i.e. Miniport driver) has written data to the buffer and
wants that data to be actually present in host memory so the NIC can
read the data from there via DMA.
So in this case, the buffer's associated cache lines are *written* to
memory.
WriteToDevice = FALSE
The software is about to pass "ownership" of the buffer's memory area
to the hardware (i.e. the NIC) so that the NIC can write received
frame data to the buffer.
In this case, the cache lines are *invalidated*, i.e. any contents of
the cache lines mapping to the buffer are *discarded*.
Note that it is important to understand that NdisFlushBuffer() needs
to be called *before* the buffer is touched by either the software or
the hardware.
Note also that it is a fault to touch a receive buffer from software
before the NIC passes ownership of that buffer back to the driver.
NdisMUpdateSharedMemory()
=========================
Actually, the following is just my guess although I think I am right:
After the NIC has finished a DMA transfer, the data that got
transferred might still be buffered (not: "cached") somewhere in
hardware. For instance, PCI bridges usually have some internal FIFO
buffer.
Thus, to make sure all data has reached its destination when a
busmaster DMA transfer is complete from the NIC's point of view, the
software should call NdisMUpdateSharedMemory() on the memory before
touching that memory.
Note that this seems to imply NdisMUpdateSharedMemory() is only
necessary for DMA transfers from the NIC to host memory because for
the opposite transfer direction (host to NIC) the NIC of course knows
when it has finished the transfer and has read all data.
Again, all of the above (both flush and update) can be completely
ignored on x86 because of the built-in cache coherency. It is still
good practice to use it.
>As for traffic buffers, it's always safe to use cached memory if
>NdisFlushBuffer is called at the right place and the right time.
Yes, as I stated above.
HTH, Stephan
Alireza Dabagh [MS]
-thanks, ali
--
This posting is provided "AS IS" with no warranties, and confers no rights.
"Stephan Wolf" <ste...@hotmail.com> wrote in message
news:3e37b7b1...@news.t-online.de...