Caching, I/O Space memory mapping and DMA
Arun Prasad
I'm having some serious of doubts on Caching, I/O Space memory
mapping and DMA data transfer on IBM PPC405 platform. Can anyone
clarify these doubts?
Do we need to invalidate the cache irrespective of our Card is
whether memory mapped or i/o mapped while accessing device registers?
And one more doubt.
Whether the memory space which we are getting after configuring
(pciAutoConfig) the PCI card (for memory mapped devices ) for I/O
space is a cacheable or non-cacheable one?
I think, it will not be a cacheable one as this memory is shared by
the Host
CPU as well as by the Card. Please do clarify me on this regard.
If we directly access the device registers (sysInbyte()
sysOutByte()), do we
need to invalidate the cache as we are accessing HW registers?
And when we need to do cache flush? (within ISR??? or in some other
places?)
One more thing,
Do we need to take care of Cache validate/invalidate if we
do not handle DMA data transfer???
Thanks in advance and have a nice time.
Regards
Arun
Note : I'm writing driver for a ethernet card on IBM PPC405
Michael Lawnick
news:e38d912c.02010...@posting.google.com...
Hi Arun,
Caching is independand from I/O- or memory mapped PCI space. If you haven't
set an address space as uncachable that is used by two masters, you need
flush/invalidate. Nevertheless I recommend to set all PCI-space to
uncachable.
For CPU-owned memory, accesses by external devices (aka DMA) there are two
ways:
- enable MMU and do a cacheDmaMalloc. This sets the allocated memory
uncachable.
- do cacheFlush before notifying the device of buffer changes (send) and a
cacheInvalidate before access after notification (aka txInterrupt) by
device. CAVE AT: buffers may not overlap in cache on concurrent accesses .
If your NIC has local memory, where it buffers data and CPU must actively
fetch it, you shouldn't have any cache troubles when disabling cache for PCI
address space of your NIC.
HTH
--
Mit freundlichen Grüßen,
Michael Lawnick
==============================================
SOFTEC GmbH Tel +49-731-96600-0
Promenade 17 Fax +49-731-96600-23
D-89073 Ulm Michael Lawnick
Germany law...@softec.de
==============================================
Dave Korn
>Hi,
>
> I'm having some serious of doubts on Caching, I/O Space memory
>mapping and DMA data transfer on IBM PPC405 platform. Can anyone
>clarify these doubts?
Hi Arun,
Your main confusion is because you are thinking of the PPC as if it was an
x86 chip. It isn't, and one big difference is that there is no separate I/O
space and memory space on
PPC: everything is mapped to memory space, and there is no I/O port space.
> Do we need to invalidate the cache irrespective of our Card is
>whether memory mapped or i/o mapped while accessing device registers?
Every card is memory mapped in PPC. So if the registers are mapped to
cacheable address space, you would have to flush the cache when writing to
them, and invalidate it before reading from them. This is tricky, which is
why it is much preferable to map device registers into non-cacheable address
space.
> Whether the memory space which we are getting after configuring
>(pciAutoConfig) the PCI card (for memory mapped devices ) for I/O
>space is a cacheable or non-cacheable one?
> I think, it will not be a cacheable one as this memory is shared by
>the Host
>CPU as well as by the Card. Please do clarify me on this regard.
PCI space should indeed always be non-cacheable. But the system cannot do
this for you. You must set up the BATs or PTEs (see manual re: sysBatDesc
and sysPhysMemDesc) that cover the address range used by the PCI bus to have
the G (guarded) and I (cache-inhibited) flags. Once you have done that,
every card will be non-cacheable no matter where in PCI space the system
decides to auto config it.
> If we directly access the device registers (sysInbyte()
>sysOutByte()), do we
>need to invalidate the cache as we are accessing HW registers?
These functions don't exist on ppc! *Everything* is memory mapped, so you
just set a pointer to point to it and read/write by using "*ptr", the same
as any other memory location. The location itself should be non-cacheable
for the device registers, but if for some reason you went crazy and wanted
to cache the entire PCI bus address space, then yes, you would need to flush
the cache before a write or invalidate it before a read.
> One more thing,
> Do we need to take care of Cache validate/invalidate if we
>do not handle DMA data transfer???
No: assuming the PCI bus address space is properly set up as
non-cacheable, then reading and writing the registers will be ok; and if the
device does not alter system RAM, then there is no need to worry about the
caches.
However I am very surprised to hear that you have an ethernet card that
has no DMA. How does the data get into the CPU's memory? Do you have to
read the data packet from the ethernet chip itself using the cpu?
> And when we need to do cache flush? (within ISR??? or in some other
>places?)
You can flush the cache any time you like - see the documentation for
cacheFlush - but the best time to do so is immediately before beginning to
write the commands to the device registers that would cause it to access the
system RAM.
DaveK
--
moderator of
alt.talk.rec.soc.biz.news.comp.humanities.meow.misc.moderated.meow
Burn your ID card! http://www.optional-identity.org.uk/
Help support the campaign, copy this into your .sig!
Proud Member of the Exclusive "I have been plonked by Davee because he
thinks I'm interesting" List
Member #<insert number here>
Arun Prasad
Thank you for your detailed reply/clarifications.
<<Your main confusion is because you are thinking of the PPC as if it
was an x86 chip.
>>Yes. You are absolutely right. I DO NOT HAVE EXPERIENCE other than
x86, though I READ a bit on Z80 and Motorola 68000 series chips.
<<It isn't, and one big difference is that there is no separate I/O
space and memory space on PPC: everything is mapped to memory space,
and there is no I/O port space.
<<Every card is memory mapped in PPC.
>>Is it so? I thought it's device/controller/card specific. If my
memory is correct, I've read from Goankar's Microprocessor book that
it is device-specific. But again that book dealt with 8085 and 8086
processors. And previously I worked on Linux Driver for SCSI Raid
Controllers with i960 processors in them and I found I/O mapped and
memory mapped cards separately and accordingly all the device access
functions varied.
It raises one more question from me. Hope you can/will answer that
too.
What happens if I have a ethernet controller which has a processor
like intel's i960 (Hope this supports I/O mapped I/O) which supports
I/O mapped I/O and if that controller is fixed on PPC system?
Will PPC still map the device in to it's memory?
<< PCI space should indeed always be non-cacheable. But the system
cannot do this for you. You must set up the BATs or PTEs (see manual
re: sysBatDesc and sysPhysMemDesc) that cover the address range used
by the PCI bus to have the G (guarded) and I (cache-inhibited) flags.
Once you have done that, every card will be non-cacheable no matter
where in PCI space the system decides to auto config it.
>>Oh. Thanks for this information. But I haven't found sysBatDesc. But
I was able to find sysPhysMemDesc. Here I'm giving the snap shot of
it.
(void *) PCI_MEMORY_START,
(void *) PCI_MEMORY_START,
PCI_MEMORY_MAP_END - PCI_MEMORY_START + 1,
VM_STATE_MASK_VALID | VM_STATE_MASK_WRITABLE |
VM_STATE_MASK_CACHEABLE |
VM_STATE_MASK_GUARDED,
VM_STATE_VALID | VM_STATE_WRITABLE |
VM_STATE_CACHEABLE_NOT |
VM_STATE_GUARDED
Can you write me on how to interpret this? Because there are two
entries are there two records are there. In this example, there are
two PCI_MEMORY_START are there and correspondingly two different sets
of flags are associated with them, one with Guarded and and non
cacheable and the other one with cacheable and unguarded. I don't know
why there are two different entries are there. Can you let me know
this?
<<> > If we directly access the device registers (sysInbyte()
sysOutByte()), do we need to invalidate the cache as we are accessing
HW registers?
<<These functions don't exist on ppc!
>>Oops... This is confusing again.
From the file /* sysLib.c - IBM 405GP Walnut system-dependent routines
*/
I could able to find sysInByte() and sysOutByte() functions and
similar PCI access routines to read/write one byte/Word/Long.
But within the definitions sysInByte() and sysOutByte() everything
treated as memory mapped only.I could able to see simple assignment
statements and EIEIO_SYNC macros only.
I guess these are only wrapper functions which simulates I/O mapped
IO. Please do clarify me if I'm wong.
<< However I am very surprised to hear that you have an ethernet card
that has no DMA.
>>I'm sorry for the confusion. Our Card has its own DMA controller and
its own Buffer/memory.
<<but the best time to do so is immediately before beginning to write
the commands to the device registers that would cause it to access the
system RAM.
>>I've made a note of it.
I have one more question. This is nothing to do with VxWorks or PPC.
Just having a curiosity to know whether the e-mail ID you have given
(<no....@my.mailbox.invalid>) is a valid one? If I send a mail, will
it reach you?
Thank you so much for your reply.
I would also like to thank Michael Lawnick (Law...@softec.de) for
his reply mail.
<<If your NIC has local memory, where it buffers data and CPU must
actively
fetch it, you shouldn't have any cache troubles when disabling cache
for PCI
address space of your NIC.
>>Oh. I. See. Yes. This is the scenario with our controller (It has
its own memory and it buffers.)
Regards
Arun
--
Arun Prasad
Software Engineer
Adaptec India Pvt. Limited
(A wholly owned subsidiary of Adaptec Inc., USA)
6-3-1086, IV Floor, Vista Grand Towers,
Rajbhavan Road, Somajiguda,
Hydereabad - 500082
Phone: (O):91-40-6661555/56/57/58
(Ext):234
Email: arun_...@adaptecin.com
"Dave Korn" <no....@my.mailbox.invalid> wrote in message news:<8bG_7.327$qS6.7...@newsr2.u-net.net>...
Michael Lawnick
as you posted your direct EMails with Dave here, I assume you want
different/additional opinions to your question. Here my 2 Cents:
As mentioned correctly by Dave, PPC always see's PCI as memory mapped, but
this does not mean, that a PPC can't do PCI-I/O. It means that there is a
memory space, where the PCI-bridge of PPC will translate all accesses into
I/O-accesses and there are other spaces, that are interpreted in other ways.
Your excerpt from sysPhysMemDesc is just _one_ entry and its correct:
VM_STATE_MASK_VALID | VM_STATE_MASK_WRITABLE |VM_STATE_MASK_CACHEABLE
|VM_STATE_MASK_GUARDED
tells which flags you want to change (_MASK_)
VM_STATE_VALID | VM_STATE_WRITABLE | VM_STATE_CACHEABLE_NOT
|VM_STATE_GUARDED
enables the entry and sets it writable, uncached and guarded - just as you
need it.
You write:
> >>I'm sorry for the confusion. Our Card has its own DMA controller and
> its own Buffer/memory.
This seems to be suspicious - any confusion possible ?
Either DMA or local buffer, both would be redundant (surely still possible,
but who wants to pay?)
On our PPC405, our scndNIC 's registers have to be set up through PCI, data
is written to CPU-memory via DMA (by the NIC). As we have shut off MMU, I
must always do invalidate before data reading and flush before telling the
NIC that there's a new packet to transmit.
If our MMU would be enabled, I could allocate memory with cacheDmaMalloc and
wouldn't need to care.
To complete your confusion about sysInByte, sysOutByte, ..., there are more
commands:
sysPciInByte, sysPciOutByte,...
The trick is:
the sysInXXX variant will write data as is, typically found on Intel arch.
sysPciInXXX will make a swap before access. This is needed, as PPC is
big-endian, but PCI is little-endian. Second issue are the instruction pipes
of PPC. As you might do two successive accesses to different PCI-registers,
but the result of the second one might be influenced by the first (e.g.
first enable device via control register, then read from status register),
it must be ensured, that the first instruction completed before the second.
This is done by a sync instruction as you might see in sysALib.s
--
Mit freundlichen Grüßen,
Michael Lawnick
==============================================
SOFTEC GmbH Tel +49-731-96600-0
Promenade 17 Fax +49-731-96600-23
D-89073 Ulm Michael Lawnick
Germany law...@softec.de
==============================================
"Arun Prasad" <arun4v...@indiatimes.com> schrieb im Newsbeitrag
news:e38d912c.02010...@posting.google.com...
Arun Prasad
Thank you for your comments/reply/clarifications.
<<as you posted your direct EMails with Dave here, I assume you want
different/additional opinions to your question. Here my 2 Cents:
>>direct EMail? No. It's not.
And yes, I am looking for additional/different opinions for my
questions/doubts. And sorry for replying at this much delay.
<<Your excerpt from sysPhysMemDesc is just _one_ entry and its
correct:
>>And now I got how to interpret it. Thanks.
<< This seems to be suspicious - any confusion possible ?
Either DMA or local buffer, both would be redundant (surely still
possible, but who wants to pay?)
>>No. that's correct only. Our ethernet card does some other
functionalities also and its bit costly too.
<<If our MMU would be enabled, I could allocate memory with
cacheDmaMalloc and wouldn't need to care.
>>I. See. We are enabling the MMU and we use cacheDmaMalloc only. So
one of our problem (cache validation/flush) solved. Is it?
And what happens if we write code with cacheDmaMalloc and if we
disable MMU?
And still I have a doubt on I/O access routines (sysInXXX and
sysOutXXX).
We use IBM PPC405GP which is Big Endian and our ethernet NIC card is
PCI based (as you know PCI is little- Endian). As I already mentioned
I find only simple assignment statements and a EIOIOSYNC macro in
sysInByte and sysOutByte routines like this
UCHAR data = (*(UCHAR *)address);
EIEIO_SYNC;
return data;
Here if I change the assignment statement like the following one for
sysInDWord ()
ULONG data = (*(ULONG *) address);
Will it work?
This is where we feel, we have to swap the bytes before making
assignments to make it Endian-safe. (I mean endian conversion required
here.)
Hoping for a clarification on this Endian Issue.
Have a nice time.
Regards
Arun
"Michael Lawnick" <Law...@softec.de> wrote in message news:<a1m65n$1ot$1...@snoopy.bndlg.de>...
Michael Lawnick
> >>I. See. We are enabling the MMU and we use cacheDmaMalloc only. So
> one of our problem (cache validation/flush) solved. Is it?
>
> And what happens if we write code with cacheDmaMalloc and if we
> disable MMU?
look into docs/vxworks/ref/cacheLib.html how to write a drive that should
work with all cache/MMU settings.
> ULONG data = (*(ULONG *) address);
>
> Will it work?
>
No, use sysPciOutLong(), defined in sysALib.s It will do the swapping within
the write command.
HTH
--
Mit freundlichen Grüßen,
Michael Lawnick
==============================================
SOFTEC GmbH Tel +49-731-96600-0
Promenade 17 Fax +49-731-96600-23
D-89073 Ulm Michael Lawnick
Germany law...@softec.de
==============================================
"Arun Prasad" <arun4v...@indiatimes.com> schrieb im Newsbeitrag
news:e38d912c.02012...@posting.google.com...