Large Arrays in DDR via PRU. Does prussdrv_map_extmem() always give contiguous physical addresses?

[Bill Gray]

Hi,

I've got PRU0 of my BBB pulling in 12 Bytes of data from a ADC with a sampling frequency of 31250hz.

I want to stash all of this data in an array that lives in DDR memory.  I would like to make the array 0x4000 bytes (16kB) large.  This would give me 1365 12 byte records and about 43ms of data to play with.

I'm currently using prussdrv_map_extmem() to hook me up with the DDR memory.  It provides me with a memory block that is 256kB, so I have plenty of room.

The question I have is... can I count on the physical address in this 256kB block of memory to be contiguous?

I'm pretty new to linux memory management, but I see that the "page size" is only 4kB... so I imagine that the virtual memory system might chop my 256kB block up into bits which might not necessarily have contiguous addresses.

If I can count on contiguous addresses, this radically simplifies the memory access code in the PRU.  If I can't... well, then I can't, and I've got to figure something else out.

When I run experiments I see that the physical addresses are contiguous, but are they guaranteed to be? or has this just been "lucky"?

Thanks,

Bill

[William Hermans]

When I run experiments I see that the physical addresses are contiguous, but are they guaranteed to be? or has this just been "lucky"?

If memory serves, I think they have to be. But, if as you say you've tested and they've come up contiguous already, then I'm very sure this can not change. Not unless they memory is somehow freed using free(), and I do not think that is possible . . .

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tyler Turcotte]

 I have been using the BeagleBone PRU and have been accessing and mapping memory this way and thus far it seems as though it is one contiguous address.  I am relatively new to Beaglebone so maybe one of the more experienced may want to step in on this one.

Once I have used prussdrv_get_phys_addr to get the physical address it seems to store to one contiguous memory and is limited to 8mb.

I believe it is allocated contiguous from uio_pruss as well.

[Charles Steinkuehler]

If you dig through the user-mode uio_pruss code, you will see that the
PRU itself, the PRU memory and control registers, and the large shared
buffer is all accessed via a mmap on the /dev/uio_pruss* device.

If you dig in the PRU kernel driver, you will see the dram shared
buffer is allocated and handled as DMA memory. The
dma_alloc_coherent() call used to allocate the memory grabs a
physically contiguous chunk, provides physical and virtual addresses,
and sets the page caching flags as needed for the architecture.

--
Charles Steinkuehler
cha...@steinkuehler.net

[Bill Gray]

Charles,

Thanks so much for a definitive answer to this question!!!

Perhaps even more than the answer, I am fascinated by the account you give of how you know this to be the case!

I would like to have a better understanding of kernel drivers and DMA memory.  As an educational exercise I would like dig through the PRU kernel driver and see this code for myself?

Could you tell me where I can find it?

Thank you very much.

Bill

--
For more options, visit http://beagleboard.org/discuss
---

You received this message because you are subscribed to a topic in the Google Groups "BeagleBoard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beagleboard/N47C7-WdsZE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beagleboard...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Bill Gray
Velkess
415 407 7356

[Soapy Smith]

Bill, I have been studying this myself, it is complex, and the learning curve looks to be pretty steep to a newbie embedded programmer.

Here's as clear of an explanation as I have been able to find:

http://processors.wiki.ti.com/index.php/PRU-ICSS_Remoteproc_and_RPMsg

I've got the PRU Cape and that combined with the Debian distro and version 4 kernel it is pretty easy to run the example code using the kernel driver.  Understanding what is going on is another story.  If you go to exploringbeaglebone.com and look in the chapter "Extra Content Linux Kernel Programming" there are 3 articles which guide you through the workings of kernel drivers.

Regards,

Greg

[William Hermans]

Could you tell me where I can find it?

http://processors.wiki.ti.com/index.php/PRU_Linux_Application_Loader_API_Guide

You can also read through the code . . . but I will mention the last time I was reading through the code, and reading up on this API, the amount of memory used is not limited to 8k or whatever is default as shipped.

Another interesting thing I realized, is that mmap() can map any register through /dev/mem/ so should be easy to access the PRU scratchpad from "userspace". Meaning, you still have the latency from Linux to memory, but PRU accesses should be much faster. Not to mention having the ability of "broadsiding" from one PRU to the other . . .granted, I'm having a hard time tryign to think of a usecase where that might be useful . . .

You received this message because you are subscribed to the Google Groups "BeagleBoard" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.

[William Hermans]

I would recommend staying away from remoteproc and rpmsg at least until it's out of experimental.

--

For more options, visit http://beagleboard.org/discuss
---

You received this message because you are subscribed to the Google Groups "BeagleBoard" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.

[John Syne]

RemoteProc and RPMSG work just fine. By experimental, it means the interface may change and currently TI are adding support for PRU/Ethernet which will mean some rework. What this means is that when you update to a newer kernel, you may have to change your code, but the change will probably be trivial. 

Regards,

John

[Soapy Smith]

remoteproc definitely has some bad behavior.  But what I am doing is just as experimental, and no motors (all data transfer) involved.

What is currently considered a reliable methodology for getting the most out of the PRUs?
Also, what about C versus assembler?  Will C alone suffice, or is there real need to do some assembler?
I've found that there are actually two different assemblers, the older pasm (apparently no longer supported), and the assembler part of clpru.
If you are just starting out in PRU work, should pasm even be considered?

Regards,

Greg

[William Hermans]

Hi Soapy,

Yeah John always seems to take things personally, or out of context. I have no problem what so ever with anyone using whatever they want. Including the OP. My comment were merely to point out that remoteproc / rpmsg are not finished, have known issues, and are a pain in the backside to initially get working.

So for someone using prussdrv, it is probably a bad idea to even start thinking about using remoteproc / rpmsg. Unless they're just experimenting . . . where then it could be a good learning experience I suppose. Me . . . I'd rather something were fully functional and well documented before I invest my time into it. remoteproc is neither of these.

[William Hermans]

remoteproc definitely has some bad behavior.  But what I am doing is just as experimental, and no motors (all data transfer) involved.

What is currently considered a reliable methodology for getting the most out of the PRUs?

Whatever works for you. Me, I opted for C in Linux of course, and ASM for the PRU's because it is the best documented. But I've experimented with it all including attempting remoteproc . . .

The assembler I decided to use was the one included with the am335x_pru_package. I have not really spend a lot of time writing PRU assembly yet, but was reading up on the registers, etc, and then got busy with other things. But I did start to understand the prussdrv driver, and API set . . .

Anyway, pick something, and go with it. If you ask me the old way of using the PRU is the best way to go right now. Later, when remoteproc / rpmsg is mainline, and perhaps well documented. Then I personally plan on giving it a more serious look. Right now, I consider it a toy . . .

[John Syne]

William, what are you talking about? Why would I take what you say personally? You make these blanket statements about a technology you say you don’t know how to use and recommend that everyone else not use this technology. If you want to stay with a dead technology, that is your call, but there is no reason for anyone to stay away from RemoteProc/RPMSG. Manufacturers other than TI have embraced this technology which open up a range of cores you can interact with, such as DSP, CortexM4, PRU, etc. Yes, I know you told me you have no interest in the x15 so this is probably not important to you and I respect that. 

Soapy, I use the drivers from Starterware and adapt them to work on the PRU, so I always use the C compiler. There is a github repo were several of the Starterware drivers have been ported to the PRU. This will save you a bunch of time. 

Regards,

John

[Soapy Smith]

Hi John, are you referring to the modules/drivers:

pruss_remoteproc.ko

virtio_rpmsg_bus.ko

rpmsg_pru.ko

The above plus some examples are already loaded and built in the latest "testing" images which use kernel 4.

pruss_remoteproc is showing up in lsmod without doing anything at all.  The Debian 8.3 based distro boots up with pruss_remoteproc inserted.

If you are referring to something different or in addition to the above, could you please post an example link to github?

Regards,

Greg

On Wednesday, February 10, 2016 at 6:57:37 PM UTC-5, john3909 wrote

[William Hermans]

William, what are you talking about? Why would I take what you say personally? You make these blanket statements about a technology you say you don’t know how to use and recommend that everyone else not use this technology. If you want to stay with a dead technology, that is your call, but there is no reason for anyone to stay away from RemoteProc/RPMSG. Manufacturers other than TI have embraced this technology which open up a range of cores you can interact with, such as DSP, CortexM4, PRU, etc. Yes, I know you told me you have no interest in the x15 so this is probably not important to you and I respect that.

So, using something consistent , that is well documented, and has been proven to work over the last several years does not make it "dead tech". It makes it something that actually works for many people. No one cares what TI adopts, except perhaps you, and TI. People care what works, with the least amount of resistance.

So hey lets put the squash on this right now. Why don't you write us some code in the next day or two that blinks the USR leds in some kind of pattern that proves the remoteproc / rpmsg is actually functional / usable. As no one cares if you can write 100 "hello world " messages into /var/log/messages . . . I can do that with a bash script and no PRU . . .

You do that, and I'll concede that remoteproc is at least semi useful.

[Soapy Smith]

http://processors.wiki.ti.com/index.php/PRU_Training:_Hands-on_Labs#LAB_6:_Blinking_LEDs_with_RPMsg_from_Linux_User_Space

Hi William, please see the above.  I have the PRU Cape, but I haven't got this far in the labs.  The other labs with remoteproc and other associated modules works so far.

Regards,

Greg

[ybea...@rehut.com]

Not to add add data points to the flames but -

That example code in that link works fine. Was able to use that sample code to
build a rpmsg simulator for other hw using the PRU. The PRU firmware creates 2
rpmsg channels that gets messages sent between then. Setup is:
Kernel driver A is a I2C driver attaches to one rpmsg channel. It reads data
from the I2C channel and forwards it to the PRU using that rpmsg channel.

Kernel driver B is rpmsg/IIO driver that attaches to the other rpmsg channel
and sends out IIO data based on the rpmsg packets from the PRU.

Having used both UIO_pruss and remote proc/rpmsg -
- remoteproc/rpmsg is a lot easier to use with standard kernel interfaces
(i.e. tying in I2C, etc).
- remoteproc/rpmsg really works better with the C compiler which adds a layer
of overhead compared to pure ASM. The UIO_pruss interface gives a raw
processor which simplifies pure ASM code.
- Passing large amounts of data using rpmsg could have considerable overhead.
For example, I would just do just pure remoteproc or stay with UIO for my
PRUSS driven video capture code. But for slower data rates, rpmsg could
simplify things as the buffer management/interrupt/driver attachment code is
done for you.
- Just keep in mind resources are limited on the PRUSS and accessing resources
outside of the PRUSS can be expensive.



On Wednesday, February 10, 2016 16:37:12 Soapy Smith wrote:
> http://processors.wiki.ti.com/index.php/PRU_Training:_Hands-on_Labs#LAB_6:_B
> linking_LEDs_with_RPMsg_from_Linux_User_Space

>
> Hi William, please see the above. I have the PRU Cape, but I haven't got
> this far in the labs. The other labs with remoteproc and other associated
> modules works so far.
>
> Regards,
> Greg
>
> On Wednesday, February 10, 2016 at 7:25:27 PM UTC-5, William Hermans wrote:

> > *William, what are you talking about? Why would I take what you say

> >
> >> personally? You make these blanket statements about a technology you say
> >> you don’t know how to use and recommend that everyone else not use this
> >> technology. If you want to stay with a dead technology, that is your
> >> call,
> >> but there is no reason for anyone to stay away from RemoteProc/RPMSG.
> >> Manufacturers other than TI have embraced this technology which open up a
> >> range of cores you can interact with, such as DSP, CortexM4, PRU, etc.
> >> Yes,
> >> I know you told me you have no interest in the x15 so this is probably
> >> not

> >> important to you and I respect that. *

Hunyue Yau
http://www.hy-research.com/

[William Hermans]

Hi William, please see the above.  I have the PRU Cape, but I haven't got this far in the labs.  The other labs with remoteproc and other associated modules works so far.

Hi Soapy,

Thanks, but I have been aware of that guide for some time now. The example, blinks a PRU cape LED, not one of the USR leds on beaglebone. That's problem #1( not everyone wants to buy yet more hardware to explore an experimental concept).

Problem #2 the guide is based on code composer studio. If you can not see the problem with that, then chances are I will probably never convince you it is a problem. For many of us though, a requirement of using CCS is a major problem.

Problem #3, there is no in depth code explanation, just setup "do x.y.z exact steps", and thats it. So, for me, no code explanation is not really the problem. The problem is no one bothers explaining how the rpmsg mechanism works in conjunction with our specific hardware. That is to say, many of us already know how the hardware works, but how do "WE" control the hardware through this software ? 1 or 2 *good* examples would go a very long ways here.

Problem #4, as ybeagle3 mentions above, performance. If we're wanting to use a PRU or two, chances are pretty good we want / need performance. Granted, there are probably ways to tweak rpmsg, but at some point one has to wonder why even bother.

Problem #5, and possibly the biggest turn off for me aside from very little documentation. This software is experimental, incomplete, and has no proven track record. Which means, no one knows how stable the software is right now. Anyone saying they know if full of it. While on the other hand, the UIO PRU based software has been around a good long time, is proven, and is probably in many commercial grade applications. Not to mention scientific applications, etc.

Anyway, I still think remoteproc / rpmsg is a really cool idea - In concept. In reality though it has a very long ways to go, and no telling if it will ever make it passed the experimental stage. Then if it does, how long will it take ?

Another thing. This hardware( beaglebone ) is an open sourced design. So who in their right mind thought it would be a good idea to demonstrate such a cool idea using a closed source toolchain / toolset ? Oh, right. The same company who says they do not support the PRU's to begin with . . .

Do also keep in mind, that I actualy am PRO TI . . .

[John Syne]

Hi Greg,

I’ve pointed this out to William before, but he doesn’t like to use CCSV6, so you are wasting your time ;-)

Hi William,

By dead technology, I mean there is no further development expected. All future efforts will focus on RemoteProc/RPMSG. For someone looking to start development with remote processor communications, why spend the time learning a technology that isn’t going anywhere? Rather, spend time learning to use a technology that will be enhanced and supported in future kernels. 

Regards,

John

[William Hermans]

Hi Greg,

I’ve pointed this out to William before, but he doesn’t like to use CCSV6, so you are wasting your time ;-)

Actually John, I was well aware of that guide long before you ever mentioned it.

Hi William,

By dead technology, I mean there is no further development expected. All future efforts will focus on RemoteProc/RPMSG. For someone looking to start development with remote processor communications, why spend the time learning a technology that isn’t going anywhere? Rather, spend time learning to use a technology that will be enhanced and supported in future kernels.

That's not for you, or even TI to decide. That is for the community to decide. Especially since TI has pretty much refused to support the PRU since day one. As for what who does when. You seem to conveniently selectively reading what I write in my post. As I've said like 500 times already. I do not care what people use. I will however come into a post like this where the OP originally posted in relation to something am335x_pru_package specific. and you or anyone for that matter jumps in suggesting to use remoteproc / rpmsg. Why ?

Because obviously this person has a time investment into the am335x_pru_package already. I'd like to try and save them from chasing yet another rabbit down yet another useless rabbit hole. As many people here use this software in products, school projects, or just something very serious in the hobbyist circles. They do not care what you think about some experimental bit of software that has yet to be proven . . .

You're like TJP or whatever his name. Everything is a nail, except instead of prusio, or whatever it is, remoteproc is your hammer.

[Soapy Smith]

Hi William, one note on CCS.  I haven't used it at all.  The compiler/assembler (clpru) and associated includes and libraries are present in the Beaglebone testing image (Debian 8, version 4 kernel).

I had to tweak add a symbolic link for the clpru, but that's it.  After that, the Makefiles included in the labs worked perfectly.  All done at the command line.  No CCS IDE required.

I wasn't excited about CCS either, but then I discovered the clpru stuff and no problems.

[John Syne]

Here is what Robert Nelson said on Dec 17, 2015: "If you want "uio_pruss" use "4.1.x-bone", while this "4.1.x-ti" branch is developing "pruss_remoteproc" which will be replacing uio_pruss…”

If you look at the development off uio_pruss, it was developed in 2011 and has a few update/fixes since then, but no new development. 

With RemoteProc/VirtualIO, it is possible to implement custom hardware interfaces on the PRU such as Profibus, Ethernet, UART, i2C, SPI, ADC, DAC, etc and make those look like a device to the Linux kernel. 

Regards,

John

[William Hermans]

Hi William, one note on CCS.  I haven't used it at all.  The compiler/assembler (clpru) and associated includes and libraries are present in the Beaglebone testing image (Debian 8, version 4 kernel).

I had to tweak add a symbolic link for the clpru, but that's it.  After that, the Makefiles included in the labs worked perfectly.  All done at the command line.  No CCS IDE required.

I wasn't excited about CCS either, but then I discovered the clpru stuff and no problems.

Hi Soapy,

Yeah there was what ? 3 different ways one could go. Well technicaly 4-5 I guess including following the gcc path. I messed with attempting to get the standalone TI C compiler, on a cross(Lubuntu 14.04 x64 ) system. No joy after I guess about a day and a half. Normally, I probably would have gotten it to work, but after playing around with that, and several other things all related to the PRU's I decided to toss in the towel.

In relation though I was able to work with a custom am335x_pru_package git, that included example code for using the USR LEDs, and a dmx light controller. The USR LED example I got working straight off, no problem, and I'd assume the dmx controller would also have worked, but I had no hardware to test.

Plus the am335x_pru_package driver uses UIO, which I find really interesting, and indeed really  easy to write a custom driver for when the need arises. Or, more likely just modify the existing one, but it's really easy to read through and understand . . .

[Charles Steinkuehler]

On 2/10/2016 3:15 PM, Bill Gray wrote:
> Charles,
>
> Thanks so much for a definitive answer to this question!!!
>
> Perhaps even more than the answer, I am fascinated by the account you give
> of how you know this to be the case!
>
> I would like to have a better understanding of kernel drivers and DMA
> memory. As an educational exercise I would like dig through the PRU kernel
> driver and see this code for myself?
>
> Could you tell me where I can find it?

The uio_pruss kernel driver is part of the BeagleBone specific
patches, here's one place to find it in raw form if you don't have a
BBB specific kernel tree handy:

https://github.com/beagleboard/linux/blob/3.8/drivers/uio/uio_pruss.c#L267-L268

...as for DMA memory handling, that's complicated. It is more complex
than generic memory handling in Linux (ie: kmalloc and friends), and
there isn't a whole lot of places it's properly talked about. My day
job involves creating hardware that streams data across the PCI/PCIe
bus via bus-mastering DMA, so I'm familiar with all the details
involved, but to truly understand what's happening you need low-level
knowledge of the system architecture, memory model & access semantics,
physical bus transaction protocols, *PLUS* a bunch of Linux specific
details about how DMA is supposed to work.

I don't mean to be trying to discourage you, but there are *LOTS* of
low-level details that have to be done *JUST RIGHT* or you get weird
subtle bugs that only crop up every blue moon (see for example the
transition of Linux from single-core to SMP, or the poor state of
early ARM drivers that didn't properly handle ARMs relaxed memory
model vs. the x86).

Anyway, if you want to start to understand DMA on Linux, the best
single reference I know of is probably still "Memory Mapping and DMA",
or chapter 15 in LDD3:

https://lwn.net/Kernel/LDD3/

...at least until LDD4 comes out (I'm still waiting for my copy).

LDD3 is pretty dated (2005!), but the concepts are still pretty much
the same. The interesting bits related to the uio_pruss driver usage
start with "DMA mappings" on page 445, with a discussion of coherent
buffers beginning on the next page.

Good luck!

--
Charles Steinkuehler
cha...@steinkuehler.net