[GSOC 2015 Weekly Report] PRUSS support for newer kernel

[Shubhangi Gupta]

Hey all

Follow project on:

https://hackaday.io/project/5837-pruss-support-for-newer-kernels

[Shubhangi Gupta]

pruss_remoteproc module code modifications:

1. Prevent pruss_remoteproc module from loading during boot. ( Since 3.14 doesn't allow device trees to be loaded after boot, probing the driver when the device is instantiated should be avoided. )

2. Prevent firmware image from loading during probe time.

3. Expose sysfile file to write f/w name from userspace.

4. Set up PRU code generation environment ( SDK, PRU-addon pkg, CCS ) and compile firmware. ( TI pruss_remoteproc driver requires resource table with code )

5. Modify code to load firmware and boot PRU when required by user.

6. Explore mailbox framework for upcoming work. Mentor discussion reveals lack of PRU code space to implement this solution. Will work towards developing custom-light weight IPC model inspired from existing frameworks.

[Shubhangi Gupta]

Week 2

Developments:
1. Exposed appropriate sysfs attributes for data length,offset,memory type and data buffer to userspace.
2. prussdrv like functions for reading and writing to DRAM0, DRAM1 and SHR-RAM work now, allowing
   a maximum of 4kB data to be written in one call.
3. Tested writing 2D integer arrays to PRU mem and subsequent read. Example code on github.
4. Dynamically boot/shutdown PRU from userspace at any time using driver-device bind/unbind function.
    Allows for rmmod pruss_remoteproc without --force option.
5. Able to provide custom sysevt->channel and channel->host mapping to 3.14 pruss_remoteproc driver

Issued faced:
1. Abandon python for userspace library. Realized python is not feasible for writing code which interacts with hardware which
has extremely limited space. C saves the day.
3. Had to decide whether to send data via sysfs or send down userspace pointer (to data buffer) to driver. Reading from sources
    advised against sending userspace pointers down to kernel. Hence former was chosen.
4. A persistent segmentation fault which took significant time to debug. Was due to dereferencing a null pointer.
5. Writing to sysfs was triggering both show/store functions. Issue resolved through mentor discussion. [ use open( ),write ( ) etc
   instead of fopen ( ) fwrite ( ). ]

Next Week:
1. Allow INTC mapping and configuration from userspace.
2. Ability to dynamically allocate larger circular buffers in DDR mem.
3. Extend userspace library to allow DDR read/write.

Code on github

[Kridner, Jason]

Was this code meant to be managed separate from the Linux tree? That doesn't quite make sense to me for the kernel patches. Is there a strategy I should read somewhere?

--
You received this message because you are subscribed to the Google Groups "BeagleBoard GSoC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard-gs...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Nelson]

Nouveau does this too, since it's setup by default as module, it's
really easy to build this quickly and test small changes on target.

When it's working, it should be easy to convert into a patch set with
individual changes...

Regards,

--
Robert Nelson
https://rcn-ee.com/

[Shubhangi Gupta]

Week 3

Developments:
1. Send sysevents ( 0-63 ) from userspace to PRU throughr driver.
2. Configure sysevent->channel and channel->host map though firmware resource table. Very easy to use method.
3. Expose driver interrupts ( ie EVTOUT from PRU->ARM ) for userspace.
4. Interrupt handling in driver, while user is notified about interrupt. User can wait for specified interrupt from PRU (indefinitely or timeout).
5. User can give callback function along with wait. ( Ex.  pruss_wait_for_interrupt ( EVTOUT1, mycallback ). mycallback is wriiten by user
It receives the EVTOUT number as argument if the interrupt was successfully received.

Issues:
1. poll function was posing trouble. It requires a dummy read on the file descriptor before polling to work correctly.
2. Considerable amount of time goes in to find the best possible way to implement something which agrees to driver coding practices. Lots
of resources scattered over internet. Collecting all important links along the way. Will write a blog post on issues faced and how they were resolved.
3. Some time to figure out resource tables of PRU firmware.

Next Week:
1. Complete work on DDR memory circular buffer.
2. Allow for custom modules to plug into remoteproc (so that advanced PRU users can use low level functions provided by pruss_remoteproc
to utilize PRU more effectively as compared to user land implementation. This might take some time.

More detailed report will be up on hackaday with links to very useful resources and critical things learnt along the way.

GitHub

[Shubhangi Gupta]

Week 4
1. Worked on user library code to allow individual PRU core boot and shutdown.
2. Wrote PRU firmware to receive interrupt from ARM and reply to ARM with another interrupt.
3. Make all three component work together : user program - driver - pru firmware.
4. Error handling in library.
5. Most importantly worked on repo to prepare it for tomorrow's evaluation. Prepared makefiles and
added build procedure to Readme. git looks clean now.

Issues faced:
1. Circular queues already part of PRU-bridge. Need to discuss thoroughly with apaar regarding integration
of these two to avoid code replication.
2. Exploring TI's progress on pru vring implementation taking time.
3. Cross-compiling 4.1 kernel is throwing errors.

GitHub

Next week:
1. PRU side of library. Try to keep code minimal.

[Shubhangi Gupta]

​Week 5

Developments:

1. Fix major bug in driver which had gone unnoticed. Word offset given to read/write routines was buggy.

2. Improved user library. Previously pru cores booted when remoteproc driver was probed. Now user has pruss_boot ( "fw_path" , PRU0/PRU1 ) and pruss_shutdown(PRU0/PRU1) routines to independently handle power to each core.

3. Able to register virtual device with vrings using remoteproc.

​4. Better example application in which ARM writes 2 values to pru memory -> interrupts pru 1 -> pru 1 adds 2 values and writes result to new location -> pru1 sends interrupt to ARM -> ARM validates result

Issues Faced:​

1. Have lagged behind a bit while trying to understand the intricacies of vring communication. Need to cover up.

​

Next week:​

1. Finish and wrap up vring based message passing from both ARM and PRU end.

GitHub

[Shubhangi Gupta]

Week 6

Developments:
1. This week was dedicated to getting the driver run on latest kernel. pruss remoteproc driver now successfully compiles and runs on 4.1.1.
Patches for kernel source will be up by tonight.

Issues faced:
1. Compiling a new kernel for beaglebone black and preparing a SD card for it is one mammoth task for someone doing it for the first time.
I'll write a blog post on that on my hackaday project page later.
2. Debugging several lines of kernel code which broke the driver when ported from 3.14 to 4.1.1

Next Week:
Get back to vrings.

[Shubhangi Gupta]

Week 7

Developments:

1. vrings are allocated automatically from remoteproc driver if they are present in firmware resource table. Currently 2 vrings are

permitted ( RX and TX ) each with 512 bytes sized buffers. Number of buffers in each ring can be maximum 512

Issues:

1. Patches for 4.1 is postponed. It was eating up necessary time. More testing showed driver could not be probed again once removed.

More files need to be patched.

2. vring's physical address in driver and that written to pru's memory are slightly different. Will have to resolve this issue asap so that messaging

passing begins successfully.

Next Week

1.Try to finish vrings and begin writing example codes.

[Shubhangi Gupta]

Week 8

Developments:
1. Virtio based vring communication now works for PRU. However things are still within the kernel.
2. Test firmware for pru cores which provide vdev info to rproc driver through resource table. Communicate
data using buffers (vrings) and inform other processor of pending data using 'kicks' which are actually sysevents.

Issues:
Single biggest issue which was most troubling was to find location of resource table within PRU DRAM.
Solution : the .resource_table SECTION specified in link (am33xx.cmd) file is written to a fixed address.

Next Week:
Expose vring to userland as char device and test performance.

[Shubhangi Gupta]

Week 9

Developments:
I am running a slightly behind schedule. Instead of exposing vrings to userspace I worked on
allowing custom callbacks for virtqueues which are executed after the kick. These callbacks
are necessary consume pending messages. Pushed code to gh.

Issues:
Lately I have been able to spend a little less time than I should have. Need to cover up.

Next Week:
Finish previous weeks objectives as fast as possible and get back to working on 4.1 patches.

On Monday, June 1, 2015 at 6:31:26 PM UTC+5:30, Shubhangi Gupta wrote:

[Shubhangi Gupta]

Week 10

Developments:
Earlier I was able to write to all 512 (or less buffers) from PRU but they were not being added back to the vring (as
free buffers) after the data was consumed by host processor. This last roadblock (hopefully) has been solved hence resulting in
successful transfers from PRU to ARM, even continuous streaming data using the vring :D

Issues:
One major optimization remains i.e I am able to get different transfer rates depending on what frequency one kicks (ie interrupts)
the ARM. Kicking the ARM after writing to each 512byte buffer followed by a kick results in stalling the ARM because of too many
interrupts within too short a time. Kicking after filling all 512 buffers have been filled in bu the PRU leads it to wait for ARM to consume
buffers first before PRU can start using them again. So, in order to attain maximum throughput, an optimum value needs to be found
out after which the ARM is kicked. The user also has freedom to decide this value on his own (in the pru firmware) depending upon his requirements.

Next Week:
Patches for 4.1 expose misc char device to stream data to user.

[Shubhangi Gupta]

Week 11

Developments:

1. Exposed vring to userspace using misc device which provides streaming data from pru to arm.

2. Proper code alignment and commenting. Not completed yet.

Issue:

Blocking read on misc device still facing synchronization issues. This involves correcting flow of pending buffer interrupts

sent by PRU->ARM and their subsequent transfer to userspace by copy_to_user.

Next Week:

Wrapping things up, example code and documentation.

[Shubhangi Gupta]

Week 12

Developments:
1. Solved major synchronization issues which was bugging me for a long time related to streaming data from PRU to ARM. copy_to_user functions sleeps
and cannot be a part of irq handler. Hunyue's suggestion to move to threaded irq handlers solved this issue. Now able to stream data to ARM using vrings without
buffers being overwritten.

2. Used kfifo to create buffer for incoming data i.e fast vrings dump data to kfifo which passes on to user through kfifo_to_user. kfifo is dynamically allocated
twice the size of vring which was specified by the user in resource table.

3. Code commenting, library docs and a blog post giving insights into vring implementation (not yet complete)
Post can be found on project wiki and hackaday page

Next Week:
Complete remaining task of code commenting and blog post.

On Monday, June 1, 2015 at 6:31:26 PM UTC+5:30, Shubhangi Gupta wrote:

[Shubhangi Gupta]

Final Report

This project aimed at making it easy for developers to write application for PRU using newer linux kernels. I believe the project status has reached very near to completion
and is currently able to provide all functionality it proposed. The course has been altered a few times during the project execution phase, since better suited alternatives
as compared to what were proposed initially. This was made possible with the help of mentor discussion and the insights I gained as I worked upon the project.

Accomplishments:

1. The initial phases involved understanding the workings of TI's pruss_remoteproc driver which was significantly different from pru_rproc driver being used till 3.8 kernel.
 This also included working on some limitations of the new driver and improving it by borrow functionality provided by pru_rproc driver.

2. In order to provide elementary data exchange between PRU and ARM, suitable routines were added to driver. To lower barrier for end user, a userspace library was written
which makes use of sysfs attributes exposed by driver. Demo app in GitHub repo along with library API can help user to get started.

3. Configuring the PRU interrupt controller followed by sending and waiting for interrupts to/from PRU is possible using user library and resource table.

4. Register vdev and vrings published in resource table using driver. Make use of vring library for PRU to set up vrings in firmware.

5. Synchronizing kicks from PRU and subsequent transfer of buffers to user was the most challenging part. Implemented control flow on vrings
 so that no buffers are dropped and continuous data streaming from PRU is achieved. Makes use of threaded irq handler.

I would like to thanks Google and Beagleboard.org for giving me wonderful opportunity to learn immeasurable amounts during this summer. Thanks to
my mentors Pantelis Antoniou Hunyue Yau and Abhishek Kumar for making this project possible. I would also like to thank Alexander Hiam, Deepak Karki
and Jason Kridner for your support at critical junctures.
Will definitely keep contributing to this as well as other open source projects.
Shubhangi Gupta

[Shubhangi Gupta]

Forgot to post the link to the blog post. The article tries to eplain how vrings work in general and how they are implemented in the PRU context.
Understanding vrings for pru

Thanks

On Monday, June 1, 2015 at 6:31:26 PM UTC+5:30, Shubhangi Gupta wrote: