Hi
i think you could start reading about DSPLINK, code engine and others (there is many info in google about it). However as I have research about it it is hard to find actual people using at this moment.
In new boards the way to access DSP is by remote-proc and rpmsg, however for Beagleboard XM there is not support as far as I know.
I manage to create an custom Linux image using and old version of yocto, meta-ti layer, with DSPLINK support, and I manage to get the DSPLINK examples to work with some modification on meta-ti recipes, however I did not manage to get my own app to do a color space transformation as i was using message passing example as a base (my code:
https://github.com/allangj/rgb2ycbcr-dsp) . I was getting a deadlock on the analisys and modification of the image pixels, but the code may work as a starting point as for increasing performance you may want to use different methods to pass information from GPP to DSP.
Also some links:
Hope this could help you as a starting point.
Regards
--Allan Granados