I don’t know if you are going to find any forum where this is precisely on topic, and if you find a definitive answer, I hope you will post back here because I would love to implement it as well! I have been researching along these lines for about a year and still have a great deal to learn. Here is some code that I am experimenting with along those lines in my own lighting controller:
https://github.com/brunchboy/afterglow/blob/master/src/afterglow/effects/color.clj#L78-L119
It doesn’t do exactly what you ask, but it is similar. Given an RGB color it first figures the standard RGBW levels you would assign, assuming that your LED wavelengths precisely match the hues red, green, and blue. Then for any additional LEDs you have of a particular hue, it figures out what level they should be illuminated at. I use this with some RGBAW+UV fixtures and it gives nice results, if not perfect. You will need to convert your LED dominant wavelengths to a color-wheel hue value to use these calculations.
-James