Hi ruds,
Here's a suggestion that could work. I haven't tried it out myself, but it could be worth a try.
The first thing you would need to do is prevent the loss of focus from the selected text in your RichTextArea when the user clicks on a cell in the colour palette. You might be able to prevent the loss of focus by calling the DOM.eventPreventDefault() method whenever the click event on the colour palette cell occurs.
The other part of the problem is somehow retrieving the selected text so that you can apply the select colour to it. This could be done by retrieving the selected text at the moment of the click on the colour palette cell. You should be able to accomplish this by writing a quick JSNI method that returns the currently selected text. As a precondition to actually changing the selected text colour, you might also want to verify that the selected text is indeed contained in the RichTextArea.
Give it a try and let us know how it goes.
Hope that helps,
-Sumit Chandel