You will need to use the listbox's underlying Java component to do this. This component can be retrieved using my FindJObj utility on the Matlab File Exchange: http://www.mathworks.com/matlabcentral/fileexchange/14317
Here is the code snippet:
hListbox = uicontrol('style','listbox', ...);
jScrollPane = findjobj(hListbox); % get the scroll-pane object
jListbox = jScrollPane.getViewport.getComponent(0);
set(jListbox, 'SelectionBackground',java.awt.Color.yellow); % option #1
set(jListbox, 'SelectionBackground',java.awt.Color(0.2,0.3,0.7)); % option #2
You can similarly change SelectionForeground and many other undocumented interesting properties.
For other interesting undocumented stuff you can do with listboxes, read here: http://undocumentedmatlab.com/blog/setting-listbox-mouse-actions/
Yair Altman
http://UndocumentedMatlab.com