Hi,
I think for now it's not possible to create such a structure. Mainly because this features has not been tested ! I'm not very suprised it does not work.
I guess the container is listening to its contained item, and when one is clicked it triggers a close of the menu, even if the conatined item IS also a container.
It may be possible to change this behaviour with a few line of code in the Radial Menu.
If you manage to do this, do not hesitate to contribute.
Cheers,
Laurent