I'm still not quite sure what you're aiming for in terms of layout, but in any case, you're probably better off not rotating the icon layer, because that's just going to confuse the button which contains it. Instead maybe experiment with rotating the layer of the Button that holds the icon? That will, I think anyway, at least avoid problems with button hit testing because it's the button's GroupLayer that does the hit testing and thus it will need the rotation so that it applies that inversely to the mouse position before doing hit testing (which is going to happen on the unrotated width/height of the underlying icon).
Then you can play with translation on the Button layer too perhaps, to achieve the layout you want, though that may be overridden by the Layout instance that's arranging the buttons, so you might have to make a custom Layout instance if you're doing something weird.