You can add a new rectangle or line. Annotate added graphic. Next, make the added Graphic invisible. Finally, select the latest Graphic and group with the group. It is a small additional step but has the advantage that you have complete control over the positioning.
Or - ungroup as you did. But immediately after ungrouping, click that outer rectangle and deselect only that rectangle, and directly group everything else. Graphics don't have a chance to move around.