The "GROUP TABS" action doesn't take into account the possible groups that may exist in the "target tabs" specified in the "Apply to" field.
In contrast, the "MOVE TABS" action does take into account those groups when creating a new window (it will create a separate window for each group in the target tabs).
So, in order to do what you need, the "GROUP TABS" action must be improved to behave analogous to the "MOVE TABS" action. i.e. a new tabstrip group should be created for each group in the target tabs.
This is indeed a useful feature to have. Thanks for suggesting. It will be added in the coming weeks.