In order to administer their group membership they have to have a
Google account to log-in to the group with. Just as you must have a
Yahoo account to administer your membership in a Yahoo group with. If
they responded to your invitation via e-mail, they shouldn't have to
"set up" their Google account to begin receiving your group, but if
they want to change their options, they probably do. If they
responded to the invitation via the web, they definitely would have
to.
Google will naturally offer for them to set up a GMail account while
they're at it, but they are not required.