Finally got back to this, only because a student was spamming the student groups again...
Actually got it to work!
Had to change affected account types to Users, not Groups as what Daniel had. Daniel, are you sure yours is working with Groups checked and not Users?
I've sent several emails to various groups and run the auditing tool, and they are not ending up in all the users (of the groups) inbox, though they do still end up in the group feed if you go to google groups site.
I did try also ticking groups again (both user and groups checked) to see if that would stop the email from going into the group feed, but it appears to not make a difference, so I have no idea what the 'groups' option actually affects.
Also, with the reject message, that doesn't seem to be coming back to my test users account, so I don't know where it ends up. Would be nice for the student to get that message.
Regardless, its stopping students from spamming each other, so that's a win for a Friday.