It will probably be tricky to change a user's role without having them log out and in again.
Their role is stored in the SMF db and while you can force a different role in chat, that's done (I think) during the call to getValidLoginUserData() which is only fired on login. You'll have to hack in a way to force their session to change and that's pretty tricky. I think what you'll need to do is hack the initMessageHandling() or the insertMessage() functions and add another check there. I think initMessageHandling() is designed for that but you'll have to test it out if it fires on every message like you want.
That check will have to consult something, like the database, to see if you've muted that user. It would be a good place to reset their group in their session. Other users won't see that the person's group has changed unless you also send out some XML update to tell everyone that someone's group has changed.
You could also do it on every chat load (ie: every poll request for messages anyone makes), even before they try to insert a message, but that would add some overhead that might slow chat down.