At one time there were a few modules with "emeritus" peers: these were
essentially formerly active peers that don't do much any more. I liked that
label because it gave people a "badge" of accomplishment, without burdening
them with daily responsibility. But I can't find any references to this any
more on the modules wiki pages. Perhaps it got dropped as part of wiki
refactoring.
Anyway, module owners have the right to drop peers. But as Stormy said, I'd
approach them first to make sure they want to be removed. I've done this
with a few build module peers and the response has typically been "yeah, I
don't wear that hat any more, please remove me."