PHPUnit isn't needed for its operation, as the source code hints:
// HACK Ugly workaround to avoid "Cannot redeclare class PHPUnit_Framework_TestResult" error
// when running text collector with PHPUnit 3.4. There really shouldn't be any dependencies
// here, but the class reflection enforces autloading of seemingly unrelated classes.
// The main problem here is the CMSMenu class, which iterates through test classes,
// which in turn trigger autoloading of PHPUnit.
$phpunitwrapper = PhpUnitWrapper::inst();
$phpunitwrapper->init();
So in the end, you have to prevent those test classes from being included in the ClassManifest.
This is usually done by implementing the "TestOnly" interface as a flag.
I think we need to refactor the retrieval from going through all PHP files,
to using ClassInfo::implementorsOf() and then looking if the file path of the class matches.
Feel free to give it a shot, otherwise I will in the next week.
BTW, phpunit is now easier to install (mainly because pear is a bit less crap than it used to be).
Still no match to composer, of course ;)