If you do build them, make sure to install them privately. You need to
make sure your binaries won't pick up system libs and that system apps
won't pick up your copy of the libs.
It might not always matter when interface versioning is sane, but
sometimes it is not (take Qt for example).
Always safer to take most paranoid approach. And the cost of space (and
build time if you distribute sources) is probably irrelevant. Depends on
you target audience I guess.