A big issue you'll run into when looking at abstracting the template engine is value injection & type inference / conversion.
SilverStripe templating is tightly coupled to ViewableData, and uses a pull based mechanism where the compiled template asks the current active context for the value in a certain form (XML, etc).
Many other templating libraries expect to be handed all the possible variables up front, rather than access them in a pull based manner. They also handle type conversion differently.
These issues can be overcome - I know of a couple of projects where people have replaced SilverStripe templates with another engine - but the result always ended up a bit frankenstein-ish. I'd be interested in any work anyone did in this area. As Sam said, it'd be good to try and decouple SSViewer from ViewableData.
I think if someone came up with a patch that implemented an elegant way of using multiple template engines we'd definitely consider merging it, as long as it didn't impact on the ease of using the default SilverStripe template engine. We're unlikely to put any development effort into this ourselves though. SilverStripe templates will I'd imagine remain the "preferred" templating engine for the foreseeable future, although if there is a specific feature from another templating language you find lacking in SilverStripe templates, I'd be keen to hear it.
Hamish Friedlander