Mathtex implements the boxmodel of TeX in order to render mathematical
expressions in a similar fashion. Sadly it does not implement any of
the advanced line-breaking, hyphenation, justification or page
breaking algorithms present in TeX.
The Scribus project, however, have implemented many of the algorithms
into their DTP package. (Optical margins, hyphenation, line breaking,
justification and even microtypography). Their code (C++ using Qt) is
also quite easy to grok.
Regards, Freddie.