It's probably not what you wish to hear, but I would recommend representing your score in some more abstract manner where transposition would be relatively easy and then translating this to VexTab. I've taken this approach using an abstract representation of the
ABC notation. If you define your data structures carefully then transposition becomes tricky but not impossible. I have an example of this in an ABC
editor and if you're unfamiliar with ABC,
here's a tutorial.