Looks like I was conflating more than one idea here, learning and inferring on more than one label type and joint inference. As far as my question goes though, I'm pretty sure that my suggestion will work - unless you override the unroll method. addFactors calls all the unrollN methods after an isAssignableFrom check and as v.asInstanceOf[T], but unroll does not. So if you tag your labels using a super type as I suggest above to put them all into an ArrayBuffer, any factors from unroll will fail to be gathered.
I noticed this because of an error with MPLP I commented on in another thread. If you are using MPLP, and the only factors you have for a variable are called using unroll, and you have more than one inference variable type, then you'll probably get an error when you infer using MPLP.
Anyway, just thought I'd share my findings :)
Josh