Ah, the problem is that Numbas has to pick a type for the free variables in the answer, in order to do the numerical comparison. It uses the expected answer for this, so when it's 'x^2', vec:a and vec:b aren't used, so it uses the default type: real number. I suppose that it could use the annotation as a hint!
I think we need to show a more useful error message. Adding a bit to explain the type of the free variables would help the question author, e.g. "No definition of 'dot' of correct type found; The variables are interpreted as: vec:a - number; vec:b - number.". That's too technical for a student though.
I thought I already had an open issue on GitHub about error messages produced by the mathematical expression part, but apparently not. I've made one now:
https://github.com/numbas/Numbas/issues/848. I'd really like to improve them, because I think it's an important part of the student's interaction with Numbas.