Hi Mike,
you are absolutely right that when working with monoclinic one has to be very careful about how to set up orthogonal coordinate system x,y,z into the non Euclidean crystal coordinate system a, b, c.In MTEX this can be specified by options when defining crystal symmetry and unit cell.
E.g. if you want to have z parallel to c and x parallel to a* (the dual coordinate system) you have to use the command
cs = symmetry('m',[aa bb cc],[alpha beta gamma],'z||c','x||a*')
where aa, bb, cc are placeholder for the length of the edges of the unit cell and alpha, beta, gamma are placeholders for the angles of the unit cell. Accordingly, you can also set x parallel to c and y parallel to b* by
cs = symmetry('m',[aa bb cc],[alpha beta gamma],'x||c','y||a*')
I hope this helps,
Ralf.