Have you verified that the machine moves the same distance per step for a full turn of the motor/pulley?
With a dial test indicator on the axis, jog the machine in steps of 0.1mm and check the movement on the DTI, do this until the motor have completed a full turn and verify that the machine Always moves an equal distance.
I'm thinking that if you have half a motor revolution worth of offset between nozzle and camera and, for example, the motor pulley isn't concentric with the motor shaft the machine won't move the distance you think it will. That could possibly explain the more or less static error on the X-axis.
Then, which Bernd also mentioned, check that the Y-axis is parallel to surface, if the distance between the camera and the surface changes - AND the camera isn't parallel with surface the image will "drift".
And check the rigidty of the whole setup. My camera mount is all metal, screwed into other metal and it's STILL fairly flimsy. Not that it moves on its own but it doesn't take much effort to push/pull/twist on the camera housing to make it go WAY out of "alignment".
/Henrik.