It could be as simple as not touching the mouthpiece at start up. Wait until all the lights stop changing after plugging it in before placing your mouth on the mouthpiece.
The center position of the joystick, and the zero pressure reading of the sip/puff sensors are calibrated during the flashing period.
If the joystick is not exactly centered when the calibration happens, when it does mechanically center, the joystick will reading will be wrong in the opposite direction. This is particularly noticeable with the mouse cursor because even the slightest offset will accumulate.
If that is not enough, you can also increase the size of the center dead-zone a little at a time until the drifting stops.
In QMP, move the center slider up a little and then click on the "Save preferences" button. The new settings will take effect after a few seconds.