I created another dataset, where the units were returned as:
position [Angstrom]
velocity [Angstrom/fs]
Force [amu * Angstrom / fs^2]
and while the error when down significantly using this dataset where I didn't have to do any unit conversions, I'm still seeing very clear drift.
Here is a plot of the absolute position error and velocity error, clearly there is still drift/errors that are not just precision errors.