I get the same with voice navigation when hiking along a gpx.
One thing is to set the tolerance distance. Outside the tolerance distance, recalculation starts. It's somewhere in the navigation settings for the profile. This is location dependant, so it relies on the gps-accuracy of your device. Mine is, uhm, suboptimal so it often places me up to 20 m off. When walking, this could mean I am on the other side of a long canal, which would warrant recalculation.
The other thing is the gpx. When cycling along a bicycle-routed gpx, you are fine, the router should have picked the correct side of roads, around roundabouts etc. If you navigate along a revered gpx, you are very often at a distance from the gpx, because you have to cycle at the correct side, sometimes even completely different paths.
Maybe something like that is part of the cause?