This morning as a test, I created a route, saved it as a GPX, and navigated it, but purposely drove on nearby parallel roads. Once I detoured, the route at first doubled back as you describe. Further down, it tried to get back to the GPX, sometimes resulting in a needless u-turn. After I passed the point where the two roads converged, so I was back on the GPX track, navigation returned to normal.
Then I tried it with again with "Calculate OsmAnd route for first and last route segment" disabled. This time on the parallel road, the route "pointed" at the GPX without backtracking. That is, it indicated I was off-course but didn't actually navigate me back to the GPX.
Neither approach is ideal, of course, and my route in a rural area isn't the same as an expressway, so this may or may not apply.