Did you change anything except using or not using the "--bundle-adjust-prefix" option? So, except using or not using bundle-adjusted cameras?
I will suggest running point2dem --errorimage on both point clouds, with and without bundle adjustment, then inspecting the obtained intersection error image. One can use gdalinfo -stats to see their stats, then colorize it with colormap with the same --min
and --max values in both cases, then open that in stereo_gui to see how much difference the bundle adjustment makes. It is meant to reduce the intersection error.
Hopefully some of these will help with those blunders. Especially mapprojection is strongly suggested for steep terrains like what you have.