One trick I use in PTGui is to look at the control point table and sort them by distance. I'll then select and delete all control points with a distance greater than say 15. This normally gets rid of any mis aligned control points and gives me a good or very good result after running the optimizer. If that still doesn't work I will go through each overlapping set of images and inspect each cp looking for the culprit. One other trick I use, especially when shooting a location with a symmetrical floor, is to throw a piece of paper or some other small object on the floor to give me a point of reference. I can then remove it in post.
HTH.
Rich