You're now to the point of the Zen of Fit Model...
If you can share your example geometry, I might be able to provide more specific and direct help.
Using FitModel, you are hand-formulating an optimization problem. You are adding degrees of freedom and also constraints that drive the objective.
If you have two variables that do (nearly) the same thing, you end up with a singular matrix -- there are infinitely many ways to achieve the exact same solution. The optimizer will spend a really long time trying to balance those two variables to no perceptible improvement.
Don't try to set up a one-shot FitModel -- instead, take it one step at a time. Approach the problem like you were trying to match the points without FitModel. I.e. you were using an 'eyeball least squares algorithm'. Adjusting until it 'looks about right'.
Here is my rough FitModel recipe...
1) Type -- Choose which component I want to use to match a given part of a geometry. Probably a Wing or a Stack -- could be a body of revolution or ellipsoid -- anything else seems unlikely. I need to get the Type correct because FitModel can't change the type.
2) Topology -- If it is a wing or a stack, then I would add the number of cross sections / sections that I am going to want to match. I need to get the Topology correct because FitModel can't change the topology of the model.
1b/2b) Type/Topology of XSecs -- I might even take this time to change the XSec types or airfoil types -- but probably not. Ellipses are probably a great choice for now for a Stack. Generic airfoils are probably a great choice right now for a Wing.
3) Visual Match -- I might spend a little time adjusting the component by hand -- this gives me a feel for which parameters I am going to want to use -- and which features in the cloud of points are going to be most useful for matching. Once you have a bunch of experience with FitModel, you may skip this step -- but it is a generally good idea.
Note: Most 'synthetic' point clouds you want to match will have a point exactly at the nose/origin of the geometry (by synthetic, I mean originating from CAD, a mesher, or some other computer model.) Most 'natural' point clouds won't have an exact point anywhere you want it, these are more challenging, I'll assume you have a synthetic cloud first (by natural, I mean originating from a laser scan or other random distribution of points).
4) Position -- I want to get the component positioned correctly. The origin of Wings is the LE root. The origin of Stacks is the front point. A synthetic cloud will likely have a single point at the component origin. You can also use a wingtip instead of a root if needed. Create a single target point -- pick the exact origin point from the cloud data. Make the target point fixed in both U and V, exactly at the nose. Typically 0,0 for a stack, or 0,0.5 for a wing. Then choose the transform variables you need -- X, Y, Z. Double check that you have chosen Abs / Rel correctly -- the ones you have chosen must be 'active'. You should have 3 DOF and 3 Equations. Click 'FitModel'. The model should 'snap such that the nose point is exactly aligned with the cloud nose point. At this point, remove the X,Y,Z variables from the control of FitModel -- their values have been set and should not be changed in the future. You can delete your target point or leave it, it won't matter much.
Note: Some points on a geometry are unique -- others are more vague. Using both kinds of points has their place in FitModel -- but you usually want to start with Unique points and move to Vague points later. Imagine a cube. Each of the eight corners of the cube are unique points. If you have a cube point cloud, you can pick out an exact corner point. You will want to match that point to the exact corner of your VSP component -- you can figure out what U,V coordinates correspond to that unique point and use the Fixed U, Fixed V target point type. Next, imagine you pick a random point that lies in the middle of one of the six faces of the cube. This is a vague point -- it is a lot like other nearby points. As a human looking at it, you can't say with confidence what the values of U,V in your VSP component match exactly to that random point. When matching it, you will want to use U,V Free. Of course, you can also pick a point along one of the cube's twelve edges -- these are each fixed in U or V (not both) and free in the other. If you are matching an edge point, then use that information in FitModel.
5) Orientation / Scale -- I want to get the component oriented in space correctly and approximately the right size. The nose is fixed, but it can be rotated about X,Y,Z in different directions. Rotating symmetrical bodies about X doesn't matter, so don't use X rotation. Vertical tails are made from wings rotated about X -- you know they are rotated 90deg, so just set that yourself. For synthetic clouds, you will probably be able to pick out another unique point to help guide you. On a body, you probably want to pick out the singular point that closes out the body (the opposite of the nose). For a Wing, you want to pick a point on the wingtip -- either LE or TE. Something that is uniquely / precisely defined. Add a target point, Fixed U, Fixed W (probably 1,0 or 1,0.5 or 1,1). Choose the required rotation variables, and choose a scale variable (span for a wing, delta X for a stack). Click FitModel, it should quickly change orientation and length to match the point. You might remove the rotation variables from FitModel control at this point -- or maybe not. You probably want to leave the scale variable under its control for now.
6) Intermediate Section Position -- For a wing, you next want to position the planform breaks -- get the span of each wing section right. For a stack, you want to position the 'hard' XSecs -- get the deltaX's right between each one. Since the overall span / length of these component types is additive, you will need to keep your original scale variable in the mix for now. I suggest you pick out a single target point on each feature -- the planform break for a wing, or the section of interest for a stack. These target points might be Fixed/Fixed (unique points) or possibly Fixed/Free (unique curves) but probably not Free/Free (vague points) at this time. Match the spans and the stack distribution. You might want to remove these variables at this time -- but you likely want to keep them in the mix for a little while longer....
6b) More Intermediate Section Position -- For a swept wing, you likely next want to adjust the leading edge sweep to match. Don't worry about the trailing edge or chords at this time. Just align the LE. For a stack, your sections probably aren't grossly offset, so you probably don't want to adjust their position relative to the construction axis just yet. Your target points from 6) may be enough for this -- just add the sweep variables you need to allow the LE to match up. If the model has substantial dihedral or twist, this may be a little more tricky at this point.
7) Intermediate Section Size -- For a wing, you next want to adjust the chords. For a Stack, you want to adjust the Widths of the XSecs. Most body type components are symmetrical right-left, so you probably don't need a Y-offset.
7b) More intermediate Section Size -- For a Stack, you now probably want to position the height, and any Z-offset of the sections. You might want to include the 'max width' parameter of the XSecCurve to keep the maximum width points aligned as above.
Note: At this point, you should have the major features of your component matched. Type, Topology, Position, Orientation, Scale, Intermediate XSec Positions and Scale. It is very likely that you do not want to change these variables going forward -- you may want to delete them from the FitModel problem.
Next you have two choices -- you can proceed to match the XSecCurves better -- or you can proceed to match the skinning between XSecs better. For a wing, you probably don't have much to do between XSecs (unless it is blended), so proceed to the XSecCurve matching. For a stack, you can go either way, but I would usually work on the skinning next.
8) Skinning Stack
8a) Topology -- go through and turn on/off your skinning variables. Set symmetry as desired, set equal angles where you need them -- allow them to be different where you want them. You likely have an XSec at the point of maximum area, so set the angle to zero at that point.
8b) TBR -- Top Bottom Right -- Your geometry is probably R/L symmetric, so you only work with the Right side. Work on _one_ of these curves at a time. I usually start with the Top. It is very beneficial to select points along the exact center line of a fuselage. There are some tricks to extract these from your mesh. Select points along each skinning section -- these points will need to be Free in the U direction. Choose your skinning variables carefully -- you usually can not select 'strength' at both ends of a segment. For a gentle curve, using both strengths is exactly the kind of thing that will drive the optimizer bonkers. If you can only use one strength value, set the other one to 1.0 to keep things balanced in general. After you've matched the 'Top' curve, delete all those target points and the variables you've selected and move on to the Bottom curve. Once you're done with that, delete everything and move on to the Right curve. The right curve can be tricky because you might not have a crisp line to work with -- you may need to match points Free/Free along the side. Once you're done with Right, delete all those points.
9) Match XSec Curves -- Go along from curve to curve -- convert your ellipse to whatever type you need to match -- SuperEllipse, RoundedRectangle, etc. Then choose points along that XSec, and fit. You likely need to make the target points Free/Free at this point. Do one XSecCurve at a time, move your way down the fuselage, or out along the wing.
10) Nuance -- At this point, you should be done. There might be some subtle things you want to go back and tweak -- but by now you should have the feel for it and should know how to go tweak things in good taste.
Note: 'Natural' point clouds can be terrible -- but so can 'Synthetic' ones. Most STL meshes have holes, overlap, gaps, tons of points where you don't need them, and lots of points where you don't care about them. It can feel like a conspiracy against you. You can also read in structured meshes (Hermite or Plot3D) and generate a point cloud from there. These meshes are generally better -- but typically come with several orders of magnitude too many points.
There are tricks for both situations....
For ugly triangulated meshes -- you can use the planar slice feature to cut the geometry at any slices you want. This is amazing for extracting a set of points down the center line of a fuselage. You can also slice a fuselage at exactly the places where you want to specify your XSecs. After you use Planar Slice, it will show the slices and the external mesh -- you can flip a switch so that it only displays the slice points. Then export that slice to a STL or TRI file. Then import that file back in. That STL or TRI file will _only_ contain that slice. Convert the slice to a point cloud and go from there. Similarly if you do X-slices to cut out fuselage station cuts.
For super-resolution structured meshes -- when you import a wireframe via Plot3D or Hermite, there are some options to manipulate the wireframe. You can invert, flip I/J, skip beginning or end points, or stride your way -- say only keep one out of every five points... This makes a great way to de-res a mesh and to extract just the row of points that you want. In addition, you can also run CompGeom to convert this to a triangle mesh or PlanarSlice to slice the wireframe to extract using the same technique as above.
If you're just playing around trying to FitModel to geometries that you generate in OpenVSP -- or you want to match a Sears Haack or Von Karman Ogive to a set of points that you wrote out with a quick Matlab program -- then you probably don't need these last couple tricks. However, if you're using some nightmare mesh from the CFD department, then you'll thank me for the tricks...
Rob