Subsurfaces are defined in the U,W coordinates on the surface of a component. The wireframe you see on-screen follows lines of constant U and W.
A straight line in U,W coordinates may be a curved line in X,Y,Z coordinates.
When you have a tapered wing like this, a constant width subsurface edge crosses over different W lines -- you want a straight line in X,Y,Z, but that requires a curved line in U,W space.
To make a constant-chord control surface, OpenVSP has to jump through some hoops. First, we run a solver to find the U,W coordinates the desired distance from the trailing edge at the start of the control surface. We repeat that solve at the end point of the control surface. If we connected those two points together with a straight line in U,W space, we would get a curved control surface edge.
To try to improve the situation, we add a couple (perhaps three) intermediate points -- running the solver on each one to find U,W coordinates the desired distance from the trailing edge.
Once we have a group of U,W points, we fit a spline curve in U,W coordinates -- with the hope that it will result in a straight line in X,Y,Z. Generally, this does a pretty good job (though it is never perfect)
In your case, since the control surface spans two wing sections with dramatically different taper, you have a really challenging case.
I would recommend you split your control surface into two -- define one for the bottom part of the vertical tail -- another for the top part. I suspect that you'll be able to get them both a lot closer to straight lines.
I tried this for a similar case -- check out the first attached image. As you can see, splitting the surface made a significant improvement in how straight the control surface edge appears. However, one trick to see if a straight line is really straight is to place your eye at one end of the line such that you are sighting down the length of the line. I've done this for the second image -- as you can see, from that perspective, this line is not straight...
The 'high frequency scallops' are controlled by the 'Num Points' control at the bottom of the control surface. The longer wavelength oscillations are determined by the number of points that OpenVSP runs the solver on to create the curve in U,W to model your control surface.
Even without the U, W coordinate transformation shenanigans, this line would likely end up curved. The surface of the wing is itself curved. In the case below, the control surface ranges from x/c = .2 to x/c = .5 (or so). The aft 20% of an airfoil is often relatively flat -- however, the middle of an airfoil may be near the point of maximum thickness. The airfoil itself is curved -- now imagine a line that is a constant distance from the TE on that curved surface. That line will always be curved.
Rob