Hi,
Probably what is happening is that it's finding a solution using a different index file.
You might see it printing at the end something like "found solution in index-5207-08" or something like that. I bet you'll find that the index scale numbers are different when you get 20 vs 30 stars.
You *could* edit your astrometry.cfg file to only use one of the index files. That's messy though!
One thing you can try would be to re-run the solve-field program, and pass in the WCS found the first time around using
solve-field --verify IMAGE.wcs [and then the usual arguments you use]
That should check the solution in all available index files and keep the best one.
Another thing you can do to make the solutions more stable is to set "--crpix-center".
I hope that helps!
-dustin