I suggest testing the Frameserver just by itself first, render from Vegas to an AVI file using frameserver, and opening that AVI file to check if it works.
I'm not sure who the author of Vegas to Handbrake script (it isn't us), so you would have to contact them for support about that script.
- Poz