As I understand, QTAutomationAgent.exe is not a part pf problem.
Hard to say something specific without looking to Java code and VBscript code. But my wild guess, the problem related to part witch is responsible for running tests one by one. You should have in some place synchronization point "wait for the end of previous test.
If the loop in VBScript, the synchronization is automatic, if the java call to VBScript, you should wait, when a previous process has been died.
Put msgbox "..." in the beginning of vbscript file. And you will see how many vbscipts are running in the same time.