When you ask for help with a syntax error, it is customary to include the input for which the syntax error was reported. It is pretty hard for us to diagnose what we cannot see. If you are sure the syntax error complaint is on the line defining the CLASSPATH, showing just that line might be enough, but if you are not sure which line the syntax error is on, show us the whole set of lines that set up the serverclass. You did that in your initial post, but it seems that you have changed some things during your experiments, so we need to see what the current lines are that create that syntax error.
I am pretty sure that Bill's suggestions both are correct: you need to name the jar file for as one of the entries in the CLASSPATH, and the value of _RLD_LIB_PATH should be a list of paths, not a list of files. I am not sure that the exact lines Bill suggested are correct. Of course, you probably need to have SET SERVER before the part that Bill showed, unless you are trying to give all the attributes in the ADD SERVER command. But beyond that, without doing a few ls commands on your system, I cannot confirm that the files are exactly where Bill suggested. But pointing the wrong place would not cause a syntax error when you are trying to create the serverclass.
The traceback from the error seems to me to say that it could not find the class com.tandem.ext.guardian.GuardianException, which was referenced at line 97 of RServer.java. I'm pretty sure com.tandem.ext.guardian.GuardianException is in JToolkit, so Bill's point about getting the jar file name into CLASSPATH is probably what will take care of that problem.
The reason that changing _RLD_LIB_PATH did not affect the error is that you have not yet gotten to the point where it tries to load the DLL. It won't do that until it has entered the JToolkit code, and you were having trouble getting CLASSPATH changed to include the tdmext.jar file. Once you fix the CLASSPATH and try to execute any of the methods in JToolkit that use something in the DLL, it will fail at that point (with a different error) if _RLD_LIB_PATH is not set properly.
I am a bit suspicious of your LINKDEPTH 16. Unless the code reading the requests from $RECEIVE reads the next request before replying to the previous request, LINKDEPTH should always be 1. That won't cause the error you are asking about, but it could cause performance to be a little lower than if it were set correctly.
Is $ZTN0.#PT3TTH2 a dynamic telnet window, or is it a permanent window that won't go away when you logoff your session? If it is dynamic, it probably would be safest to change it to a name that will always exist. This also would not be the cause of the error you have, but I believe that Pathway will get an error when it tries to start the JVM if the HOMETERM does not exist at that time.