Carl,
There are several different server components, so the first step is figuring out which one is running out of memory.
If it is the web server, you can modify the top-level build.xml file to increase the heap. You would do this in the "run-without-wrapper" target where it starts Java. Alternately, you can build the server using "ant package", and then run it with a command like:
java -jar dist/Wonderland.jar -Xmx900m my.run.properties
This will start the web server using the setting from my.run.properties with 900 MB of Java heap.
If the Darkstar server is running out of memory, you can modify the run script used to start Darkstar, which is at wonderland/core/build-tools/run-scripts/run-server.xml. You can add arguments to the java invocation in the core-run-server target.
It would be great to know why the server is running out of memory. The only known memory leak in the web server is to do with screen sharing. Are you using the screen sharer module?