I usually use Windows 7 for ModelSim but I have a Windows 8 system so I tested the download/install/compile/run and everything worked as expected. I'm now sure what to tell you. The behavior you are seeing looks like some kind of memory allocation failure so my guess is that there is something unique about your system that is causing this.
My suggestion would be to uninstall and download the student edition again. Check the new download file with the old one to see if they are the same size. When you reinstall make sure you use all the defaults (location/icon/path modification/etc.). Once you have the new license file saved to the root modelsim directory test again and see if it works. If not I'm not sure what to do.