Most probably there is enough memory available to start Java. Please read
FAQ to find out how to check that you are using 64-bit Java and how to fix memory problems.
You can get problem-specific message if you run the software from command window.
Please note that because Planet file contains currently more than 26 Gb of map data, you need at least 4 Gb of memory for the program to process it successfully.