That error message means that Maxent is running out of ram, see this
text taken from the help tag on Maxent:
Memory Issues
If the environmental layers are very large files, you may get "out of
memory" or "heap space" errors when you try to run the program. There
are a number of ways to address this problem.
First, make sure that you are clicking on the maxent.bat file, rather
than the maxent.jar file.
Second, make sure that java is being given close to the maximum memory
available on your computer. The maxent.bat file (or any command-line
invocation) should begin "java -mxXXXm", where XXX is a little less
than the number of megabytes of memory in your computer (e.g., use the
flag "-mx900m" if you have a gigabyte of memory). It shouldn't equal
the amount of memory in your computer, otherwise "thrashing" will
occur as the last of the memory is consumed. An exception is on
Microsoft Windows systems with multiple gigabytes of memory: Windows
cannot give java the large contiguous block of memory it desires, so
unfortunately you are limited to a maximum of about 1.3 gigabytes.
Third, you can create SWD-format files (described above) containing
the environmental conditions at the sample localities and a random set
of background pixels (for example, using a GIS) so that the maxent
program doesn't need to load the large environmental layers files. If
you provide the input in this format, you'll probably want to project
the resulting model onto your original environmental layers, so you
should give their location in the "projection directory". The
projection process is memory-efficient, as it doesn't need to hold all
the environmental variables in memory at the same time.
If none of those work, you may have to make your model less memory
intensive by using less environmental layers or lower resolution
environmental layers...