There is insufficient memory for the Java Runtime Environment to continue

[samcm...@gmail.com]

Hi,

I am looking to compare a set of downsampled vcf files to a truth set using rtg-tools vcfeval. I am running the command:

rtg vcfeval --baseline $truth_set --calls $downsampled_calls --template $output_sdf --output $output_dir

The data is genome-wide human data in gvcf format. The truth set is 1.6GB and the baseline set varies with coverage, but the largest one is around 846mb. 

I am requesting a large amount of memory from the compute node (LSF) - up to 248GB. 

Each time, a variety of jobs fail with the Java error:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f0c00480000, 17997234176, 0) failed; error='Cannot allocate memory' (errno=12)
The operating system did not make requested memory available to the JVM.  Try removing other jobs on this machine, adjusting allocated memory appropriate to currently available memory, or adjusting command parameters to reduce memory requirements. More information is contained in the file: ./hs_err_pid30143.log

I have tried increasing the amount of memory up to 248Gb, but the error persists. Strangely, the larger downsampled call sets seem to be able to finish, but the smaller ones don't.

I suppose the error comes from the way Java is accessing memory, rather than there not being enough memory. Or perhaps it is to do with the memory each thread is using?

Could you suggest some ways I could fix the error?

Thanks. 

[Sam Morris]

Btw, if I run rtg version:

Product: RTG Tools 3.10.1   (*** The latest release is 3.11 ***)

Core Version: 6b0f9ef (2019-05-02)

RAM: 291.3GB of 364.1GB RAM can be used by rtg (80%)

CPU: Defaulting to 12 of 12 available processors (100%)

JVM: Java HotSpot(TM) 64-Bit Server VM 1.8.0_181

License: No license file required

Contact: sup...@realtimegenomics.com

[Len Trigg]

Hi Sam,

That type of out of memory error generally occurs when Java thinks it is allowed to use more memory than it actually is permitted to use according to the operating system (i.e. LSF constraints). Also, often when Java has a ton of memory available, it defers garbage collection, even if the active memory use can be way lower, and that could be pushing you over the LSF limit.

Try running with a Java memory heap limit that is safely less than what you request from LSF. e.g. "rtg RTG_MEM=200g vcfeval ..."

Cheers,

Len.

--
You received this message because you are subscribed to the Google Groups "RTG Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rtg-users+...@realtimegenomics.com.
To view this discussion on the web visit https://groups.google.com/a/realtimegenomics.com/d/msgid/rtg-users/e2b1c130-43ba-433f-b605-7007ea6798a9o%40realtimegenomics.com.