java.lang.OutOfMemoryError

[Fuchuan Han]

Hello,

I attempted the following code to convert a matrix of size 945*7M into chgeno format. However, after running for almost 8 days, it encountered an error. Is there any alternative solution? options(java.parameters = "-Xmx800G")
geno <- genotypes(file = biggeno, format = "biparental")
cat("** Genotypes completed \n")
cat("** Storing genotypes locally \n")
geno$java <- .jserialize(geno$java)
saveRDS(geno, file = "./945sample_withoutLDfiltered_geno.rds")

Error message: ** [ 2024-05-01 12:32:30 ] Genotypes in progress
** Genotypes completed
** Storing genotypes locally
Error in .jcall("RJavaClassLoader", "[B", "toByte", .jcast(o, "java.lang.Object")) :
  java.lang.OutOfMemoryError: Required array length 2147483639 + 512 is too large
Calls: .jserialize -> .jcall -> .jcheck
Execution halted

[Herman De Beukelaer]

Hi Fuchuan,

That's a huge dataset and from the error message you get, it seems you have hit the maximum array length in Java. There is no way around this. I am wondering though if Core Hunter really needs this level of detail in the dataset. What evaluation measure are you planning to use to sample a core from these genotypes? If you are using one of the distance-based measures (i.e. the default E-NE or perhaps the alternative A-NE) then you could precompute a distance matrix (945 x 945) which will be much smaller than the full genotype data. If you pass the distance matrix to Core Hunter instead of the original data, the results should be the same.

By default Core Hunter uses Modified Rogers' distance to compute the distance matrix so if you search of an R package that can compute a MR distance matrix from your genotypes, then you should be able to pass that matrix to Core Hunter, bypassing the Java memory limits.

Let me know if this works for you.

Herman

Op vrijdag 10 mei 2024 om 11:11:31 UTC+2 schreef Fuchuan Han: