Problem with gl.run.structure using large data
Comlan Arnaud Gouda
Hi all,
I am coming to a problem with running STRUCTURE analysis using gl.run.structure command with DArT package.
I firstly convert my data (9K samples with 27K SNPs) in vcf format to genlight object
Then used the script below to run the STRUCTURE analysis in window system (32 RAM memory)
library(dartR)
gl11b <- readRDS("mygenlightdata.RDS")
sr <- gl.run.structure(
gl11b,
k.range = 2:10,
num.k.rep = 10,
burnin = 10000,
numreps = 100000,
noadmix = FALSE,
exec = "D:/Essaie/structure.exe",
plot.out = TRUE,
save2tmp = TRUE)
Unfortunately, I am getting the following output:
Reading file "gtypes.created.on.2023-06-07.12.14.28.structureRun/gtypes.created.on.2023-06-07.12.14.28.structureRun.k1.r1_mainparams".
datafile is
gtypes.created.on.2023-06-07.12.14.28.structureRun/gtypes.created.on.2023-06-07.12.14.28.structureRun.k1.r1_data
Reading file "gtypes.created.on.2023-06-07.12.14.28.structureRun/gtypes.created.on.2023-06-07.12.14.28.structureRun.k1.r1_extraparams".
Note: RANDOMIZE is set to 1. The random number generator will be initialized using the system clock, ignoring any specified value of SEED.
Error in assigning memory (not enough space?)
Exiting the program due to error(s) listed above.
Error in FUN(X[[i]], ...) :
Error running STRUCTURE. Error code 1 returned.
Pls Any ideas as to what I can fix would be really appreciated!!
Thanks,
Arnaud
Bernd.Gruber
Hi Arnaud,
Based on your folder name I assume you are running this on Windows. There is a memory limitation on R on Windows (less so on Linux/Macos), hence I assume you need to use a different OS, or you may try to increase memory using R mechanism (memory.limit()) , but this is no longer supported after R 4.2 if I am not mistaken.
So most likely you need to shift to a Linux based R.
Cheers, Bernd
==============================================================================
Dr Bernd Gruber )/_
_.--..---"-,--c_
Professor Ecological Modelling \|..' ._O__)_
Tel: (02) 6206 3804 ,=. _.+ _ \..--( /
Fax: (02) 6201 2328 \\.-''_.-' \ ( \_
Institute for Applied Ecology `''' `\__ /\
Faculty of Science and Technology ')
University of Canberra ACT 2601 AUSTRALIA
Email: bernd....@canberra.edu.au
WWW: bernd-gruber
Australian Government Higher Education Provider Number CRICOS #00212K
NOTICE & DISCLAIMER: This email and any files transmitted with it may contain
confidential or copyright material and are for the attention of the addressee
only. If you have received this email in error please notify us by email
reply and delete it from your system. The University of Canberra accepts
no liability for any damage caused by any virus transmitted by this email.
==============================================================================
--
You received this message because you are subscribed to the Google Groups "dartR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
dartr+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dartr/d0e533c8-2006-41a2-a5c3-54e423643384n%40googlegroups.com.
Jose Luis Mijangos
Running Structure in such a large dataset and with those settings requires a lot of computing power, the analysis of your dataset would take a very long time probably months, on a personal computer.
Some options are:
- Run Structure on a high-performance computing cluster, see this paper: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1593-0
- Give a try to the program fastStructure (https://rajanil.github.io/fastStructure/) but it is a bit of trouble to get it to run. You could use the function gl2faststructure to convert a genlight object into faststructure format. Results could be processed using Clumpak (http://clumpak.tau.ac.il/).
- You could filter very stringently your dataset and subsample loci (using function gl.subsample.loci and choosing the most informative loci i.e. method = "pic"). 1,000 loci would do depending on the level of genetic structure of your dataset, you could have a look at the below articles to give you an idea about how many loci to use.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3797491/
https://onlinelibrary.wiley.com/doi/full/10.1111/1755-0998.12650
Cheers,
Luis
Bernd.Gruber
Hi Jose,
I would try to run a subset of your data set first (e.g. only 100 loci to see if it works).
But Luis is right most likely you would need to run it on a cluster or fun faststructure with is faster (but different though the name indicates it is the same).
Cheers, Bernd
==============================================================================
Dr Bernd Gruber )/_
_.--..---"-,--c_
Professor Ecological Modelling \|..' ._O__)_
Tel: (02) 6206 3804 ,=. _.+ _ \..--( /
Fax: (02) 6201 2328 \\.-''_.-' \ ( \_
Institute for Applied Ecology `''' `\__ /\
Faculty of Science and Technology ')
University of Canberra ACT 2601 AUSTRALIA
Email: bernd....@canberra.edu.au
WWW: bernd-gruber
Australian Government Higher Education Provider Number CRICOS #00212K
NOTICE & DISCLAIMER: This email and any files transmitted with it may contain
confidential or copyright material and are for the attention of the addressee
only. If you have received this email in error please notify us by email
reply and delete it from your system. The University of Canberra accepts
no liability for any damage caused by any virus transmitted by this email.
==============================================================================
From: da...@googlegroups.com <da...@googlegroups.com> On Behalf Of Jose Luis Mijangos
Sent: Thursday, 8 June 2023 12:27
To: dartR <da...@googlegroups.com>
--
You received this message because you are subscribed to the Google Groups "dartR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
dartr+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dartr/f5035157-e154-4536-bf8f-0eab3fbe67abn%40googlegroups.com.
GOUDA
Hello Luis and Bernd,
Thank you for all the suggestions you have given me to solve the
problem,
I will try to use the FastStructure program.
Cheers,,
Arnaud
--
You received this message because you are subscribed to a topic in the Google Groups "dartR" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dartr/-lbeVqV-ofc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dartr+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dartr/f5035157-e154-4536-bf8f-0eab3fbe67abn%40googlegroups.com.
Cdlt, Arnaud.
Hewan Demissie
gl,
k.range = 1:5,
num.k.rep = 10,
burnin = 30, # 30,000
numreps = 100, # 1,000,000
exec = "D:/research/Avocado/GBS-Avocado/STRUCTURE",
noadmix=FALSE)
out_evanno4 <- gl.evanno(out)
qmat4 <- gl.plot.structure(out, K=2, colors_clusters = c("#1F78B4", "#33A02C"))
Bernd.Gruber
Hi,
Assuming you are running structure under windows you need to provide the full path to the exe file e.g.
D:/research/Avocado/GBS-Avocado/STRUCTURE/structure.exe
If under macos or linux you need to do
./research/Avocado/GBS-Avocado/STRUCTURE/structure
Then it should run,
Cheers, Bernd
==============================================================================
Dr Bernd Gruber )/_
_.--..---"-,--c_
Professor Ecological Modelling \|..' ._O__)_
Tel: (02) 6206 3804 ,=. _.+ _ \..--( /
Fax: (02) 6201 2328 \\.-''_.-' \ ( \_
Institute for Applied Ecology `''' `\__ /\
Faculty of Science and Technology ')
University of Canberra ACT 2601 AUSTRALIA
Email: bernd....@canberra.edu.au
WWW: bernd-gruber
Australian Government Higher Education Provider Number CRICOS #00212K
NOTICE & DISCLAIMER: This email and any files transmitted with it may contain
confidential or copyright material and are for the attention of the addressee
only. If you have received this email in error please notify us by email
reply and delete it from your system. The University of Canberra accepts
no liability for any damage caused by any virus transmitted by this email.
==============================================================================
--
You received this message because you are subscribed to the Google Groups "dartR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
dartr+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dartr/15d7c1be-1664-423c-9201-9dee3dae2210n%40googlegroups.com.
Hewan Demissie
Jose Luis Mijangos
> # filtering loci with all missing data
> t1 <- gl.filter.allna(platypus.gl)
> # running Structure in MACOS
> res <- gl.run.structure(t1, exec = "./structure")
> # running Structure in Windows
> res <- gl.run.structure(t1, exec = "/structure.exe")