Too slow when i use MLM, how to speed up? TasselBlas for linux?

[Zhonglin Wang]

hi, I'm confused when I use MLM, since it took so long. I've already wait about 26 hours, but progress is only 10%. But before I use MLM, I've tried GLM, and finish in 30 minutes.

I searched a lot, some conversations mentioned about TasselBlas.In fact, I found one INFO log about TasselBlas, "[Thread-16] INFO net.maizegenetics.matrixalgebra.Matrix.DoubleMatrixFactory - TasselBlas library for system-specific BLAS/LAPACK not found. Using system-independent EJML for DoubleMatrix operations."

So I tried blas and lapack in linux, got libblas.a and liblapack.a. But it do not work. Even thougn I got libblas.so and liblapack.so, still got that INFO log.

Finally, I make a libTasselBlas.so by changing name from liblapack.so, the INFO log changed, but I got an exception like this.

"[Thread-14] INFO net.maizegenetics.pipeline.TasselPipeline - net.maizegenetics.analysis.association.WeightedMLMPlugin: time: 十二月 14, 2021 21:08:26: progress: 100%

Exception in thread "Thread-14" java.lang.UnsatisfiedLinkError: net.maizegenetics.matrixalgebra.Matrix.BlasDoubleMatrix.singularValueDecompositionDgesdd(CII[DI[D[DI[DI)I

at net.maizegenetics.matrixalgebra.Matrix.BlasDoubleMatrix.singularValueDecompositionDgesdd(Native Method)

at net.maizegenetics.matrixalgebra.decomposition.BlasSingularValueDecomposition.<init>(BlasSingularValueDecomposition.java:47)

at net.maizegenetics.matrixalgebra.Matrix.BlasDoubleMatrix.columnRank(BlasDoubleMatrix.java:489)

at net.maizegenetics.stats.EMMA.EMMAforDoubleMatrix.<init>(EMMAforDoubleMatrix.java:122)

at net.maizegenetics.analysis.association.CompressedMLMusingDoubleMatrix.computeZKZ(CompressedMLMusingDoubleMatrix.java:709)

at net.maizegenetics.analysis.association.CompressedMLMusingDoubleMatrix.solve(CompressedMLMusingDoubleMatrix.java:323)

at net.maizegenetics.analysis.association.WeightedMLMPlugin.processData(WeightedMLMPlugin.java:251)

at net.maizegenetics.plugindef.AbstractPlugin.performFunction(AbstractPlugin.java:111)

at net.maizegenetics.plugindef.AbstractPlugin.dataSetReturned(AbstractPlugin.java:2017)

at net.maizegenetics.plugindef.AbstractPlugin.fireDataSetReturned(AbstractPlugin.java:1918)

at net.maizegenetics.plugindef.AbstractPlugin.fireDataSetReturned(AbstractPlugin.java:1934)

at net.maizegenetics.analysis.data.CombineDataSetsPlugin.performFunction(CombineDataSetsPlugin.java:65)

at net.maizegenetics.analysis.data.CombineDataSetsPlugin.dataSetReturned(CombineDataSetsPlugin.java:126)

at net.maizegenetics.plugindef.ThreadedPluginListener.run(ThreadedPluginListener.java:29)"

I just want to know is there a way to speed up MLM? Is TasselBlas only for MAC?Why?

thanks

[Peter Bradbury]

TasselBlas is only for the Mac because the library has to be compiled for specific operating systems and libraries. All Mac's come with a linear algebra library (veclib), which made it easier to create a library that would work. To create a linux shared library, the c wrapper in the TASSEL repository would have to be modified then re-compiled and linked to the correct library on the linux system. You are welcome to tackle that if you are a C programmer. Lately, as a different approach to speeding up MLM, I have been looking at an R package called MM4LMM, which is fast and flexible (but not particularly user friendly). I have posted an R notebook for doing a simple mixed model analysis using that and rTASSEL. See https://bitbucket.org/bucklerlab/linear_model_notebooks/src/master/RNotebooks/MLM-association.Rmd. The notebook shows how to analyze one of the traits in the tutorial dataset, so would need to be modified to handle other data. But it could serve as a starting point.

One important difference with that approach used in the notebook is how missing data is handled. MM4LMM needs incidence matrices for the SNPs with no missing data. To generate those, I set made missing data an additional genotype class.

[Fausto Rodríguez Zapata]

Sorry Peter, I coul not take a look at the notebook. It seems it's private. Igot the following message from bitbucket:

We can't let you see this page

To access this page, you may need to log in with another account. You can also return to the previous page or go back to your dashboard.

[Peter Bradbury]

I just changed the repository setting to public (which I thought I had done). Try again.

Peter

[Fausto Rodríguez Zapata]

Thanks!