MeDeCom (DeComPipeline) install issue

[Peter Mcerlean]

Dear CompEpigen Team,

I've been trying to install MeDeCom as part of the DecompPipeline via the recommendation on the Github page and had issues with the DecompPipeline/FactorViz packages not locating it.

I tried all sorts including installing the binary manually and R CMD INSTALL in terminal but was still getting errors (below).

After some investigation I was able to determine the error is with MeDeCom 1.0.0 as using version MeDeCom 0.3.0 everything seems to load fine.

However, I haven't run any analysis with MeDeCom 0.3.0 as I should naturally be using the latest version?

Apologizes if this forum isn't he correct place for a development issue!

Best,

Peter

 

> library(MeDeCom)

 

…everything good until…

 

Error: package or namespace load failed for ‘MeDeCom’ in dyn.load(file, DLLpath = DLLpath, ...):

 unable to load shared object '/Library/Frameworks/R.framework/Versions/3.6/Resources/library/MeDeCom/libs/MeDeCom.so':

  dlopen(/Library/Frameworks/R.framework/Versions/3.6/Resources/library/MeDeCom/libs/MeDeCom.so, 6): Symbol not found: ____chkstk_darwin

  Referenced from: /Library/Frameworks/R.framework/Versions/3.6/Resources/library/MeDeCom/libs/MeDeCom.so

  Expected in: /usr/lib/libSystem.B.dylib

 in /Library/Frameworks/R.framework/Versions/3.6/Resources/library/MeDeCom/libs/MeDeCom.so

 

Re install from github

 

> devtools::install_github("lutsik/MeDeCom")

Downloading GitHub repo lutsik/MeDeCom@master

Skipping 1 packages not available: RnBeads

✓  checking for file ‘/private/var/folders/ls/g6jy40g97xn6482d5tr5vfy80000gs/T/RtmpLxCd6p/remotesf93c46ed127d/lutsik-MeDeCom-d740fe6/DESCRIPTION’ ...

─  preparing ‘MeDeCom’:

✓  checking DESCRIPTION meta-information ...

─  cleaning src

─  checking for LF line-endings in source and make files and shell scripts

─  checking for empty or unneeded directories

─  looking to see if a ‘data/datalist’ file should be added

─  building ‘MeDeCom_1.0.0.tar.gz’

  

* installing *source* package ‘MeDeCom’ ...

** using staged installation

** libs

clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG  -I"/Library/Frameworks/R.framework/Versions/3.6/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.6/Resources/library/RcppEigen/include" -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include `/Library/Frameworks/R.framework/Resources/bin/Rscript -e "Rcpp:::CxxFlags()"` `/Library/Frameworks/R.framework/Resources/bin/Rscript -e "RcppEigen:::CxxFlags()"` -I. -std=c++11 -fPIC  -Wall -g -O2  -c HCLasso.cpp -o HCLasso.o

HCLasso.cpp:47:10: fatal error: 'omp.h' file not found

#include <omp.h>

         ^~~~~~~

1 error generated.

make: *** [HCLasso.o] Error 1

ERROR: compilation failed for package ‘MeDeCom’

* removing ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library/MeDeCom’

* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library/MeDeCom’

Error: Failed to install 'MeDeCom' from GitHub:

  (converted from warning) installation of package ‘/var/folders/ls/g6jy40g97xn6482d5tr5vfy80000gs/T//RtmpLxCd6p/filef93c49311100/MeDeCom_1.0.0.tar.gz’ had non-zero exit status

[Michael Scherer]

Hi Peter,

This forum is definitely the right place for this issue. However, I don't have an idea what the issue might be, since the installation works fine in my settings (Debian 10, R-4.0.0 and Debian 7, R-3.6.1). Could you give us some more information about your setting? Please note that, for Windows systems, we have a dedicated branch on GitHub (devtools::install_github("lutsik/MeDeCom",ref="windows")). For MacOS, you should use the binary available from GitHub (https://github.com/lutsik/MeDeCom/releases/download/v1.0.0/MeDeCom_1.0.0.tgz).

In case none of these installations are working, we also created a Docker container comprising MeDeCom, DecompPipeline and FactorViz available from https://hub.docker.com/r/mscherer/medecom. As a last instance, this should solve all installation issues.

Best,

Michael

[Peter Mcerlean]

Hi Micheal, 

Thanks for the reply. 

I'm on a MAC, OS Sierra and R 3.6.3 via R Studio v1.2.5033. 

I thought that it might have been an update issue but when I found it was restricted to MeDeCom I was equally perplexed. And yes, I did try the binary version but still threw up the error. Should I try running it with MeDeCom 0.3.0?

I also thought about the Docker container so may just go ahead a use it. 

Also, my MAC has 2.2GHz processor and 16GB memory, do you think its capable of running 44 EPIC array samples through the pipeline?

Best,

Peter

[Michael Scherer]

Hi Peter,

Ok, this is odd. We found that the binary version was working properly in our hands. But thanks for reporting this, we will look further into this. The core functionality of MeDeCom is also available in version 0.3.0, just some interpretation functions, as we report in our preprint (https://www.biorxiv.org/content/10.1101/853150v2) are not available in the older version. You might think about running the analysis on 0.3.0 and using the Docker container for the interpretation.

Yes, your MAC should be sufficient to run such an analysis, although we made better experience when using the framework on larger compute servers (with e.g. 128GB of main memory). Let us know if you face further issues.

Best,

Michael

[Peter Mcerlean]

Hi Michael, 

So I updated to R 4.0.0 and found I was getting this error:

dyld: lazy symbol binding failed: Symbol not found: _utimensat

  Referenced from: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libR.dylib (which was built for Mac OS X 10.13)

  Expected in: /usr/lib/libSystem.B.dylib

So seems my MAC OS could be the issue?

I ended up bitting the bullet and went the Docker route but am running into other, probably novice issues.

I can use links provided in terminal and launch the libraries well enough:

library(RnBeads)

library(MeDeCom)

library(DecompePipeline)

library(FactorViz)

But since I already have a "rnb.set" file, I'm jumping straight into the analysis but seems my files aren't being recognized?

> ref.set <- load.rnb.set("/Users/.../rnbSet_preprocessed")

Error in load.rnb.set("/Users/.../rnbSet_preprocessed") : 

  invalid value for path; the path does not exist

> getwd()

[1] "/"

> setwd ("/Users/../Decomp")

Error in setwd("/Users/PFM/Decomp") : cannot change working directory

The path and file are good as they can be loaded into R studio (reverted back to R 3.6.3). I suspect I'm not making a directory to point the files/analysis too?

Best,

Peter

[Michael Scherer]

Hi Peter,

Indeed, it looks like a MacOS thing. Sorry that I cannot help you further on that, since I don't have access to a MAC.

For the Docker environment, you will have to specify your local folder structure to Docker, since it constitutes a subsystem, which does not have access to your PC's folder structure. This can be done using the following command:

docker run mscherer/medecom -v <PC_path>:/mnt/win

I would like to refer to the Docker documentation for further information about Docker.

Best,

Michael

[Peter Mcerlean]

Hi Michael,

So I ran the pipeline using MeDeCom 0.3.0/FactorViz etc all within R studio and it seemed to work ok with some interesting LMCs identified (attached).  

FYI, for my 44 EPIC array samples using default K/lambda settings the analysis took around 3.5 days (gulp!).

I've been wanting to see how my results compare to reference data sets (WGBS) but running the run.refbased option is giving me an error (below). 

I'm aware of missing values due to EPIC vs WGBS comparisons but don't think thats the issue.

I've played around a bit with some of the parameters (e.g. opt.method) but still throwing up same error. 

Another issue with my MAC OS version?

Best,

Peter

cg_subsets = 5000 most var CpGs

> ref.set

Object of class RnBeadRawSet

      44 samples

  784669 probes

of which: 784669 CpG, 0 CpH, and 0 rs

Region types:

  237884 regions of type tiling

   32060 regions of type genes

   41708 regions of type promoters

   24887 regions of type cpgislands

Intensity information is present

Detection p-values are present

Bead counts are present

Quality control information is present

Summary of normalization procedures:

The methylation data was normalized with method wm.dasen.

No background correction was performed.

>Blue.set

Object of class RnBiseqSet

      13 samples

  832953 methylation sites

Region types:

  247394 regions of type tiling

   43385 regions of type promoters

Coverage information is present

>Medecom.BlueP.test <-run.refbased(ref.set, Ks = 5, lambdas=0.0001, cg_subsets = cg_subset, opt.method = "MeDeCom.cppTAfact", temp.dir = NULL, ref.base = "local", most.var = NULL, NCORES = 4, cluster.settings = NULL, ref.set = Blue.set , id.col = "CellType",save.restricted.sites = FALSE)

2020-06-02 16:55:40     6.4  STATUS                 STARTED Imputation procedure knn 

2020-06-02 16:59:42     3.2  STATUS                 COMPLETED Imputation procedure knn 

2020-06-02 17:03:42     4.9  STATUS                 STARTED Imputation procedure knn 

Error: C stack usage  7971012 is too close to the limit

In addition: Warning message:

In knnimp(x, k, maxmiss = rowmax, maxp = maxp) :

  31376 rows with more than 50 % entries missing;

 mean imputation used for these rows

[Michael Scherer]

Hi Peter,

That really looks interesting. Please note that MeDeCom tests for different combinations of the regularizer lambda, the number of components K, and also performs cross-validation, which all contribute to the long running time. 3.5 days are not unusual for such an analysis.

The issue that you report is most likely insufficient main memory for the analysis that MeDeCom wants to perform, which is data imputation in this step. You should be able to solve this issue by setting an RnBeads options, in this case

rnb.options(enforce.memory.management=TRUE)

Hope that solves the issue.

[Peter Mcerlean]

Hi Michael,

Thanks for the suggestion. Using the 'enforce memory' option worked but unfortunately produced to errors when trying to plot (below). 

I read around a bit on the error and found it could be due to too many zero values so removed them in my reference set and reran the analysis to same effect. 

Upon further inspection I actually noticed that the analysis was not restricted to the 5K CpGs but all the EPIC probes. I tried various ways to select 5K CpGs and using either the cg_subset and most.var options. It seemed the error came up with the most.var option.

I'm not sure if the two issues are related? 

Thanks again for your continued help!

Best,

Peter

> rnb.options(enforce.memory.management=TRUE)

> Medecom.BlueP.test <-run.refbased(ref.set, Ks = 5, lambdas=0.0001, cg_subsets = cg_subset, opt.method = "MeDeCom.cppTAfact", temp.dir = NULL, ref.base = "local", most.var = NULL, NCORES = 4, cluster.settings = NULL, ref.set = Blue.set , id.col = "CellType",save.restricted.sites = FALSE)

2020-06-04 09:05:19     2.3    INFO                     Low memory imputation not compatible with method knn, switched to mean.cpgs

2020-06-04 09:05:19     2.3  STATUS                     STARTED Low memory footprint version of imputation

2020-06-04 09:05:24     3.9  STATUS                     COMPLETED Low memory footprint version of imputation

2020-06-04 09:13:32     4.9    INFO                     No low memory footprint imputation available for matrix

[Main:] checking inputs

[Main:] preparing data

[Main:] preparing jobs

[Main:] 11 factorization runs in total

[Main:] 11 runs complete

[Main:] finished all jobs. Creating the object

> Medecom.BlueP.test

$MeDeComSet

An object of class MeDeComSet

Input data set:

776688 CpGs

44 methylomes

Experimental parameters:

k values: 5

lambda values: 1e-04

        ....etc.

        ....

        ....until end.....

[76,] 0.841269841 0.848484848 0.87500000 0.866666667 0.70967742

 [ reached getOption("max.print") -- omitted 776612 rows ] <- All epic probes?

>plotLMCs(Medecom.BlueP.test, K=5, lambda=0.0001, type="dendrogram")

Error in (function (classes, fdef, mtable)  : 

  unable to find an inherited method for function ‘getLMCs’ for signature ‘"list"’

Removing NAs

> meth.data <- meth(Blue.set)

> has.missing <- apply(meth.data,1,function(x)any(is.na(x)))

> BlueP.NAs.removed <- remove.sites(BlueP.test,has.missing)

> BlueP.NAs.removed

Object of class RnBiseqSet

      13 samples

  661849 methylation sites

Region types:

  217439 regions of type tiling

   39651 regions of type promoters

Coverage information is present

CpG selection issue:

most.var = 5000

or

CpGs <-5000

most.var = CpGs

Running same parameters as above produces this:

Error in get.i.vector(probelist, rownames(object@sites)) : 

  specified logical vector is of unexpected length

[Michael Scherer]

Hi Peter,

Thanks for reporting this. This might be a bug in the run.refbased function, and I will look further into this and let you know about any updates.

For the removing sites method, it looks as if the vector that you specify does not have the same length as the number of CpGs in the object. Are the Blue.set and the BlueP.test set the same things?




Sorry for not being able to solve the issue.


[Peter Mcerlean]

Hi Michael,

No worries. 

Datasets used in the analysis are below. 

Let me know if you'd like me to share any of the data. 

Best,

Peter

> Blue.set (WBGS data)

Object of class RnBiseqSet

      13 samples

  832953 methylation sites

Region types:

  247394 regions of type tiling

   43385 regions of type promoters

Coverage information is present

> ref.set (EPIC Arrays)

Object of class RnBeadRawSet

      44 samples

  784669 probes

of which: 784669 CpG, 0 CpH, and 0 rs

Region types:

  237884 regions of type tiling

   32060 regions of type genes

   41708 regions of type promoters

   24887 regions of type cpgislands

Intensity information is present

Detection p-values are present

Bead counts are present

Quality control information is present

Summary of normalization procedures:

The methylation data was normalized with method wm.dasen.

No background correction was performed.

> Medecom.BlueP.test (medecom output)

$MeDeComSet

An object of class MeDeComSet

Input data set:

776688 CpGs

44 methylomes

Experimental parameters:

k values: 5

lambda values: 1e-04

$RefMeth....etc....