Re: PBDDmat with a matrix ~9.68GB !

[Ostrouchov, George]

Hi MinhDQ,

Thanks for providing more information about what you are doing. I cc back to the rbigdataprogramming group so others can see this.

I assume that you are running the code below in batch with "mpirun –np 10 Rscript your_code_file.r"   or a similar command on your server. This runs 10 copies of your code and readRDS() reads the entire file in each copy. This means about 10x9.68GB multiplied by copies that R likes to make before the distributed object starts construction. It makes sense that you would run out of memory on a 128GB server.

A possible quick fix might be to put 

if(comm.rank() == 0) {

  M <- readRDS(“ …" )

  sM <- as.matrix(M)

}

else sM <- NULL

So that only rank 0 reads the data before distributing it to other ranks.

Without a parallel file system you are still serial for reading the entire file. Since I/O is by far the slowest operation in your code, you may not see much speedup. With a parallel file system you would read different parts of a file in parallel by different cores.

George

From: <ankan...@gmail.com> on behalf of MinhDQ <minhd...@gmail.com>
Date: Wednesday, January 6, 2016 at 4:57 AM
To: "Ostrouchov, George" <gost...@utk.edu>
Subject: PBDDmat with a matrix ~9.68GB !

Dear George Ostrouchov,

I am using your package R-pbd (PBDDmat in this case) to distributed a matrix of about 9.68 GB with 2302607280 elements. Then, I had error in allocating memory when I try to distribute that matrix to apply colSums function on that. 

Here are my codes:

library(pbdDMAT,quiet = T)

pr <- 2 # row

pc <- 2 # column

init.grid (NPROW = pr, NPCOL = pc)

M <- readRDS("../../datal/pfizer_vs_hs_9_9_metahit_raw_smart_shared_reads_rounded.rda")

sM <- as.matrix(M)

dM <- distribute(sM, bldim = c(256,256))

rs <- colSums(dM)

comm.print(rs)

finalize()

and implemented on our server of 10 cores with 128 GB RAM running on CentOS 6.7

I posted on R-pbd google group with no answer and I've already read the post "problem with distributing a 50kx50k matrix" but I really did not get main idea of that post. 

So, could you show me more details how to distributed such a huge matrix like that with R-pbd package?

Thanks in advanced,

Best regards,

MinhDQ

--------------------------------------------------------------
Msc Dao Quang Minh (Mr.)
---------------Contact Imformation------------------
- Mobile: +84 9 04 32 47 27 | +84 9 36 03 02 86
- Home phone: +84 4 38 43 10 70
- Office phone: +84 4 38 68 23 55

- France phone: +33 6 52 90 35 22
- Website: http://hpcc.hust.edu.vn/~minhdq
- Email: minh.d...@hust.edu.vn
--------------------------------------------------------------

[MinhDQ]

Dear George Ostrouchov,

I think you didn't get my idea. Following errors are showed when distribute() function executes, not for as.matrix() fucntion.

xxmr2d:out of memory

Error in base.rpdgemr2d(x = dx@Data, descx = descx, descy = descy) : 

  exit mr2d_malloc

Calls: distribute -> dmat.reblock -> base.rpdgemr2d

Best,

[MinhDQ]

Dear George, 

I still get problem related to allocate memory when I read a huge matrix (9,68GB) only on rank 0 and then distribute that matrix.

I would like to ask you some advises about how to run in parallel way a very big matrix with R-pbd, If it is possible, could you show me in examples.

Thank you so much,

Waiting for your reply

MinhDQ 

--------------------------------------------------------------
Msc Dao Quang Minh (Mr.)
---------------Contact Imformation------------------
- Mobile: +84 9 04 32 47 27 | +84 9 36 03 02 86
- Home phone: +84 4 38 43 10 70
- Office phone: +84 4 38 68 23 55

- France phone: +33 6 52 90 35 22
- Website: http://hpcc.hust.edu.vn/~minhdq
- Email: minh.d...@hust.edu.vn
--------------------------------------------------------------

[MinhDQ]

Dear George, 

Could you spend your time to reply my email, please?

Thanks a lot!

MinhDQ

--------------------------------------------------------------
Msc Dao Quang Minh (Mr.)
---------------Contact Imformation------------------
- Mobile: +84 9 04 32 47 27 | +84 9 36 03 02 86
- Home phone: +84 4 38 43 10 70
- Office phone: +84 4 38 68 23 55

- France phone: +33 6 52 90 35 22
- Website: http://hpcc.hust.edu.vn/~minhdq
- Email: minh.d...@hust.edu.vn
--------------------------------------------------------------

[Ostrouchov, George]

Dear MinDQ,

If you are still having difficulty getting the data, consider looking at the memuse package.  Its functions can help you examine your memory use, object sizes, and available memory in each process at different stages of your code. There can be many causes of the out of memory error you are observing.

George Ostrouchov

From: <ankan...@gmail.com> on behalf of MinhDQ <minhd...@gmail.com>