using "demo.ncvar_put_dmat" function

[Dale Wang]

Hello everyone,

     I tried to use "demo.ncvar_put_dmat()" function provided in pbdDEMO to store my own ddmatrix object to a NCDF4 file but I met a problem.
    When I called "demo.ncvar_put_dmat()" function, I met the following error message:

     

Using 2x2 for the default grid size

Error in demo.ncvar_put_2D(nc, varid, vals, verbose = verbose, comm = comm) :
  dim(vals) and count are not consistent.
Calls: demo.ncvar_put_dmat -> demo.ncvar_put_2D
Execution halted

    I wrote my code following the demo file 'nc4_demo.r'(github page here). I modified the demo code to store a simple 4*3 matrix . My code is provided as below(also provided in the attachment):

library(pbdDEMO, quietly = TRUE)
library(pbdNCDF4, quietly = TRUE)

# -------------------------------------
# Write and read NetCDF4 file in ddmatrix
# -------------------------------------

### initial grid
init.grid()
if(comm.size() != 4){
  comm.stop("This example requries 4 processors.")
}

### divide data into ddmatrix
x <- matrix(c(1,2,3,4,5,6,7,8,9,10,11,12),nrow=4,ncol=3)
dx <- as.ddmatrix(x)

# define dimension and variable
dimRow <- ncdim_def("dimRow","1",vals=seq(1,4))
dimCol <- ncdim_def("dimCol","1",vals=seq(1,3))
var.def <- ncvar_def("Data","1",list(dimRow=dimRow,dimCol=dimCol),NULL)

### parallel write
file.name <- "nc4_dmat.nc"
nc <- nc_create_par(file.name, var.def)
demo.ncvar_put_dmat(nc, "Data", dx)
nc_close(nc)
if(comm.rank() == 0){
  ncdump(file.name)
}

finalize()

I ran the program with : mpirun -np 4 Rscript ./my_own_write.r I got the error message before.

Is there something wrong in my own code? or is it a bug?

Thank you for your discussion!

                                                        Dale Wang

 

[Wei-Chen Chen]

Fixed at https://github.com/snoweye/pbdDEMO

[Dale Wang]

Thank you Wei-Chen, I see your commit at github. And write part is OK now.

Unfortunately, there is also a bug in "demo.ncvar_get_dmat()".
After wring to the NCDF4 file, we can read the file back with the follwing code:(Attached a new version my_own_write.r)

### parallel write
file.name <- "my.nc"
nc <- nc_create_par(file.name, var.def)
demo.ncvar_put_dmat(nc, "Data", dx,verbose=FALSE)

nc_close(nc)
if(comm.rank() == 0){
  ncdump(file.name)
}

### parallel read
nc <- nc_open_par(file.name)
if(comm.rank() == 0)
{
    print(nc)
}
new.dx <- demo.ncvar_get_dmat(nc,"Data",bldim=bldim(dx),ICTXT=dmat.ictxt(dx))

In my environment, the program will hangs at the following code in "ncvar_2d.r"(in pbdDEMO/R/ncvar_2d.r line 161 ):

 ### parallel read
  pbdNCDF4::nc_var_par_access(nc, varid)
  comm.print("Start",all.rank=T) # I use it to debug
  vals <- try(pbdNCDF4::ncvar_get(nc, varid, start = start, count = count,
                                   verbose = verbose, signedbyte = signedbyte,
                                   collapse_degen = collapse_degen),
              silent = TRUE)
  comm.print("end") # I use it to debug

and I only get "Start" and No "end".

Can you check the problem? Thank you very much.
                                             Dale


在 2014年11月30日星期日UTC+8上午9时41分59秒，Wei-Chen Chen写道：

[Wei-Chen Chen]

I will fix the bugs until I get time ... steps as next ...
1. Add fake count to ncdf4 parallel get method and get a penny.
2. Convert vals to right matrix format fit in ictxt 1 with the penny if needed.

On Saturday, November 29, 2014 8:56:06 PM UTC-5, Dale Wang wrote:

Thank you Wei-Chen, I see your commit at github. And write part is OK now.

Unfortunately, there is also a bug in "demo.ncvar_get_dmat()".
After wring to the NCDF4 file, we can read the file back with the follwing code:(Attached a new version my_own_write.r)

### parallel write
file.name <- "my.nc"
nc <- nc_create_par(file.name, var.def)
demo.ncvar_put_dmat(nc, "Data", dx,verbose=FALSE)

nc_close(nc)
if(comm.rank() == 0){
  ncdump(file.name)
}

### parallel read
nc <- nc_open_par(file.name)
if(comm.rank() == 0)
{
    print(nc)
}
new.dx <- demo.ncvar_get_dmat(nc,"Data",bldim=bldim(dx),ICTXT=dmat.ictxt(dx))

In my environment, the program will hangs at the following code in "ncvar_2d.r"(in pbdDEMO/R/ncvar_2d.r line 161 ):

 ### parallel read
  pbdNCDF4::nc_var_par_access(nc, varid)
  comm.print("Start",all.rank=T) # I use it to debug
  vals <- try(pbdNCDF4::ncvar_get(nc, varid, start = start, count = count,
                                   verbose = verbose, signedbyte = signedbyte,
                                   collapse_degen = collapse_degen),
              silent = TRUE)
  comm.print("end") # I use it to debug

and I only get "Start" and No "end".

Can you check the problem? Thank you very much.
                                             Dale


在 2014年11月30日星期日UTC+8上午9时41分59秒，Wei-Chen Chen写道：

Fixed at https://github.com/snoweye/pbdDEMO

[Wei-Chen Chen]

Fixed.
As usual at https://github.com/snoweye/pbdDEMO.
More examples at the end of Ch10 vignettes.

[ZhaoKang Wang]

Thank you Wei-Chen, from the github commit log, it is a rather big patch. I will try it again these days :)

                         Dale(Zhaokang)

 

------------------ Original ------------------

From:  "Wei-Chen Chen";<wcc...@gmail.com>;

Date:  Mon, Dec 8, 2014 07:07 AM

To:  "rbigdataprogramming"<rbigdatap...@googlegroups.com>;

Subject:  [RBigData] Re: using "demo.ncvar_put_dmat" function

--
Programming with Big Data in R
Simplifying Scalability
http://r-pbd.org/
---
You received this message because you are subscribed to the Google Groups "RBigDataProgramming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rbigdataprogram...@googlegroups.com.
To post to this group, send email to rbigdatap...@googlegroups.com.
Visit this group at http://groups.google.com/group/rbigdataprogramming.
To view this discussion on the web visit https://groups.google.com/d/msgid/rbigdataprogramming/28c80e38-f6c5-4182-9918-fb6a1fe7647e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.