This doesn't look normal to me.
>
> Looking with "top" at the R process, I realized that it uses only
> about 0.3-5.0 %CPU although it uses only about 24 %MEM.
> Do you know why it does not use 99 %CPU?
Are you working toward a shared file system or a local disk? Working
with files over a shared files system can be a killer.
If you look at the verbose output, you can see after each chunk how
much time fit() spend in reading, fitting, and writing, respectively.
What values to you see?
>
> Looking at the source code, the time-consuming step seems to be:
> # Fit model using affyPLM code
> if (!is.null(w)) {
> fit <- .Call("R_wrlm_rma_default_model", y, psiCode, psiK, w,
> PACKAGE=rlmPkg);
> } else {
> fit <- .Call("R_rlm_rma_default_model", y, psiCode, psiK,
> PACKAGE=rlmPkg);
> }
I'm not sure what you mean by this: you mean that you'd except that
this should be where you would spend most of the time, but you don't?
>
> Furthermore, I found the following mail:https://stat.ethz.ch/pipermail/
> bioconductor/2007-September/019286.html
> Could this be the reason?
It could be, but I doubt it. I don't think that kicks in that hard
already at 338 arrays. You could also try to compare with using
AvgCnPlm instead of RlmCnPlm and see if there is a big difference.
>
> What can I do to increase the processing time?
Run more things in the background ;) Seriously, as I mentioned in
earlier threads, handling the single-probe CN units separately by
immediately transfer their intensities to chip effects would speed up
things a lot. If done correctly, that will come down to copying
values from the input CEL files to the output chip-effect CEL files;
right now those are processed through the complete PLM fitting
mechanism.
Cheers
Henrik
Oh, I thought that was provided after each chunk, but I guess that is
only the information of ETA.
> However, I did run another job with only 22 samples and for these the
> output is:
>
> Fitting chunk #27 of 27...done
> Total time for all units across all 22 arrays: 5656.53s == 94.28min
> Total time per unit across all 22 arrays: 0.02s/unit
> Total time per unit and array: 1.08ms/unit & array
> Total time for one array (238378 units): 4.29min = 0.07h
> Total time for complete data set: 94.28min = 1.57h
> Fraction of time spent on different tasks: Fitting: 7.3%, Reading:
> 62.7%, Writing: 29.6% (of which 71.24% is for encoding/writing chip-
> effects), Explicit garbage collection: 0.4%
> Fitting model of class RmaCnPlm:...done
>
> As you see, most time is spent with reading/writing, nevertheless the
> total time for one array was only 4.29min and not 50min as in the case
> of 338 samples.
It is hard to compare this figures with the 338 array data set,
because of the differ chunk sizes etc. However, from even from the
above 22 array data set, it looks to me that you spend a fair bit of
time (62.7%) in reading the data. If you compare this to what you get
if you work toward a local drive, you'll probably see that it will be
much faster. Note that the "writing" fraction is mostly spend in the
unwrapping/wrapping (encoding/writing) part, and not per se for the
actual writing to file. The writing to file is very roughly
(100-71.24)% of 29.6%.
Another reason for the processing time of 338 arrays seems
"exponentially" slower than for 22 arrays, could be that there are
more file-cache misses when more files (and more disk space) have to
be accessed in each chunk. With 22 arrays, the file cache might get
more read/write cache hits.
See the thread 'Parallelizing the Total Copy Number Analysis (GWS6)
Options' and my reply on April 8, 2008.
In order to do this, it is important to understand how the PLM
model(s) work. When you do that you will also understand that there
are no probe-affinity parameters to fit for the single-probe units, so
the whole modeling vanishes. This will lead you to a simple "model"
where the chip-effect estimate of a single-probe CN units is identical
to the intensity of the CN probe. Thus, you can just copy the value
over regardless of PLM.
> Is there also a possibility for me to do things in parallel?
Yes, but that should not be the first goal. The main gain will be the
skipping of the fitting mechanism for all CN probes (which are 50% of
the units on the new arrays).
/Henrik
On Tue, Jul 8, 2008 at 4:57 AM, cstratowa
<Christian...@vie.boehringer-ingelheim.com> wrote:
>
>
Use the 'ram' argument instead. The 'moreUnits' argument is obsolete
and there for backward compatibility.
The 'bytesPerChunk' is just a scaling constant. Don't consider it an
estimate of the number of bytes per chunk, at most a number
proportional to the number of bytes per chunk, but it depends a lot on
the fitting algorithm called and that varies with PLMs (and
'flavor':s). It also depends on the number of probes in a unit, which
depends on the specific CDF etc. So you cannot really predict the
number bytes.
Anyway, if you use ram > 1, more units per chunk will be fitted, and
with ram < 1, less units per will be fitted, than what is the default
(ram = 1). So, if you see that you are not using all of your RAM, you
can increase the 'ram' argument. This will use more memory. It will
also decrease the I/O overhead of opening and closing files for
reading, because this is done fewer times when there are less chunks.
In each chunk, a larger portion of the CEL files are also
read/writing, which will improve the I/O speed *per unit* as well (the
reason/details are in affxparser::updateCel()). However, I think
there are properties that also vary with hardware system and file
system. My guts tells me though that a larger value of 'ram' will
speed up things. There is probably an upper limit when the marginal
gain vanish. Basically it is a trial an error to find out. You're
welcome to post some summary data here, if you do a comparison.
Cheers
Henrik
On Wed, Jul 9, 2008 at 6:58 AM, cstratowa
<Christian...@vie.boehringer-ingelheim.com> wrote:
>
> Dear Henrik
>
> Thank you.
>
> Meanwhile our sysadmin has included a server with 32GB RAM as one node
> in the cluster, which I will use for the normalization step.
>
> I will test different settings of the "ram" parameter, however, this
> will take some time. I will let you know.
>
> Currently, I am running 85 samples on the server node, but the output
> is sometimes strange. When fitting the chunks, some arrays are
> sometimes missing, e.g. Array #1 and Array #2. The output looks as
> follows:
I never seen this. Is the problem only for the verbose output? Are
the estimates still alright?
/Henrik