Problem running GLAD with ChromosomeExplorer on Cluster nodes
cstratowa
We are trying to run GLAD on different cluster nodes using a perl
script which distributes an R-script to different cluster nodes. As
mentioned in an earlier discussion thread, we are facing the problem
of data overwrite when using ChromosomeExplorer, see:
http://groups.google.com/group/aroma-affymetrix/browse_thread/thread/2d3a7c10d512c539#
I did follow your advice to use tags:
> ce <- ChromosomeExplorer(glad, tags="*,foo")
and this seemed to solve the problem, e.g. the following directories
were created:
./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,1
./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,2
./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,3
./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,5
However, when running GLAD for six chips on six cluster nodes (one for
each chip), four of the tasks (1,2,3,5) actually completed, but the
others had some file/directory conflicts - see below. The order the
subtasks finished was arbitrary (e.g. subtask 1 was one of the last
ones to complete).
Tasks 4 and 6 failed probably due to the same problem:
4: Exception: Could not create file path: reports/includes/images
6: Exception: Could not create file path: reports/includes/js/
ChromosomeExplorer4
Do you have any idea why I get this message?
Why do only 2 of 6 tasks have this problem?
Do you have any idea how to solve this problem?
Here are the truncated outputs of the six shell scripts:
(Note: method "computeCN()" is my S4 class calling glad)
# - - - - - - - - - - - - - - - - - - - - - - -
Output of 1st shell script "sge_task.sh.o432.1":
### computing CN for array CEU_NA06985 ...
Loading required package: GLAD
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
Generating ChromosomeExplorer report...done
# - - - - - - - - - - - - - - - - - - - - - - -
Output of 2nd shell script "sge_task.sh.o432.2":
### computing CN for array CEU_NA06991 ...
Loading required package: GLAD
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
Generating ChromosomeExplorer report...done
# - - - - - - - - - - - - - - - - - - - - - - -
Output of 3rd shell script "sge_task.sh.o432.3":
### computing CN for array CEU_NA06993 ...
Loading required package: GLAD
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
Generating ChromosomeExplorer report...done
# - - - - - - - - - - - - - - - - - - - - - - -
Output of 4th shell script "sge_task.sh.o432.4":
### computing CN for array Molm13 ...
Loading required package: GLAD
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Error in list("computeCN(aroma, model = model, arrays = arrays[i],
chromosomes = NULL, ref" = <environment>, :
[2008-06-03 16:20:56] Exception: Could not create file path: reports/
includes/images
at throw(Exception(...))
at throw.default("Could not create file path: ", parent)
at throw("Could not create file path: ", parent)
at getWritablePathname.Arguments(static, file = "dummy-not-created",
path = pa
at getWritablePathname(static, file = "dummy-not-created", path =
path, ...)
at method(static, ...)
at Arguments$getWritablePath(to, mkdirs = TRUE, absolutePath =
FALSE)
at copyDirectory.default(file, to = filePath(to, basename), ...,
recursive = T
at copyDirectory(file, to = filePath(to, basename), ..., recursive =
TRUE)
at copyDirectory.default(from = srcPath, to = destPath, recursive =
TRUE, over
at copyDirectory(from = srcPath, to = destPath, recursive = TRUE,
overwrite =
at addIncludes.Explorer(this, ..., force = force)
at addIncludes(this, ..., force = force)
at setup.ChromosomeExplorer(this, ..., verbose = less(verbose))
at setup(this, ..., verbose
Calls: computeCN ... getWritablePathname.Arguments -> throw ->
throw.default -> throw -> throw.Exception
In addition: Warning messages:
1: In dir.create(pathname) : 'reports/includes/images' already exists
2: In dir.create(pathname) : 'reports/includes/images' already exists
Generating ChromosomeExplorer report...done
Execution halted
# - - - - - - - - - - - - - - - - - - - - - - -
Output of 5th shell script "sge_task.sh.o432.5":
### computing CN for array NCIH292 ...
Loading required package: GLAD
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
Generating ChromosomeExplorer report...done
# - - - - - - - - - - - - - - - - - - - - - - -
Output of 6th shell script "sge_task.sh.o432.6":
### computing CN for array Ref103 ...
Loading required package: GLAD
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Error in list("computeCN(aroma, model = model, arrays = arrays[i],
chromosomes = NULL, ref" = <environment>, :
[2008-06-03 16:20:56] Exception: Could not create file path: reports/
includes/js/ChromosomeExplorer4
at throw(Exception(...))
at throw.default("Could not create file path: ", parent)
at throw("Could not create file path: ", parent)
at getWritablePathname.Arguments(static, file = "dummy-not-created",
path = pa
at getWritablePathname(static, file = "dummy-not-created", path =
path, ...)
at method(static, ...)
at Arguments$getWritablePath(to, mkdirs = TRUE, absolutePath =
FALSE)
at copyDirectory.default(file, to = filePath(to, basename), ...,
recursive = T
at copyDirectory(file, to = filePath(to, basename), ..., recursive =
TRUE)
at copyDirectory.default(file, to = filePath(to, basename), ...,
recursive = T
at copyDirectory(file, to = filePath(to, basename), ..., recursive =
TRUE)
at copyDirectory.default(from = srcPath, to = destPath, recursive =
TRUE, over
at copyDirectory(from = srcPath, to = destPath, recursive = TRUE,
overwrite =
at addIncludes
Calls: computeCN ... getWritablePathname.Arguments -> throw ->
throw.default -> throw -> throw.Exception
In addition: Warning messages:
1: In dir.create(pathname) :
'reports/includes/js/ChromosomeExplorer4' already exists
2: In dir.create(pathname) :
'reports/includes/js/ChromosomeExplorer4' already exists
Generating ChromosomeExplorer report...done
Execution halted
I do not have a sessionInfo for the cluster, but it should be the same
as:
> sessionInfo()
R version 2.6.1 (2007-11-26)
x86_64-unknown-linux-gnu
locale:
C
attached base packages:
[1] tools stats graphics grDevices utils datasets
methods
[8] base
other attached packages:
[1] biasnp_0.2.10 Biobase_1.16.1
aroma.affymetrix_0.9.1
[4] aroma.apd_0.1.3 R.huge_0.1.5
digest_0.3.1
[7] aroma.light_1.6.0 affxparser_1.10.1
R.rsp_0.3.3
[10] R.cache_0.1.7 R.utils_1.0.2
R.oo_1.4.3
[13] R.methodsS3_1.0.0
loaded via a namespace (and not attached):
[1] rcompgen_0.1-17
Best regards
Christian
Henrik Bengtsson
On Wed, Jun 4, 2008 at 1:42 AM, cstratowa
<Christian...@vie.boehringer-ingelheim.com> wrote:
>
> Dear Henrik
>
> We are trying to run GLAD on different cluster nodes using a perl
> script which distributes an R-script to different cluster nodes. As
> mentioned in an earlier discussion thread, we are facing the problem
> of data overwrite when using ChromosomeExplorer, see:
> http://groups.google.com/group/aroma-affymetrix/browse_thread/thread/2d3a7c10d512c539#
>
> I did follow your advice to use tags:
>> ce <- ChromosomeExplorer(glad, tags="*,foo")
>
> and this seemed to solve the problem, e.g. the following directories
> were created:
> ./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,1
> ./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,2
> ./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,3
> ./reports/TestBatch/ACC,-XY,QN,RMA,A+B,FLN,-XY,5
Ok.
>
>
> However, when running GLAD for six chips on six cluster nodes (one for
> each chip), four of the tasks (1,2,3,5) actually completed, but the
> others had some file/directory conflicts - see below. The order the
> subtasks finished was arbitrary (e.g. subtask 1 was one of the last
> ones to complete).
>
> Tasks 4 and 6 failed probably due to the same problem:
> 4: Exception: Could not create file path: reports/includes/images
> 6: Exception: Could not create file path: reports/includes/js/
> ChromosomeExplorer4
>
> Do you have any idea why I get this message?
The *Explorer write common files to reports/includes/ each "run".
They are always the same. The clash is that your hosts try to
create/write the same directories/files at the same time (or when the
file systems have locked these). In this case, I don't think you have
to worry.
> Why do only 2 of 6 tasks have this problem?
By pure chance. Has nothing to do with aroma.affymetrix.
> Do you have any idea how to solve this problem?
Don't think it is a problem. Just try the *Explorer and see if it works.
Cheers
Henrik
Henrik Bengtsson
ok, now I see what your problem is. So, when you call process() on
the ChromosomeExplorer, an race-condition exception might occur while
creating the reports/includes/ files causing process() to
interrupt/quit before it calls fit() on the GladModel and plotts the
results. Since the setup of reports/includes/ is "orthogonal" to the
latter, one could let the Explorer to ignore these kind of errors.
I've updated the Explorer class with a alpha-version method
setParallelSafe(). After installing the patch:
library(aroma.affymetrix);
downloadPackagePatch("aroma.affymetrix");
call setParallelSafe(ce, TRUE) before calling process() on each of the
machines. That will make the Explorer to ignore "non-important"
exceptions that may occur due to parallel race conditions.
Tech details: Note that it this is very much a quick-and-dirty alpha
implementation; a much better solution would be to use a mutex
mechanism handling race conditions, but that is current not available
in R.
Let us know if this helps
Henrik
On Thu, Jun 5, 2008 at 7:36 AM,
<christian...@boehringer-ingelheim.com> wrote:
> Dear Henrik
>
> First, I am trying to reply in this way, since today it is for some reason
> impossible for me to signin to the aroma.affymetrix google group: I have
> cleared browser cash and cookies, reset the password, etc. as suggested by
> google help, but nothing worked.
>
> I would appreciate if you could reply to the following thread even though I
> could not put my question there:
> http://groups.google.com/group/aroma-affymetrix/browse_thread/thread/1ce7b462
> 1731e457#
> Thank you in advance
> Christian
>
> So here is my reply:
>
> I must admit that I do NOT understand your answer.
> As you see from the output, tasks 4 and 6 throw an exception, which means
> that GLAD did NOT compute the copy numbers for these two tasks, at least no
> data for the corresponding chips were stored in the subdirectory of
> "gladData" and the same is true for the "reports" subdirectory. Since no data
> for these two chips were computed, opening *Explorer does not solve the
> problem.
>
> Best regards
> Christian
cstratowa
Thank you very much for your fast patch, great!
Since we have to upgrade to aroma.affymetrix 0.9.2 and apply your
patch on all machines including the cluster, it may take same time
until I will be able to respond to let you know if your patch could
solve our problem. But I will keep you informed.
Best regards
Chrsitian
> > Thank you in advance
> > Christian
>
> > So here is my reply:
>
> > I must admit that I do NOT understand your answer.
> > As you see from the output, tasks 4 and 6 throw an exception, which means
> > that GLAD did NOT compute the copy numbers for these two tasks, at least no
> > data for the corresponding chips were stored in the subdirectory of
> > "gladData" and the same is true for the "reports" subdirectory. Since no data
> > for these two chips were computed, opening *Explorer does not solve the
> > problem.
>
> > Best regards
> > Christian
>
> On Wed, Jun 4, 2008 at 12:44 PM, Henrik Bengtsson <h...@stat.berkeley.edu> wrote:
> > Hi.
>
> > On Wed, Jun 4, 2008 at 1:42 AM, cstratowa
> > <Christian.Strat...@vie.boehringer-ingelheim.com> wrote:
>
> >> Dear Henrik
>
> >> We are trying to run GLAD on different cluster nodes using a perl
> >> script which distributes an R-script to different cluster nodes. As
> >> mentioned in an earlier discussion thread, we are facing the problem
> >> of data overwrite when using ChromosomeExplorer, see:
>
> read more »- Hide quoted text -
>
> - Show quoted text -
cstratowa
I am afraid that your patch did not work, or there is a new problem.
Here is the relevant code fragment, which is called by each node:
"computeCN" <- function(object, model="GLAD",explorer=TRUE, ...)
{
if (model == "GLAD") {
model <- GladModel(cesList, refList);
} else if (model == "CBS") {
model <- CbsModel(cesList, refList);
}#if
if (explorer) {
ce <- ChromosomeExplorer(model, tags=tags);
setParallelSafe(ce, status=TRUE);
tmp <- process(ce);
} else {
tmp <- fit(model);
}#if
return(object);
}#computeCN
Only two tasks compute GLAD as expected and store the results in
directory gladData.
In three tasks I get an error (see output below):
Exception: Failed to rename temporary file: plmData/TestPerl,etc....
In one task I get an error (see output below):
Error in readCel(pathname, readIntensities = FALSE, readStdvs =
FALSE,:
Cannot read CEL file. File not found: plmData/TestPerl,etc...
Could this be a new problem, since I did not get these error messages
before?
Here are the truncated outputs of the six shell scripts:
### computing CN for array CEU_NA06985_STY ...
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
chromosomes = NULL, ref" = <environment>, :
plmData/TestPerl,ACC,-XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-
intensities-median-mad,def1a76532f24e3899d90fcec50fa3ac.CEL.tmp ->
plmData/TestPerl,ACC,-XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-
intensities-median-mad,def1a76532f24e3899d90fcec50fa3ac.CEL
at throw(Exception(...))
at throw.default("Failed to rename temporary file: ", tmpPathname, "
-> ", pat
at throw("Failed to rename temporary file: ", tmpPathname, " -> ",
pathname)
at createFrom.AffymetrixCelFile(df, filename = filename, path =
getPath(this),
at createFrom(df, filename = filename, path = getPath(this), methods
= "create
at getAverageFile.AffymetrixCelSet(ces, force = force, verbose =
less(verbose)
at NextMethod(generic = "getAverageFile", object = this, indices =
indices, ..
at getAverageFile.ChipEffectSet(ces, force = force, verbose =
less(verbose))
at NextMethod(generic = "getAverageFile", object = this, ...)
at
Calls: computeCN ... createFrom.AffymetrixCelFile -> throw ->
In addition: Warning messages:
2: In dir.create(pathname) : 'reports/TestPerl' already exists
3: In dir.create(pathname) :
'gladData/TestPerl,ACC,-XY,QN,RMA,A+B,FLN,-XY' already exists
4: In dir.create(pathname) :
'gladData/TestPerl,ACC,-XY,QN,RMA,A+B,FLN,-XY' already exists
5: In dir.create(pathname) :
'gladData/TestPerl,ACC,-XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty'
already exists
6: In dir.create(pathname) :
'gladData/TestPerl,ACC,-XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty'
Generating ChromosomeExplorer report...done
Execution halted
# - - - - - - - - - - - - - - - - - - - - - - -
### computing CN for array CEU_NA06993_STY ...
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
Generating ChromosomeExplorer report...done
# - - - - - - - - - - - - - - - - - - - - - - -
### computing CN for array V2800 ...
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
Generating ChromosomeExplorer report...
Loading required package: Cairo
FALSE, :
Cannot read CEL file. File not found: plmData/TestPerl,ACC,-
XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-intensities-median-
mad,def1a76532f24e3899d90fcec50fa3ac.CEL
Calls: computeCN ... NextMethod -> getAverageFile.AffymetrixCelSet ->
readCel
In addition: Warning messages:
1: In dir.create(pathname) : 'reports/TestPerl' already exists
2: In dir.create(pathname) : 'reports/TestPerl' already exists
Execution halted
Christian
On Jun 10, 8:27 am, cstratowa <Christian.Strat...@vie.boehringer-
cstratowa
> sessionInfo()
R version 2.6.1 (2007-11-26)
x86_64-unknown-linux-gnu
locale:
C
attached base packages:
base
other attached packages:
[1] biasnp_0.2.11 aroma.affymetrix_0.9.3
aroma.apd_0.1.3
[4] R.huge_0.1.5 affxparser_1.10.1
aroma.core_0.9.3
[7] sfit_0.1.5 aroma.light_1.8.1
digest_0.3.1
[10] matrixStats_0.1.2 R.rsp_0.3.4
R.cache_0.1.7
[13] R.utils_1.0.2 R.oo_1.4.3
R.methodsS3_1.0.1
loaded via a namespace (and not attached):
[1] rcompgen_0.1-17
cstratowa
Now I am trying to run GLAD on the cluster w/o using
ChromosomeExplorer.
Of the six tasks only three are running GLAD as expected, the other
three tasks result in following errors:
# - - - - - - - - - - - - - - - - - - - - - - -
### computing CN for array CEU_NA06991_STY ...
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
No reference available.
Calculating average chip effects...
*** caught bus error ***
address 0x46791000, cause 'non-existent physical address'
Traceback:
1: .Call("R_affx_get_cel_file", filename, readHeader,
readIntensities, readXY, readXY, readPixels, readStdvs,
readOutliers, readMasked, indices, as.integer(verbose), PACKAGE =
"affxparser")
2: readCel(pathname, readIntensities = FALSE, readStdvs = FALSE,
readPixels = TRUE)
3: getAverageFile.AffymetrixCelSet(ces, force = force, verbose =
less(verbose), indices = "remaining")
4: NextMethod(generic = "getAverageFile", object = this, indices =
indices, ...)
5: getAverageFile.ChipEffectSet(ces, force = force, verbose =
less(verbose))
6: NextMethod(generic = "getAverageFile", object = this, ...)
7: getAverageFile.SnpChipEffectSet(ces, force = force, verbose =
less(verbose))
8: NextMethod(generic = "getAverageFile", object = this, ...)
9: getAverageFile.CnChipEffectSet(ces, force = force, verbose =
less(verbose))
10: getAverageFile(ces, force = force, verbose = less(verbose))
11: getReferenceSetTuple.CopyNumberChromosomalModel(this, verbose =
verbose)
12: getReferenceSetTuple(this, verbose = verbose)
13: fit.CopyNumberSegmentationModel(model, arrays = arrays,
chromosomes = chromosomes, force = force, verbose = verbose)
14: fit(model, arrays = arrays, chromosomes = chromosomes, force =
force, verbose = verbose)
15: .local(object, ...)
16: computeCN(aroma, model = model, arrays = arrays[i], chromosomes =
NULL, reference = NULL, explorer = xplor, tags = tags, verbose =
T)
17: computeCN(aroma, model = model, arrays = arrays[i], chromosomes =
NULL, reference = NULL, explorer = xplor, tags = tags, verbose =
T)
aborting ...
/net/uce/sge/default/spool/beo-50/job_scripts/1074: line 2: 26783 Bus
error (core dumped) R --no-save --args "path:/net/bioinf/
stratowa/aromacluster" "file:TestPerl2.Rdata" "model:GLAD:FALSE"
"array" $SGE_TASK_ID <batch.aroma.model.R
# - - - - - - - - - - - - - - - - - - - - - - -
### computing CN for array V2367 ...
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
No reference available.
Calculating average chip effects...
chromosomes = NULL, ref" = <environment>, :
plmData/TestPerl2,ACC,-XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-
intensities-median-mad,def1a76532f24e3899d90fcec50fa3ac.CEL.tmp ->
plmData/TestPerl2,ACC,-XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-
at throw(Exception(...))
at throw.default("Failed to rename temporary file: ", tmpPathname, "
-> ", pat
at throw("Failed to rename temporary file: ", tmpPathname, " -> ",
pathname)
at createFrom.AffymetrixCelFile(df, filename = filename, path =
getPath(this),
at createFrom(df, filename = filename, path = getPath(this), methods
= "create
at getAverageFile.AffymetrixCelSet(ces, force = force, verbose =
less(verbose)
at NextMethod(generic = "getAverageFile", object = this, indices =
indices, ..
at getAverageFile.ChipEffectSet(ces, force = force, verbose =
less(verbose))
at NextMethod(generic = "getAverageFile", object = this, ...)
throw.default -> throw -> throw.Exception
Building tuple of reference sets...done
# - - - - - - - - - - - - - - - - - - - - - - -
### computing CN for array V2799 ...
Loading required package: aws
Loading required package: tcltk
Loading Tcl/Tk interface ... done
[1] "Have fun with GLAD"
No reference available.
Calculating average chip effects...
terminate called after throwing an instance of
'affymetrix_calvin_exceptions::UnableToOpenFileException'
/net/uce/sge/default/spool/beo-52/job_scripts/1074: line 2: 24314
Aborted (core dumped) R --no-save --args "path:/net/
bioinf/stratowa/aromacluster" "file:TestPerl2.Rdata"
"model:GLAD:FALSE" "array" $SGE_TASK_ID <batch.aroma.model.R
The error for the 4th shell script is the same as reported earlier
today when using ChromosomeExplorer, however the other two errors are
new.
Any solutions?
Best regards
Christian
On Jun 11, 12:47 pm, cstratowa <Christian.Strat...@vie.boehringer-
Henrik Bengtsson
the patch I provided was only for the very basic race conditions in
ChromosomeExplorer, but not in preceeding models/algorithms, e.g.
GladModel, AvgCnPlm etc.
The whole paralizing of aroma.affymetrix (and other computations) is
complicated and the only safe solution to deal with race conditions
like the ones you are reporting is to work with a semaphore/mutex
mechanism. Such a solution is not available in R and until that it
available it does not make sense to put much effort to work around it
in aroma.affymetrix. I don't think the R language itself will provide
this feature in a long time. Instead I think "we" have to to create
"mutex" package that provides the necessary features. The challenges
is to make such a package to work on all platforms and more
importantly be bullet proof.
FYI, I've looked into options like using RSQLite to provide the mutex
functionality, but unfortunately it turns out that SQLite itself is
not parallel safe and depends heavily on the underlying OS. It seems
that the easiest solution is to have one dedicated server handing out
semaphores via a TCP/IP connection (opening and closing connections
are race-condition safe to the best of my knowledge). However, such a
solution is not immediately transparent to aroma.affymetrix, so I've
procrastinated that path. One can also work with lock files as
semaphores, but they are not bullet proof and there are cases where
two processes can lock the same file - it is better than nothing but
not safe.
To summarize, don't expect parallel computation in aroma.affymetrix
until there exist a well tested "mutex" package in R.
Cheers
Henrik
cstratowa
This is sad, thus my current trial is to create directories for each
chip and create symlinks to directories rawData, annotationData and
later to plmData.
However, I am not sure if a symlink to plmData will help, or if I will
get the following error again:
Exception: Failed to rename temporary file: plmData/TestPerl2,ACC,-
XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-intensities-median-
mad,def1a76532f24e3899d90fcec50fa3ac.CEL.tmp -> plmData/TestPerl2,ACC,-
XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-intensities-median-
mad,def1a76532f24e3899d90fcec50fa3ac.CEL
file created?
Does this ".average-intensities-median-mad,xxx.CEL" file contain the
average intensities of the normalized CEL-files?
Is there a way to create this ".average-intensities-median-
mad,xxx.CEL" file with a defined method after normalization, so that
GLAD "knows" that this average file exists already and does not to try
to create this file itself?
Best regards
Christian
>
> read more »- Hide quoted text -
Henrik Bengtsson
On Mon, Jun 16, 2008 at 7:07 AM, cstratowa
<Christian...@vie.boehringer-ingelheim.com> wrote:
>
> Dear Henrik
>
> This is sad, thus my current trial is to create directories for each
> chip and create symlinks to directories rawData, annotationData and
> later to plmData.
> However, I am not sure if a symlink to plmData will help, or if I will
> get the following error again:
>
> Exception: Failed to rename temporary file: plmData/TestPerl2,ACC,-
> XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-intensities-median-
> mad,def1a76532f24e3899d90fcec50fa3ac.CEL.tmp -> plmData/TestPerl2,ACC,-
> XY,QN,RMA,A+B,FLN,-XY/Mapping250K_Sty/.average-intensities-median-
> mad,def1a76532f24e3899d90fcec50fa3ac.CEL
>
> Can you explain what this error means? When and why is a temporary
> file created?
> Does this ".average-intensities-median-mad,xxx.CEL" file contain the
> average intensities of the normalized CEL-files?
> Is there a way to create this ".average-intensities-median-
> mad,xxx.CEL" file with a defined method after normalization, so that
> GLAD "knows" that this average file exists already and does not to try
> to create this file itself?
Yes, that file contains the average of the CEL files in a data set.
It is actually when getAverageFile() of AffymetrixCelSet calls
createFrom() of AffymetrixCelFile, when the temporary file is created
and exists. In your case, the different hosts happens to be running
createFrom() at the same time. The createFrom() method creates a
valid CEL files either by copying a template CEL file or creating it
from scratch. It also takes an argument 'clear' which will make sure
the CEL files has all zeros if clear=TRUE. If a template CEL file is
used, clear=TRUE has to "update" the CEL with all zeros, if created
from scratch, this is not needed. However, since createFrom() can be
a two-step process, working with a temporary file is the only way to
"guarantee" that you get what you requested (in case of interrupts
etc).
Ideally, getAverageFile() of AffymetrixCelSet would also work with a
temporary file first and when complete rename it to the final name.
That would the method more "atomic", i.e. either complete or fail.
Right now, I think it is possible to interrupt getAverageFile() and
get a incomplete file. Yet another thing on the todo list.
/Henrik
cstratowa
Thank you for your explanation.
Meanwhile I managed to run my program on a cluster in the following
way
- normalization is done on one node first
- the directory structure is replicated for each sample
- symlinks are created in each replicated directory to the normalized
CEL-files
- GLAD analyis is done in parallel in the replicated directories using
one node per sample
- results are copied from the replicated directories to the main sub-
directories
- replicated directories are deleted
Best regards
Christian
>
> read more »- Hide quoted text -
>