RHEL 5.3 - Error on Feature Picking

[Christian Paredes]

Hi,

I am trying to setup Corra on a two node "cluster", and have installed
Corra + Tomcat6 successfully. I'm running OSCAR 5.1rc1 on my head
node with OpenPBS. However, when I attempt to submit the job for
feature picking, the job fails:

Creating input file: /proteomics/analysis/Corra/users/cparedes/test/
param.def
DEBUG (Jul 24, 2009 2:48:48 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Submitting Corra Job with 6 child jobs
INFO (Jul 24, 2009 2:48:48 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor

---------------------------------------------------------------------------
COMMAND: ssh -x -i ~/.ssh/id_dsa corra@cluster qsub /proteomics/
analysis/Corra/users/cparedes/test/PBS/cparedes_test_SHFE_0.sh
---------------------------------------------------------------------------

INFO (Jul 24, 2009 2:48:49 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Job submitted: 21
INFO (Jul 24, 2009 2:48:49 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor

---------------------------------------------------------------------------
COMMAND: ssh -x -i ~/.ssh/id_dsa corra@cluster qsub /proteomics/
analysis/Corra/users/cparedes/test/PBS/cparedes_test_SHFE_1.sh
---------------------------------------------------------------------------

INFO (Jul 24, 2009 2:48:50 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Job submitted: 22
INFO (Jul 24, 2009 2:48:50 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor

---------------------------------------------------------------------------
COMMAND: ssh -x -i ~/.ssh/id_dsa corra@cluster qsub /proteomics/
analysis/Corra/users/cparedes/test/PBS/cparedes_test_SHFE_2.sh
---------------------------------------------------------------------------

INFO (Jul 24, 2009 2:48:50 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Job submitted: 23
INFO (Jul 24, 2009 2:48:50 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor

---------------------------------------------------------------------------
COMMAND: ssh -x -i ~/.ssh/id_dsa corra@cluster qsub /proteomics/
analysis/Corra/users/cparedes/test/PBS/cparedes_test_SHFE_3.sh
---------------------------------------------------------------------------

INFO (Jul 24, 2009 2:48:51 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Job submitted: 24
INFO (Jul 24, 2009 2:48:51 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor

---------------------------------------------------------------------------
COMMAND: ssh -x -i ~/.ssh/id_dsa corra@cluster qsub /proteomics/
analysis/Corra/users/cparedes/test/PBS/cparedes_test_SHFE_4.sh
---------------------------------------------------------------------------

INFO (Jul 24, 2009 2:48:51 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Job submitted: 25
INFO (Jul 24, 2009 2:48:51 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor

---------------------------------------------------------------------------
COMMAND: ssh -x -i ~/.ssh/id_dsa corra@cluster qsub /proteomics/
analysis/Corra/users/cparedes/test/PBS/cparedes_test_SHFE_5.sh
---------------------------------------------------------------------------

INFO (Jul 24, 2009 2:48:51 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
Job submitted: 26
DEBUG (Jul 24, 2009 2:49:34 PM)
org.systemsbiology.apps.corragui.server.executor.superhirn.FeModuleExecutor
Checking output of FE
ERROR (Jul 24, 2009 2:49:34 PM)
org.systemsbiology.apps.corragui.server.executor.superhirn.FeModuleExecutor
FAILED: Feature Picking cparedes_test_SHFE
ERROR (Jul 24, 2009 2:49:34 PM)
org.systemsbiology.apps.corragui.server.executor.superhirn.FeaturePickingExecutor
FAILED: Feature Picking SuperHirn_feature_picking
ERROR (Jul 24, 2009 2:49:34 PM)
org.systemsbiology.apps.corragui.server.executor.TrackableExecutor
checkOutput() failed on job: cparedes_test_SHFE

I have followed the directions very carefully and made sure that
the .xml conf files were copied correctly into the appropriate
folders. Also, I've generated a public / private SSH key pair for the
tomcat user on the head node, as well as created the tomcat and corra
users on the single slave node that I have set up so far. Further,
I've copied the entire /proteomics folder that was referenced in the
log file above into the slave node, plus made sure all of the owners
and permissions are equivalent to what's on the head node.

I'm at a loss at what else I have to do to get Corra to work, I'll be
happy to provide any other information as needed. Thanks in advance!

[Mark Christiansen]

Hi Christian,

Have you configured corra_webapp.properties?  Specifically, the fields below are necessary to run the shell scripts.

corra.scheduler.ssh.host=
corra.scheduler.ssh.user=
corra.scheduler.ssh.idfile=

If you have configured this file, I would suggest running the shell scripts that were generated from within their directory as the same user specified above.  By manually running the shell scripts, or looking at the PBS log file, you should be able to further troubleshoot why these shell scripts are failing.

Regards,

Mark

[Christian Paredes]

Cool, thanks for the tip! I think I've gotten the problem a little
bit more narrowed down now, but I'm running into permissions-hell. I
found out that I had the SGID's setup properly on /proteomics/analysis/
Corra/users directory, but since all of the subdirectories were owned
by tomcat, the corra user couldn't write in them properly. Here's
what I have so far in my directories (I've tried setting up the
permissions manually on the cparedes directory, but I'm going to go
through and reproduce what I ran into by using a new user, cparedes2:)

/proteomics/analysis/Corra/users
[root@cluster users]# ls -la
total 20
drwsrwsr-x 5 corra tomcat 4096 Jul 24 15:49 .
drwxr-xr-x 5 root root 4096 Jul 23 17:23 ..
drwxrwxr-x 4 corra tomcat 4096 Jul 24 12:58 cparedes
drwxr-sr-x 2 tomcat tomcat 4096 Jul 24 13:04 cparedes2
drwxr-sr-x 2 tomcat tomcat 4096 Jul 24 15:49 cparedes3

/proteomics/analysis/Corra/users/cparedes2
[root@cluster cparedes2]# ls -la
total 12
drwxr-sr-x 3 tomcat tomcat 4096 Jul 24 16:07 .
drwsrwsr-x 5 corra tomcat 4096 Jul 24 15:49 ..
drwxr-sr-x 2 tomcat tomcat 4096 Jul 24 16:07 New_Project

/proteomics/analysis/Corra/users/cparedes2/New_Project
[root@cluster New_Project]# ls -la
total 48
drwxr-sr-x 3 tomcat tomcat 4096 Jul 24 16:08 .
drwxr-sr-x 3 tomcat tomcat 4096 Jul 24 16:07 ..
-rw-r--r-- 1 tomcat tomcat 279 Jul 24 16:07 alignment_setup.xml
-rw-r--r-- 1 tomcat tomcat 11 Jul 24 16:08 alignment.status
-rw-r--r-- 1 tomcat tomcat 4033 Jul 24 16:08 corra.log
-rw-r--r-- 1 tomcat tomcat 935 Jul 24 16:08 feature_picking_setup.xml
-rw-r--r-- 1 tomcat tomcat 7 Jul 24 16:08 feature_picking.status
-rw-r--r-- 1 tomcat tomcat 1649 Jul 24 16:08 param.def
drwxr-sr-x 3 tomcat tomcat 4096 Jul 24 16:08 PBS
-rw-r--r-- 1 tomcat tomcat 2030 Jul 24 16:07 project_setup.xml
-rw-r--r-- 1 tomcat tomcat 505 Jul 24 16:07 statistics_setup.xml
-rw-r--r-- 1 tomcat tomcat 11 Jul 24 16:08 statistics.status

/proteomics/analysis/Corra/users/cparedes2/New_Project/PBS
[root@cluster PBS]# ls -la
total 36
drwxr-sr-x 3 tomcat tomcat 4096 Jul 24 16:08 .
drwxr-sr-x 3 tomcat tomcat 4096 Jul 24 16:08 ..
-rwxrwxrwx 1 tomcat tomcat 602 Jul 24 16:08
cparedes2_New_Project_SHFE_0.sh
-rwxrwxrwx 1 tomcat tomcat 602 Jul 24 16:08
cparedes2_New_Project_SHFE_1.sh
-rwxrwxrwx 1 tomcat tomcat 602 Jul 24 16:08
cparedes2_New_Project_SHFE_2.sh
-rwxrwxrwx 1 tomcat tomcat 600 Jul 24 16:08
cparedes2_New_Project_SHFE_3.sh
-rwxrwxrwx 1 tomcat tomcat 600 Jul 24 16:08
cparedes2_New_Project_SHFE_4.sh
-rwxrwxrwx 1 tomcat tomcat 600 Jul 24 16:08
cparedes2_New_Project_SHFE_5.sh
drwxr-sr-x 2 tomcat tomcat 4096 Jul 24 16:08 output

And nothing is inside ./output/. So I set the permissions on output
to corra:tomcat, which allowed the scripts to write into the output
directory, then I found out that the script cannot write into /var/
spool/pbs/mom_priv/jobs. So I made that owned by corra:tomcat + chmod
775, but that didn't work either!

-- Christian

On Jul 24, 3:40 pm, Mark Christiansen

<mchristian...@systemsbiology.org> wrote:
> Hi Christian,
>
> Have you configured corra_webapp.properties?  Specifically, the fields below
> are necessary to run the shell scripts.
>
> corra.scheduler.ssh.host=
> corra.scheduler.ssh.user=
> corra.scheduler.ssh.idfile=
>
> If you have configured this file, I would suggest running the shell scripts
> that were generated from within their directory as the same user specified
> above.  By manually running the shell scripts, or looking at the PBS log
> file, you should be able to further troubleshoot why these shell scripts are
> failing.
>
> Regards,
>
> Mark
>

[Christian Paredes]

Argh. Okay, so I've done a little bit of research into the SUID bit,
which I was hoping would automatically set the owner of every file and
directory in /proteomics/analysis/Corra/users as corra:tomcat.
However, on Linux, this seems to have no effect whatsoever on user
ownership; only the SGID bit has an effect on the group ownership.
Thus, every file / directory generated by Corra is owned by
tomcat:tomcat, rather than corra:tomcat.

-- christian

[Christian Paredes]

More findings: each of the .sh files that are generated by Corra in
the project/PBS directory seem to be missing the necessary reference
to the param.def file that SuperHirn needs:

/proteomics/analysis/Corra/users/cparedes/test/PBS/cparedes_SHFE_0.sh:

#!/bin/sh

#PBS -N cparedes_test_SHFE_0
#PBS -j oe
#PBS -o /proteomics/analysis/Corra/users/cparedes/test/PBS/output/cparedes_test_SHFE_0.oe
#PBS -l walltime=200:00:00
#PBS -l vmem=4294967296
#PBS -p 100

#change to working directory
cd /proteomics/analysis/Corra/users/cparedes/test
echo `pwd`

echo Queue ${PBS_QUEUE}
echo Job Id ${PBS_JOBID}


echo "/proteomics/analysis/Corra/bin/SuperHirn -FE
/proteomics/mzxml//080731APHDLAg4_01.mzXML >> SuperHirn.log"

time /proteomics/analysis/Corra/bin/SuperHirn -FE
/proteomics/mzxml//080731APHDLAg4_01.mzXML >> SuperHirn.log


The last two lines require an additional parameter, -PARAM
/path/to/param.def, customized for each project directory. For
example, in this case, since my test project directory is located at
/proteomics/analysis/Corra/users/cparedes/test, Corra should've added
the following at the end of each line:

-PARAM /proteomics/analysis/Corra/users/cparedes/test/param.def

I came to this conclusion after looking at the SuperHirn.log file
generated in corra's home directory:

********************************************************************
***** Welcome to S U P E R H I R N *****
********************************************************************


ERROR reading file '' in file_reader::open_file().

ERROR reading file '' in file_reader::open_file().

***
ERROR opening file '', read_param.cc::113
***
*** PROGRAM EXIT::incomplete parameter input ***


-- Christian

[Christian Paredes]

I think I might've fixed it. I had to expose the /proteomics
directory in NFS and mount it on all of my nodes. SpecArray is
working so far, I'll have to see if SuperHirn will work as well.

-- Christian

> On Fri, Jul 24, 2009 at 5:19 PM, Christian Paredes<merc...@gmail.com> wrote:
> > Argh.  Okay, so I've done a little bit of research into the SUID bit,
> > which I was hoping would automatically set the owner of every file and
> > directory in /proteomics/analysis/Corra/users as corra:tomcat.
> > However, on Linux, this seems to have no effect whatsoever on user
> > ownership; only the SGID bit has an effect on the group ownership.
> > Thus, every file / directory generated by Corra is owned by
> > tomcat:tomcat, rather than corra:tomcat.
>
> > -- christian
>

> ...
>
> read more »