[Rocks-Discuss] SGE state problems and state always end in Eqw for all submitted jobs
Cumberland, Lonnie
I hope that you are doing well today.
We have recently set up the Rocks cluster which seems to be working well except for some problems with SGE.
It seems that we can submit jobs via qsub and they even run, but after running everything changes the qstate state from "t" to "Eqw" and there are no output or error files produced which could tell us what is happening.
I walked through the "roll-sge-usersguide.pdf" with the MPI example and can see the job migrate to the various nodes via the "qstat -f" command but still ends in the "Eqw" state.
Not sure what is happening here. Any ideas?
Thanks and have a great day,
Lonnie Cumberland, Prof.
Physicist
(301) 975-6869 (Office)
(313) 333-2935 (Cell)
(301) 926-7416 (Fax)
National Institute of Standards and Technology
Radiation and Biomolecular Physics Division (682)
Radiation Physics Group (245), Room C106
ADDRESS:
100 Bureau Drive, Stop 682
Gaithersburg, MD 20899
EMAIL: lonnie.c...@nist.gov<mailto:lonnie.c...@nist.gov>
________________________________
CONFIDENTIALITY NOTICE - This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify Lonnie Cumberland by telephone or at Lonnie.C...@nist.gov and delete the original transmission and its attachments without reading or saving in any manner.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120127/2039e751/attachment.html
Xavier
Can't say exactly what is wrong, but clearly you jump from t(ransfert)
to E(rror)qw, without running anythings in between (r state).
How do you submit your job? do you have a script? can you post it?
Xavier
--
Universidade da Madeira
CCM - Centro de Ciencias Matematicas
Campus Universitario da Penteada
9000-390 Funchal, Madeira Island
Portugal
Pat Haley
Hi,
I notice that you mention using an MPI example. Have you tried
running a simple serial job (not using MPI), just to isolate
the problem?
I ask because the way our cluster is set-up, we needed to
set ssh keys to be able to run parallel (MPI or PVM) jobs.
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: pha...@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-222B http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Felipe
Are you using using ldap?
If so just to try restart sgeexec on your nodes:
/etc/init.d/sgeexecd stop
/etc/init.d/sgeexecd start
Regards,
----------------------------------
Felipe Formiga
NSCAD Microeletronica - UFRGS/CEI
Phone: +55 (51) 3308-7024
http://www.nscad.org.br
----------------------------------
----lonnie.c...@nist.gov wrote ----
Greetings All,
EMAIL: lonnie.c...@nist.gov<mailto:lonnie.c...@nist.gov>
Felipe
"qstat -j <jub_number_id>"
Regards,
----------------------------------
Felipe Formiga
NSCAD Microeletronica - UFRGS/CEI
Phone: +55 (51) 3308-7024
http://www.nscad.org.br
----------------------------------
----lonnie.c...@nist.gov wrote ----
Greetings All,
EMAIL: lonnie.c...@nist.gov<mailto:lonnie.c...@nist.gov>
Hung-Sheng Tsao (laoTsao)
Sent from my iPad
Cumberland, Lonnie
We are using the 5.4.3 version of Rocks Linux.
I tried the serial version using a simple script:
-------------------------- sleep.sh
#!/bin/bash
#
#$ -cwd
#$ -j y
#$ -S /bin/bash
#
date
sleep 10
date
as well as the mpi version
---------------------------------- mpi-ring.qsub
#!/bin/bash
#
#$ -cwd
#$ -j y
#$ -S /bin/bash
#
/opt/openmpi/bin/mpirun $HOME/test/mpi-ring
-------------------------------------------------------------------------
Both examples give the same "Eqw" state but seem to go through the transfer "t" state first.
The examples have come from the file "roll-sge-usersguide.pdf" of which there are a number of them on the web and one can be found here:
http://grid.ateneo.edu/roll-documentation/sge/4.3/roll-sge-usersguide.pdf
We have mapped in the Windows Active Directory as our domain manager for user authentication which seems to work really well.
Thanks and have a great day,
Lonnie Cumberland, Prof.
Physicist
(301) 975-6869 (Office)
(313) 333-2935 (Cell)
(301) 926-7416 (Fax)
National Institute of Standards and Technology
Radiation and Biomolecular Physics Division (682)
Radiation Physics Group (245), Room C106
ADDRESS:
100 Bureau Drive, Stop 682
Gaithersburg, MD 20899
EMAIL: lonnie.c...@nist.gov
CONFIDENTIALITY NOTICE - This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify Lonnie Cumberland by telephone or at Lonnie.C...@nist.gov and delete the original transmission and its attachments without reading or saving in any manner.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: roll-sge-usersguide.pdf
Type: application/pdf
Size: 182407 bytes
Desc: roll-sge-usersguide.pdf
Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120127/eecd1167/roll-sge-usersguide.pdf
Cumberland, Lonnie
[lcumber@spartan test]$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
-------------------------------------------------------------------------------- ---------------------------------
20 0.55500 mpi-ring.q lcumber Eqw 01/27/2012 12:14:56 60
[lcumber@spartan test]$ qstat -j 20
==============================================================
job_number: 20
exec_file: job_scripts/20
submission_time: Fri Jan 27 12:14:56 2012
owner: lcumber
uid: 31907
group: domain
gid: 10513
sge_o_home: /export/home/NIST/lcumber
sge_o_log_name: lcumber
sge_o_path: /usr/kerberos/bin:/usr/java/latest/bin:/usr/local/bi n:/bin:/usr/bin:/opt/bio/ncbi/bin:/opt/bio/mpiblast/bin/:/opt/bio/hmmer/bin:/opt /bio/EMBOSS/bin:/opt/bio/clustalw/bin:/opt/bio/tcoffee/bin:/opt/bio/phylip/exe:/ opt/bio/mrbayes:/opt/bio/fasta:/opt/bio/glimmer/bin://opt/bio/glimmer/scripts:/o pt/bio/gromacs/bin:/opt/bio/gmap/bin:/opt/bio/tigr/bin:/opt/bio/autodocksuite/bi n:/opt/bio/wgs/bin:/opt/eclipse:/opt/ganglia/bin:/opt/ganglia/sbin:/opt/maven/bi n:/opt/rocks/bin:/opt/rocks/sbin:/opt/condor/bin:/opt/condor/sbin:/opt/gridengin e/bin/lx26-amd64:/export/home/NIST/lcumber/bin
sge_o_shell: /bin/bash
sge_o_workdir: /state/partition1/home/NIST/lcumber/test
sge_o_host: spartan
account: sge
cwd: /export/home/NIST/lcumber/test
merge: y
mail_list: lcu...@spartan.local
notify: FALSE
job_name: mpi-ring.qsub
jobshare: 0
shell_list: NONE:/bin/bash
env_list:
script_file: mpi-ring.qsub
parallel environment: orte range: 60
error reason 1: can't get password entry for user "lcumber". Either the user does not exist or NIS error!
scheduling info: (Collecting of scheduler job information is turned o ff)
We have just set up the head node with Windows Active Directory for user authentication on the NIST domain and perhaps there is a problem there.
Does each node have to somehow be set up to Windows Active Directory as well?
Thanks and have a great day,
Lonnie Cumberland, Prof.
Physicist
(301) 975-6869 (Office)
(313) 333-2935 (Cell)
(301) 926-7416 (Fax)
National Institute of Standards and Technology
Radiation and Biomolecular Physics Division (682)
Radiation Physics Group (245), Room C106
ADDRESS:
100 Bureau Drive, Stop 682
Gaithersburg, MD 20899
EMAIL: lonnie.c...@nist.gov
> From: npaci-rocks-dis...@sdsc.edu [mailto:npaci-rocks-
> discussio...@sdsc.edu] On Behalf Of Felipe
> Sent: Friday, January 27, 2012 11:42 AM
> To: Discussion of Rocks Clusters
> Subject: Re: [Rocks-Discuss] SGE state problems and state always end in Eqw
> for all submitted jobs
>
Rayson Ho
Rayson
=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/
Scalable Grid Engine Support Program
http://www.scalablelogic.com/
Bart Brashers
> authentication on the NIST domain and perhaps there is a problem there.
Ideally, I would like to make the users by hand, but have the AD server handle authentication. That would keep me from having to change my password in two places (Windows and Linux) every time.
If you have a way of doing that, I'd be very appreciative if you could post it.
There's some stuff on the wiki, but I couldn't parse well enough in the limited amount of time I had...
Bart
________________________________
This message contains information that may be confidential, privileged or otherwise protected by law from disclosure. It is intended for the exclusive use of the Addressee(s). Unless you are the addressee or authorized agent of the addressee, you may not review, copy, distribute or disclose to anyone the message or any information contained within. If you have received this message in error, please contact the sender by electronic reply to em...@environcorp.com and immediately delete all copies of the message.
Xavier
rocks run host " ps aux | grep sge_execd"
Are you sure your execution deamon are running?
--
Cumberland, Lonnie
rocks run host " ps aux | grep sge_execd"
shows to be running on each node.
Luca Clementi
if you do a
qstat -j <jobID>
at the end of the long output it should tell you why the job is in E
(error) state.
You can get the job id from the output of the:
qstat
Sincerely,
Luca
Rayson Ho
*If* all you need is to run code on the cluster for a few users (or just your
own account), then I think the easiest would be to create the lcumber user
"manually" on Rocks (similar to syncing users from an external NIS server - but
I don't know enough of AD to dump the user ids... so you need to figure out
this part if you want to automate this process).
http://www.rocksclusters.org/roll-documentation/base/5.4/faq-sysadmin.html#FOUR11-NIS
And let me reiterate, as long as the user account is there on the nodes, then
SGE can run jobs on them for the user (in your case, as "lcumber"). You can
have Windows AD handling authentication on the front-end. As long as the
"lcumber" account is there on the nodes, and as long as SGE can switch the user
id to lcumber (as root, run "su - lcumber" to make sure - if su can do it, then
SGE can do it as well) on the nodes, then SGE is happy to run jobs for you.
evm
I've been mucking around in building ROCKS clusters for a few years
now. Mostly on the backburner and with so many other things going on
that I really don't get a chance to gain the masetry of building ROCKS.
Always under the gun to get something going....
I need to install Intel fortran (we have licenses) and an OFED. I have
Mellanox OFED for Linux and the Rolls working group Intel 12 complier to
install.
Any thoughts on this as a good direction?
(also, what is a good US conference to go to to come up to speed on
Running and admin of a rocks cluster)
regards, Ethan
--
Ethan VanMatre
Research Systems Engineer
Center for Coastal Margin Observation and Prediction
Oregon Health Science University
503-748-1157
e...@stccmop.org
"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."
one way is check out the triton roll of ofed and intel
http://git.rocksclusters.org/git/gitweb.cgi
http://git.rocksclusters.org/git/gitweb.cgi?p=triton/intel/.git;a=summary
http://git.rocksclusters.org/git/gitweb.cgi?p=triton/ofed/.git;a=summary
of course you need rocks source e.g. 5.4.3 to make roll
the other options check out rocks+
http://www.stackiq.com/download/
regards
On 1/27/2012 3:33 PM, evm wrote:
> Hi All,
>
> I've been mucking around in building ROCKS clusters for a few years
> now. Mostly on the backburner and with so many other things going on
> that I really don't get a chance to gain the masetry of building
> ROCKS. Always under the gun to get something going....
>
> I need to install Intel fortran (we have licenses) and an OFED. I have
> Mellanox OFED for Linux and the Rolls working group Intel 12 complier
> to install.
>
> Any thoughts on this as a good direction?
>
> (also, what is a good US conference to go to to come up to speed on
> Running and admin of a rocks cluster)
>
> regards, Ethan
>
--
Hung-Sheng Tsao Ph D.
Founder& Principal
HopBit GridComputing LLC
cell: 9734950840
http://laotsao.blogspot.com/
http://laotsao.wordpress.com/
http://blogs.oracle.com/hstsao/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available
Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120127/a1e7936d/laotsao.vcf
Felipe
Maybe it will help you:
http://blog.cundy.ca/2010/04/01/rocks-cluster-5-3-ldap-authentication/
Regards
----------------------------------
Felipe Formiga
NSCAD Microeletronica - UFRGS/CEI
Phone: +55 (51) 3308-7024
http://www.nscad.org.br
----------------------------------
----rayso...@yahoo.com wrote ----
Lonnie,
http://www.rocksclusters.org/roll-documentation/base/5.4/faq-sysadmin.html#FOUR11-NIS
Rayson