[slurm-users] Seff error with Slurm-18.08.1
Miguel A. Sánchez
I have updated my slurm from the 17.11.0 version to the 18.08.1. With
the previous version, the 17.11.0 version, the seff tool was working
fine but with the 18.08.1 version, when I try to run the seff tool I
receive the next error message:
# ./seff 5555
perl: error: plugin_load_from_file:
dlopen(/usr/local/slurm-18.08.2/lib/slurm/accounting_storage_slurmdbd.so):
/usr/local/slurm-18.08.2/lib/slurm/accounting_storage_slurmdbd.so:
undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for
accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for
accounting_storage/slurmdbd
perl: error: plugin_load_from_file:
dlopen(/usr/local/slurm-18.08.2/lib/slurm/accounting_storage_slurmdbd.so):
/usr/local/slurm-18.08.2/lib/slurm/accounting_storage_slurmdbd.so:
undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for
accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for
accounting_storage/slurmdbd
Job not found.
#
Both Slurm installations has been compiled from sources in the same
computer but only the seff that was compiled in the 17.11.0 version
works fine. To compile the seff tool, from the source Slurm tree:
cd contrib
make
make install
I think the problem is in the perlapi. Could it be a bug? Any Idea about
how can I fix this problem? Thanks a lot.
--
Miguel A. Sánchez Gómez
System Administrator
Research Programme on Biomedical Informatics - GRIB (IMIM-UPF)
Barcelona Biomedical Research Park (office 4.80)
Doctor Aiguader 88 | 08003 Barcelona (Spain)
Phone: +34/ 93 316 0522 | Fax: +34/ 93 3160 550
e-mail: miguelang...@upf.edu
Mike Cammilleri
I'm also interested in this issue since I've come across the same error today. We built Slurm-18.08.1 with the contribs packages on Ubuntu Bionic and seff is also complaining with
Mike Cammilleri
Systems Administrator
Department of Statistics | UW-Madison
1300 University Ave | Room 1280
608-263-6673 | mi...@stat.wisc.edu
Sent: Tuesday, October 23, 2018 10:26 AM
To: slurm...@schedmd.com
Subject: [slurm-users] Seff error with Slurm-18.08.1
Baker D.J.
Hello Mike et al,
This is a known bug in slurm v18.08*. We installed the initial release a short while ago and came across this issue very quickly. We actually use this script at the end of the job epilog to report job efficiency to users, and so it is real shame that it is now broken! The good new is that I am assured by SchedMD that the bug has been fixed in v18.08.3. Having said that we will probably live with this issue rather than disrupt users with another upgrade so soon . We have found a few other minor bugs in the new version of slurm, however I am glad to say that none of them are "life threatening".
If you're keen to have a work around in the meantime then please feel free to use our replacement script, "seff_new" -- a copy is attached with this email. It's not the most elegant of scripts, however it does work.
Best regards,
David
Sent: 05 November 2018 21:39
To: slurm...@schedmd.com; Slurm User Community List
Subject: Re: [slurm-users] Seff error with Slurm-18.08.1
Chris Samuel
> The good new is that I am assured by SchedMD that the bug has been fixed
> in v18.08.3.
commit 3d85c8f9240542d9e6dfb727244e75e449430aac
Author: Danny Auble <d...@schedmd.com>
Date: Wed Oct 24 14:10:12 2018 -0600
Handle symbol resolution errors in the 18.08 slurmdbd.
Caused by b1ff43429f6426c when moving the slurmdbd agent internals.
Bug 5882.
> Having said that we will probably live with this issue
> rather than disrupt users with another upgrade so soon .
should it? We just flip a symlink and the users see the new binaries,
libraries, etc immediately, we can then restart daemons as and when we
need to (in the right order of course, slurmdbd, slurmctld and then
slurmd's).
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
Mike Cammilleri
Thanks for this. We'll try the workaround script. It is not mission-critical but our users have gotten accustomed to seeing these metrics at the end of each run and its nice to have. We are currently doing this in a test VM environment, so by the time we actually do the upgrade to the cluster perhaps the fix will be available then.
Mike Cammilleri
Systems Administrator
Department of Statistics | UW-Madison
1300 University Ave | Room 1280
608-263-6673 | mi...@stat.wisc.edu
Sent: Tuesday, November 6, 2018 5:03 AM
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] Seff error with Slurm-18.08.1
Miguel A. Sánchez
Hi and thanks for all your answers and sorry for the delay in my answer. Yesterday I have installed in the controller machine the Slurm-18.08.3 to check if with this last release the Seff command is working fine. The behavior has improve but I still receive a error message:
# /usr/local/slurm-18.08.3/bin/seff 1694112
Use of uninitialized value $lmem in numeric lt (<) at
/usr/local/slurm-18.08.3/bin/seff line 130, <DATA> line
624.
Job ID: 1694112
Cluster: XXXXX
User/Group: XXXXX
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 01:39:33
CPU Efficiency: 4266.43% of 00:02:20 core-walltime
Job Wall-clock time: 00:01:10
Memory Utilized: 0.00 MB (estimated maximum)
Memory Efficiency: 0.00% of 3.91 GB (3.91 GB/node)
[root@hydra ~]#
And due to this problem, any job shows me as memory utilized the
value of 0.00 MB.
With slurm-17.11.1 is working fine:
# /usr/local/slurm-17.11.0/bin/seff 1694112
Job ID: 1694112
Cluster: XXXXX
User/Group: XXXXX
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 01:39:33
CPU Efficiency: 4266.43% of 00:02:20 core-walltime
Job Wall-clock time: 00:01:10
Memory Utilized: 2.44 GB
Memory Efficiency: 62.57% of 3.91 GB
[root@hydra bin]#
Miguel A. Sánchez Gómez System Administrator Research Programme on Biomedical Informatics - GRIB (IMIM-UPF) Barcelona Biomedical Research Park (office 4.80) Doctor Aiguader 88 | 08003 Barcelona (Spain) Phone: +34/ 93 316 0522 | Fax: +34/ 93 3160 550 e-mail: miguelang...@upf.edu
Marcus Wagner
this is because SchedMD changed the stats field. There exists no more rss_max, cmp. line 225 of seff.
You need to evaluate the field stats{tres_usage_in_max}, and there the value after '2=', but this is the memory value in bytes instead of kbytes, so this should be divided by 1024 additionally.
Best
Marcus
-- Marcus Wagner, Dipl.-Inf. IT Center Abteilung: Systeme und Betrieb RWTH Aachen University Seffenter Weg 23 52074 Aachen Tel: +49 241 80-24383 Fax: +49 241 80-624383 wag...@itc.rwth-aachen.de www.itc.rwth-aachen.de
Paddy Doyle
It looks like we can use the api to avoid having to manually parse the '2='
value from the stats{tres_usage_in_max} value.
I've submitted a bug report and patch:
https://bugs.schedmd.com/show_bug.cgi?id=6004
The minimal changes needed would be in the attched seff.patch.
Hope that helps,
Paddy
On Thu, Nov 08, 2018 at 11:54:59AM +0100, Marcus Wagner wrote:
> Hi Miguel,
>
>
> this is because SchedMD changed the stats field. There exists no more
> rss_max, cmp. line 225 of seff.
> You need to evaluate the field stats{tres_usage_in_max}, and there the value
> after '2=', but this is the memory value in bytes instead of kbytes, so this
> should be divided by 1024 additionally.
>
>
> Best
> Marcus
>
> On 11/08/2018 11:06 AM, Miguel A. Sánchez wrote:
> >
> > Hi and thanks for all your answers and sorry for the delay in my answer.
> > Yesterday I have installed in the controller machine the Slurm-18.08.3
> > to check if with this last release the Seff command is working fine. The
> > behavior has improve but I still receive a error message:
> >
> >
> > # /usr/local/slurm-18.08.3/bin/seff 1694112
> > /usr/local/slurm-18.08.3/bin/seff line 130, <DATA> line 624.*
> > > *From:* slurm-users <slurm-use...@lists.schedmd.com> on
> > > *To:* slurm...@lists.schedmd.com
> > > *Subject:* Re: [slurm-users] Seff error with Slurm-18.08.1
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
Marcus Wagner
just something learned again ;)
Best
Marcus
Miguel A. Sánchez
Miguel A. Sánchez Gómez
System Administrator
Research Programme on Biomedical Informatics - GRIB (IMIM-UPF)
Barcelona Biomedical Research Park (office 4.80)
Doctor Aiguader 88 | 08003 Barcelona (Spain)
Phone: +34/ 93 316 0522 | Fax: +34/ 93 3160 550
e-mail: miguelang...@upf.edu