[slurm-users] Is sacct not handling quotes properly?

[David Henkemeyer]

I am seeing what I think might be a bug with sacct.  When I do the following:

> sbatch --export=NONE --wrap='uname -a' --exclusive

Submitted batch job 2869585

Then, I ask sacct for the SubmitLine, as such:

> sacct -j 2869586 -o "SubmitLine%-70"
SubmitLine
----------------------------------------------------------------------
sbatch --export=NONE --wrap=uname -a --exclusive

As you can see, the quotes around 'uname -a' are gone.  Hence, that submit line is not a valid sbatch commandline:

> sbatch --export=NONE --wrap=uname -a --exclusive
sbatch: error: Batch job submission failed: Invalid job array specification

Like I said, I suspect this is a bug with sacct.  But I want to make sure.  Can I somehow "peek" inside the accounting DB to see if the quotes are there?  

Perhaps there is an interaction with my bash shell, that's stripping the quotes?  This seems unlikely to me.

Thanks in advance!

David

[Michael Jennings]

On Wednesday, 04 May 2022, at 10:00:57 (-0700),

David Henkemeyer wrote:

>I am seeing what I think might be a bug with sacct. When I do the
>following:
>
>

>*> sbatch --export=NONE --wrap='uname -a' --exclusive*
>*Submitted batch job 2869585*

>
>Then, I ask sacct for the SubmitLine, as such:
>
>
>
>

>*> sacct -j 2869586 -o
>"SubmitLine%-70"SubmitLine----------------------------------------------------------------------sbatch
>--export=NONE --wrap=uname -a --exclusive*

>
>As you can see, the quotes around 'uname -a' are gone. Hence, that submit
>line is not a valid sbatch commandline:
>
>

>*> sbatch --export=NONE --wrap=uname -a --exclusivesbatch: error: Batch job
>submission failed: Invalid job array specification*

>
>Like I said, I suspect this is a bug with sacct. But I want to make sure.
>Can I somehow "peek" inside the accounting DB to see if the quotes are
>there?
>
>Perhaps there is an interaction with my bash shell, that's stripping the
>quotes? This seems unlikely to me.

It's not actually unlikely; that's exactly what's happening. Check
out the BASH man page section on EXPANSION and look for the term
"quote removal." It's part of shells' normal operations. :-)

The real question is in how the command is being stored; are you
seeing 4 words or 5? Shells like BASH internally track token sets in
terms of "words," and those words can have embedded spaces. But
embedded spaces and word-separation spaces are indistinguishable
without additional metadata.

In other words, what you're really asking are two questions:
- Does the accounting DB store the submit line in a way that
preserves embedded spaces in arguments?
- If so, can that value be retrieved in a way that quotes those
arguments with embedded spaces (possibly by quoting *all*
arguments) in a way that can be reused as shell input?

We can see this in action with the following example using BASH's
support for arrays. Just like with `$*` vs. `$@`, array expansion
unquoted or inside double-quotes expands to a single word with
elements separated by the first character of `$IFS` (space by default)
when `$*` or `${ARRAYNAME[*]}` is used but to individual separate
words instead when `$@` or `${ARRAYNAME[@]}` is used. However, it's
impossible to see the difference between embedded single spaces inside
a word/element and the single spaces BASH puts between them when
they're displayed:

$ (declare -ax TESTME=( 1 2 '3 4 5' 6 7 8 ) ; echo ${TESTME[*]} ; echo "${TESTME[@]}" ; declare -p TESTME)
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
declare -ax TESTME=([0]="1" [1]="2" [2]="3 4 5" [3]="6" [4]="7" [5]="8")

Notice that the 1st 2 lines look identical to one another, even though
internally they're not. As you can see in the final line, there are
still spaces in the 3rd array element (`[2]="3 4 5"`).

The distinction becomes clearer if more than one space at a time is
embedded into the 3rd element, like so:

$ (declare -ax TESTME=( 1 2 '3 4 5' 6 7 8 ) ; echo ${TESTME[*]} ; echo "${TESTME[@]}" ; declare -p TESTME)
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
declare -ax TESTME=([0]="1" [1]="2" [2]="3 4 5" [3]="6" [4]="7" [5]="8")

So if you want to find the answers to the above questions (at least
the 1st one), pass 'uname -a' instead of 'uname -a' and see what
you get back from your `sacct` query! :-)

HTH,
Michael

--
Michael E. Jennings <m...@lanl.gov> - [PGPH: he/him/his/Mr] -- hpc.lanl.gov
HPC Systems Engineer -- Platforms Team -- HPC Systems Group (HPC-SYS)
Strategic Computing Complex, Bldg. 03-2327, Rm. 2341 W: +1 (505) 606-0605
Los Alamos National Laboratory, P.O. Box 1663, Los Alamos, NM 87545-0001

[David Henkemeyer]

Thank you, Michael!  In fact, it appears as though Slurm is storing the entire commandline as a single "word":
(! 1111)-> sacct -j 2871474 -o "SubmitLine%-100"
SubmitLine
----------------------------------------------------------------------------------------------------

sbatch --export=NONE --wrap=uname   -a --exclusive

So, its storing properly, now I need to see if I can figure out how to preserve/add the quotes on the way out of the DB...

David