[slurm-users] Change something in user's script using job_submit.lua plugin

[Paulo Jose Braga Estrela]

Hi,

Is it possible to change something in user’s sbatch script by using a job_submit plugin? To be more specific, using Lua job_submit plugin.

 

I’m trying to do the following in job_submit.lua when a user changes job’s partition to “cloud” partition, but the script got executed without modification.

 

function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)

 

        if job_desc.partition == "cloud" then

                slurm.log_info("slurm_job_modify: Bursting job %u from uid %u to the cloud...",job_rec.job_id,modify_uid)

               

                script = job_rec.script

                slurm.log_info("Script BEFORE change: %s",script)

 

                -- changing user command to another command

                script = string.gsub(script,"local command","cloud command")

 

                slurm.log_info("Script AFTER change %s",script)

                -- The script variable is really changed

 

                job_rec.script = script

                slurm.log_info("Job RECORD SCRIPT %s",job_rec.script)

                -- The job record also got changed, but the EXECUTED script isn’t changed at all. It runs without modification.

        end

 

        return slurm.SUCCESS

end

 

 

 

 

PAULO ESTRELA

 

Analista de Sistemas Sênior

PETROBRAS - Tecnologia da Informação e Telecomunicações

Suporte e Operação de Soluções de Alto Desempenho (HPC)

TIC/OI/SSH/SAD

 

O emitente desta mensagem é responsável por seu conteúdo e endereçamento e deve observar as normas internas da Petrobras. Cabe ao destinatário assegurar que as informações e dados pessoais contidos neste correio eletrônico somente sejam utilizados com o grau de sigilo adequado e em conformidade com a legislação de proteção de dados e privacidade aplicável. A utilização das informações e dados pessoais contidos neste correio eletrônico em desconformidade com as normas aplicáveis acarretará a aplicação das sanções cabíveis.

The sender of this message is responsible for its content and address and must comply with Petrobras' internal rules. It is up to the recipient to ensure that the information and personal data contained in this email are only used with the appropriate degree of confidentiality and in compliance with applicable data protection and privacy legislation. The use of the information and personal data contained in this e-mail in violation of the applicable rules will result in the application of the applicable sanctions.

El remitente de este mensaje es responsable por su contenido y dirección y debe cumplir con las normas internas de Petrobras. Corresponde al destinatario asegurarse de que la información y los datos personales contenidos en este correo electrónico solo se utilicen con el grado adecuado de confidencialidad y de conformidad con la legislación aplicable en materia de privacidad y protección de datos. El uso de la información y datos personales contenidos en este correo electrónico en contravención de las normas aplicables dará lugar a la aplicación de las sanciones correspondientes.

PÚBLICA

[Ole Holm Nielsen]

Hi Paulo,

Which Slurm version do you have, and did you set this in slurm.conf:
JobSubmitPlugins=lua ?

Perhaps you may find some useful information in this Wiki page:
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#job-submit-plugins

/Ole

On 26-10-2023 19:07, Paulo Jose Braga Estrela wrote:
> Is it possible to change something in user’s sbatch script by using a
> job_submit plugin? To be more specific, using Lua job_submit plugin.
>
> I’m trying to do the following in job_submit.lua when a user changes
> job’s partition to “cloud” partition, but the script got executed
> without modification.
>
> function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
>
>         if job_desc.partition == "cloud" then
>
>                 slurm.log_info("slurm_job_modify: Bursting job %u from
> uid %u to the cloud...",job_rec.job_id,modify_uid)
>
>                 script = job_rec.script
>
>                 slurm.log_info("Script BEFORE change: %s",script)
>
>                 -- changing user command to another command
>
>                 script = string.gsub(script,"local command","cloud
> command")
>
>                 slurm.log_info("Script AFTER change %s",script)
>
>                 -- The script variable is really changed
>
>                 job_rec.script = script
>
>                 slurm.log_info("Job RECORD SCRIPT %s",job_rec.script)
>
>                 -- The job record also got changed, but the EXECUTED
> script isn’t changed at all. It runs without modification.
>
>         end
>
>         return slurm.SUCCESS
>
> end
>

> *PAULO ESTRELA*

[Paulo Jose Braga Estrela]

Hi Ole,

Yes, the script is running and changing other fields like comment, partition, account is working fine. The only problem seems to be the script field of job_rec. I'm using Slurm 20.11.9 from EPEL repository for RHEL 8. Thank you for sharing your Wiki. I've accessed it before. It's really useful for HPC engineers.

Best regards,


PÚBLICA
-----Mensagem original-----
De: slurm-users <slurm-use...@lists.schedmd.com> Em nome de Ole Holm Nielsen
Enviada em: sexta-feira, 27 de outubro de 2023 03:31
Para: slurm...@lists.schedmd.com
Assunto: Re: [slurm-users] Change something in user's script using job_submit.lua plugin

[Ole Holm Nielsen]

Hi Paulo,

Maybe what you see is due to a bug then? You might try to update Slurm
to see if has been fixed.

You should not use the Slurm RPMs from EPEL - I think offering these
RPMs was a mistake.

Anyway you ought to upgrade to the latest Slurm 23.02.6 since a serious
security issue was fixed a couple of weeks ago. Older Slurm versions
are all affected! Perhaps this Wiki guide can help you upgrade to the
latest RPM: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/

/Ole

[Paulo Jose Braga Estrela]

Yes, reading the sources I found that _update_job function in job_mgr.c is responsible for calling job_submit_plugin_modify function. After calling it, _update_job validates and apply the changes made by the plugin function to many job record fields but don’t touch the script field. So, for now we can't change a job script using job_submit plugin. I'll see if it was done intentionally to prevent something from going wrong or if it was just forgotten.

By the way, thanks for trying to help me, Ole!



PÚBLICA
-----Mensagem original-----
De: Ole Holm Nielsen <Ole.H....@fysik.dtu.dk>
Enviada em: sábado, 28 de outubro de 2023 03:48
Para: Slurm User Community List <slurm...@lists.schedmd.com>
Cc: Paulo Jose Braga Estrela <paulo....@petrobras.com.br>
Assunto: Re: RES: [slurm-users] Change something in user's script using job_submit.lua plugin

Hi Paulo,

Maybe what you see is due to a bug then? You might try to update Slurm to see if has been fixed.

You should not use the Slurm RPMs from EPEL - I think offering these RPMs was a mistake.

Anyway you ought to upgrade to the latest Slurm 23.02.6 since a serious security issue was fixed a couple of weeks ago. Older Slurm versions are all affected! Perhaps this Wiki guide can help you upgrade to the latest RPM: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/

/Ole


On 27-10-2023 13:13, Paulo Jose Braga Estrela wrote:
> Yes, the script is running and changing other fields like comment, partition, account is working fine. The only problem seems to be the script field of job_rec. I'm using Slurm 20.11.9 from EPEL repository for RHEL 8. Thank you for sharing your Wiki. I've accessed it before. It's really useful for HPC engineers.
>
> Best regards,
>
>
> PÚBLICA
> -----Mensagem original-----
> De: slurm-users <slurm-use...@lists.schedmd.com> Em nome de Ole
> Holm Nielsen Enviada em: sexta-feira, 27 de outubro de 2023 03:31
> Para: slurm...@lists.schedmd.com
> Assunto: Re: [slurm-users] Change something in user's script using
> job_submit.lua plugin
>
> Hi Paulo,
>
> Which Slurm version do you have, and did you set this in slurm.conf:
> JobSubmitPlugins=lua ?
>
> Perhaps you may find some useful information in this Wiki page:

> https://wiki/
> .fysik.dtu.dk%2FNiflheim_system%2FSlurm_configuration%2F%23job-submit-
> plugins&data=05%7C01%7Cpaulo.estrela%40petrobras.com.br%7Cba06d0f43b16
> 4fdf6d8e08dbd781ce98%7C5b6f62419a574be48e501dfa72e79a57%7C0%7C0%7C6383
> 40724794687584%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l
> uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lthsF5GAYkNM
> jcFHVzERVO0stoth%2F8XZCssW%2B6jLGBw%3D&reserved=0