% Allocated CPUs Used vs CPU_LOAD on Slurm cluster

[Pietro Marchesi]

I am running a Nextflow pipeline on a Slurm cluster, and am confused as to how Nextflow computes the percentage of CPU used versus the CPU_LOAD that I get in Slurm. 

Monitoring the state of the nodes during execution with sinfo, 

sinfo -N --Format=nodehost,cpusstate,cpusload,memory,freemem,statecompact

I get that the CPU_LOAD is always quite low (around 1). If I then look at the Nextflow report, the % Allocated CPUs Used is between 1000 and 2000 for those same jobs. 

In the pipeline I only ask for 1 cpu per job, although on this system I get a full node allocated per job, so that is 256 cores (64 cores. 4 hw threads), so if Nextflow computes its measure based on the requested resources (instead of allocated) then this may account for at least some of the difference. 

Does anyone have any experience on this?

Cheers, 

Pietro

[Paolo Di Tommaso]

Hi Pietro, 

Could you report also the "cpu raw usage", maybe there's something odd with the formatting. 

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Steve]

I think I am experiencing a similar issue.

In the 'Allocated CPU' section, the y axis only goes up to 100 and many of the processes are off the scale:

<img width="760" alt="screen shot 2018-02-17 at 1 09 48 pm" src="https://user-images.githubusercontent.com/10505524/36344268-a7d77b5e-13e5-11e8-8f6a-19fc1fb1c8ba.png">
https://user-images.githubusercontent.com/10505524/36344268-a7d77b5e-13e5-11e8-8f6a-19fc1fb1c8ba.png


This is not the case on the 'Raw Usage' section though

<img width="751" alt="screen shot 2018-02-17 at 1 10 27 pm" src="https://user-images.githubusercontent.com/10505524/36344276-bc847994-13e5-11e8-88e6-601df99f87bd.png">
https://user-images.githubusercontent.com/10505524/36344276-bc847994-13e5-11e8-88e6-601df99f87bd.png

On Friday, February 16, 2018 at 8:00:00 AM UTC-5, Paolo Di Tommaso wrote:

Hi Pietro, 

Could you report also the "cpu raw usage", maybe there's something odd with the formatting. 

p

On Fri, Feb 16, 2018 at 10:54 AM, Pietro Marchesi <pietroma...@gmail.com> wrote:

I am running a Nextflow pipeline on a Slurm cluster, and am confused as to how Nextflow computes the percentage of CPU used versus the CPU_LOAD that I get in Slurm. 

Monitoring the state of the nodes during execution with sinfo, 

sinfo -N --Format=nodehost,cpusstate,cpusload,memory,freemem,statecompact

I get that the CPU_LOAD is always quite low (around 1). If I then look at the Nextflow report, the % Allocated CPUs Used is between 1000 and 2000 for those same jobs. 

In the pipeline I only ask for 1 cpu per job, although on this system I get a full node allocated per job, so that is 256 cores (64 cores. 4 hw threads), so if Nextflow computes its measure based on the requested resources (instead of allocated) then this may account for at least some of the difference. 

Does anyone have any experience on this?

Cheers, 

Pietro

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

Yes, I think here there's a bug. We have refactored that chart that will be available in the next release. 

You may want to give a try to the latest snapshot by using this command: 

NXF_VER=0.28.0-SNAPSHOT nextflow run ...etc

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Pietro Marchesi]

Thanks Paolo, I will try the snapshot as soon as possible. 

Best, 

P