Absurdly high reported CPU usage and possible underflow

[Antonio Delgado Peris]

Dear all,

In the last months, we've had a few cases (<10) of jobs reporting absurdly huge values for RemoteSysCpu, or RemoteUserCpu, or both. In all cases, the values fit a value pattern like 18,446,744,???,???. That is remarkably close to 2**64 divided by 1e6. That could point to an underflow, i.e. a negative time in microseconds.

We've seen this with WNs running in 24.12 and 25.0 series.

After investigating four recent cases, we've seen that in all cases there was an ssh-to-job connection, and an error is produced when trying to remove the cgroup associated to it, like:

ProcFamilyDirectCgroupV2::trimCgroupTree error removing cgroup system.slice/htcondor/_pool_condo...@b9p20p2191.cern.ch/sshd.slice: Device or resource busy

Could this be causing a negative result when updating cgroups cpu usage, which is then casted into unsigned int as ~2**64/1e6? For example, if the cpu counter for the sshd cgroup was moved to 0 upon the failed deletion, then previous-new usage would be negative.

What do you think?

Cheers,

    Antonio