how to prevent signal 9 given to vacating job?

[Carsten Aulbert]

Hi all,

I'm currently stuck on a problem which I thought should be easy to solve:

I want to configure a node where interactive jobs are preferred over
regular ones and I want to give the interactive jobs at least a day to
run and, say, 150s for regular jobs to allow them to set-up the GPU or
whatever.

That part is probably already solved by

MaxJobRetirementTime = ifThenElse(InteractiveJob =?= True, 86400, 150)
RANK = (InteractiveJob =?= True)

But now I want to give any job a grace time of 180s to receive SIGHUP
and shut down properly - possibly by writing a check point, however, I'm
currently lost how to do that as MachineMaxVacateTime seems to be
counted backwards from the MaxJobRetirementTime and thus does not apply
beyond an activity time of MaxJobRetirementTime - MachineMaxVacateTime.

After several iterations I'm stuck at

WANT_VACATE = True
MachineMaxVacateTime = 180
PREEMPT = (Activity == "Busy") && (Target.Rank > my.Rank)
KILL = (Activity == "Vacating") && ((time - EnteredCurrentActivity) > 180)

which still does not work. The KILL was simply a hail Mary attempt as
KILL was always "false" before. But still, jobs are directly killed via

07/08/26 10:42:05 (D_ALWAYS) Got vacate code=1019 subcode=0
reason=Preempted for a Priority user
07/08/26 10:42:05 (D_ALWAYS) ShutdownFast all jobs.
07/08/26 10:42:05 (D_ALWAYS:2) in VanillaProc::ShutdownFast()
07/08/26 10:42:05 (D_ALWAYS:2) ProcFamilyDirectCgroupV2::kill_family for
pid 56634 cgroup
system.slice/condor.service/htcondor/_local_condor_...@ahost.slice
07/08/26 10:42:05 (D_ALWAYS:2) ProcFamilyDirectCgroupV2::suspend for pid
56634 for root pid 808334642
in cgroup
system.slice/condor.service/htcondor/_local_condor_...@ahost.slice
07/08/26 10:42:05 (D_ALWAYS:2) cgroupv2 killing with signal 9 to pid
56636 in cgroup
system.slice/condor.service/htcondor/_local_condor_...@host.slice/_local_condor_...@host.scope

Anyone with some idea how I could get what I want here?

Cheers and thanks a lot in advance

Carsten


--
Dr. Carsten Aulbert, Max Planck Institute for Gravitational Physics,
Callinstraße 38, 30167 Hannover, Germany, Phone +49 511 762 17185

[Joachim Meyer]

Hi Carsten,

maybe this thread gives you some idea what's going on: https://www-auth.cs.wisc.edu/lists/htcondor-users/2026-March/msg00079.shtml

To my knowledge, when using partitionable slots (which you seem to use given the slot1_1 naming), the preemption mechanism does not respect any of the retiring time, .. knobs. Static slots allow for a whoole lot more tuning..

My work-around for giving a group priority access to a node is the following:

EP:

> MaxJobRetirementTime = 0
> START = Target.MaxJobRetirementTime == 0 || AcctGroup =?= "prio_group"

Negotiator:

> # My == Machine, Target == Job
> PREEMPTION_REQUIREMENTS = My.Machine =?= "special_node.hpc.uni-saarland.de" && Target.AcctGroup =?= "prio_group" && !regexp("prio_group\..+@", My.AccountingGroup) && (time() - Target.EnteredCurrentStatus) >= 3600
> PREEMPTION_RANK = ifThenElse(isUndefined(RemoteUserPrio), 0, (RemoteUserPrio * 1000000)) - ifThenElse(isUndefined(TotalJobRunTime), 0, TotalJobRunTime) - CPUs - 24*GPUs

I.e. only start jobs on the "special_node" that explicitly say they are fine with being killed without warning (MaxJobRetirementTime = 0).
On the negotiator, preempt only non-priority jobs & only after the job had more than 1hr since entering the running state.
So every job gets a guaranteed min-runtime before being interrupted.. but they still have to checkpoint proactively, not only on SIGTERM/SIGHUP.

Best,
- Joachim

Am 08.07.26 um 12:57 schrieb 'Carsten Aulbert' via HTCondor Users:

--

Joachim Meyer
HPC-Koordination & Support

Universität des Saarlandes FR Informatik | HPC

Postanschrift: Postfach 15 11 50 | 66041 Saarbrücken

Besucheranschrift: Campus E1 3 | Raum 4.03 66123 Saarbrücken

T: +49 681 302-57522 jme...@cs.uni-saarland.de www.uni-saarland.de

[Carsten Aulbert]

Hi Joachim,

On 7/8/26 14:27, Joachim Meyer wrote:
> maybe this thread gives you some idea what's going on: https://www-
> auth.cs.wisc.edu/lists/htcondor-users/2026-March/msg00079.shtml

darn, missed that earlier this year. Thanks for the pointer!

> To my knowledge, when using partitionable slots (which you seem to use
> given the slot1_1 naming), the preemption mechanism does not respect any
> of the retiring time, .. knobs. Static slots allow for a whoole lot more
> tuning..

Oh well, then users need to tag their jobs which can "live" with being
killed without notice to harness every last cycle they can get.

Thanks a lot

Carsten

PS: @ Condor team - while looking into this I noticed a small typo in
condor_q -bet -reverse -machine where it writes `requirments` at the
end. No big deal, but one of your test cases has the same typo ;-)

https://github.com/search?q=repo%3Ahtcondor%2Fhtcondor++requirment+&type=code

src/condor_tests/job_xform_factory.run in praticular