Weka with HTCondor

[Ram Ban]

Hi all,

I am running Htcondor for a large time with partition able slots, Now I want to try weka filesystem to use fast fs, Weka fs requires minimum 1 cpu from on box which cant be used by other processes(in dpdk mode). 

I have tried giving NUM_CPUS = N-1 assuming htcondor will use 0 to N-2 cores and pinning weka on N-1 core, but htcondor randomly keeps any one core free not specific one 

Is there any way around to fix this problem?

From documentation I thought of creating a static slot with 1 core and 0 ram(dummy slot) but then I didn't find a way to change affinity so that condor never uses the weka core 

I can also set specific core to be used by weka.

Thanks and Regards 

Raman

[Ram Ban]

Thanks a lot for quick response, 

I will try this.

Just a small doubt if I have 16 core machine, currently for computer heavy workloads I use Count_Disable_hyperthreads to use only 8 cores at once, will using cset reduce it too 7?

As if my understanding is correct then it will be 15 cores remaining?

Regards 

Raman

On Thu, Aug 6, 2026, 22:05 Todd, Allen <Allen...@sig.com> wrote:

Sounds like you want to use “cset”

 

 

   Typical uses of cset include

       Setting up and managing a simple shielded CPU environment

           The concept of shielded cpus is that a certain number of cpus are partitioned off on the system and

           only processes that are of interest are run on these cpus (i.e., inside the shield).

 

           For a simple shielded configuration, one typically uses three cpusets: the root set, a system set and a

           user set. Cset includes a super command that implements this strategy and lets you easily manage it.

           See cset-shield(1) for more details.

---

IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

[Ram Ban]

Hi Allen,

Also cursor suggest me to use STARTD_ASSIGNED_AFFINITY variable to give core list by looking on GitHub code and remove ASSIGN_CPU_AFFINITY? Is it safe to use this ?

Thanks and Regards 

Raman

[Emily Kooistra]

Hi Raman,

Beter move is then to use isolcpus to isolate 1 cpu core for weka, and
place the NUM_CPUS to N-1, and place with taskset (or the weka tooling).
the processes on the isolated CPU core. THen the linux scheduler can't
use that core anymore and HTCONDOR (Assuming you don't use CPU pinning),
will not be scheduled on those cores anymore.

Emily Kooistra

> --
> Archives for older messages are found at https://www-auth.cs.wisc.edu/
> lists/htcondor-users/ <https://www-auth.cs.wisc.edu/lists/htcondor-users/>
> ---
> You received this message because you are subscribed to the Google
> Groups "HTCondor Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to htcondor-user...@g-groups.wisc.edu
> <mailto:htcondor-user...@g-groups.wisc.edu>.
> To view this discussion visit https://groups.google.com/a/g-
> groups.wisc.edu/d/msgid/htcondor-users/CAG-
> J6sh5YjhyHok6ZkxNxPdfMbsoqk%3DFdYTR%3DEedQgXz-P2uag%40mail.gmail.com
> <https://groups.google.com/a/g-groups.wisc.edu/d/msgid/htcondor-users/
> CAG-J6sh5YjhyHok6ZkxNxPdfMbsoqk%3DFdYTR%3DEedQgXz-
> P2uag%40mail.gmail.com?utm_medium=email&utm_source=footer>.