Isolated CPU and cold start

[r r]

Hi,

let's look at the example:

The system is running with the following kernel parameters: 

isolcpus=10, nohz_full=10, nohz=on, idle=poll, intel_pstate=disable. 

We have a thread T that uses Thread.onSpinWait() while polling a lock-free shared queue. In this context, the task interval refers to the time elapsed between adding consecutive tasks to the queue.

When thread T is pinned to CPU #10 and the task interval is set to 1ms, the average task execution time is 100 µs. However, when the task interval is increased to 40ms on the same pinned core, the average execution time significantly degrades to 250 µs.

In contrast, when thread T is unpinned, the performance remains much more consistent. At a 1ms task interval, the average execution time is 110 µs, and it only slightly increases to 120 µs when the interval is extended to 40ms.

[r r]

Do you know how it can be explained?

[Jean-Philippe BEMPEL]

Hi,

You should not  use isolcpus, but use cpuset instead. 

isolcpus disables task load balancing from the scheduler on the isolated cpu, which is not the case using cpuset.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/mechanical-sympathy/4fbbd936-a784-45b2-a58e-61a292aad704n%40googlegroups.com.

[r r]

Thanks,

Do you think that it could explain difference? 

> isolcpus disables task load balancing from the scheduler on the isolated cpu

Why it is a problem? It is exactly what I want: I want to isolate CPU to be exclusive for my critical threads. What is the point to enable task load balancing here? 

[Mark E. Dawson, Jr.]

Do you have a baseline for how your isolated core should perform using a tool like 'osnoise'?

[r r]

No. Actually, after some retests my observation is that it happens regardless the thread is pinned or not.

So, 

When thread T is pinned to CPU #10 and the task interval is set to 1ms, the average task execution time is 100 µs. However, when the task interval is increased to 40ms on the same pinned core, the average execution time significantly degrades to 250 µs. If T is not pinned, the result is same.

[Li Guang Sheng (GS)]

Hi,

It can be due to cpu C state. When the CPU is isolated, there is no task assigned to it when the pinned thread is not running thus increases the chance for it to go to deep sleep state, it is quite costly to wake up from deep sleep state. For non isolated cores, they are probably consistently running tasks thus lower chance to go to deep sleep state.

You can try to disable c state in bios and see if it helps.

Best Regards,

GS

[r r]

Thread  T is alway running - it has loops in manner: 

while(!isInterrupted()) { 

  checkForWork();

  noIdle

 } 

or 

while(!isInterrupted()) { 

  checkForWork();

  Thread.onSpinWait() // PAUSE on X86

}

In both cases I see this difference.

C-state should not be the case because of kernel option ilde=poll.

[Francesco Nigro]

Use Linux perf stat after writing a jmh bench and inspect the assembly to make sure what's goin on

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/mechanical-sympathy/9c15417b-8c8c-4b76-b63e-1f9d38d507e0n%40googlegroups.com.

[Peter]

Can you please add more details? How many sockets/cores? What hardware brand? What distro/OS version? Is it a regular vs real time kernel?  What does pidstat show that your app is doing? Specifically how many cores is this app using? Are there other apps running on the same host? Are you doing anything with cgroups? Are you using a tuned profile? 

Note that some of the low latency OS features depend upon matching kernel settings with BIOS settings - which is why I ask which hardware vendor? Correct power savings behavior depends on having correct BIOS settings.

[Peter]

Average task execution time is less interesting that seeing raw latency data - what does this app do? Is it listening to market data, customer orders, doing rescheduled work? Are you using specialized (SolarFlare/Mellanox NICs? 100 µs is a long time with Skylake and newer hardware.

[Peter]

I agree that one possible likely explanation is the one previously offered by deep.b - deep cstates. Another could be needing to configure irqbalance to mask core 10 to avoid interrupts being scheduled on it.  

The kernel setting idle=poll won't work unless the BIOS settings match. For example, with HP hardware you need to use ILO to set Minimum Processor Idle Power Core C-State to No C-States. IBM, Dell have their own equivalent tools. HP have a BIOS level configuration, "Workload Profile" that is similar to what a tuned profile does in the kernel. Changing this profile from the default to the Low Latency profile (and then Custom) will bundle a bunch of useful stuff like disabling hyperthreading, VT-d, setting the CPU power regulator and energy bias to maximum performance, enabling Turbo mode, etc.

HPE document all of this in great detail in a whitepaper Configuring and tuning HPE ProLiant Servers for low-latency applications

It might be interesting to see the implementation of checkForWork()