Interrupts and hyperthreading.

[Peter Veentjer]

Hi,

I'm reading the following book "Developing High-Frequency Trading Systems". My goal is not to write any high-frequency trading systems, but to get some insights into the domain and learn as much as possible from the applied techniques. It is a Packt book and they are not known for their quality. But it is written by 3 engineers with a combined HFT experience of almost 50 years.

On page 62 of the book, they make the following claim. If you are using a hyper-threading and there is an interrupt or a system call on one logical core, then the hyper-sibling will stall as well because both need access to the kernel (mode switch).

I don't believe this is correct. Each logical core has its own architectural state; so its own set of architectural registers and its own APIC including an interrupt descriptor table. The current privilege level is stored in the first 2 bits of the CS register and since every logical core has its own copy of that, the hyper siblings should be able to run independently no matter if there is a mode switch.

Of course, disabling hyper-threading will lead to improved performance of a single core, because it doesn't need to share any resources like rob, line fill buffers, store buffers, load buffets, execution units, reservation stations, caches, etc. So that is a valid reason to disable hyper-threading.

I'm by no means an X86 expert and certainly not a high-frequency trading expert. So perhaps I'm missing something.

Regards,

Peter.

[peter royal]

is it possible the book was accurate at the time of writing but hardware has since improved?

-pete 

-- 

(peter.royal|osi)@pobox.com - http://fotap.org/~osi

On Jul 1, 2022, at 8:54 PM, Peter Veentjer <alarm...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAGuAWdChA2kzo1RXHUB3Zh1H5GonVs82BhkCzG7ap4Q1PaUvdg%40mail.gmail.com.

[Wojciech Kudla]

> If you are using a hyper-threading and there is an interrupt or a system call on one logical core, then the hyper-sibling will stall as well because both need access to the kernel (mode switch).

This is kind of correct. That is, for cases where the kernel implements microarchitectural data sampling mitigations (think spectre).

Almost all low latency shops run their apps with SMT disabled so this does not apply but when using HT and with mitigations enabled the kernel has to synchronize ring 0 transitions.

Se more here:

https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/intel-analysis-microarchitectural-data-sampling.html

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/473A9790-D9B6-46DA-8C8C-8FD98957EB76%40pobox.com.

[Peter Veentjer]

@peter royal: the book is released this year. So it should be up to date.

@wojchech: thanks for your answer.

Does the CPU only need to serialize the transition, or does it need to serialize the interrupt/systemcall while it is in ring 0?

The impact of the first would be a lot smaller than the second. But the second seems to be more in line with what the authors are describing.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAHNMKAqzLGqRmuZThB9YrFaAiAgx-4zSxjFiwbRnFkyQu-Hn3w%40mail.gmail.com.

[Mark Dawson]

I'd hope these authors aren't referring to behavior under mitigation circumstances since *all* HFT firms disable cpu mitigation schemes from the kernel boot parameter list as a standard procedure.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAHNMKAqzLGqRmuZThB9YrFaAiAgx-4zSxjFiwbRnFkyQu-Hn3w%40mail.gmail.com.

[Mark Dawson]

Peter, I agree with you. I don't believe this is true for the same reasons you outlined. It almost reads like a throwaway statement that would've been easily corrected if properly reviewed and unrushed. 

Also, please correct me if I am wrong but, looking at the Table Of Contents on Amazon, it seems like each topic gets only 1 or 2 pages of treatment on average. This leads me to believe that this is a pretty cursory overview of HFT topics. Frankly, any one of the areas involving low-latency coding techniques & configuration could take several pages of background explanation, code examples, benchmarks + graphs, etc. to give it its full due. And I think such treatment is warranted considering that HFT (and Gaming, for that matter) often appear like a foreign world for devs coming from other industries (heck, the C++ community even gave us our own standards committee with SG14). Clocking in at only 280 total pages, I'm beginning to think I may be onto something here with this book characterization.

[Andrew Hunter]

On Sat, Jul 2, 2022 at 9:42 AM Mark Dawson <medaw...@gmail.com> wrote:

I'd hope these authors aren't referring to behavior under mitigation circumstances since *all* HFT firms disable cpu mitigation schemes from the kernel boot parameter list as a standard procedure.

Without expressing any opinion on this particular claim, it's important to realize that serious HFT practitioners are *extremely* secretive. They don't talk about anything they consider important publicly, and go to legends both to keep their own secrets and infer as much as they can about their competitors from public statements.

You should be extremely skeptical of any claims like "All HFTs X". I haven't read the book OP mentions but I'm doubtful you should infer much from it.

[Mark Dawson]

Andrew,

Yes, you are correct about the secrecy of the HFT industry, in general. As a matter of fact, when I worked at Jump Trading we used to have to deliver presentations at Linux Kernel/Tracing Summits under a completely different company name (that has changed since 2013 or so when Jump joined the Linux Foundation).

However, aside from secret trading strategies and esoteric protocol exchange handling techniques, OS configuration guidelines are pretty standard across the board. In fact, Red Hat publishes a new Low Latency Guidelines document with every release primarily targeted at our domain. Our representation in the kernel community ushered a lot of the advancements that we have today in the Adaptive Ticks (i.e. nohz_full) infrastructure.

Also, general techniques around pre-faulting memory, pinning threads, cache warming, and avoiding runtime overhead via template metaprogramming/constexpr, etc. - these are all things that you'll see members of HFT firms speak about freely and openly at tech conferences. We *all* do these things on the portions of our trading that still runs in software.

But no one is gonna talk about their secret sauce, of course. That secret trigger they discovered which works perfectly for their FPGA-based strategies. Things of that sort. But I'm willing to bet a nice chunk of change that no one in the industry who cares about latency is leaving the cpu security vulnerability mitigations enabled (which are, also, addressed in Red Hat's Low Latency Tuning guides).

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CANf_6Th9Dwg6Cv4ccbNiiVfBa1UbyRV2fbSpCeahpcEdUc-i8A%40mail.gmail.com.

[Wojciech Kudla]

@Peter

> Does the CPU only need to serialize the transition, or does it need to serialize the interrupt/systemcall while it is in ring 0?

Sadly yes, the kernel does need to temporarily "idle" the sibling thread while the other is in ring 0. This is described as transitioning from state 6a/6b to state 4 in the Intel doc I cited. In practice it's a bit more complicated because what happens depends on whether the thread in kernel mode is accessing "secret data" or not.

Regardless, the sibling thread is stuck in the IPI routine until the other one exits the kernel. When that happens, CPU's microarchitectural data buffers are cleared with VERW instruction and both siblings are allowed to resume execution.

Hope this helps.

It seems like this thread branched off to a general discussion about the quality of the cited book wrt the culture of secrecy in our field. When I was entering this industry many years ago I was put off by this and sometimes found it much harder to make progress in my work because you start dealing with problems you won't find blog posts about or an easy solution on SO. With time I learned to appreciate that since it often costs us a lot of frustration-filled effort and grey hair to work out the secret sauce on our own and this is a big part of one's value as a professional.

I think it's totally fair to not want to give away something like that for free (you'll see the same problem in the world of magicians). Not even mentioning how reliant the HFT industry is on staying ahead of the pack in terms of technology and innovation.

Just my two cents...

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAFvqqVe9E-_hZWYCmTpubNPpq58RnV%3DZTdMSESCcLP3YN_Uu7A%40mail.gmail.com.

[Avi Kivity]

It's true for IBM POWER, where all of a core's threads have to be in kernel mode or user mode and cannot intermix.

It's not true for x86. I also don't believe there is a stall during the mode transition, otherwise a thread that continuously issues null system calls could stall its sibling. That should be easy to test.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAGuAWdChA2kzo1RXHUB3Zh1H5GonVs82BhkCzG7ap4Q1PaUvdg%40mail.gmail.com.

[Wojciech Kudla]

Thanks Avi, that's interesting. I was under the impression that the Intel doc was referring to x86 specifically.

I'm a bit confused about the discrepancy between what how that document describes the mitigation and your comment.

Could you point to a resource or Kernel source file that sheds more light on this?

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/5e74f6e3-d0fe-452d-fa9e-146860767785%40scylladb.com.

[Mark Dawson]

Wojciech,

Just to be clear, you're illuminating a particular set of circumstances that only occurs within cpu security vulnerability mitigation routines, correct?

In other words, in the instance that an engineer has configured his/her system for low noise/jitter/latency, this serialization at kernel mode switch time among sibling threads should not occur, correct?

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAHNMKArHU1XMWAJSVcdwfVvgqTrbDcF2vgPLhvtAUSLgyoCkuA%40mail.gmail.com.

[Wojciech Kudla]

Yes, this absolutely does not apply when mitigations are disabled or when running with no SMT, so proper low latency shops never face this issue.

However I'm confused about Avi's comment as he seems to contradict what Intel are saying here:

https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/intel-analysis-microarchitectural-data-sampling.html

In the section about ring 0 transitions and IPIs.

I always trust Avi's enormous experience so hope he finds a little time to chime in and explain where my understanding is wrong.

W.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAFvqqVfD3t-8ptvwkyFzFGDT9MNaa9duJbCsLv7mpfGjgxUyCA%40mail.gmail.com.

[Mark Dawson]

I read Avi's response as simply concurring with mine - i.e., that under *normal* low-latency operation, siblings do not hinder each other during kernel mode switching. In other words, refuting the assertion made in the book.

But I'll let him respond directly to you.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAHNMKAp%3D-3mD27bHbANN_oD1fngdNzUP9VX4ojUZtTBFG9KmVw%40mail.gmail.com.