Re: Some troubles with birq site

[pkun]

Hello

The using of hyperthreading is not useless.
The tests show the hyperthreading speed up an irq processing. We use
platforms with a big amount of LAN interfaces and our system was highly
loaded.

When I disable a hyperthreading (use first thread of CPU and don't use
second thread of this CPU) then the summary speed is slower. And we
did'n see any examples when the system with ht is slower than system
without ht using. The ht is recommended to use.

So the real tests show that scaling.txt is not right about ht.
You can use birq-1.1.3 to see it yourself. The --ht option enables the
using of ht. The birq without this option will not use ht. It really
works for birq-1.1.3 and it hasn't the same effect for birq-1.2.0.

The cause of problem to definitely disable ht in birq-1.2.0 is a
behaviour of linux kernel. It has a problem with initial irq allocation.
While system initialization linux puts irqs to first CPU. CPU has 256
slots for irq handling. Some of this slots are used by service irqs. So
when the all the slots are busy linux will use second CPU. And so on. If
CPU irq table is full then you can't move another irq to this CPU (there
is no free slot for it). The second problem is the irq affinity settings
will be applied when the irq really arrived. Before this moment the irq
is located within old CPU table and is not removed from it. In the same
time it will reserve a slot within new CPU table. So the moving of the
inactive irqs can pollute the CPU's tables. I try to minimize the irq
moving.

So the new birq don't change initial affinity for all irqs on start (The
old birq did it). It get the current irq affinity. It can change the irq
affinity if CPU is overloaded and if irq has at least one interrupt
while current interval (iteration period). It allows to move only active
irqs.

The new birq without --ht option will get current affinities. The
current affinity mostly use ht. Birq will not change this affinity
unless the correspondent CPU is overloaded and irq is active. So you
think the disabling of ht is not working. Really it works. It will move
irq to the first CPU thread only. But second threads already have irqs
on birq start.

In our test we had a CPUs with busy irq tables. The most irqs were
passive and CPU was idle but we can't move active irqs to this idle CPU.
This is the serious linux kernel problem for big platforms with a large
amount of LAN interfaces. Note each interface has a several queues. Each
queue has its own irq.

Please, write to birq mailing list http://groups.google.com/group/birq.
It's not a demand but someone can ask the same questions. It's not easy
for me to write the identical long messages in english twice. :)

02.06.2015 00:42, Pavel Odintsov пишет:
> Hello!
>
> Only one small question. Do you have any ideas about option for
> binding irqs to only physical cores (not hyper threaded)?
>
> Because according to
> https://www.kernel.org/doc/Documentation/networking/scaling.txt its
> enough useless:
> Per-cpu load can be observed using the mpstat utility, but note that
> on processors with hyperthreading (HT), each hyperthread is
> represented as a separate CPU. For interrupt handling, HT has shown no
> benefit in initial tests, so limit the number of queues to the number
> of CPU cores in the system.
>
> I have tried --ht flag but looks like it did not work correctly for my
> i7 3820 (8 logical, 4 physical cores) CPU:
>
> ----[ 17:39:51 ]----------------------------------------------------------------
> IRQ 0 * [IO-APIC-edge] timer
> IRQ 1 * [IO-APIC-edge] i8042
> IRQ 6 * [IO-APIC-edge] floppy
> IRQ 8 * [IO-APIC-edge] rtc0
> IRQ 9 000000ff [IO-APIC-fasteoi] acpi
> IRQ 10 000000ff [IO-APIC-fasteoi] uhci_hcd:usb2, uhci_hcd:usb3, virtio0
> IRQ 11 000000ff [IO-APIC-fasteoi] uhci_hcd:usb1, ehci_hcd:usb4, eth0
> IRQ 12 * [IO-APIC-edge] i8042
> IRQ 14 * [IO-APIC-edge] ata_piix
> IRQ 15 * [IO-APIC-edge] ata_piix
> IRQ 49 000000ff [PCI-MSI-edge] eth5-TxRx-0
> IRQ 50 000000ff [PCI-MSI-edge] eth5-TxRx-1
> IRQ 51 000000ff [PCI-MSI-edge] eth5-TxRx-2
> IRQ 52 000000ff [PCI-MSI-edge] eth5-TxRx-3
> IRQ 53 000000ff [PCI-MSI-edge] eth5-TxRx-4
> IRQ 54 000000ff [PCI-MSI-edge] eth5-TxRx-5
> IRQ 55 000000ff [PCI-MSI-edge] eth5-TxRx-6
> IRQ 56 000000ff [PCI-MSI-edge] eth5-TxRx-7
> IRQ 57 000000ff [PCI-MSI-edge] eth5
> CPU0 package 0, core 0, irqs 2, old 0.93%, load 50.00%
> IRQ 14, [00000001], weight 0, intr 0, ata_piix
> IRQ 55, [00000001], weight 0, intr 45981, eth5-TxRx-6
> CPU1 package 1, core 0, irqs 3, old 33.12%, load 22.96%
> IRQ 1, [00000002], weight 0, intr 0, i8042
> IRQ 15, [00000002], weight 0, intr 0, ata_piix
> IRQ 56, [00000002], weight 0, intr 52965, eth5-TxRx-7
> CPU2 package 2, core 0, irqs 3, old 40.67%, load 60.71%
> IRQ 6, [00000004], weight 0, intr 0, floppy
> IRQ 49, [00000004], weight 0, intr 37855, eth5-TxRx-0
> IRQ 57, [00000004], weight 0, intr 58, eth5
> CPU3 package 3, core 0, irqs 2, old 73.52%, load 2.75%
> IRQ 8, [00000008], weight 0, intr 0, rtc0
> IRQ 50, [00000008], weight 0, intr 51539, eth5-TxRx-1
> CPU4 package 4, core 0, irqs 2, old 59.90%, load 48.57%
> IRQ 9, [00000010], weight 0, intr 0, acpi
> IRQ 51, [00000010], weight 0, intr 43839, eth5-TxRx-2
> CPU5 package 5, core 0, irqs 2, old 16.00%, load 39.24%
> IRQ 10, [00000020], weight 0, intr 0, uhci_hcd:usb2, uhci_hcd:usb3, virtio0
> IRQ 52, [00000020], weight 0, intr 46478, eth5-TxRx-3
> CPU6 package 6, core 0, irqs 2, old 32.88%, load 49.46%
> IRQ 11, [00000040], weight 0, intr 67, uhci_hcd:usb1, ehci_hcd:usb4, eth0
> IRQ 53, [00000040], weight 0, intr 44915, eth5-TxRx-4
> CPU7 package 7, core 0, irqs 2, old 53.72%, load 62.90%
> IRQ 12, [00000080], weight 0, intr 0, i8042
> IRQ 54, [00000080], weight 0, intr 40224, eth5-TxRx-5
> ----[ 17:39:56 ]----------------------------------------------------------------
>
> On Mon, Jun 1, 2015 at 2:25 PM, Pavel Odintsov <pavel.o...@gmail.com> wrote:
>> Hello!
>>
>> First of all, my great thanks for this nice toolkit! :) Secondly,
>> please check your site, it did not work: https://src.libcode.org/birq
>>
>> --
>> Sincerely yours, Pavel Odintsov
>
>