Chromebook 3 (ARM) event counter IRQs and CPU affinity issues ...

[Bill Smith]

This might be a bit esoteric for a Chrome OS discussion - but I am not sure where else to post this question.

I have been developing a kernel module for the Linux 3.4 kernel that uses the cycle counter and the event counters in the Exynos 5's PMU to generate and respond to interrupts on counter overflow events.

From what I have been able to figure out so far, IRQ 170 corresponds to the overflow counters on CPU 0 and IRQ 340 corresponds to the the overflow counters on CPU 1.

I have set up interrupt handlers for these IRQs and set their affinity hint for their correct CPUs, 

As expected, all overflow interrupts on CPU 0 are sent to the IRQ handler on CPU 0 - so my IRQ handler is working correctly for this core.

However, all of the overflow interrupts on CPU 1 are also sent to CPU 0 - and CPU 0 cannot actually read the event counter overflow flags on CPU 1.  So the CPU 0 IRQ handler cannot actually correctly respond to the counter overflow interrupt generated on CPU 1.  Hence IRQ 340 never gets handled and the kernel complains about the unhandled IRQ and  disables the 340 IRQ for CPU 1.

In looking at the Exynos 5 IRQ definitions in <kernel>/arch/arm/mach-exynos/include/mach/, Samsung is using a "COMBINER_IRQ" type.  Using a not very scientific survey (via "cat /proc/interrupts") - all of the COMBINER_IRQ types appear to only generate IRQs on CPU 0.  

In fact - it looks like almost all of the interrupts are handled by default on CPU 0.

Is there a hardware reason why IRQ 340 for CPU 1 is handled only on CPU 0?

Or is this due to how Samsung (and Google?) chose to implement the Linux IRQ interface for the Exynos 5?

For IRQ events that are due to CPU specific interrupts (such as the overflow events associated with the PMU event counters), you would expect and hope that you could handle the IRQ on the correct CPU.

Any deeper insight or pointers to more detailed information here would be appreciated.

Thanks.

- Bill

[Sonny Rao]

On Thu, Jun 13, 2013 at 6:01 PM, Bill Smith <bi...@smith-online.org> wrote:
> This might be a bit esoteric for a Chrome OS discussion - but I am not sure
> where else to post this question.
>
> I have been developing a kernel module for the Linux 3.4 kernel that uses
> the cycle counter and the event counters in the Exynos 5's PMU to generate
> and respond to interrupts on counter overflow events.
>

Is this upstream 3.4 or the Chrome-OS tree? I'm assuming Chrome-OS
since upstream 3.4 doesn't really support Exynos 5250 at all.

There was a bug at one point which is very similar to what you're
reporting where we just didn't see any PMU interrupts for CPU1. A few
fixes went into our tree:

commit 2ba000de62ef4df1e627cb67090526e4fb9a1453
Author: Olof Johansson <ol...@chromium.org>
Date: Fri Jun 22 15:11:50 2012 -0700

ARM: exynos: add pmu nodes to exynos5250 dts

commit a879586ba71e64df4d0a3f2321704fc359283a65
Author: Olof Johansson <ol...@chromium.org>
Date: Fri Jun 22 17:33:42 2012 -0700

CHROMIUM: ARM: exynos: arm-pmu configuration through dts

commit 22915d86cb9114de012979a0ba38bcf042a28a77
Author: Olof Johansson <ol...@chromium.org>
Date: Fri Jun 22 17:34:28 2012 -0700

CHROMIUM: ARM: exynos: handle setaffinity on combined interrupts

And after those three went in, I'm pretty sure we saw the interrupts
happening on CPU1.
If you run: taskset -c 1 perf record sha1sum /dev/zero &
and look at /proc/interrupts, do you see anything happening ?
I see this:

CPU0 CPU1
...
340: 0 1912 COMBINER arm-pmu

so I'm seeing the pmu interrupts going to cpu1



> Thanks.
>
> - Bill
>
> --
> --
> Chromium OS discuss mailing list: chromium-...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-discuss?hl=en
>
>
>

[Bill Smith]

What you described here makes some sense, but it is not clear if it is exactly the issue I have.

I am actually on ChrUbuntu with 3.4.0-5-chromebook as the kernel, so it is not clear if these patches were pushed into the kernel and device tree at the time I installed it.

I tried your "taskset" test and I do see IRQ event counts for 340 on CPU 1 and likewise I see event counts for 170 when I pin it to CPU 0. 

Does this mean the kernel I am using has these patches applied?

I had previously noticed that perf did appear to work correctly, but the perf tool support is tightly integrated into the kernel and uses "irq_set_affinity" to associate its IRQ handler with the CPU instance.  The "irq_set_affinity" function is not exported as a symbol from the kernel and thus is not accessible by kernel modules.  Instead,  kernel modules have to use "irq_set_affinity_hint" since it is actually exported form the kernel.  

The upshot of this is that the "irq_set_affinity_hint" kernel function does not appear to actually work as needed (at all?).

This appears to mean that I currently cannot implement a kernel module with an IRQ handler per CPU to handle local IRQ events. Is that really correct?

Did the patch above somehow not apply to "irq_set_affinity_hint"?

Should I be able to use "irq_set_affinity_hint" to associate my IRQ handler with a specific CPU core?

I really appreciate any insights you have to offer here.

Thanks.