Disable IRQ sharing
Niki Estner
If I install Windows 2000 in a default way (with ACPI) on Pentium 4
computers it assigns IRQ 9 to this special card - and to all the others,
too.
Of course, a well-implemented driver would accept this and use the shared
IRQ, but this one doesn't, and this is a high-performance, high-throughput
card, so it really should have its own IRQ anyway.
The vendor tells me the only way to prevent this is to install the PC with
the "PC-Standard" option - i.e. disable ACPI and all the nice little things
that come with it (power off on shutdown, standby mode, USB...).
Is there a way to dedicate an own IRQ (if one is available) exculsively to
one PCI adapter/driver?
If this can be done from within the driver, I'm quite confident the vendor
would do it if I could tell him how.
Or is it possible to prevent IRQ sharing without turning of ACPI completely?
Niki Estner
Eliyas Yakub
>computers it assigns IRQ 9 to this special card - and to all the others,
>too.
Here is a long story on why it's so from the developer who designed this
scheme:
-----Original Message-----
From: Jake Oshins
Sent: Sunday, December 30, 2001 6:19 AM
To: NT Developers Interest List
This is a stupid story. And I'm embarrassed to tell you the truth. But here
it is.
The early ACPI machines were mostly laptops. And the laptops of that
generation had most of their devices either embedded in the chipset or on
the ISA bus. The PCI or AGP buses were used only for video, and to connect
the north bridge with the south bridge. (In Intel's chipset terms, the North
bridge has all the fast gates of the chipset, including the memory
controller, AGP and in that generation, the PCI bus generation logic. The
south bridge contains all the slow gates, including the IDE controller, the
ISA bridge, all the PC legacy stuff and probably a USB controller. Today,
the south bridge probably also has audio and a few other random odds and
ends.) Because the laptops of that era had all of their devices on the ISA
bus, interrupt sharing worked poorly. If you bought a mid-'90s laptop from
IBM or Toshiba, the serial port and possibly IR would be disabled. There
would be a utility packaged with the machine that allowed you to turn on
your serial or IR, but at the cost of the bi-directional parallel port, or
one of the PCMCIA slots, since there just weren't enough IRQs in the machine
to guarantee that all of the peripherals worked, especially if you filled
both PCMCIA slots with combo cards.
I once debugged a Toshiba 750CDT in a docking station that had two PCMCIA
cards plugged into the machine, two PCMCIA cards plugged into the slots in
dock, two ISA cards in the dock and an extra IDE device in the dock, too.
This meant that the total demand on the machine was 20 IRQs, when only 16
were actually available.
(As an aside, I've been trying to convince Intel to put APIC interrupt
controllers, which would allow many more IRQs, in their laptop chipsets
since 1997. My predecessor had been trying since '94. They may actually
manage it soon.)
Along comes ACPI. When you turn on ACPI in a machine, it suddenly switches
all the power management logic in the machine from delivering its interrupts
as BIOS-visible, non-vectored System Management Interrupts over to
OS-visible, vectored interrupts. And that interrupt is delivered
level-triggered, active-low, which means that it can be shared with a PCI
interrupt.
Now consider that these early ACPI machines were already over-committed in
terms of interrupts. There was no way to make them work with PCI devices
spread out on lots of IRQs. So I just made the code collapse all the
sharable devices onto the ACPI interrupt, which was fixed in the chipset by
Intel at IRQ 9. By doing it this way, I could hide the fact that ACPI had
just created a demand for one more IRQ. (If you use a non-Intel chipset that
has ACPI coming in on some other IRQ, you'll see all the PCI devices in
Win2K go to that IRQ, not 9.)
Further complicating this story was that I was trying to get ACPI machines
to work back in 1997, when the people working on Plug and Play in Win2K
hadn't yet gotten their stuff going yet. At time, it wasn't possible to move
a device from one set of resources to another after it had been started.
This meant that any IRQ solution that I came up with had to work from the
first try, so it had to be conservative.
The everything-on-IRQ-9 solution worked. It got the machines to run, as long
as none of the device drivers mis-handled their ISRs. (Later, this turned
out to be a huge debugging problem, since when you chain eight or nine
devices, you'll get somebody who fails.) The solution wasn't optimal, but it
did work. I meant to go back and change it later, before we shipped Windows
2000.
A couple of years passed. I had been working on multi-processor problems and
on other aspects of ACPI. It got close to the time to ship Windows 2000 and
somebody brought up the old question of IRQ stacking. I worked up a
more-elegant solution, one that spread out interrupts on most machines. By
that time, Plug and Play had been mostly completed, and that wasn't a
bottleneck any more. But the test team told me that they wouldn't let me put
it into the product, since they didn't have time to re-test the thousands of
machines that had already been tested with the old algorithm.
At the time, I thought that this was somewhat ridiculous. I thought that my
code would work just fine. I thought that their fears were un-justified. But
I was overruled, and I just put the code into what became Windows XP,
letting Windows 2000 ship with the simple, safe, yet frustrating stacking.
This is a good point in the story to explain that, in ACPI machines, the IRQ
steering is accomplished by interpreting BIOS-supplied P-code called ASL.
The IRQ routers are completely abstracted by the BIOS. The OS doesn't need
to know about the actual hardware. The old IRQ steering code in Win9x, which
was dropped into the non-ACPI HAL in Win2K, had to have code specific to
each chipset, which meant that it didn't work when new chipsets were
shipped. It was also written in a way that it assumed that there were
exactly four IRQs coming from PCI. ACPI machines sometimes have many more.
(This is the reason that you don't see the IRQ steering tab in ACPI
machines. It just wasn't flexible enough and we didn't have time to re-do
it.)
What we discovered with Windows XP was that all of those ACPI machines that
had been tested with their IRQs stacked on IRQ 9 tended to fail when you
spread the IRQs out. A typical example of a failure would work like this:
WinXP doesn't need the IRQ for the parallel port unless you're using one of
the extended modes. So the parallel driver releases its IRQ until it's
needed. The IRQ choosing logic (called an IRQ
"arbiter") would move a PCI device onto the parallel IRQ. This action
depends on re-programming the chipset so that the parallel port isn't actual
ly triggering the IRQ. This is supposed to happen by interpreting even more
BIOS P-code that manipulates the chipset, since there is no standard for
parallel port configuration.
If your chipset comes from Intel, this probably works, since the mere act of
setting a PCI device to an IRQ also disconnects that IRQ from the ISA bus.
But if your chipset comes from VIA or ALi, there is another step involved.
The problem is that nearly all of the BIOS P-code out there is copied from
old Intel example code. So they are almost all missing the extra step
necessary in VIA and ALi machines.
If the BIOS fails to stop the IRQ coming from the parallel port, the machine
hangs, since the parallel port, which sends its IRQs active-high,
edge-triggered, will ground the interrupt signal in the passive state. And
grounding an interrupt which is enabled active-low, level-triggered will
cause an endless stream of interrupts.
The parallel port is just an example. Pick any device that is in the legacy
SuperIO chip and the story repeats itself.
In Windows XP, I made a bunch of changes. In machines without cardbus
controllers, (which don't have the IRQ problems created by PCMCIA,) it will
try to keep the PCI devices on the IRQs that the BIOS used during boot. If
the BIOS didn't set the device up, then any IRQ may be chosen. But if your
machine has a VIA chipset, or if it has a BIOS that we know to be broken,
then we fall back to the Win2K-style stacking behavior. The unfortunate
truth is that you guys on this list mostly build your own machines, rather
than buying them from reputable manufacturers, which means that you guys own
the machines with broken BIOSes and VIA chipsets. So even with WindowsXP,
you'll see the same old stacking behavior.
One notable addendum is that any machine with an APIC interrupt controller,
and thus more than 16 IRQs, will spread interrupts out, even in Win2K. In
the past, this was mostly limited to SMP machines. But any desktop machine
shipping today that gets the Windows logo has to have an APIC. (This was
another reason that I hadn't gone back to re-write this code earlier. Intel
had promised that all machines would have APICs by 1998. If this had
materialized, then none of you would have had any complaints by now.) I'm
actually currently working on software for some future NT that will let an
administrator configure the machine in any way he or she desires.
--
-Eliyas
This posting is provided "AS IS" with no warranties, and confers no rights.
"Niki Estner" <niki....@cube.net> wrote in message
news:#H0h72S5BHA.1796@tkmsftngp02...
Maxim S. Shatskih
> IRQ, but this one doesn't, and this is a high-performance, high-throughput
> card, so it really should have its own IRQ anyway.
No, this cannot justify the lame hardware design not conforming to the PCI spec. SCSI HBAs are even more high-performance, and do
share the IRQ fine.
> The vendor tells me the only way to prevent this is to install the PC with
> the "PC-Standard" option - i.e. disable ACPI and all the nice little things
> that come with it (power off on shutdown, standby mode, USB...).
> Is there a way to dedicate an own IRQ (if one is available) exculsively to
> one PCI adapter/driver?
Under w2k with an ACPI uniprocessor HAL on a mobo without APIC support - NO ways.
Do one of the following:
- remove ACPI
- use a mobo with APIC (SMP mobos are such)
- upgrade to XP which promised to fix this
- buy a better card.
> Or is it possible to prevent IRQ sharing without turning of ACPI completely?
No, HALACPI is written this way.
Max
Jake Oshins
some more:
-----Original Message-----
From: Jake Oshins
Sent: Sunday, January 27, 2002 2:48 PM
Subject: RE: Distributing Interrupts over different vectors
From what you write, I think that you're confused about a bunch of things.
Let me do some explaining:
First, IRQL is different from IRQ. IRQL is the priority level at which your
interrupt is delivered. Many different vectors may map to the same IRQL.
This doesn't mean that you're sharing interrupts. It just means that some
interrupts will be masked (on one of the processors) while a device is
servicing it's interrupt. In practice this doesn't mean much. Don't pay much
attention to IRQL, as it doesn't tell you what you think it tells you. (You
also use the word "vector" below. That's something completely separate,
which I won't cover here.)
Second, IRQ is a representation of which interrupt line will be triggered.
The PCI spec says that each device interrupts by driving a line called INTA#
(or perhaps INTB#, INTC# or INTD#, but the differences are meaningless here)
to ground. The guy who designed HALMPS (my former boss) decided to encode
the "IRQ" of a PCI device as (DeviceNumber << 2 | (IntPin - 1)). This means
that, for instance, a device with a PCI device number of 6 that triggers
INTA# (which yeilds and Interrupt Pin Number of 1) would get encoded as (6
<< 2 | 0), or 24. This means that there is exactly one IRQ 24 on every PCI
bus, using HALMPS. If there are two PCI buses, and both of them have a
device number 6, then you'll get two IRQ 24s, even if they aren't sharing.
Furthermore, you might have two different PCI devices that are sharing. The
motherboard may wire them together. With HALMPS, you'll never see the same
IRQ for them, since they'll have different PCI Device numbers.
These two things make it completely impossible to tell if PCI devices are
sharing interrupts using HALMPS. There is no user interface that will tell
you. (You can find out with the debugger. Just type either "!arbiter 4" or
"!idt". These will make it clear.) If your hardware guy insisted that the
devices aren't sharing, then he's probably right. They aren't.
That's one of the reasons why, when I designed HALACPI, HALAACPI and
HALMACPI, I changed the representation of IRQ for those HALs. For them, IRQ
is the number of the input on the interrupt controller. If you're using an
8259 PIC interrupt controller (and consequently HALACPI) then you'll see the
same IRQ representation you saw in DOS. If you're using APIC interrupt
controllers (and consequently either HALAACPI or HALMACPI) then the
representation of IRQ starts with the primary I/O APIC, which frequently has
20 inputs. The first 16 get mapped to IRQ 0-15, which are the ISA inputs.
The next four usually get used for PCI interrupts. If you're lucky enough to
have multiple I/O APICs, you may see much higher IRQ numbers. (There were a
few major simplifications in this description. If anybody wants to go into
greater detail, post specific questions to this list.)
With these HALs, it's quite easy to see if you're sharing interrupts. If the
IRQ is the same, then your devices are plugged into the same input on the
interrupt controller, and they are sharing.
- Jake
-----Original Message-----
Subject: Distributing Interrupts over different vectors
Date: Sat, 26 Jan 2002 21:25:48 -0500
I have a new piece of hardware running NT-E, with 6 PCI busses.
This is a Dual P4 running HalMps. I have 10 high interrupt devices,
and I'd like to avoid interrupt sharing for performance reasons.
Bus 0 has 4 devices, and the other busses have 2 devices each.
I'm getting something like 4 IRQL 16, 4 IRQL 17, and 1 each IRQL 24 and 25,
i.e., the IRQL
starts back at 16 on each bus. (The IRQL is as reported by Administrative
Tools\Windows NT Diagnostics).
I've look through the HAL source, and have a very rough idea how it assigns
the vectors.
The hardware guys insist that the hardware can avoid interrupt sharing, and
that is also my understanding of the APIC.
Questions:
- Is this IRQL as reported 1:1 with the APIC Vector, and therefore I'm
really getting interrupt sharing?
- Given I can change the HAL and BIOS, is it feasible to avoid interrupt
sharing?
- Is there a description of how I'd avoid interrupt sharing some where?
- Would I need to modify the HAL, or is this a BIOS problem?
- I ended up in the HAL at some configuration files that look like they turn
into data tables inside the HAL. Is that the place
I should be looking? Is there a description somewhere?
Thanks,
-DH
--
Jake Oshins
Windows Kernel Group
This posting is provided "AS IS" with no warranties, and confers no rights.
"Eliyas Yakub" <eli...@microsoft.com> wrote in message
news:3cbc3032$1...@news.microsoft.com...
Jake Oshins
-----Original Message-----
From: Jake Oshins
Sent: Tuesday, January 29, 2002 3:48 PM
Subject: RE: How to register ISA interrupts that sit behind a PCI bridge?
Your description is vague in a couple of critical areas. You don't say
whether the "PCI bridge" is actually a PCI compliant bridge with a Type 1
header. I suspect that it's not. What you are probably describing is a
device that attaches legacy logic to the PCI bus, with a type 0 header that
you get to fill in with your specifics.
This design can't work unless you have a latch somewhere in your PCI logic
that takes the edge-triggered signals from the legacy stuff and then asserts
the PCI level-triggered interrupt in a PCI-compliant level-triggered mode.
Along with the latch, you need a couple of other things. First, you need to
gate the PCI interrupt behind some software-visible register, so that
interrupts cannot possibly be asserted unless your driver unmasks them. You
also need a status bit that allows software to see that the latch is set.
Lastly, you need a bit in a software-visible register that clears the latch.
If you try to pass the edge-triggered interrupts straight through to the
INTA# pin in the PCI slot, then you'll get into a bunch of weird edge-cases.
First, all PCI interrupts are, by definition within the PCI spec,
level-triggered and shareable. This means that your PCI card may be plugged
into a motherboard that forces it to share interrupts with some other PCI
device. In order to share successfully, you have implement the same protocol
as the other device.
If you don't want to change the hardware, you've still got problems. Even if
your current product definition says that you control the motherboard, the
BIOS and all of your custom devices, (which could get you around the
problems listed above, since you might not be required to share with another
PCI device,) then you'll still have trouble because all of the interrupt
logic in NT assumes that PCI devices are PCI compliant, which yours wouldn't
be. And it would fail to route interrupts appropriately, since the decisions
that it makes are based on the PCI spec. There are no APIs that say "Please
connect my interrupt. It's coming from a PCI device. But it's not really a
PCI-compliant device." It wouldn't be practical to add such an API, as there
are just too many permutations of PCI-non-compliance. Even if that feat were
attempted, it would be impossible to test, as you'd have to build hardware
with every possible spec violation. The matrix would be huge.
On to the next topic. You don't say if you intend to expose generic plug-in
ISA devices or whether you have a controlled set of a few that you're trying
to deal with. If you're looking for something that amounts to a generic ISA
bridge, intending to run the stock drivers for those ISA devices, you're
still going to fail, as most ISA devices can't share an interrupt, and
certainly haven't been tested with level-triggered semantics. You'd be
better off to re-do your design using a stock cardbus bridge, representing
your ISA devices as PCMCIA devices. This is well-known territory.
If, on the other hand, you're trying to glue a few well-controlled ISA
devices onto a modern PCI bus, then your task will be a little easier.
You'll need to write a new device driver, probably a bus driver, that
handles interrupts for each of the child devices. Those children will
register their ISRs with the parent bus driver, not with the IO manager. The
parent bus driver will register the one and only PCI interrupt through the
IO manager and then call the child ISRs. (This is exactly what the PCMCIA
bus driver does when the child devices are using PCI-style interrupts.
- Jake Oshins
-----Original Message-----
Subject: How to register ISA interrupts that sit behind a PCI bridge?
Date: Sun, 27 Jan 2002 21:45:50 -0800
We have a somewhat odd setup where we have a few ISA cards that share a
single edge-triggered interrupt. These are not connected to the motherboard
directly, but actually sit behind a device which then converts it to PCI
(such as the PLX / Tundra products), and a single level-triggered interrupt.
Since the devices share an interrupt, they are obviously shared, but are
they edge- or level-triggered? From the motherboard's point of view, it is
level-triggered. But since the actual devices are edge-triggered, maybe
that is how they should be registered?
This makes a difference due to the different algorithms used by NT to handle
shared interrupts. On shared level-triggered devices, the ISRs are called
until one of them returns TRUE. On shared edge-triggered devices, the ISRs
are called until ALL of them have returned FALSE.
Due to that, I think that registering them as edge-triggered would ensure
the correct behavior and avoid loss of interrupts. However, how will NT
react when we registered a PCI interrupt as edge-triggered?
(The hardware is not functional yet, so we can't just try this out...)
--
Jake Oshins
Windows Kernel Group
This posting is provided "AS IS" with no warranties, and confers no rights.
"Eliyas Yakub" <eli...@microsoft.com> wrote in message
news:3cbc3032$1...@news.microsoft.com...