Msi-x Pcie

[Karriem Drewery]

Ihave added MSI-X support to an implementation of the Altera PCIe core we're already testing in the lab. My understanding, is that with MSI-X enabled in the megawizard and my vector table offset set appropriately, I have created the vector table in my own register set at that offset. I then, as I understand it, trigger an MSI-X interrupt simply by issueing a single DW PCIe Memory Write to the appropriate MSI-X address with the appropriate data, as specified in the vector table. As I understand it, the specific MSI I/O to the core isn't used for MSI-X, and you just do a normal memory write.

Look at the other thread, I gave you a link to the correct MSI-X description (here it is again ( -x_ecn.pdf)). Your description is most probably from the PCI 3.0 pre-release which incorporates an early version of msi-x. I have that pre-release, too, so I understand your confusion :)

The address looks strange indeed. Can you trap the write accesses to these registers and look what address Linux actually writes, on the AST level (as write data)? Is your MSI-X Table handler prepared for all kinds of write accesses, especially if they are aggregated?

Hello! It's seems that I'm made everything ok and interrupts came to CPU. But problem appear - MSI-X is working only on 1 PC (I have tried on other PC and on 2 servers). I saw on Signal TAP that MSI-X write-packet is formed exactly like on 1st PC...

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

And the Linux kernel uses a method called Device Tree overlays to do it. On the Pi 5 (and other Pis), these overlays are stored as .dtb files inside the /boot/firmware directory, and there's an overlay for every major Raspberry Pi hardware model.

I've had to modify the dtb files in the past to increase the PCIe BAR space for early GPU testing on the Compute Module 4. And recently I've had to mess with how the PCIe address space is set up for testing certain devices on the Raspberry Pi 5.

I consider myself an absolute noob at Device Trees in Linux... but I've now done this enough times that I'd like a simple reference and most of the ones out there assume you are an expert-level wizard in all things Linux/hardware!

However, after reading the forum post. I noticed that the trick was to get the pcie@110000 device to use itself as an msi-parent. And that address had changed from 0x66 in your docs. To 0x6d in my case.

Your comment saved me. I had it worked completely but I decided to reinstall pi5 os. But this time I couldnt get it to work. turns out mine also changed to 0x6d.

If I may ask, How were you able to figure that out in case it ever changes again

In the parent article, the forum post was linked where the author figured this out. And I read through that.

The actual change you're making to the dtb is changing the msi-parent from &mip1 to &pcie1.

So, if anyone reading this in future encounters a similar issue. The information you need is in the .dts file. You will need to find the &pcie1 address that's being used, and update pcie@110000's mis-parent to use that address

LinkedIn and 3rd parties use essential and non-essential cookies to provide, secure, analyze and improve our Services, and to show you relevant ads (including professional and job ads) on and off LinkedIn. Learn more in our Cookie Policy.

A lot of work was done in constructing the PCI Express specification to give backwards compatibility to PCI configurations such that operating system code for PCI could function when enumerating and configuring systems which now had PCIe components. However, operating systems with PCIe aware software can have access to extended capability status and configuration. The original PCI configuration space was for 256 bytes. This is now extended to 4096 bytes, with the first 256 bytes for PCI and the rest for PCIe extended capabilities. Furthermore, within the 256 byte PCI space, the first 64 bytes are fully PCI compatible registers, with the other 192 bytes used for PCIe capabilities that can be accessed by legacy PCI OS code. The diagram below summarises this arrangement:

Since the configurations space is designed to be compatible with legacy OS code, and must function as such, the extended capabilities are all optional and meant to enhance functionality for operating systems that can make uses of them.

Both type 0 and type 1 configurations have a set of common registers in the PCI compatible region (0 to 3Fh). The diagram below shows these common registers and their relative position in the configuration space.

The device ID and vendor ID are read-only registers that uniquely identify the function. The vendor ID is assigned by the PCI-SIG and is different for each vendor, but the device ID is set by the vendor to identify the function. The revision ID field is also set by the vendor to identify hardware and/or firmware versions.

The class code field is a PCI register for identifying the type of function, with different numbers representing different classes of functionality. For example, a class code of 02h is a network controller or 01h is a mass storage device. These are defined in the PCI Code and ID Assignment Specification.

The cache line size register is usually programmed by the operating system to the implemented cache line size. However, in PCIe devices although a read/write register for compatibility, it has no effect on the device. Similarly, the master latency timer is not used for PCIe and is hardwired to 0.

The header type identifies whether the space is type 0 or type 1, thus defining the layout of the rest of the type specific registers. The BIST register allows control of any built-in-self-test of the function. Bit 7 indicates whether a BIST capability is available, and bit 6 is written to 1 to start the test if available. A result is returned in bits 0 to 3.

Type 0 configuration spaces are for endpoints. Beyond the common registers described above, this type of configurations space is mainly given over to defining base address registers (BARs), but with a few extra registers thrown in. The diagram below shows the layout of the PCI compatible region for a type 0 configuration space.

There are six base address registers which are used to define regions of memory mapping that the device can be assigned. This can be up to six individual 32 bits address regions, or even/odd pairs can be formed for 64-bit address regions. The lower 4 bits define characteristics of the address:

If the BAR is for I/O, bit 1 is reserved and bits 3:2 are used as part of the naturally aligned 32-bit address. If bits 2:1 of an even BAR register indicates a 64 address, then the following BAR register is the upper bits (63:32) of the address. The operating system software can determine how much space that the device is requesting to be reserved in the address space by writing all 1s to the BAR registers. The implementation hardwires the address bits to 0 for the requested space. For example, if wishing to reserve 8Mbytes of address space, bits 22:4 will be hardwired to 0, and read back as such. The minimum that can be reserved is 128 bytes. Once the software has determined the requests space it can allocate this in the memory map and set the upper bits of the BAR to the relevant address, naturally aligned to the requested space size. Normally the BAR prefetchable bit would be set unless the device has regions where reading a location would have side effects. It is strongly discouraged to design devices with this characteristic, and to make all regions prefetchable.

The subsystem ID and subsystem vendor ID are similar to that of the common registers but are used to differentiate a specific variant. For example, a common driver for network cards with the same device ID/vendor ID combinations might wish to know further information, such as different chips used on the particular device, to alter its behaviour to match any slight variations in configuration or control.

The expansion ROM base address is similar to other BARs and is used to locate, in the memory map, code held in a ROM on the device that can be executed during BIOS initialisation. The BIOS will initialise this BAR but then hand off execution to this code, usually having copied the code to main memory. The code will have device specific initialisation routines.

The secondary latency timer is not used in PCIe and is tied to 0. The secondary status register is basically a mirror of the common status register but without the interrupt status or capabilities list flag bits.

The type 1 configurations space registers also have a set of base/limit pairs, split over multiple registers, which define an upper and lower boundary for the memory and I/O regions. The memory region is split into non-prefetchable and prefetchable regions (see above). If a TLP is received by the link from upstream, and it fits between the base and limit values (for the relevant type) of the link, it will be forwarded on that downstream port.

Having looked at the PCI compatible type 0 and type 1 capabilities headers, there are other capability registers that may (or should) be included in the PCI configuration space (i.e. between 00h and FFh). These capabilities are listed below

Each capability structure can be located anywhere in the PCI configuration space, aligned to 32-bits, and will have a capability ID in the first byte to identify the type of structure. This is then followed by a next capability pointer byte that links to then next structure. A value of 00h for the next capability pointer marks the end of the linked list of capabilities.

The power management capability structure, after the common ID and next capability pointer, has a power management capabilities register (PMC) which is a set of read-only fields indicating whether a PME clock is required for generating power management events (hardwired to 0 for PCIe), whether device specific initialisation is required, the maximum 3.3V auxiliary current required, whether D1 and D2 power states are supported and which power states can assert the power management event signal. In addition, this structure has a power management control/status register (PMCSR). A two-bit field indicates what the current power state is (D3hot to D0) or, when written to, set to a new power state. Another bit indicates whether a reset is performed when software changes the power state to D0. Power management events can be enabled. A data select field selects what information is shown in the data register (the last byte of the power management capabilities structure) if that register is implemented. This optional information is for reporting power dissipated or consumed for each of the power states. Finally there is a PME_Status bit that shows the state of the PME# signal, regardless of whether PME is enabled or not.

3a8082e126