Anyone using MSI-X?

[Valentine Sinitsyn]

Hi all,

I faced a strange issue regarding MSI-X on my board.

A USB host (xchi_hcd) enables 5 MSI-X vectors out of 8.

The problem is other three vectors remain unmasked (at least, according
to dev->msi_vectors in Jailhouse), even if they store all-zeros. The PCI
spec says all vectors must be masked on reset (which sounds sane).

What's this? A bug in the kernel/driver? A config issue? Has anyone
tested Jailhouse in conjunction with MSI-X?

Best,
Valentine

[Valentine Sinitsyn]

Just looked into the MSI-X table. It really has all vectors unmasked
even before Jailhouse gets into play.

Looks like a BIOS/hardware issue to me. If it is so, shall we work
around it in the Jailhouse somehow?

Valentine

[Jan Kiszka]

On 2016-07-05 13:21, Valentine Sinitsyn wrote:
> Hi all,
>
> I faced a strange issue regarding MSI-X on my board.
>
> A USB host (xchi_hcd) enables 5 MSI-X vectors out of 8.
>
> The problem is other three vectors remain unmasked (at least, according
> to dev->msi_vectors in Jailhouse), even if they store all-zeros. The PCI

(you mean dev->msix_vectors, I suppose)

> spec says all vectors must be masked on reset (which sounds sane).
>
> What's this? A bug in the kernel/driver? A config issue? Has anyone
> tested Jailhouse in conjunction with MSI-X?

Yes, in various setups. Worked fine so far.

What is the exact problem you see? Does Jailhouse fail when enabling,
thus trying to migrate those vectors to remapped mode?

Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[Valentine Sinitsyn]

On 05.07.2016 22:51, Jan Kiszka wrote:
> On 2016-07-05 13:21, Valentine Sinitsyn wrote:
>> Hi all,
>>
>> I faced a strange issue regarding MSI-X on my board.
>>
>> A USB host (xchi_hcd) enables 5 MSI-X vectors out of 8.
>>
>> The problem is other three vectors remain unmasked (at least, according
>> to dev->msi_vectors in Jailhouse), even if they store all-zeros. The PCI
>
> (you mean dev->msix_vectors, I suppose)

Sure, dev->msix_vectors.

>
>> spec says all vectors must be masked on reset (which sounds sane).
>>
>> What's this? A bug in the kernel/driver? A config issue? Has anyone
>> tested Jailhouse in conjunction with MSI-X?
>
> Yes, in various setups. Worked fine so far.
>
> What is the exact problem you see? Does Jailhouse fail when enabling,
> thus trying to migrate those vectors to remapped mode?

Yes. When iommu_map_interrupt() tries to validate these all-zero
entries, it fails. This is expected, of course, but the precise reason
is destination CPU 0 (all-zeros entry obviously can't reference anything
else) doesn't exists on this board.

The expected behavior would be not to remap these entries, of course.
However, they aren't masked (which is what I'd expect), and adding
another "address != 0" check seems like a hack.

Valentine

[Jan Kiszka]

We are pretty strict right now with the configuration migration during
jailhouse enable. We could relax that, only warn about "blocking invalid
vector X of device Y", do not configure a route and continue. When a
guest actively programs such things later on, failing is likely still
better (because that would be a guest bug, or worse).

[Valentine Sinitsyn]

On 06.07.2016 11:28, Jan Kiszka wrote:
> On 2016-07-05 20:49, Valentine Sinitsyn wrote:

[...]

>> The expected behavior would be not to remap these entries, of course.
>> However, they aren't masked (which is what I'd expect), and adding
>> another "address != 0" check seems like a hack.
>
> We are pretty strict right now with the configuration migration during
> jailhouse enable. We could relax that, only warn about "blocking invalid
> vector X of device Y", do not configure a route and continue. When a

Yes, this makes sense - probably I'll implement this. Do you think a
BIOS update could fix the issue on this particular board (haven't tried
yet)?

Valentine

[Jan Kiszka]

On 2016-07-06 08:44, Valentine Sinitsyn wrote:
> On 06.07.2016 11:28, Jan Kiszka wrote:
>> On 2016-07-05 20:49, Valentine Sinitsyn wrote:
> [...]
>>> The expected behavior would be not to remap these entries, of course.
>>> However, they aren't masked (which is what I'd expect), and adding
>>> another "address != 0" check seems like a hack.
>>
>> We are pretty strict right now with the configuration migration during
>> jailhouse enable. We could relax that, only warn about "blocking invalid
>> vector X of device Y", do not configure a route and continue. When a
> Yes, this makes sense - probably I'll implement this. Do you think a
> BIOS update could fix the issue on this particular board (haven't tried
> yet)?

I wouldn't bet on the BIOS only. Maybe the device [firmware] already
fails to reset the mask bits properly.