[PATCH v2 00/11] KVM: x86/xen: Clean up shared info / vcpu info handling

[David Woodhouse]

KVM reads kvm->arch.xen.long_mode in a dozen places to decide whether the
guest's shared_info and vcpu_info are in native or compat layout. The guest
can change that flag at any time from another vCPU, and several code paths
read it more than once, assuming the two reads agree.

The most serious case is kvm_xen_set_evtchn_fast(), where max_evtchn_port()
and the bitmap layout selection each call it separately. If the mode changes
in between, a port accepted by the 64-bit range check (up to 4095) can be
handled with the 32-bit layout, giving a port_word_bit of up to 127 — and
test_and_set_bit() then runs off the end of the 8-byte evtchn_pending_sel,
into the adjacent vcpu_id and timer_virq fields.

Related, the vcpu_info is guest-controlled and only 4-byte aligned, but the
generic test_and_set_bit()/set_bit() helpers use 64-bit locked operations on
x86-64. On a host with split_lock_detect=fatal that is a guest-triggerable
panic.

Patches 10 and 11 are new bugfixes in v2. One bug I found while working
on the SRCU conversion of the gfn_to_pfn_cache locking, and syzbot found
the other. These could potentially each be submitted separately, but
I've elected to round them up here to slightly reduce the number of
patch series in flight.

So:

- Patches 1-3 are preparatory renames and a kvm_xen_has_64bit_shinfo()
helper which does the READ_ONCE() in one place.

- Patches 4-5 latch the mode once on entry to kvm_xen_set_evtchn_fast()
and kvm_xen_schedop_poll(), so the range check and the layout selection
cannot disagree.

- Patch 6 enforces 4-byte alignment of the vcpu_info registration. Note
this deliberately does *not* require 8-byte alignment even in 64-bit
mode: a guest may validly register a 4-byte aligned vcpu_info while in
32-bit mode and only later switch to 64-bit, and rejecting that would
break migration of such guests.

- Patches 7-9 use 32-bit atomics where the address is guest-controlled,
and then replace the open-coded asm with the atomic*() APIs.

- Patch 10 takes kvm->srcu in __kvm_xen_has_interrupt(), which has been
calling kvm_gpc_check() — and thus dereferencing kvm->memslots — without
it since 2022.

- Patch 11 fixes a use-after-free in the gfn_to_pfn_cache refresh path
for HVA-based caches (e.g. a shared_info page registered with
KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA), reported by syzbot. The refresh
retry check relies on kvm->mmu_invalidate_seq, which is only advanced
for invalidations overlapping a memslot; an HVA-based cache need not
be backed by a memslot at all, so an invalidation of the cached HVA
which starts and ends entirely within the HVA->PFN lookup window is
invisible to it, and the refresh publishes a mapping of a freed page.
Add a dedicated sequence count advanced by every invalidation.


v2:
- Add patch 10 (kvm->srcu in __kvm_xen_has_interrupt()).
- Add patch 11 (dedicated invalidation sequence for HVA-based caches).
- Take Sean's version of the 32-bit atomics patch, and his follow-up
replacing the remaining asm blobs with atomic*() APIs.
- Use GEN_BINARY_RMWcc() rather than open-coding the btsl.
- Explain why the pending_bits access in kvm_xen_set_evtchn_fast() does
not need the same treatment as the vcpu_info one (it is in the page
aligned per-VM shared_info).
- Don't require 8-byte alignment of vcpu_info in 64-bit mode; always
accept 4-byte alignment, so as not to break migration of guests which
registered while in 32-bit mode.
- Add Closes: links for the reported issues.
- Cast to u64 before the >> 32 in the unaligned evtchn_pending_sel
handling; evtchn_pending_sel is unsigned long, so the shift was
undefined on 32-bit even though the branch is unreachable there
(kernel test robot).
- Rebase onto kvm-x86/next; the mode-aware kvm_<reg>_read() helpers which
landed in the meantime subsume most of what patch 1 was doing by hand.

v1: https://lore.kernel.org/all/2026060514303...@infradead.org/

David Woodhouse (8):
KVM: x86/xen: Rename 'longmode' to 'is_64bit' in hypercall handling
KVM: x86/xen: Introduce kvm_xen_has_64bit_shinfo() macro
KVM: x86/xen: Rename max_evtchn_port() to kvm_max_evtchn_port()
KVM: x86/xen: Latch shinfo mode in kvm_xen_schedop_poll()
KVM: x86/xen: Enforce 4-byte alignment of vcpu_info registration
KVM: x86/xen: Use 32-bit locked bts for vcpu_info evtchn_pending_sel
KVM: x86/xen: Take kvm->srcu in __kvm_xen_has_interrupt()
KVM: pfncache: use a dedicated invalidation sequence for cache refresh

Hyunwoo Kim (1):
KVM: x86/xen: Latch shinfo mode in kvm_xen_set_evtchn_fast()

Sean Christopherson (2):
KVM: x86/xen: Use 32-bit atomics if vCPU's evtchn_pending_sel isn't aligned
KVM: x86/xen: Use atomic*() APIs instead of open coded equivalents

arch/x86/kvm/xen.c | 170 ++++++++++++++++++++++++++++++-----------------
arch/x86/kvm/xen.h | 5 ++
include/linux/kvm_host.h | 2 +
virt/kvm/kvm_main.c | 10 +++
virt/kvm/pfncache.c | 18 ++---
5 files changed, 135 insertions(+), 70 deletions(-)

[David Woodhouse]

From: David Woodhouse <dw...@amazon.co.uk>

Rename in preparation for adding a variant that takes a latched bool
argument for use in paths that need a consistent snapshot of the
shinfo mode.

No functional change.

Assisted-by: Kiro:claude-opus-4.6-1m
Signed-off-by: David Woodhouse <dw...@amazon.co.uk>
---
arch/x86/kvm/xen.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index e234c2a192c3..00894afea9be 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -1423,7 +1423,7 @@ static int kvm_xen_hypercall_complete_userspace(struct kvm_vcpu *vcpu)
return kvm_xen_hypercall_set_result(vcpu, run->xen.u.hcall.result);
}

-static inline int max_evtchn_port(struct kvm *kvm)
+static inline int kvm_max_evtchn_port(struct kvm *kvm)
{
if (kvm_xen_has_64bit_shinfo(kvm))
return EVTCHN_2L_NR_CHANNELS;
@@ -1530,7 +1530,7 @@ static bool kvm_xen_schedop_poll(struct kvm_vcpu *vcpu, bool is_64bit,
}

for (i = 0; i < sched_poll.nr_ports; i++) {
- if (ports[i] >= max_evtchn_port(vcpu->kvm)) {
+ if (ports[i] >= kvm_max_evtchn_port(vcpu->kvm)) {
*r = -EINVAL;
goto out;
}
@@ -1817,7 +1817,7 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
WRITE_ONCE(xe->vcpu_idx, vcpu->vcpu_idx);
}

- if (xe->port >= max_evtchn_port(kvm))
+ if (xe->port >= kvm_max_evtchn_port(kvm))
return -EINVAL;

rc = -EWOULDBLOCK;
@@ -1979,7 +1979,7 @@ int kvm_xen_setup_evtchn(struct kvm *kvm,
struct kvm_vcpu *vcpu;

/*
- * Don't check for the port being within range of max_evtchn_port().
+ * Don't check for the port being within range of kvm_max_evtchn_port().
* Userspace can configure what ever targets it likes; events just won't
* be delivered if/while the target is invalid, just like userspace can
* configure MSIs which target non-existent APICs.
@@ -1988,8 +1988,8 @@ int kvm_xen_setup_evtchn(struct kvm *kvm,
* can be restored *independently* of other things like creating vCPUs,
* without imposing an ordering dependency on userspace. In this
* particular case, the problematic ordering would be with setting the
- * Xen 'long mode' flag, which changes max_evtchn_port() to allow 4096
- * instead of 1024 event channels.
+ * Xen 'long mode' flag, which changes kvm_max_evtchn_port() to allow
+ * 4096 instead of 1024 event channels.
*/

/* We only support 2 level event channels for now */
@@ -2026,7 +2026,7 @@ int kvm_xen_hvm_evtchn_send(struct kvm *kvm, struct kvm_irq_routing_xen_evtchn *
struct kvm_xen_evtchn e;
int ret;

- if (!uxe->port || uxe->port >= max_evtchn_port(kvm))
+ if (!uxe->port || uxe->port >= kvm_max_evtchn_port(kvm))
return -EINVAL;

/* We only support 2 level event channels for now */
@@ -2136,7 +2136,7 @@ static int kvm_xen_eventfd_assign(struct kvm *kvm,

case EVTCHNSTAT_interdomain:
if (data->u.evtchn.deliver.port.port) {
- if (data->u.evtchn.deliver.port.port >= max_evtchn_port(kvm))
+ if (data->u.evtchn.deliver.port.port >= kvm_max_evtchn_port(kvm))
goto out_noeventfd; /* -EINVAL */
} else {
eventfd = eventfd_ctx_fdget(data->u.evtchn.deliver.eventfd.fd);
@@ -2254,7 +2254,7 @@ static int kvm_xen_setattr_evtchn(struct kvm *kvm, struct kvm_xen_hvm_attr *data
if (data->u.evtchn.flags == KVM_XEN_EVTCHN_RESET)
return kvm_xen_eventfd_reset(kvm);

- if (!port || port >= max_evtchn_port(kvm))
+ if (!port || port >= kvm_max_evtchn_port(kvm))
return -EINVAL;

if (data->u.evtchn.flags == KVM_XEN_EVTCHN_DEASSIGN)
--
2.55.0

[David Woodhouse]

From: David Woodhouse <dw...@amazon.co.uk>

Replace test_and_set_bit() on vcpu_info->evtchn_pending_sel with an
explicit 'lock btsl' in kvm_xen_set_evtchn_fast(). The generic
test_and_set_bit() uses a 64-bit locked operation ('lock btsq') on
x86-64, and the address of the per-vCPU info is guest-controlled and
only required to be 32-bit aligned, so an 8-byte access can generate a
split-lock #AC exception.

Since evtchn_pending_sel is at most 64 bits wide and port_word_bit
ranges 0-63, a 32-bit 'lock btsl' suffices for both native and compat
vcpu_info layouts, and only requires the 4-byte alignment that is
already guaranteed by the registration path.

This also eliminates the bogus cast of compat_vcpu_info's 32-bit
evtchn_pending_sel to 'unsigned long *' which was the original source
of the split-lock hazard.

Note, KVM reuses the local gpc. The atomic accesses to pending_bits is
on the page-aligned per-VM shared_info structure, i.e. doesn't need the
same treatment as the access is guaranteed to be 64-bit aligned.

Fixes: 14243b387137 ("KVM: x86/xen: Add KVM_IRQ_ROUTING_XEN_EVTCHN and event channel delivery")
Reported-by: sashi...@kernel.org
Closes: https://lore.kernel.org/all/2026060419355...@smtp.kernel.org
Suggested-by: Sean Christopherson <sea...@google.com>
Signed-off-by: David Woodhouse <dw...@amazon.co.uk>
Assisted-by: Claude:claude-mythos-5
---
arch/x86/kvm/xen.c | 47 ++++++++++++++++++++++++++++++++--------------
1 file changed, 33 insertions(+), 14 deletions(-)

diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 959d79eef0ce..c935651906ec 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -1815,7 +1815,7 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
{
struct gfn_to_pfn_cache *gpc = &kvm->arch.xen.shinfo_cache;
bool has_64bit_shinfo = kvm_xen_has_64bit_shinfo(kvm);
- unsigned long *pending_bits, *mask_bits;
+ unsigned long *pending_bits, *mask_bits, vi_pending_sel_ofs;
struct kvm_vcpu *vcpu;
unsigned long flags;
int port_word_bit;
@@ -1848,11 +1848,18 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
pending_bits = (unsigned long *)&shinfo->evtchn_pending;
mask_bits = (unsigned long *)&shinfo->evtchn_mask;
port_word_bit = xe->port / 64;
+
+ vi_pending_sel_ofs = offsetof(struct vcpu_info, evtchn_pending_sel);
} else {
struct compat_shared_info *shinfo = gpc->khva;
pending_bits = (unsigned long *)&shinfo->evtchn_pending;
mask_bits = (unsigned long *)&shinfo->evtchn_mask;
port_word_bit = xe->port / 32;
+
+ vi_pending_sel_ofs = offsetof(struct compat_vcpu_info, evtchn_pending_sel);
+
+ /* test_and_set_bit() needs 64-bit alignment, but that's OK */
+ BUILD_BUG_ON(offsetof(struct compat_shared_info, evtchn_pending) & 7);
}

/*
@@ -1868,6 +1875,8 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
rc = -ENOTCONN; /* Masked */
kvm_xen_check_poller(vcpu, xe->port);
} else {
+ bool old;
+
rc = 1; /* Delivered to the bitmap in shared_info. */
/* Now switch to the vCPU's vcpu_info to set the index and pending_sel */
read_unlock_irqrestore(&gpc->lock, flags);
@@ -1884,19 +1893,29 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
goto out_rcu;
}

- if (has_64bit_shinfo) {
- struct vcpu_info *vcpu_info = gpc->khva;
- if (!test_and_set_bit(port_word_bit, &vcpu_info->evtchn_pending_sel)) {
- WRITE_ONCE(vcpu_info->evtchn_upcall_pending, 1);
- kick_vcpu = true;
- }
- } else {
- struct compat_vcpu_info *vcpu_info = gpc->khva;
- if (!test_and_set_bit(port_word_bit,
- (unsigned long *)&vcpu_info->evtchn_pending_sel)) {
- WRITE_ONCE(vcpu_info->evtchn_upcall_pending, 1);
- kick_vcpu = true;
- }
+ /*
+ * Explicitly use a 32-bit btsl instead of test_and_set_bit(),
+ * which would use btsq on x86-64. The vcpu_info is guest-
+ * controlled and only required to be 32-bit aligned, so a
+ * 64-bit access could generate a split-lock #AC.
+ *
+ * Note, this does not apply to the test_and_set_bit() on
+ * pending_bits above: that is in the per-VM shared_info, which
+ * is page aligned, so the access is guaranteed to be 64-bit
+ * aligned.
+ */
+ old = GEN_BINARY_RMWcc(LOCK_PREFIX "btsl",
+ *(u32 *)(gpc->khva + vi_pending_sel_ofs),
+ c, "Ir", port_word_bit);
+ if (!old) {
+ struct vcpu_info *vi = gpc->khva;
+
+ /* No need for compat handling */
+ BUILD_BUG_ON(offsetof(struct vcpu_info, evtchn_upcall_pending) !=
+ offsetof(struct compat_vcpu_info, evtchn_upcall_pending));
+
+ WRITE_ONCE(vi->evtchn_upcall_pending, 1);
+ kick_vcpu = true;
}

/* For the per-vCPU lapic vector, deliver it as MSI. */
--
2.55.0

[David Woodhouse]

From: Sean Christopherson <sea...@google.com>

Replace the open coded atomic asm blobs in the Xen event injection code
with equivalent atomic{,64}_xxx() operations. Casting the event channel
to atomic types is ugly, but not as ugly as asm blobs.

No functional change intended.

Signed-off-by: Sean Christopherson <sea...@google.com>
Signed-off-by: David Woodhouse <dw...@amazon.co.uk>
---
arch/x86/kvm/xen.c | 35 ++++++++++++-----------------------
1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 419b07fdaa3a..99f3ffa64fdb 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -662,12 +662,12 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
/* Now gpc->khva is a valid kernel address for the vcpu_info */
if (kvm_xen_has_64bit_shinfo(v->kvm)) {

struct vcpu_info *vi = gpc->khva;

+ void *vi_pending_sel = &vi->evtchn_pending_sel;

- if (IS_ALIGNED((unsigned long)&vi->evtchn_pending_sel, sizeof(u64)))
- asm volatile(LOCK_PREFIX "orq %[src], %[dst]\n"
- : [dst] "+m" (vi->evtchn_pending_sel)
- : [src] "r" (evtchn_pending_sel));
- else
+ if (IS_ALIGNED((unsigned long)vi_pending_sel, sizeof(u64))) {
+ atomic64_or(evtchn_pending_sel, vi_pending_sel);
+ } else {
+ atomic_or(evtchn_pending_sel, vi_pending_sel);
/*
* The cast keeps the shift well-defined on 32-bit,
* where evtchn_pending_sel is 32 bits wide and this
@@ -675,28 +675,17 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
* kvm_xen_has_64bit_shinfo(), which is gated on
* IS_ENABLED(CONFIG_64BIT)).
*/
- asm volatile(LOCK_PREFIX "orl %[src_lo], %[dst_lo]\n"
- LOCK_PREFIX "orl %[src_hi], %[dst_hi]\n"
- : [dst_lo] "+m" (vi->evtchn_pending_sel),
- [dst_hi] "+m" (*(((u32 *)&vi->evtchn_pending_sel) + 1))
- : [src_lo] "r" ((u32)evtchn_pending_sel),
- [src_hi] "r" ((u32)((u64)evtchn_pending_sel >> 32)));
-
- asm volatile(LOCK_PREFIX "andq %1, %0\n"
- : "+m" (v->arch.xen.evtchn_pending_sel)
- : "r" (~evtchn_pending_sel));
+ atomic_or((u64)evtchn_pending_sel >> 32,
+ vi_pending_sel + 4);
+ }
+
+ atomic64_andnot(evtchn_pending_sel, (void *)&v->arch.xen.evtchn_pending_sel);
WRITE_ONCE(vi->evtchn_upcall_pending, 1);
} else {
- u32 evtchn_pending_sel32 = evtchn_pending_sel;
struct compat_vcpu_info *vi = gpc->khva;

- asm volatile(LOCK_PREFIX "orl %0, %1\n"
- "notl %0\n"
- LOCK_PREFIX "andl %0, %2\n"
- : "=r" (evtchn_pending_sel32),
- "+m" (vi->evtchn_pending_sel),
- "+m" (v->arch.xen.evtchn_pending_sel)
- : "0" (evtchn_pending_sel32));
+ atomic_or(evtchn_pending_sel, (void *)&vi->evtchn_pending_sel);
+ atomic_andnot(evtchn_pending_sel, (void *)&v->arch.xen.evtchn_pending_sel);
WRITE_ONCE(vi->evtchn_upcall_pending, 1);
}

--
2.55.0

[David Woodhouse]

From: David Woodhouse <dw...@amazon.co.uk>

kvm_xen_schedop_poll() validates port numbers against
kvm_max_evtchn_port() and then calls wait_pending_event() which reads
the shinfo mode again to select the bitmap layout.

Latch kvm_xen_has_64bit_shinfo() once and pass it to both
max_evtchn_port() and wait_pending_event().

As with the previous fix to kvm_xen_set_evtchn_fast(), this is
harmless in practice for the same reasons: the inconsistency can only
corrupt fields in the guest's own shared_info page, and the same
corruption can occur anyway if the mode changes immediately after the
latch.

Fixes: d518b9d0fc80 ("KVM: x86/xen: handle PV spinlocks slowpath")
Assisted-by: Kiro:claude-opus-4.6-1m

Signed-off-by: David Woodhouse <dw...@amazon.co.uk>
---

arch/x86/kvm/xen.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 9edfdc585337..e249a1b1d446 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -1436,8 +1436,8 @@ static inline int kvm_max_evtchn_port(struct kvm *kvm)
return max_evtchn_port(kvm_xen_has_64bit_shinfo(kvm));
}

-static bool wait_pending_event(struct kvm_vcpu *vcpu, int nr_ports,
- evtchn_port_t *ports)
+static bool wait_pending_event(struct kvm_vcpu *vcpu, bool has_64bit_shinfo,
+ int nr_ports, evtchn_port_t *ports)
{
struct kvm *kvm = vcpu->kvm;

struct gfn_to_pfn_cache *gpc = &kvm->arch.xen.shinfo_cache;

@@ -1452,7 +1452,7 @@ static bool wait_pending_event(struct kvm_vcpu *vcpu, int nr_ports,
goto out_rcu;

ret = false;
- if (kvm_xen_has_64bit_shinfo(kvm)) {
+ if (has_64bit_shinfo) {
struct shared_info *shinfo = gpc->khva;

pending_bits = (unsigned long *)&shinfo->evtchn_pending;

} else {
@@ -1477,6 +1477,7 @@ static bool wait_pending_event(struct kvm_vcpu *vcpu, int nr_ports,

static bool kvm_xen_schedop_poll(struct kvm_vcpu *vcpu, bool is_64bit,

u64 param, u64 *r)
{
+ bool has_64bit_shinfo = kvm_xen_has_64bit_shinfo(vcpu->kvm);
struct sched_poll sched_poll;
evtchn_port_t port, *ports;
struct x86_exception e;
@@ -1535,7 +1536,7 @@ static bool kvm_xen_schedop_poll(struct kvm_vcpu *vcpu, bool is_64bit,

}

for (i = 0; i < sched_poll.nr_ports; i++) {

- if (ports[i] >= kvm_max_evtchn_port(vcpu->kvm)) {
+ if (ports[i] >= max_evtchn_port(has_64bit_shinfo)) {

*r = -EINVAL;
goto out;
}

@@ -1548,7 +1549,7 @@ static bool kvm_xen_schedop_poll(struct kvm_vcpu *vcpu, bool is_64bit,

set_bit(vcpu->vcpu_idx, vcpu->kvm->arch.xen.poll_mask);

- if (!wait_pending_event(vcpu, sched_poll.nr_ports, ports)) {
+ if (!wait_pending_event(vcpu, has_64bit_shinfo, sched_poll.nr_ports, ports)) {
kvm_set_mp_state(vcpu, KVM_MP_STATE_HALTED);

if (sched_poll.timeout)
--
2.55.0