Re: [PATCH v4] x86/paravirt: Disable virt spinlock on bare metal

[maobibo]

Chenyu,

Sorry to bother you, I am porting pv spinlock to LoongArch platform, I
do not know the history about function virt_spin_lock().

When CONFIG_PARAVIRT_SPINLOCKS is enabled, there is pv_enabled() before
virt_spin_lock(), it seems that virt_spin_lock is never called in this
condition.
if (pv_enabled())
goto pv_queue;

if (virt_spin_lock(lock))
return;

When CONFIG_PARAVIRT_SPINLOCKS is disabled, there is no pv qspinlock
compiled, so what is the usage about function virt_spin_lock()? Is it
to switch spinlock method from test-set spinlock to qspinlock? why is
there such requirement for spinlock switching method?

Regards
Bibo Mao


On 2024/7/29 下午2:52, Chen Yu wrote:
> The kernel can change spinlock behavior when running as a guest. But
> this guest-friendly behavior causes performance problems on bare metal.
> So there's a 'virt_spin_lock_key' static key to switch between the two
> modes.
>
> In current code, the static key is always enabled by default when
> running in guest mode. The key is disabled for bare metal (and in
> some guests that want native behavior).
>
> Large performance regression is reported when running encode/decode
> workload and BenchSEE cache sub-workload on the bare metal.
> Bisect points to commit ce0a1b608bfc ("x86/paravirt: Silence unused
> native_pv_lock_init() function warning"). When CONFIG_PARAVIRT_SPINLOCKS
> is disabled, the virt_spin_lock_key is incorrectly set to true on bare
> metal. The qspinlock degenerates to test-and-set spinlock, which
> decrease the performance on bare metal.
>
> Set the default value of virt_spin_lock_key to false. If booting in
> a VM, enable this key. Later during the VM initialization, if other
> high-efficient spinlock is detected(paravirt-spinlock eg), the
> virt_spin_lock_key is disabled. According to the description above,
> the final effect will be as followed:
>
> X86_FEATURE_HYPERVISOR Y Y Y N
> CONFIG_PARAVIRT_SPINLOCKS Y Y N Y/N
> PV spinlock Y N N Y/N
>
> virt_spin_lock_key N N Y N
>
> To summarize, the virt_spin_lock_key is disabled on the bare metal
> no matter what other condidtion is. And the virt_spin_lock_key is
> also disabled when other spinlock mechanism is detected in the VM
> guest.
>
> Fixes: ce0a1b608bfc ("x86/paravirt: Silence unused native_pv_lock_init() function warning")
> Suggested-by: Dave Hansen <dave....@linux.intel.com>
> Suggested-by: Qiuxu Zhuo <qiuxu...@intel.com>
> Suggested-by: Nikolay Borisov <nik.b...@suse.com>
> Reported-by: Prem Nath Dey <prem.n...@intel.com>
> Reported-by: Xiaoping Zhou <xiaopi...@intel.com>
> Reviewed-by: Nikolay Borisov <nik.b...@suse.com>
> Signed-off-by: Chen Yu <yu.c...@intel.com>
> ---
> v3->v4:
> Refine the commit log.
> Added Reviewed-by tag from Nikolay.
> v2->v3:
> Change the default value of virt_spin_lock_key from true to false.
> Enable this key when it is in the VM, and disable it when needed.
> This makes the code more readable. (Nikolay Borisov)
> Dropped Reviewed-by because the code has been changed.
> v1->v2:
> Refine the commit log per Dave's suggestion.
> Simplify the fix by directly disabling the virt_spin_lock_key on bare metal.
> Collect Reviewed-by from Juergen.
> ---
> arch/x86/include/asm/qspinlock.h | 4 ++--
> arch/x86/kernel/paravirt.c | 7 +++----
> 2 files changed, 5 insertions(+), 6 deletions(-)
>
> diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h
> index a053c1293975..a32bd2aabdf9 100644
> --- a/arch/x86/include/asm/qspinlock.h
> +++ b/arch/x86/include/asm/qspinlock.h
> @@ -66,13 +66,13 @@ static inline bool vcpu_is_preempted(long cpu)
>
> #ifdef CONFIG_PARAVIRT
> /*
> - * virt_spin_lock_key - enables (by default) the virt_spin_lock() hijack.
> + * virt_spin_lock_key - disables (by default) the virt_spin_lock() hijack.
> *
> * Native (and PV wanting native due to vCPU pinning) should disable this key.
> * It is done in this backwards fashion to only have a single direction change,
> * which removes ordering between native_pv_spin_init() and HV setup.
> */
> -DECLARE_STATIC_KEY_TRUE(virt_spin_lock_key);
> +DECLARE_STATIC_KEY_FALSE(virt_spin_lock_key);
>
> /*
> * Shortcut for the queued_spin_lock_slowpath() function that allows
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index 5358d43886ad..fec381533555 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -51,13 +51,12 @@ DEFINE_ASM_FUNC(pv_native_irq_enable, "sti", .noinstr.text);
> DEFINE_ASM_FUNC(pv_native_read_cr2, "mov %cr2, %rax", .noinstr.text);
> #endif
>
> -DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key);
> +DEFINE_STATIC_KEY_FALSE(virt_spin_lock_key);
>
> void __init native_pv_lock_init(void)
> {
> - if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) &&
> - !boot_cpu_has(X86_FEATURE_HYPERVISOR))
> - static_branch_disable(&virt_spin_lock_key);
> + if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
> + static_branch_enable(&virt_spin_lock_key);
> }
>
> static void native_tlb_remove_table(struct mmu_gather *tlb, void *table)
>

[maobibo]

On 2024/7/30 下午4:46, Chen Yu wrote:
> Hi Bibo,

>
> On 2024-07-30 at 09:21:45 +0800, maobibo wrote:
>> Chenyu,
>>
>> Sorry to bother you, I am porting pv spinlock to LoongArch platform, I do
>> not know the history about function virt_spin_lock().
>>
>> When CONFIG_PARAVIRT_SPINLOCKS is enabled, there is pv_enabled() before
>> virt_spin_lock(), it seems that virt_spin_lock is never called in this
>> condition.
>

> Right, if CONFIG_PARAVIRT_SPINLOCKS is enabled, virt_spin_lock() will not be
> checked at all and go directly to pv_queue lock section.

>
>> if (pv_enabled())
>> goto pv_queue;
>>
>> if (virt_spin_lock(lock))
>> return;
>>
>> When CONFIG_PARAVIRT_SPINLOCKS is disabled, there is no pv qspinlock
>> compiled, so what is the usage about function virt_spin_lock()?
>> Is it to switch spinlock method from test-set spinlock to qspinlock?
>> why is there such requirement for spinlock switching method?
>

> Firstly, according to the commit 2aa79af64263 ("locking/qspinlock: Revert
> to test-and-set on hypervisors"), virt_spin_lock() was originally introduced
> to avoid the over-preemption of native qspinlock(when paravirt-spinlock is
> disabled).
Chenyu,

Thanks for the information. It seems that test-and-set spinlock is
suggested rather than qspinlock on VM when paravirt-spinlock is disabled.

Regards
Bibo Mao
>
> Secondly, there is user requirement to use native qspinlock over test-set lock
> in the VM. For example, nopvspin parameter is parsed in xen_init_spinlocks().
> commit 9043442b43b1 ("locking/paravirt: Use new static key for controlling call
> of virt_spin_lock()") was introduced to adjust the key from user boot up commandline.
> so the key switches between test-set and the native qspinlock. I've Cced
> Juergen and Peter in case I misunderstood.
>
> thanks,
> Chenyu
>