failed to set tss addr: File exists (os error 17)

[Mikko Koivisto]

Colleague of mine is trying to run cuttlefish virtual device on Azure. The start fails with:

the architecture failed to build the vm

Caused by: failed to set tss addr: File exists (os error 17)

Azure machine is of type Standard_D2s_v3 running Ubuntu2004. In more detail:

# uname -a
Linux myVM 6.2.0-1019-azure #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# cat /proc/cpuinfo

processor    : 0

vendor_id    : GenuineIntel

cpu family    : 6

model        : 85

model name    : Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz

....

flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear

vmx flags    : vnmi invvpid tsc_offset vtpr ept vpid

What's wrong?

[Mikko Koivisto]

Seems that EEXIST is returned when the address range intersects with already mapped address. And in this case the address range seems to start from 0xfffbd000 and has size PAGE_SIZE * 3.

[Frederick Mayle]

Can you share the crosvm invocation (the cmdline args)?

--
You received this message because you are subscribed to the Google Groups "crosvm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crosvm-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crosvm-dev/9ce51ce5-6fa5-4e82-8092-a4f35aa52bf9n%40chromium.org.

[Mikko Koivisto]

I manually parsed this from launcher.log that combines output of cuttlefish host side processes:

/cuttlefish/bin/crosvm

--extended-status

run

--socket=/tmp/cf_avd_0/cvd-1/internal/crosvm_control.sock

--no-usb

--core-scheduling=false

--gpu=backend=2D

--gpu-display=mode=windowed[720,1280],dpi=[320,320],refresh-rate=30

--wayland-sock=/tmp/cf_avd_0/cvd-1/internal/frames.sock

--rw-pmem-device=/cuttlefish/cuttlefish.single/hwcomposer-pmem

--mem=4096

--cpus=2

--rwdisk=/cuttlefish/cuttlefish.single/overlay.img

--rwdisk=/cuttlefish/cuttlefish.single/persistent_composite.img

--rwdisk=/cuttlefish/cuttlefish.single/sdcard.img

--multi-touch=/tmp/cf_avd_0/cvd-1/internal/touch_0.sock:720:1280

--rotary=/cuttlefish/cuttlefish.single/internal/rotary.sock

--keyboard=/tmp/cf_avd_0/cvd-1/internal/keyboard.sock

--switches=/tmp/cf_avd_0/cvd-1/internal/switches.sock

--net=tap-fd=89,mac="00:1a:11:e0:cf:00"

--net=tap-fd=91,mac="00:1a:11:e1:cf:00"

--net=tap-fd=93

--rw-pmem-device=/cuttlefish/cuttlefish.single/access-kregistry

--pstore=path=/cuttlefish/cuttlefish.single/pstore,size=2097152

--seccomp-policy-dir=/cuttlefish/usr/share/crosvm/x86_64-linux-gnu/seccomp

--cid=3

--serial=hardware=virtio-console,num=1,type=file,path=/cuttlefish/cuttlefish.single/internal/kernel-log-pipe,console=true

--serial=hardware=serial,num=1,type=file,path=/cuttlefish/cuttlefish.single/internal/kernel-log-pipe,earlycon=true

--serial=hardware=virtio-console,num=2,type=sink

--serial=hardware=virtio-console,num=3,type=file,path=/cuttlefish/cuttlefish.single/internal/logcat-pipe

--serial=hardware=virtio-console,num=4,type=file,path=/cuttlefish/cuttlefish.single/internal/keymaster_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/keymaster_fifo_vm.in

--serial=hardware=virtio-console,num=5,type=file,path=/cuttlefish/cuttlefish.single/internal/gatekeeper_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/gatekeeper_fifo_vm.in

--serial=hardware=virtio-console,num=6,type=file,path=/cuttlefish/cuttlefish.single/internal/bt_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/bt_fifo_vm.in

--serial=hardware=virtio-console,num=7,type=sink

--serial=hardware=virtio-console,num=8,type=sink

--serial=hardware=virtio-console,num=9,type=file,path=/cuttlefish/cuttlefish.single/internal/confui_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/confui_fifo_vm.in

--serial=hardware=virtio-console,num=10,type=file,path=/cuttlefish/cuttlefish.single/internal/uwb_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/uwb_fifo_vm.in

--serial=hardware=virtio-console,num=11,type=file,path=/cuttlefish/cuttlefish.single/internal/oemlock_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/oemlock_fifo_vm.in

--serial=hardware=virtio-console,num=12,type=file,path=/cuttlefish/cuttlefish.single/internal/keymint_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/keymint_fifo_vm.in

--serial=hardware=virtio-console,num=13,type=sink

--serial=hardware=virtio-console,num=14,type=file,path=/cuttlefish/cuttlefish.single/internal/sensors_fifo_vm.out,input=/cuttlefish/cuttlefish.single/internal/sensors_fifo_vm.in

--serial=hardware=virtio-console,num=15,type=sink

--serial=hardware=virtio-console,num=16,type=sink

--sound=/tmp/cf_avd_0/cvd-1/internal/audio_server.sock

--shared-dir=/cuttlefish/cuttlefish.single/shared:shared:type=fs

--bios=/cuttlefish/bootloader

--no-smt

[Mikko Koivisto]

It seems that

ioctl(7, KVM_SET_TSS_ADDR, 0xfffbd000) = 0    

When bootloader file size is 64kB
ioctl(7, KVM_SET_TSS_ADDR, 0xfffbd000)  = -1 EEXIST (File exists)

When bootloader file size is 1MB (the default u-boot in cuttlefish)

The bootloader size does not cause problems if enable_unrestricted_guest has been set since that makes vmx_set_tss_addr return 0 right away. That explains differences on different host setups (i.e. Azure vs GCP for example)
So the fix would be either to decrease Cuttlefish boot loader size or give more room for bios in crosvm.

[Daniel Verkamp]

On Tue, Feb 6, 2024 at 11:28 PM 'Mikko Koivisto' via crosvm-dev
<crosv...@chromium.org> wrote:
>
> It seems that
>
> ioctl(7, KVM_SET_TSS_ADDR, 0xfffbd000) = 0
>
> When bootloader file size is 64kB
> ioctl(7, KVM_SET_TSS_ADDR, 0xfffbd000) = -1 EEXIST (File exists)
>
> When bootloader file size is 1MB (the default u-boot in cuttlefish)
>
> The bootloader size does not cause problems if enable_unrestricted_guest has been set since that makes vmx_set_tss_addr return 0 right away. That explains differences on different host setups (i.e. Azure vs GCP for example)
> So the fix would be either to decrease Cuttlefish boot loader size or give more room for bios in crosvm.

Hi Mikko,

Thanks for the report. I suspect the non-enable_unrestricted_guest
case has not been tested, since it has been available on all Intel
CPUs with virtualization support since around 2010. Is there a
scenario where you find it useful to disable Unrestricted Guest for
some reason, or is this on a CPU that really does not support it?

I think even if we adjust the TSS region to avoid conflicts with the
BIOS, crosvm will probably not boot a BIOS without unrestricted guest
support on Intel VMX, since we also do not set up identity map region
(KVM_SET_IDENTITY_MAP_ADDR), which I believe is necessary to enter
real mode on VMX when unrestricted guest mode is not available. This
should be fixable if such a configuration is actually useful, though.

Thanks,
-- Daniel

[Mikko Koivisto]

We are asking about Azure unrestricted guest on another forum

https://learn.microsoft.com/en-us/answers/questions/1526675/enabling-unrestricted-guest-vmx-feature-on-azure-d

And yes, the tss problem goes away by setting the address lower but then the next error is about

failed to start vcpu 1: failed to create vcpu: File exists (os error 17)