Experience report: bhyve virtio-blk & passthru hangs, & nvme works

[Mathias Picker]

Hi all,

just to have this somewhere searchable:

I just configured a machine with a Supermicro X11DPI-NT
motherboard, two
Xeon Silver 4114, 128Gb RAM and two RTX 3090.

It's for experimenting with LLMs, using bhyve with the GraKa
passed
through.

This installed relatively easy, but on downloading large models
from
hugingface the system always stopped, showing multiple errors like
this
on the console:

ixl0: ixl_process_adminq: Unable to allocate memory for Admin
Queue event!
ixl1: ixl_process_adminq: Unable to allocate memory for Admin
Queue event!

This lead me in the wrong direction, the culprit was the disk. I
used a
sparse zvol and virtuio-blk.
With bhyve started AND ONLY WITH ACTIVE PASSTHRU this lead to
hanging
path from the zvol to the host, e.g. with the VΜ w. passthru up
running

$ mdo dd if=/dev/zvol/zroot/vm/llm/disk0 of=/dev/null bs=1M
status=progress

on the FreeBSD host (!) would hang after 7-11Gb.

That was puzzling. the same dd with /dev/nda0 worked just fine.

So I just tried to change the vm-bhyve config for this vm from
virtio-blk to nvme and voila, everything seems to work fine now.

dd works for 200Gb and the download from huggingface is not at
50Gb.

Hope this helps someone,

Mathias



--
Mathias Picker
Geschäftsführer
Mathias...@virtual-earth.de

virtual earth Gesellschaft für Wissens re/prä sentation mbH
http://www.virtual-earth.de/ HRB126870
sup...@virtual-earth.de Westendstr. 142
089 / 1250 3943