[PATCH 0/3] scripts: small run.py enhancements
73 views
Skip to first unread message
Fotis Xenakis
Mar 6, 2021, 4:32:05 PM3/6/21
to osv...@googlegroups.com, Fotis Xenakis
This series includes some minor run.py enhancements. All patches are
independent, please feel free to apply as/if you see fit.
Fotis Xenakis (3):
scripts: add kernel command-line options to run.py
scripts: enhance virtiofsd functionality in run.py
scripts: add virtio-fs DAX option to run.py
scripts/run.py | 63 +++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 49 insertions(+), 14 deletions(-)
--
2.30.1
independent, please feel free to apply as/if you see fit.
Fotis Xenakis (3):
scripts: add kernel command-line options to run.py
scripts: enhance virtiofsd functionality in run.py
scripts: add virtio-fs DAX option to run.py
scripts/run.py | 63 +++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 49 insertions(+), 14 deletions(-)
--
2.30.1
Fotis Xenakis
Mar 6, 2021, 4:32:58 PM3/6/21
to osv...@googlegroups.com, Fotis Xenakis
Thia adds the --mount-fs, --ip and --bootchart options.
Signed-off-by: Fotis Xenakis <fo...@windowslive.com>
---
scripts/run.py | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/scripts/run.py b/scripts/run.py
index eb0a99ff..abecda3e 100755
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -73,6 +73,15 @@ def set_imgargs(options):
if options.hypervisor == 'qemu_microvm':
execute = '--nopci ' + execute
+ if options.mount_fs:
+ execute = ' '.join('--mount-fs=%s' % m for m in options.mount_fs) + ' ' + execute
+
+ if options.ip:
+ execute = ' '.join('--ip=%s' % i for i in options.ip) + ' ' + execute
+
+ if options.bootchart:
+ execute = '--bootchart ' + execute
+
options.osv_cmdline = execute
if options.kernel or options.hypervisor == 'qemu_microvm' or options.arch == 'aarch64':
return
@@ -576,6 +585,12 @@ if __name__ == "__main__":
help="virtio-fs device tag")
parser.add_argument("--virtio-fs-dir", action="store",
help="path to the directory exposed via virtio-fs mount")
+ parser.add_argument("--mount-fs", default=[], action="append",
+ help="extra mounts (forwarded to respective kernel command line option)")
+ parser.add_argument("--ip", default=[], action="append",
+ help="static ip addresses (forwarded to respective kernel command line option)")
+ parser.add_argument("--bootchart", action="store_true",
+ help="bootchart mode (forwarded to respective kernel command line option")
cmdargs = parser.parse_args()
cmdargs.opt_path = "debug" if cmdargs.debug else "release" if cmdargs.release else "last"
--
2.30.1
Signed-off-by: Fotis Xenakis <fo...@windowslive.com>
---
scripts/run.py | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/scripts/run.py b/scripts/run.py
index eb0a99ff..abecda3e 100755
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -73,6 +73,15 @@ def set_imgargs(options):
if options.hypervisor == 'qemu_microvm':
execute = '--nopci ' + execute
+ if options.mount_fs:
+ execute = ' '.join('--mount-fs=%s' % m for m in options.mount_fs) + ' ' + execute
+
+ if options.ip:
+ execute = ' '.join('--ip=%s' % i for i in options.ip) + ' ' + execute
+
+ if options.bootchart:
+ execute = '--bootchart ' + execute
+
options.osv_cmdline = execute
if options.kernel or options.hypervisor == 'qemu_microvm' or options.arch == 'aarch64':
return
@@ -576,6 +585,12 @@ if __name__ == "__main__":
help="virtio-fs device tag")
parser.add_argument("--virtio-fs-dir", action="store",
help="path to the directory exposed via virtio-fs mount")
+ parser.add_argument("--mount-fs", default=[], action="append",
+ help="extra mounts (forwarded to respective kernel command line option)")
+ parser.add_argument("--ip", default=[], action="append",
+ help="static ip addresses (forwarded to respective kernel command line option)")
+ parser.add_argument("--bootchart", action="store_true",
+ help="bootchart mode (forwarded to respective kernel command line option")
cmdargs = parser.parse_args()
cmdargs.opt_path = "debug" if cmdargs.debug else "release" if cmdargs.release else "last"
--
2.30.1
Fotis Xenakis
Mar 6, 2021, 4:33:52 PM3/6/21
to osv...@googlegroups.com, Fotis Xenakis
scripts/run.py | 43 ++++++++++++++++++++++++++++++-------------
1 file changed, 30 insertions(+), 13 deletions(-)
diff --git a/scripts/run.py b/scripts/run.py
index abecda3e..71c6cdd7 100755
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -260,17 +260,25 @@ def start_osv_qemu(options):
for a in options.pass_args or []:
args += a.split()
+ virtiofsd = None
if options.virtio_fs_dir:
- try:
- # Normally virtiofsd exits by itself but in future we should probably kill it if it did not
- subprocess.Popen(["virtiofsd", "--socket-path=/tmp/vhostqemu", "-o",
- "source=%s" % options.virtio_fs_dir, "-o", "cache=always"], stdout=devnull, stderr=devnull)
- except OSError as e:
- if e.errno == errno.ENOENT:
- print("virtiofsd binary not found. Please install the qemu-system-x86 package that comes with it (>= 4.2) and is in path.")
- else:
- print("OS error({0}): \"{1}\" while running virtiofsd {2}".
- format(e.errno, e.strerror, " ".join(args)))
+ virtiofsd_cmdline = ["virtiofsd", "--socket-path=/tmp/vhostqemu", "-o",
+ "source=%s" % options.virtio_fs_dir]
+ if options.dry_run:
+ print(format_args(virtiofsd_cmdline))
+ else:
+ try:
+ virtiofsd = subprocess.Popen(virtiofsd_cmdline, stdout=devnull,
+ stderr=devnull)
+ except OSError as e:
+ if e.errno == errno.ENOENT:
+ print("virtiofsd binary not found. Please install the qemu-system-%s package "
+ "that comes with it (>= 4.2) and is in path." % options.arch,
+ file=sys.stderr)
+ else:
+ print("OS error(%d): \"%s\" while running virtiofsd %s" % \
+ (e.errno, e.strerror, " ".join(args)), file=sys.stderr)
+ sys.exit()
try:
# Save the current settings of the stty
@@ -290,12 +298,21 @@ def start_osv_qemu(options):
sys.exit("qemu failed.")
except OSError as e:
if e.errno == errno.ENOENT:
- print("'%s' binary not found. Please install the qemu-system-x86 package." % qemu_path)
+ print("'%s' binary not found. Please install the qemu-system-%s package." % \
+ (qemu_path, options.arch), file=sys.stderr)
else:
- print("OS error({0}): \"{1}\" while running qemu-system-{2} {3}".
- format(e.errno, e.strerror, options.arch, " ".join(args)))
+ print("OS error(%d): \"%s\" while running qemu-system-%s %s" %
+ (e.errno, e.strerror, options.arch, " ".join(args)), file=sys.stderr)
finally:
cleanups()
+ # Clean up the spawned virtiofsd, if any
+ if virtiofsd is not None:
+ if virtiofsd.poll() is None:
+ virtiofsd.terminate()
+ try:
+ virtiofsd.wait(5)
+ except subprocess.TimeoutExpired:
+ virtiofsd.kill()
def start_osv_xen(options):
if options.hypervisor == "xen":
--
2.30.1
1 file changed, 30 insertions(+), 13 deletions(-)
diff --git a/scripts/run.py b/scripts/run.py
index abecda3e..71c6cdd7 100755
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -260,17 +260,25 @@ def start_osv_qemu(options):
for a in options.pass_args or []:
args += a.split()
+ virtiofsd = None
if options.virtio_fs_dir:
- try:
- # Normally virtiofsd exits by itself but in future we should probably kill it if it did not
- subprocess.Popen(["virtiofsd", "--socket-path=/tmp/vhostqemu", "-o",
- "source=%s" % options.virtio_fs_dir, "-o", "cache=always"], stdout=devnull, stderr=devnull)
- except OSError as e:
- if e.errno == errno.ENOENT:
- print("virtiofsd binary not found. Please install the qemu-system-x86 package that comes with it (>= 4.2) and is in path.")
- else:
- print("OS error({0}): \"{1}\" while running virtiofsd {2}".
- format(e.errno, e.strerror, " ".join(args)))
+ virtiofsd_cmdline = ["virtiofsd", "--socket-path=/tmp/vhostqemu", "-o",
+ "source=%s" % options.virtio_fs_dir]
+ if options.dry_run:
+ print(format_args(virtiofsd_cmdline))
+ else:
+ try:
+ virtiofsd = subprocess.Popen(virtiofsd_cmdline, stdout=devnull,
+ stderr=devnull)
+ except OSError as e:
+ if e.errno == errno.ENOENT:
+ print("virtiofsd binary not found. Please install the qemu-system-%s package "
+ "that comes with it (>= 4.2) and is in path." % options.arch,
+ file=sys.stderr)
+ else:
+ print("OS error(%d): \"%s\" while running virtiofsd %s" % \
+ (e.errno, e.strerror, " ".join(args)), file=sys.stderr)
+ sys.exit()
try:
# Save the current settings of the stty
@@ -290,12 +298,21 @@ def start_osv_qemu(options):
sys.exit("qemu failed.")
except OSError as e:
if e.errno == errno.ENOENT:
- print("'%s' binary not found. Please install the qemu-system-x86 package." % qemu_path)
+ print("'%s' binary not found. Please install the qemu-system-%s package." % \
+ (qemu_path, options.arch), file=sys.stderr)
else:
- print("OS error({0}): \"{1}\" while running qemu-system-{2} {3}".
- format(e.errno, e.strerror, options.arch, " ".join(args)))
+ print("OS error(%d): \"%s\" while running qemu-system-%s %s" %
+ (e.errno, e.strerror, options.arch, " ".join(args)), file=sys.stderr)
finally:
cleanups()
+ # Clean up the spawned virtiofsd, if any
+ if virtiofsd is not None:
+ if virtiofsd.poll() is None:
+ virtiofsd.terminate()
+ try:
+ virtiofsd.wait(5)
+ except subprocess.TimeoutExpired:
+ virtiofsd.kill()
def start_osv_xen(options):
if options.hypervisor == "xen":
--
2.30.1
Fotis Xenakis
Mar 6, 2021, 4:34:31 PM3/6/21
to osv...@googlegroups.com, Fotis Xenakis
scripts/run.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/scripts/run.py b/scripts/run.py
index 71c6cdd7..6a81e98b 100755
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -178,9 +178,10 @@ def start_osv_qemu(options):
"-drive", "file=%s,if=none,id=hd1" % (options.cloud_init_image)]
if options.virtio_fs_tag:
+ dax = (",cache-size=%s" % options.virtio_fs_dax) if options.virtio_fs_dax else ""
args += [
"-chardev", "socket,id=char0,path=/tmp/vhostqemu",
- "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s" % options.virtio_fs_tag,
+ "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s%s" % (options.virtio_fs_tag, dax),
"-object", "memory-backend-file,id=mem,size=%s,mem-path=/dev/shm,share=on" % options.memsize,
"-numa", "node,memdev=mem"]
@@ -602,6 +603,8 @@ if __name__ == "__main__":
+ help="DAX window size for virtio-fs device (disabled if not specified)")
2.30.1
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/scripts/run.py b/scripts/run.py
index 71c6cdd7..6a81e98b 100755
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -178,9 +178,10 @@ def start_osv_qemu(options):
"-drive", "file=%s,if=none,id=hd1" % (options.cloud_init_image)]
if options.virtio_fs_tag:
+ dax = (",cache-size=%s" % options.virtio_fs_dax) if options.virtio_fs_dax else ""
args += [
"-chardev", "socket,id=char0,path=/tmp/vhostqemu",
- "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s" % options.virtio_fs_tag,
+ "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s%s" % (options.virtio_fs_tag, dax),
"-object", "memory-backend-file,id=mem,size=%s,mem-path=/dev/shm,share=on" % options.memsize,
"-numa", "node,memdev=mem"]
@@ -602,6 +603,8 @@ if __name__ == "__main__":
help="virtio-fs device tag")
parser.add_argument("--virtio-fs-dir", action="store",
help="path to the directory exposed via virtio-fs mount")
+ parser.add_argument("--virtio-fs-dax", action="store",
parser.add_argument("--virtio-fs-dir", action="store",
help="path to the directory exposed via virtio-fs mount")
+ help="DAX window size for virtio-fs device (disabled if not specified)")
parser.add_argument("--mount-fs", default=[], action="append",
help="extra mounts (forwarded to respective kernel command line option)")
parser.add_argument("--ip", default=[], action="append",
--
2.30.1
Waldek Kozaczuk
Mar 13, 2021, 11:08:29 AM3/13/21
to OSv Development
Hi,
The other 2 patches work fine but this one fails when I test it. What am I doing wrong?
sudo PATH=build/downloaded_packages/qemu/build/tools/virtiofsd:$PATH ./scripts/run.py --virtio-fs-tag=myfs --virtio-fs-dir=$(pwd)/build/export --qemu-path build/downloaded_packages/qemu/build/qemu-system-x86_64 --virtio-fs-dax=1024M
qemu-system-x86_64: -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=1024M: Property 'vhost-user-fs-pci.cache-size' not found
qemu failed.
sudo PATH=build/downloaded_packages/qemu/build/tools/virtiofsd:$PATH ./scripts/run.py --virtio-fs-tag=myfs --virtio-fs-dir=$(pwd)/build/export --qemu-path build/downloaded_packages/qemu/build/qemu-system-x86_64 --virtio-fs-dax=1024M --dry-run
/home/wkozaczuk/projects/osv-master/scripts/../scripts/imgedit.py setargs /home/wkozaczuk/projects/osv-master/build/last/usr.img "--rootfs=virtiofs /cli"
virtiofsd \
--socket-path=/tmp/vhostqemu \
-o source=/home/wkozaczuk/projects/osv-master/build/export
build/downloaded_packages/qemu/build/qemu-system-x86_64 \
-m 2G \
-smp 4 \
-vnc :1 \
-gdb tcp::1234,server,nowait \
-device virtio-blk-pci,id=blk0,drive=hd0,scsi=off,bootindex=0 \
-drive file=/home/wkozaczuk/projects/osv-master/build/last/usr.img,if=none,id=hd0,cache=none,aio=native \
-chardev socket,id=char0,path=/tmp/vhostqemu \
-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=1024M \
-object memory-backend-file,id=mem,size=2G,mem-path=/dev/shm,share=on \
-numa node,memdev=mem \
-netdev user,id=un0,net=192.168.122.0/24,host=192.168.122.1 \
-device virtio-net-pci,netdev=un0 \
-device virtio-rng-pci \
-enable-kvm \
-cpu host,+x2apic \
-chardev stdio,mux=on,id=stdio,signal=off \
-mon chardev=stdio,mode=readline \
-device isa-serial,chardev=stdio
I am using QEMU 5.2 built from sources.
Waldek
Commit Bot
Mar 13, 2021, 11:23:23 AM3/13/21
to osv...@googlegroups.com, Fotis Xenakis
From: Fotis Xenakis <fo...@windowslive.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
scripts: add kernel command-line options to run.py
---
diff --git a/scripts/run.py b/scripts/run.py
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
scripts: add kernel command-line options to run.py
Thia adds the --mount-fs, --ip and --bootchart options.
Signed-off-by: Fotis Xenakis <fo...@windowslive.com>
Message-Id: <VI1PR03MB3773A99749...@VI1PR03MB3773.eurprd03.prod.outlook.com>
Signed-off-by: Fotis Xenakis <fo...@windowslive.com>
---
diff --git a/scripts/run.py b/scripts/run.py
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -73,6 +73,15 @@ def set_imgargs(options):
if options.hypervisor == 'qemu_microvm':
execute = '--nopci ' + execute
+ if options.mount_fs:
+ execute = ' '.join('--mount-fs=%s' % m for m in options.mount_fs) + ' ' + execute
+
+ if options.ip:
+ execute = ' '.join('--ip=%s' % i for i in options.ip) + ' ' + execute
+
+ if options.bootchart:
+ execute = '--bootchart ' + execute
+
options.osv_cmdline = execute
if options.kernel or options.hypervisor == 'qemu_microvm' or options.arch == 'aarch64':
return
@@ -576,6 +585,12 @@ def main(options):
+++ b/scripts/run.py
@@ -73,6 +73,15 @@ def set_imgargs(options):
if options.hypervisor == 'qemu_microvm':
execute = '--nopci ' + execute
+ if options.mount_fs:
+ execute = ' '.join('--mount-fs=%s' % m for m in options.mount_fs) + ' ' + execute
+
+ if options.ip:
+ execute = ' '.join('--ip=%s' % i for i in options.ip) + ' ' + execute
+
+ if options.bootchart:
+ execute = '--bootchart ' + execute
+
options.osv_cmdline = execute
if options.kernel or options.hypervisor == 'qemu_microvm' or options.arch == 'aarch64':
return
Commit Bot
Mar 13, 2021, 11:23:24 AM3/13/21
to osv...@googlegroups.com, Fotis Xenakis
From: Fotis Xenakis <fo...@windowslive.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
scripts: enhance virtiofsd functionality in run.py
Signed-off-by: Fotis Xenakis <fo...@windowslive.com>
Message-Id: <VI1PR03MB3773DE8B4F...@VI1PR03MB3773.eurprd03.prod.outlook.com>
---
diff --git a/scripts/run.py b/scripts/run.py
Waldek Kozaczuk
Mar 13, 2021, 11:29:28 AM3/13/21
to OSv Development
Hey,
I have applied 3 of your patches.
As I was doing and testing it I realized we do not have any Wiki page that describes how to use virtio-fs with OSv. Would you have a bit of time to add a page that does so - a simple step-by-step cookbook - how to build with export and fs options, and then run with various new options (DAX, no dax, etc), need to use sudo, which version of QEMU, etc - https://github.com/cloudius-systems/osv/wiki? Maybe also link to your QEMU article?
Right now we have some info scattered acros some commit messages but nothing in one place.
I would really appreciate it,
Waldek
On Saturday, March 6, 2021 at 4:34:31 PM UTC-5 Fotis Xenakis wrote:
Waldek Kozaczuk
Mar 13, 2021, 11:53:28 AM3/13/21
to OSv Development
BTW is DAX enabled by default or you have to explicitly enable it?
Fotis Xenakis
Mar 13, 2021, 1:58:57 PM3/13/21
to OSv Development
Unfortunately DAX support has yet to be merged to upstream QEMU (I
was under the impression that it had been staged for QEMU 5.2.0, but
alas it is not part of it).
So, to use it, one still needs to use the virtio-fs downstream version which includes the necessary patches [1].
The virtio-fs team are actively pursuing its upstreaming, but I 'm afraid I don't have a clear view as to how long that might still take.
Regarding the patch, do you think it would be better to hold it back until there's proper support upstream?
Fotis Xenakis
Mar 13, 2021, 2:04:03 PM3/13/21
to OSv Development
You are absolutely right, some proper guide and documentation is necessary, thank you for the suggestion!
I shall take care of it upon first chance.
Fotis Xenakis
Mar 13, 2021, 2:07:05 PM3/13/21
to OSv Development
On Saturday, March 13, 2021 at 6:53:28 PM UTC+2 jwkoz...@gmail.com wrote:
BTW is DAX enabled by default or you have to explicitly enable it?
If the --virtio-fs-dax option to run.py is not specified, the DAX window is disabled, so you have to explicitly enable it (trying to be backwards compatible and not intrusive).
Fotis Xenakis
Mar 26, 2021, 1:44:30 PM3/26/21
to OSv Development
Update: the wiki page is here: https://github.com/cloudius-systems/osv/wiki/virtio-fs
All questions/feedback and of course contributions welcome!
Waldek Kozaczuk
Mar 30, 2021, 2:23:24 PM3/30/21
to OSv Development
Fotis,
Thanks a lot!
I have updated this page to include the information about ARM - virtio-fs works on this arch as well which is great!
The virtio-fs support is probably the single most exciting new feature added to OSv in the last couple of years! I am not aware of any other unikernel supporting it.
Thanks a lot,
Waldek
PS. It would be nice if some other hypervisors (thinking of firecracker) supported virtio-fs as well.
Fotis Xenakis
Mar 31, 2021, 10:54:57 AM3/31/21
to OSv Development
On Tuesday, March 30, 2021 at 9:23:24 PM UTC+3 jwkoz...@gmail.com wrote:
Fotis,Thanks a lot!I have updated this page to include the information about ARM - virtio-fs works on this arch as well which is great!
It is awesome to hear that! What would be mildly surprising is if DAX also worked on ARM (on the host/QEMU side as well, given it's not yet fully stable).
The virtio-fs support is probably the single most exciting new feature added to OSv in the last couple of years! I am not aware of any other unikernel supporting it.
It sure has a great value to dev effort ratio! The only other unikernel supporting it is toro AFAIK: https://github.com/torokernel/torokernel
Thanks a lot,WaldekPS. It would be nice if some other hypervisors (thinking of firecracker) supported virtio-fs as well.
Indeed, it seems a great fit for the lightweight VMMs. Firecracker was close to adding it, but eventually did not: https://github.com/firecracker-microvm/firecracker/pull/1351
Yet, cloud-hypervisor does support it, notably with the contributions of upstream virtio-fs devs: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/v0.14.0/docs/fs.md
Fotis Xenakis
Mar 31, 2021, 10:56:27 AM3/31/21
to OSv Development
Bumping on this: I guess we should wait for QEMU here?
Waldek Kozaczuk
Mar 31, 2021, 11:14:32 AM3/31/21
to OSv Development
Sorry. I missed it. Let me build that specific version of QEMU with DAX support and test. I will also try it on ARM to see if it works.
Matias Vara
Mar 31, 2021, 2:39:29 PM3/31/21
to Fotis Xenakis, OSv Development
Hello,
I think it may be interesting to have a look at libkrun (https://github.com/containers/libkrun). As far I know, it supports virtio-fs.
Matias
--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/538fe958-fb89-4286-8cb5-a0dc4664ae72n%40googlegroups.com.
Commit Bot
Apr 1, 2021, 10:38:58 AM4/1/21
to osv...@googlegroups.com, Fotis Xenakis
From: Fotis Xenakis <fo...@windowslive.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
scripts: add virtio-fs DAX option to run.py
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
Signed-off-by: Fotis Xenakis <fo...@windowslive.com>
Message-Id: <VI1PR03MB3773F8FBFC...@VI1PR03MB3773.eurprd03.prod.outlook.com>
---
diff --git a/scripts/run.py b/scripts/run.py
--- a/scripts/run.py
+++ b/scripts/run.py
@@ -178,9 +178,10 @@ def start_osv_qemu(options):
"-drive", "file=%s,if=none,id=hd1" % (options.cloud_init_image)]
if options.virtio_fs_tag:
+ dax = (",cache-size=%s" % options.virtio_fs_dax) if options.virtio_fs_dax else ""
args += [
"-chardev", "socket,id=char0,path=/tmp/vhostqemu",
- "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s" % options.virtio_fs_tag,
+ "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s%s" % (options.virtio_fs_tag, dax),
"-object", "memory-backend-file,id=mem,size=%s,mem-path=/dev/shm,share=on" % options.memsize,
"-numa", "node,memdev=mem"]
@@ -602,6 +603,8 @@ def main(options):
+++ b/scripts/run.py
@@ -178,9 +178,10 @@ def start_osv_qemu(options):
"-drive", "file=%s,if=none,id=hd1" % (options.cloud_init_image)]
if options.virtio_fs_tag:
+ dax = (",cache-size=%s" % options.virtio_fs_dax) if options.virtio_fs_dax else ""
args += [
"-chardev", "socket,id=char0,path=/tmp/vhostqemu",
- "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s" % options.virtio_fs_tag,
+ "-device", "vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=%s%s" % (options.virtio_fs_tag, dax),
"-object", "memory-backend-file,id=mem,size=%s,mem-path=/dev/shm,share=on" % options.memsize,
"-numa", "node,memdev=mem"]
Waldek Kozaczuk
Apr 1, 2021, 10:49:14 AM4/1/21
to OSv Development
The intel version of DAX works fine. The ARM not, I get this using this qemu (which seems like a year old) - https://gitlab.com/virtio-fs/qemu/-/tree/virtio-fs-dev:
sudo PATH=/home/wkozaczuk/projects/osv/build/downloaded_packages/qemu/build/tools/virtiofsd:/home/wkozaczuk/projects/osv/build/downloaded_packages/qemu/build:$PATH ./scripts/run.py --virtio-fs-tag=myfs --virtio-fs-dir=$(pwd)/build/export --virtio-fs-dax=1024M --api -V --arch=aarch64 -c 1
OSv v0.55.0-239-g47601e6e
getauxval() stubbed
PSCI: version 0.2 detected.
1 CPUs detected
getauxval() stubbed
Firmware vendor: Unknown
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
eth0: ethernet address: 52:54:00:12:34:56
virtio-blk: Add blk device instances 0 as vblk0, devsize=18869248
random: virtio-rng registered as a source.
virtio-fs: Detected device with tag: [] and num_queues: 0
virtio-fs: Detected DAX window with length 1073741824
random: <Software, Yarrow> initialized
VFS: unmounting /dev
VFS: mounting virtiofs at /virtiofs
[virtiofs] Error opening device!
failed to mount /virtiofs, error = No such device or address
Could not mount virtiofs root filesystem.
Even the regular - non-DAX - version does not work:
sudo PATH=/home/wkozaczuk/projects/osv/build/downloaded_packages/qemu/build/tools/virtiofsd:/home/wkozaczuk/projects/osv/build/downloaded_packages/qemu/build:$PATH ./scripts/run.py --virtio-fs-tag=myfs --virtio-fs-dir=$(pwd)/build/export --api -V --arch=aarch64 -c 1
OSv v0.55.0-239-g47601e6e
getauxval() stubbed
PSCI: version 0.2 detected.
1 CPUs detected
getauxval() stubbed
Firmware vendor: Unknown
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
Version 1 not supported!
eth0: ethernet address: 52:54:00:12:34:56
virtio-blk: Add blk device instances 0 as vblk0, devsize=18869248
random: virtio-rng registered as a source.
qemu-system-aarch64: wrong value for queue_enable 0
qemu-system-aarch64: wrong value for queue_enable 0
virtio-fs: Detected device with tag: [] and num_queues: -1
Assertion failed: sched::preemptable() (arch/aarch64/mmu.cc: page_fault: 46)
[backtrace]
0x00000000400e8c10 <__assert_fail+32>
0x000000004020bb10 <page_fault+220>
0x000000004020b828 <???+1075886120>
0x0000000040208c64 <virtio::fs::req_done()+100>
0x0000000040208e5c <???+1075875420>
0x00000000402e72b0 <thread_main_c+32>
0x000000004020b784 <thread_main+0>
So it looks the version of QEMU from https://gitlab.com/virtio-fs/qemu/-/tree/virtio-fs-dev is somewhat outdated as far as ARM support goes. The non-DAX virtiofs work on ARM using the regular released version of qemu - 5.1.0, as I stated in other emails.
BTW you can build qemu from sources using a script we have in place:
./scripts/download_and_build_qemu.sh virtio-fs-dev https://gitlab.com/virtio-fs/qemu.git
Waldek Kozaczuk
Apr 1, 2021, 11:09:02 AM4/1/21
to OSv Development
Updated the wiki as well. Feel free to look at it to verify if anything is off.
I noticed that using '--qemu-path' to specify the location of qemu executable is not enough for virtiofsd as it is often located in another directory. So one may be better of explicitly adding both to the PATH or maybe we can do something in run.py to automate it.
Fotis Xenakis
Apr 2, 2021, 3:14:02 PM4/2/21
to OSv Development
Thank you for taking the time to try it out Waldek!
It would be somewhat surprising if DAX worked on ARM, at least given the devs are not focused on it.
Not really sure what the current difference to stable QEMU is, apart from DAX, which would also justify the non-DAX version not working.
BTW, this is not based on such an old QEMU version: the latest non-virtio-fs commits are from Jan 6, 2021, so after QEMU 5.2.0.
Also, thanks for the heads up for the new script, I had totally missed that up to now.
Finally, thank you for the prompt update to the wiki page, it seems spot on to me!
There was I think a mention of having to add virtiofsd to the PATH, but it's good to stress it out as much as possible: it's probably the easiest point to miss.
Fotis Xenakis
Apr 2, 2021, 3:30:00 PM4/2/21
to OSv Development
Hello Matias,
Thank you for chiming in,
libkrun indeed seems very promising and, very importantly, pretty valuable from an end-user perspective if it becomes practical!
To my eyes, virtio-fs makes most sense in such setups: it's kind of the equivalent of bind mounts in Docker terminology.
On a side note, it's started by some very prominent people in the lightweight virtualization community.
P.S. It's great to have such, more general, discussions or even sharing resources and perspectives. I wouldn't want to hijack this thread,
but feel free to start a fresh one if you 'd like to continue this discussion.
Reply all
Reply to author
Forward
0 new messages