Re: Converting qcow2 image on the fly to raw format

[Adam Litke]

Adding some kubevirt developers to the thread.  Thanks guys for the information!  I think this could work perfectly for on the fly conversion of qcow2 images to raw format on our PVCs.

On Mon, Jul 9, 2018 at 9:47 PM Nir Soffer <nso...@redhat.com> wrote:

On Mon, Jul 9, 2018 at 8:06 PM Richard W.M. Jones <rjo...@redhat.com> wrote:

On Mon, Jul 09, 2018 at 07:02:50PM +0200, Kevin Wolf wrote:
> Am 09.07.2018 um 18:52 hat Richard W.M. Jones geschrieben:
> > On Mon, Jul 09, 2018 at 07:38:05PM +0300, Nir Soffer wrote:
> > > We are discussing importing VM images to KubVirt. The goal is to be
> > > able to import an existing qcow2 disk, probably some appliance stored
> > > on http server, and and convert it to raw format for writing to storage.
> > >
> > > This can be also useful for for oVirt for importing OVA, since we like to
> > > pack
> > > disks in qcow2 format inside OVA, but the user may like to use raw disks, or
> > > for uploading existing disks.
> > >
> > > Of course converting the image using qemu-img is easy, but requires
> > > downloading the image to temporary disk. We would like to avoid temporary
> > > disks, or telling users to convert the image.
> > >
> > > Base on the discussion we had here:
> > > https://lists.ovirt.org/archives/list/us...@ovirt.org/thread/GNAVJ253FP65QUSOONES5XZGRIDX5ABC/#YMLSEGU7PN3MX5MUORGEGGAQLLSL4KKJ
> > >
> > > I think this is impossible since qcow2 is not built for streaming. But both
> > > Richard and Eric suggested some solutions.
> > >
> > > The flow is:
> > >
> > >     qcow2 image -- http --> importer -> raw file
> > >
> > > Is it possible to implement the importer using qemu-img and qemu-nbd,
> > > or maybe nbdkit?
> >
> > Strictly speaking streaming qcow2 to raw is not possible.  However
> > placing an overlay on top of the original remote image will allow
> > streaming to raw with only a modest amount of local storage consumed.
> >
> > You can demonstrate this fairly easily:
> >
> > $ qemu-img create -f qcow2 -b 'json: { "file.driver": "https", "file.url": "https://uk-mirrors.evowise.com/fedora/releases/28/Cloud/x86_64/images/Fedora-Cloud-Base-28-1.1.x86_64.qcow2", "file.timeout": 10000 }' /var/tmp/overlay.qcow2
> > $ qemu-img convert -p -f qcow2 -O raw overlay.qcow2 fedora.img
>
> This overlay stays empty, so it's pretty pointless and you could just
> directly point 'qemu-img convert' to https and the real image.

Right, indeed.  I was copying what virt-v2v does without thinking
about it enough.  virt-v2v needs the overlay because it actually wants
to write into it, and it does copy-on-read for the first phase (not
the final ‘qemu-img convert’).

Thanks, I just tested the simple:

    qemu-img convert -p -f qcow2 -O raw http://localhost/orig.qcow2 converted.raw

And it just works :-)

I got timeouts trying to download from

https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-28-20180625.1/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180625.1.x86_64.qcow2

I guess we need to use 

'json: { "file.driver": "http", "file.url": "url...", "file.timeout": 10000 }'

To change timeout? Where is these and other options documented?

I did also some timings, using sever on local network with 1g nic.

$ time wget http://local.server/Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 

...

Length: 638043136 (608M) [application/octet-stream]Saving to: ‘Fedora-AtomicHost-28-20180625.1.x86_64.qcow2’

Fedora-AtomicHost-28-20180625.1.x86_64.qcow2    100%[=====================================================================================================>] 608.49M   107MB/s    in 5.6s    

2018-07-09 21:38:39 (108 MB/s) - ‘Fedora-AtomicHost-28-20180625.1.x86_64.qcow2’ saved [638043136/638043136]

real 0m5.941s

user 0m0.183s

sys 0m1.185s

$ time qemu-img convert -p -f qcow2 -O raw http://local.server/Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 converted.raw

    (100.00/100%)

real 0m14.217s

user 0m5.235s

sys 0m2.343s

$ time qemu-img convert -p -f qcow2 -O raw Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 converted.raw

    (100.00/100%)

real 0m11.909s

user 0m4.728s

sys 0m1.595s

So converting on the fly is even little faster then downloading to temporary file and converting.

Nir

--

Adam Litke

[Marcus Sorensen]

This is also the goal of the CDI project, right?

https://github.com/kubevirt/containerized-data-importer/blob/master/README.md 

[Daniel P. Berrangé]

On Wed, Jul 11, 2018 at 02:17:18PM +0300, Adam Litke wrote:
> Adding some kubevirt developers to the thread. Thanks guys for the
> information! I think this could work perfectly for on the fly conversion
> of qcow2 images to raw format on our PVCs.

FYI if you are intending to accept qcow2 images from untrustworthy sources
you must take special care to validate the image in a confined environment.
It is possible to construct malicious images that inflict a denial of
service attack on CPU or memory or both, even when merely opening the image
to query its metadata. This has been reported as a CVE against OpenStack
in the past:

https://bugs.launchpad.net/ossa/+bug/1449062

Recommendation is to run 'qemu-img info' to extract the metadata and sanity
check results eg no backing file list, not unreasonable size, etc. When
running 'qemu-img info' apply process limits of 30 secs CPU time, and 1 GB
address space.

Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

[Roman Mohr]

On Monday, July 16, 2018 at 10:56:38 AM UTC+2, Daniel P. Berrangé wrote:

On Wed, Jul 11, 2018 at 02:17:18PM +0300, Adam Litke wrote:
> Adding some kubevirt developers to the thread.  Thanks guys for the
> information!  I think this could work perfectly for on the fly conversion
> of qcow2 images to raw format on our PVCs.

FYI if you are intending to accept qcow2 images from untrustworthy sources
you must take special care to validate the image in a confined environment.
It is possible to construct malicious images that inflict a denial of
service attack on CPU or memory or both, even when merely opening the image
to query its metadata. This has been reported as a CVE against OpenStack
in the past:

  https://bugs.launchpad.net/ossa/+bug/1449062

Recommendation is to run 'qemu-img info' to extract the metadata and sanity
check results eg no backing file list, not unreasonable size, etc. When
running 'qemu-img info' apply process limits of 30 secs CPU time, and 1 GB
address space.

We are affected. Created https://github.com/kubevirt/kubevirt/issues/1318.

Thanks for the hint,

Roman

[Richard W.M. Jones]

On Thu, Jul 19, 2018 at 04:25:28PM +0300, Nir Soffer wrote:

> On Mon, Jul 16, 2018 at 11:56 AM Daniel P. Berrangé <berr...@redhat.com>
> wrote:
>
> > On Wed, Jul 11, 2018 at 02:17:18PM +0300, Adam Litke wrote:
> > > Adding some kubevirt developers to the thread. Thanks guys for the
> > > information! I think this could work perfectly for on the fly conversion
> > > of qcow2 images to raw format on our PVCs.
> >
> > FYI if you are intending to accept qcow2 images from untrustworthy sources
> > you must take special care to validate the image in a confined environment.
> > It is possible to construct malicious images that inflict a denial of
> > service attack on CPU or memory or both, even when merely opening the image
> > to query its metadata. This has been reported as a CVE against OpenStack
> > in the past:
> >
> > https://bugs.launchpad.net/ossa/+bug/1449062
> >
> > Recommendation is to run 'qemu-img info' to extract the metadata and sanity
> > check results eg no backing file list, not unreasonable size, etc. When
> > running 'qemu-img info' apply process limits of 30 secs CPU time, and 1 GB
> > address space.
> >
>

> Thanks for the suggestion.
>
> We currently do not limit the qemu-img process in any way, but it sounds
> like
> a good idea.
>
> We also don't verify the size of the image, this should be fixed.
>
> What we do currently is:
> 1. Mark image as illegal in oVirt metadata - prevents using the image by
> oVirt.
> 2. Expose the image via http
> 3. Wait until the user completes the upload
> 4. Unexposed the image, so no more data can be written.
> 5. Run qemu-img info /path/to/image (running as vdsm, but without any limit)
> 6. Verify format with oVirt metada - it must be same as specified in oVirt
> 7. Verify backing file with ovirt metadata - it must be same as specified
> in oVirt
> (no backing file or volume UUID)
> 8. Verify that qcow2 compat is compatible with the storage domain
> 9. If all checks are ok, mark the image as legal.
>
> The image is deleted on verification failure.
>
> This is the code if someone like to check:
> -
> https://github.com/oVirt/vdsm/blob/857df825f8c1e9030f7c1e46e6c59cb63546d7c9/lib/vdsm/storage/hsm.py#L1535
> -
> https://github.com/oVirt/vdsm/blob/857df825f8c1e9030f7c1e46e6c59cb63546d7c9/tests/storage/hsm_test.py#L49

Apart from putting some ulimits around the qemu-img info command I
think this should be fine.

A few weeks ago I spent some time looking closely at the qemu qcow2
code and trying to write a custom malicious qcow2 file to break it,
but I didn't get anywhere.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Richard W.M. Jones]

On Thu, Jul 19, 2018 at 09:50:00PM +0300, Nir Soffer wrote:
> On Mon, Jul 16, 2018 at 11:56 AM Daniel P. Berrangé <berr...@redhat.com>
> wrote:

> ...

>
> > Recommendation is to run 'qemu-img info' to extract the metadata and sanity
> > check results eg no backing file list, not unreasonable size, etc. When
> > running 'qemu-img info' apply process limits of 30 secs CPU time, and 1 GB
> > address space.
> >
>

> Can you explain the values of cpu seconds and addres space?
>
> Also, I tried to use prlimit --rss=N, and the value seems to be ignored. Is
> this
> a bug in prlimit?

This is the code in OpenStack:

https://github.com/openstack/nova/blob/50f40854b04351fb622fd8b68b374a8fe8ca2070/nova/virt/images.py#L74

https://github.com/openstack/oslo.concurrency/blob/a9d728b71e47540fd248a6bc2d301fdfa9a988ce/oslo_concurrency/processutils.py#L194
(see prlimit)

HTH,

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Richard W.M. Jones]

This is the code in libguestfs if you prefer something in C:

https://github.com/libguestfs/libguestfs/blob/3a87c4bb441430c9cef9223e67d10bf51a4e865c/lib/info.c#L150-L160

https://github.com/libguestfs/libguestfs/blob/3a87c4bb441430c9cef9223e67d10bf51a4e865c/lib/command.c#L585-L600

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org

[Richard W.M. Jones]

I did the original work using AFL to fuzz qemu-img and find
problematic images. From that work Dan & I suggested some fairly low
limits (10 seconds IIRC). See:

https://bugs.launchpad.net/qemu/+bug/1462944
https://bugs.launchpad.net/qemu/+bug/1462949

A lot more problematic images were found (at least 16), but I cannot
recall if we filed bugs for all of them. Note the images do not need
to be qcow2, since someone can upload any old thing to your service
and cause you problems.

On Thu, Jul 19, 2018 at 11:00:14PM +0300, Nir Soffer wrote:
> The 30 seconds cpu_time time limit confuses me; it was added in:
> https://github.com/openstack/nova/commit/011ae614d5c5fb35b2e9c22a9c4c99158f6aee20
>
> The patch references this bug:
> https://bugs.launchpad.net/nova/+bug/1705340

It looks as if those original limits were too low and they have been
increased. For RHV I think you should go with the same settings that
OpenStack is using.

> The bug show that qemu-img info took more then 8 seconds real time:
>
> /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:355
> 2017-07-19 19:47:42.849 7 DEBUG oslo_concurrency.processutils
> [req-7ed3314d-1c11-4dd8-b612-f8d9c022417f
> ff236d57a57dd42cb5811c998e30fca1a76233873b9f08330f725fb639c8b025
> 9776d48734a24c23a4aef51cb78cc269 - - -] CMD "/usr/bin/python2 -m
> oslo_concurrency.prlimit --as=1073741824 --cpu=16 -- env LC_ALL=C
> LANG=C qemu-img info
> /var/lib/nova/instances/_base/41ebff725eab55d368f97bc79ddeea47df894145.part"
> returned: 0 in 8.639s execute
>
> How the size of the snapshot can slow down reading the qcow2 header?

Since the original bugs were found using fuzzing I didn't spend much
time working out exactly why the code was slow in certain situations.

> Is this relevant for qemu-img-rhev 2.10?

Yes, for all versions of qemu-img.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com

virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Daniel P. Berrangé]

On Thu, Jul 19, 2018 at 09:50:00PM +0300, Nir Soffer wrote:
> On Mon, Jul 16, 2018 at 11:56 AM Daniel P. Berrangé <berr...@redhat.com>
> wrote:
> ...
>

> > Recommendation is to run 'qemu-img info' to extract the metadata and sanity
> > check results eg no backing file list, not unreasonable size, etc. When
> > running 'qemu-img info' apply process limits of 30 secs CPU time, and 1 GB
> > address space.
> >
>

> Can you explain the values of cpu seconds and addres space?

Initially the values were (informed) guesswork. After testing real world
examples we settled on these values as providing protection from DOS,
while not causing valid images to be rejected

[Daniel P. Berrangé]

On Thu, Jul 19, 2018 at 09:39:35PM +0100, Richard W.M. Jones wrote:
> I did the original work using AFL to fuzz qemu-img and find
> problematic images. From that work Dan & I suggested some fairly low
> limits (10 seconds IIRC). See:
>
> https://bugs.launchpad.net/qemu/+bug/1462944
> https://bugs.launchpad.net/qemu/+bug/1462949
>
> A lot more problematic images were found (at least 16), but I cannot
> recall if we filed bugs for all of them. Note the images do not need
> to be qcow2, since someone can upload any old thing to your service
> and cause you problems.
>
> On Thu, Jul 19, 2018 at 11:00:14PM +0300, Nir Soffer wrote:
> > The 30 seconds cpu_time time limit confuses me; it was added in:
> > https://github.com/openstack/nova/commit/011ae614d5c5fb35b2e9c22a9c4c99158f6aee20
> >
> > The patch references this bug:
> > https://bugs.launchpad.net/nova/+bug/1705340
>
> It looks as if those original limits were too low and they have been
> increased. For RHV I think you should go with the same settings that
> OpenStack is using.

Yes, real world usage found our original limit was too low for certain
valid images, so we increased it to 30 seconds.