[Discuss] LVM vs File system file for KVM Virtual Machines?
ma...@mohawksoft.com
vs btrfs or zfs, does anyone want to discuss the pros and cons of LVM
"device" for a virtual machine vs a file on a file system for a virtual
machine?
So, do you create a 30G file on a file system, like EXT3, jfs, or xfs and
use that or do you create a 30G LVM device and use it directly? There are
some benefits to using LVM and with the 3.x kernel, you could even use a
"thin provisioned" device.
Which do you suspect would be more resource efficient? Which do you think
would have faster I/O?
I've set up two systems, one on a jfs file system and one on an "old
style" LVM partition. (Fully allocated). I don't see much of a difference.
I suspect the LVM based system should be more efficient because it does
not have to go through the intermediate file system layer to get to the
device layer. Internally, the VM sees the LVM device as its own device.
_______________________________________________
Discuss mailing list
Dis...@blu.org
http://lists.blu.org/mailman/listinfo/discuss
Derek Atkins
Think about it this way: when using a file on the file system the VM has
a whole extra layer of indirection that it has to go through, because it
has to go through the FS layer in the VM, then the block layer in the VM,
then the VM system storage layer, and then the FS layer in the host, then
block layer in the host.. Whereas if it's just an LVM container then you
can bypass the FS layer in the host completely. So I would always expect
the direct LVM container to be faster and more resource efficient.
-derek
--
Derek Atkins 617-623-3745
de...@ihtfp.com www.ihtfp.com
Computer and Internet Security Consultant
ma...@mohawksoft.com
>
> Think about it this way: when using a file on the file system the VM has
> a whole extra layer of indirection that it has to go through, because it
> has to go through the FS layer in the VM, then the block layer in the VM,
> then the VM system storage layer, and then the FS layer in the host, then
> block layer in the host.. Whereas if it's just an LVM container then you
> can bypass the FS layer in the host completely. So I would always expect
> the direct LVM container to be faster and more resource efficient.
I think that was my assumption to begin with. Great minds think alike!
That being said, there is caching in the file system layer that we
wouldn't get with LVM, but that may hinder more than help. There may be a
little more I/O involved with a file because not only would the VM be
managing the file system, the host would have to manage the meta-data for
the file. Not too much, I don't think, because the file based vm would be
more or less fixed in size.
Richard Pieri
really the least concern.
- I can make snapshots of running systems with LVM2. Can't do it at all
with image files.
- I can easily resize LVM-backed file systems. Image files typically
require conversion and writing new image files.
- The DRBD driver works seamlessly with LVM2-backed domains. It doesn't
work at all with image files because image files aren't block devices.
- I can directly mount LVM2-backed file systems on the host assuming the
host knows how to read/write those file systems. Image files require
mucking around with loopback devices.
There are other conveniences but these are the ones that I typically
rely on with my Xen domains.
--
Rich P.
Edward Ned Harvey
> bounces+blu=nedhar...@blu.org] On Behalf Of Derek Atkins
>
> Think about it this way: when using a file on the file system the VM has
> a whole extra layer of indirection that it has to go through, because it
> has to go through the FS layer in the VM, then the block layer in the VM,
> then the VM system storage layer, and then the FS layer in the host, then
> block layer in the host.. Whereas if it's just an LVM container then you
> can bypass the FS layer in the host completely. So I would always expect
> the direct LVM container to be faster and more resource efficient.
Agreed, but how significant? Not at all. You can easily see this - if you
create a file, and start doing benchmarks against it, using something like
iozone, bonnie, etc. Then you actually measure performance limited only by
the disk hardware in the system. All the extra filesystem overhead amounts
to something negligible.
Your main advantage, I think, is in the way you
create/delete/expand/shrink/interact with the device. When you use LVM, all
that stuff is clear and simple and difficult to destroy with a simple
type-o. And with LVM, all that stuff is instant, or near-instant. But if
you are using a 30G file for the backend storage... You've got to wait for
the system to create a 30G file. Unless you create it sparsely, in which
case you're still going to wait for it - but the waiting will be spread out
and aggregate during runtime.
I see very little, if any, argument in favor of using the file. I suppose,
if you wanted to run your virtual head on some other machine, then you could
export your file via NFS instead of iscsi, and that gives you one extra
degree of freedom if you're using a file. Also, if you got something like a
snapshotting storage server, you could migrate your file over to that
server, and benefit by snapshots.
All in all, I don't see any huge compelling reasons for either one. Both
ways are fine, with small advantages in favor of each one, for specific
circumstances.
Edward Ned Harvey
> bounces+blu=nedhar...@blu.org] On Behalf Of
>
> wouldn't get with LVM,
Think about it like this:
On reads: The guest OS will read some file. If it's a partial file, or
it's fragmented, then the guest OS is smart enough to know how much to
readahead, and which sectors. Unfortunately, the host OS doesn't know any
of that, and the host OS might be doing readahead for useless sectors. But
the host OS is only going to do readahead when/if the disks were otherwise
going to be idle. So there's no benefit and no harm, regardless of whether
you have LVM or file backing.
On async writes: The guest OS will be buffering these anyway. The guest OS
writes them to "disk." The host OS now buffers them. They're queued up and
going to disk, in the background, everything is smooth. No benefit and no
harm either way, whether you have LVM or file backing.
On sync writes: The guest OS writes something to "disk" and issues the
flush command to the "device." The host OS then interprets this, and issues
the actual flush command to the actual device. It returns exactly the same
as the guest would have thought. No benefit and no harm either way, whether
you have LVM or file backing.
I say it's neutral either way.
Mark Woodward
>
> I see very little, if any, argument in favor of using the file. I suppose,
> if you wanted to run your virtual head on some other machine, then you could
> export your file via NFS instead of iscsi, and that gives you one extra
> degree of freedom if you're using a file. Also, if you got something like a
> snapshotting storage server, you could migrate your file over to that
> server, and benefit by snapshots.
is that you have a "file" that can be copied and backed up easily. It is
relocatable in that it cam be moved from one file system to another and
it can bring its size with it. The LVM device is more or less dependent
on the LVM create command and the admin ensuring that a properly sized
device is created if you want to move it to a different machine. I guess
you could us dd to copy the device (or a snapshot) to a file to move it
around, but you see the issue.
There are definitely a list of pros and cons. It's interesting to see
what other people think. So far, I think the "pros" go to LVM.
Jay Kramer
Jay
> ______________________________**_________________
> Discuss mailing list
> Dis...@blu.org
> http://lists.blu.org/mailman/**listinfo/discuss<http://lists.blu.org/mailman/listinfo/discuss>