My first type/provider to manage LVM volumes - Hoping for suggestions
Daniel Kerwin
i wrote a simple type and provider to manage logical volumes with
puppet. This is my first attempt and it may not match the puppet
styles and best practices. I'm hoping for your improvement
suggestions.
Type:
=====
Puppet::Type.newtype(:logicalvolume) do
@doc = "Manage logical volumes"
ensurable
newparam(:lvname) do
desc "The logcal volumes name"
validate do |value|
unless value =~ /^[a-z0-9]+/
raise ArgumentError , "%s is not a valid lv name" %
value
end
end
isnamevar
end
newproperty(:size) do
desc "The size in M or G"
validate do |value|
unless value =~ /^[0-9]+[MGTPE]/
raise ArgumentError , "%s is not a valid lv size" %
value
end
end
end
newparam(:vg) do
desc "The volumevg to create the volume in"
validate do |value|
unless value =~ /^[a-z0-9]+/
raise ArgumentError , "%s is not a valid lv name" %
value
end
end
end
end
Provider:
========
Puppet::Type.type(:logicalvolume).provide(:logicalvolume) do
include Puppet::Util
def size
lvm_size_units = { "M" => 1024, "G" => 1048576, "T" =>
1073741824, "P" => 1099511627776, "E" => 1125899906842624 }
lvm_size_units_match = lvm_size_units.keys().join('|')
output = Puppet::Util.execute(["lvs", "--noheading", "/dev/
#{resource[:vg]}/#{resource[:name]}", "-o", "size"])
if output =~ /^\s+(\d+)\.\d+
(#{lvm_size_units_match.downcase})/
size = $1 + $2.capitalize
debug("Size of /dev/#{resource[:vg]}/#{resource[:name]} is
#{size}")
return size
else
fail("Failed to parse current volume size!")
end
end
def size=(size)
lvm_size_units = { "M" => 1024, "G" => 1048576, "T" =>
1073741824, "P" => 1099511627776, "E" => 1125899906842624 }
lvm_size_units_match = lvm_size_units.keys().join('|')
resizeable = false
c_size = self.size()
if size =~ /(\d+)(#{lvm_size_units_match})/
n_size_bytes = $1.to_i
n_size_unit = $2
end
if c_size =~ /(\d+)(#{lvm_size_units_match})/
c_size_bytes = $1.to_i
c_size_unit = $2
end
## Check if given requested size matches extend size of VG
output = Puppet::Util.execute(["vgdisplay", "-c",
resource[:vg]])
vg_extend = output.split(':')[12].to_i
if n_size_bytes * lvm_size_units[n_size_unit] % vg_extend != 0
fail("Cannot extend to size #{size} because VG extend is
#{vg_extend}KB")
end
## Veritfy that it's a extend: Only extends are allowed
if lvm_size_units[c_size_unit] < lvm_size_units[n_size_unit]
resizeable = true
elsif lvm_size_units[c_size_unit] ==
lvm_size_units[n_size_unit]
if n_size_bytes > c_size_bytes
resizeable = true
end
end
if not resizeable
fail("Cannot change size of /dev/#{resource[:vg]}/
#{resource[:name]} as #{size} < #{c_size}")
else
output = Puppet::Util.execute(["lvextend", "-L", size, "/
dev/#{resource[:vg]}/#{resource[:name]}"])
debug(output)
return true
end
end
def create
output = Puppet::Util.execute(["lvcreate", "-L",
resource[:size], "-n", resource[:name], resource[:vg]])
debug(output)
return true
end
def destroy
output = Puppet::Util.execute(["lvchange", "-an", "/dev/
#{resource[:vg]}/#{resource[:name]}"])
output = Puppet::Util.execute(["lvremove", "-f", "/dev/
#{resource[:vg]}/#{resource[:name]}"])
debug(output)
return true
end
def exists?
output = Puppet::Util.execute(["lvs", "--noheading", "/dev/
#{resource[:vg]}/#{resource[:name]}"], :failonfail => false, :combine
=> true)
debug(output)
if output.include?("One or more specified logical volume(s)
not found.")
return nil
else
return true
end
end
end
Thanks,
Daniel
Luke Kanies
> Hi list,
>
> i wrote a simple type and provider to manage logical volumes with
> puppet. This is my first attempt and it may not match the puppet
> styles and best practices. I'm hoping for your improvement
> suggestions.
Hi Daniel,
I don't quite have the remaining brainpower today to review your code,
but we're just about to publish an LVM module for managing physical
volumes, volume groups, logical volumes, and filesystems. It doesn't
have the resizing built in like yours does, but it's otherwise pretty
comprehensive. It'd be great if we could join efforts.
--
I take my children everywhere, but they always find their way
back home. --Robert Orben
---------------------------------------------------------------------
Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199
Daniel
> On Feb 17, 2010, at 1:49 PM, Daniel Kerwin wrote:
>
>> Hi list,
>>
>> i wrote a simple type and provider to manage logical volumes with
>> puppet. This is my first attempt and it may not match the puppet
>> styles and best practices. I'm hoping for your improvement
>> suggestions.
>
>
> Hi Daniel,
>
> I don't quite have the remaining brainpower today to review your code, but
> we're just about to publish an LVM module for managing physical volumes,
> volume groups, logical volumes, and filesystems. It doesn't have the
> resizing built in like yours does, but it's otherwise pretty comprehensive.
> It'd be great if we could join efforts.
>
Hi Luke,
thanks for your reply.
That sounds great. Is the code already publicly available? I'd be glad
to help you. Just let me know if there's anything i can do.
--
Cheers,
Daniel
Luke Kanies
We haven't published it yet, likely will next week.
--
Silence is a text easy to misread.
-- A. A. Attanasio, 'The Eagle and the Sword'
Daniel
> On Feb 19, 2010, at 5:06 AM, Daniel wrote:
>
>> On Thu, Feb 18, 2010 at 11:27 PM, Luke Kanies <lu...@reductivelabs.com>
>> wrote:
>>>
>>> On Feb 17, 2010, at 1:49 PM, Daniel Kerwin wrote:
>>>
>>>> Hi list,
>>>>
>>>> i wrote a simple type and provider to manage logical volumes with
>>>> puppet. This is my first attempt and it may not match the puppet
>>>> styles and best practices. I'm hoping for your improvement
>>>> suggestions.
>>>
>>>
>>> Hi Daniel,
>>>
>>> I don't quite have the remaining brainpower today to review your code,
>>> but
>>> we're just about to publish an LVM module for managing physical volumes,
>>> volume groups, logical volumes, and filesystems. It doesn't have the
>>> resizing built in like yours does, but it's otherwise pretty
>>> comprehensive.
>>> It'd be great if we could join efforts.
>>>
>>
>> Hi Luke,
>>
>> thanks for your reply.
>>
>> That sounds great. Is the code already publicly available? I'd be glad
>> to help you. Just let me know if there's anything i can do.
>
> We haven't published it yet, likely will next week.
>
Ok. If you're looking for an early adopter i can make some tests and
give you feedback.
Thanks.
> --
> Silence is a text easy to misread.
> -- A. A. Attanasio, 'The Eagle and the Sword'
> ---------------------------------------------------------------------
> Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.
>
>
--
Cheers,
Daniel
Luke Kanies
Ok, we've just published our LVM module:
http://www.gnu.org/licenses/gpl-2.0.txt
As you'll see, there's a lot of overlap.
We'll still be making some changes that we owe our customer, but I'd
definitely like to hear anything you'd like to see done differently.
--
A complex system that works is invariably found to have evolved from a
simple system that works. -- John Gaule
Luke Kanies
Or rather:
http://github.com/reductivelabs/puppet-lvm
Oops.
--
Never esteem anything as of advantage to you that will make you break
your word or lose your self-respect. -- Marcus Aurelius Antoninus
Daniel
know if i have suggestions.
Daniel
1) Your mkfs statements may not always succeed. When you use reiserfs
for example mkfs.reiserfs will wait for user input when the parameter
"-f" isn't supplied.
2) Resizing of logical volumes is a important feature of LVM. Reiserfs
(and i think ext3/etx4 too?) can be online resized without service
interruption. My (poor) provider supports it and it works pretty good.
What about a parameter like "autoextend" for the logicalvolume and
filesystem type? Depending on the FS puppet can extend the volume and
the filesystem.
3) Removing physical volumes may lead to problems but i need to make
some tests to trace possible problems
I'll deploy it on my test systems to see if there's something else i've missed.
Luke Kanies
> i had a quick look at the code and here are my sugestions:
>
> 1) Your mkfs statements may not always succeed. When you use reiserfs
> for example mkfs.reiserfs will wait for user input when the parameter
> "-f" isn't supplied.
Ah, I hadn't caught that.
> 2) Resizing of logical volumes is a important feature of LVM. Reiserfs
> (and i think ext3/etx4 too?) can be online resized without service
> interruption. My (poor) provider supports it and it works pretty good.
> What about a parameter like "autoextend" for the logicalvolume and
> filesystem type? Depending on the FS puppet can extend the volume and
> the filesystem.
Yeah, resizing is one of the things we'd planned to add but hadn't
gotten around to yet.
> 3) Removing physical volumes may lead to problems but i need to make
> some tests to trace possible problems
We tested it and it only attempts it if it's safe - otherwise it fails
saying you need to run pvmove or whatever.
> I'll deploy it on my test systems to see if there's something else
> i've missed.
Thanks.
--
Silence is a text easy to misread.
-- A. A. Attanasio, 'The Eagle and the Sword'
Daniel
> On Feb 23, 2010, at 4:44 AM, Daniel wrote:
>
>> i had a quick look at the code and here are my sugestions:
>>
>> 1) Your mkfs statements may not always succeed. When you use reiserfs
>> for example mkfs.reiserfs will wait for user input when the parameter
>> "-f" isn't supplied.
>
> Ah, I hadn't caught that.
>
>> 2) Resizing of logical volumes is a important feature of LVM. Reiserfs
>> (and i think ext3/etx4 too?) can be online resized without service
>> interruption. My (poor) provider supports it and it works pretty good.
>> What about a parameter like "autoextend" for the logicalvolume and
>> filesystem type? Depending on the FS puppet can extend the volume and
>> the filesystem.
>
> Yeah, resizing is one of the things we'd planned to add but hadn't gotten
> around to yet.
If i can help you with this issue please let me know. I'd be glad to
contribute even if my ruby skills aren't really big.
> Silence is a text easy to misread.
> -- A. A. Attanasio, 'The Eagle and the Sword'
> ---------------------------------------------------------------------
> Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.
>
>
--
Cheers,
Daniel
rodjek
> On Feb 19, 2010, at 9:26 AM, Daniel wrote:
>
>
>
> > On Fri, Feb 19, 2010 at 6:00 PM, Luke Kanies
> > <l...@reductivelabs.com> wrote:
> >> On Feb 19, 2010, at 5:06 AM, Daniel wrote:
>
I've got a bit of a major concern with the filesystem type. Mainly,
what would happen if the superblock on my filesystem went bad? mount -
f --guess-fstype <dev> would presumably return that I don't have a
valid filesystem and before you know it, Puppet'd give me a nice shiny
new filesystem.
Perhaps if the filesystem type had one-shot functionality similar to
exec's refreshonly...
Luke Kanies
It would be excellent if you ported your resizing code over to our
module.
>>> 3) Removing physical volumes may lead to problems but i need to make
>>> some tests to trace possible problems
>>
>> We tested it and it only attempts it if it's safe - otherwise it
>> fails
>> saying you need to run pvmove or whatever.
>>
>>> I'll deploy it on my test systems to see if there's something else
>>> i've
>>> missed.
>>
>> Thanks.
--
The major difference between a thing that might go wrong and a thing
that cannot possibly go wrong is that when a thing that cannot possibly
goes wrong goes wrong it usually turns out to be impossible to get at
or repair. -- Douglas Adams, Mostly Harmless
Luke Kanies
> I've got a bit of a major concern with the filesystem type. Mainly,
> what would happen if the superblock on my filesystem went bad?
> mount -
> f --guess-fstype <dev> would presumably return that I don't have a
> valid filesystem and before you know it, Puppet'd give me a nice shiny
> new filesystem.
>
> Perhaps if the filesystem type had one-shot functionality similar to
> exec's refreshonly...
Hrm, you're right, that does seem potentially dangerous. Is there a
better way to test for the current filesystem?
Or should we maybe skip the filesystem type entirely and just
initialize the volume as a given filesystem when the volume itself is
created? E.g., something like:
logical_volume { foo: vg => bar, fs => ext3, ensure => present }
So when you create the lv, you initialize the fs, but otherwise you
don't mess with it.
--
Ah, but I am more perceptive than most of the universe. Especially
the parts of the universe that are vacuum. -- James Alan Gardner
Daniel
submitting? Seems like your git repo is read-only.
Luke Kanies
modules; at this point, either just send me a pull request via github,
or email me the patches or something.
--
Susskind's Rule of Thumb:
Don't ask what they think. Ask what they do.
Daniel
code tomorrow and will send you my results. Thanks
> Susskind's Rule of Thumb:
> Don't ask what they think. Ask what they do.
> ---------------------------------------------------------------------
> Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199
>
Marcin Owsiany
>
> Or should we maybe skip the filesystem type entirely and just initialize
> the volume as a given filesystem when the volume itself is created?
> E.g., something like:
>
> logical_volume { foo: vg => bar, fs => ext3, ensure => present }
>
> So when you create the lv, you initialize the fs, but otherwise you
> don't mess with it.
Makes sense, although you should not get rid of the FS type. It will
still be useful for changing filesystem type (including online migration
between compatible filesystems like ext2->ext3).
--
Marcin Owsiany <mar...@owsiany.pl> http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216
"Every program in development at MIT expands until it can read mail."
-- Unknown
Daniel
but may not be perfect. I tested it about 2 hours and and extended a
lot of volumes without any issues. The filesystem part is currently
ignored.
--
Cheers,
Daniel
Luke Kanies
> On Wed, Feb 24, 2010 at 09:24:55AM -0800, Luke Kanies wrote:
>>
>> Or should we maybe skip the filesystem type entirely and just
>> initialize
>> the volume as a given filesystem when the volume itself is created?
>> E.g., something like:
>>
>> logical_volume { foo: vg => bar, fs => ext3, ensure => present }
>>
>> So when you create the lv, you initialize the fs, but otherwise you
>> don't mess with it.
>
> Makes sense, although you should not get rid of the FS type. It will
> still be useful for changing filesystem type (including online
> migration
> between compatible filesystems like ext2->ext3).
Is there, then, a consistent, safe way to determine the filesystem type?
--
Reality is that which, when you stop believing in it, doesn't go
away. -- Philip K. Dick, "How to Build a Universe"
Daniel
> On Feb 25, 2010, at 12:13 AM, Marcin Owsiany wrote:
>
>> On Wed, Feb 24, 2010 at 09:24:55AM -0800, Luke Kanies wrote:
>>>
>>> Or should we maybe skip the filesystem type entirely and just initialize
>>> the volume as a given filesystem when the volume itself is created?
>>> E.g., something like:
>>>
>>> logical_volume { foo: vg => bar, fs => ext3, ensure => present }
>>>
>>> So when you create the lv, you initialize the fs, but otherwise you
>>> don't mess with it.
>>
>> Makes sense, although you should not get rid of the FS type. It will
>> still be useful for changing filesystem type (including online migration
>> between compatible filesystems like ext2->ext3).
>
> Is there, then, a consistent, safe way to determine the filesystem type?
>
Maybe file i another option can help.
# file -s /dev/mapper/rootvg-tmplv
/dev/mapper/rootvg-tmplv: ReiserFS V3.6 block size 4096 (mounted or
unclean) num blocks 128000 r5 hash
I'm not sure if this works when the superblock is corrupt. Maybe i can
do some testing on this subject
> --
> Reality is that which, when you stop believing in it, doesn't go
> away. -- Philip K. Dick, "How to Build a Universe"
> ---------------------------------------------------------------------
> Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199
>
Nicolas Szalay
| Is there, then, a consistent, safe way to determine the filesystem type?
Not really, gpart is quite good at this iirc.
Marcin Owsiany
> On Thu, Feb 25, 2010 at 6:05 PM, Luke Kanies <lu...@reductivelabs.com> wrote:
> > On Feb 25, 2010, at 12:13 AM, Marcin Owsiany wrote:
> >
> >> On Wed, Feb 24, 2010 at 09:24:55AM -0800, Luke Kanies wrote:
> >>>
> >>> Or should we maybe skip the filesystem type entirely and just initialize
> >>> the volume as a given filesystem when the volume itself is created?
> >>> E.g., something like:
> >>>
> >>> logical_volume { foo: vg => bar, fs => ext3, ensure => present }
> >>>
> >>> So when you create the lv, you initialize the fs, but otherwise you
> >>> don't mess with it.
> >>
> >> Makes sense, although you should not get rid of the FS type. It will
> >> still be useful for changing filesystem type (including online migration
> >> between compatible filesystems like ext2->ext3).
> >
> > Is there, then, a consistent, safe way to determine the filesystem type?
>
> Maybe file i another option can help.
>
> # file -s /dev/mapper/rootvg-tmplv
> /dev/mapper/rootvg-tmplv: ReiserFS V3.6 block size 4096 (mounted or
> unclean) num blocks 128000 r5 hash
That's also approximately the method I used in my provider for the filesystem type.
> I'm not sure if this works when the superblock is corrupt. Maybe i can
> do some testing on this subject
I'm not sure it matters. There could always be a situation where your
filesystem is broken enough for file/mount/kernel to not be able to
recognize it, but for the data to still be in a recoverable state (at
least partially). The point is that we don't want puppet to do anything
harmful (like recreate the filesystem) in that case. Then again I don't
know if there is anything that you can do to a filesystem without
potentially harming it..
Luke Kanies
This sounds basically intractable - there's no dependable way to know
if there's a filesystem in place, which to me means there's no way to
know if we should create a filesystem. Right?
Even recording the fact that we've created an FS before isn't
foolproof - if someone removes the state.yaml file, we lose that
recorded information.
--
Experience is the name everyone gives to their mistakes.
-- Oscar Wilde
Daniel Kerwin
Recovery from alternative superblocks hasn't really helped me in the
past.
I think a combination of live determination and maybe the state file
is ok. Right now I use exec calls to create and verify existing
filesystems and it's better to have a standard puppet type to do it.
In my oppinion Puppet should have this feature and the user can decide
if he wants to use it.
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.
>
>
--
Sent from Google Mail for mobile | mobile.google.com
Cheers,
Daniel
Luke Kanies
Puppet. For now it's just going to be in our github account, but
we'll have our module site up soon enough, and that's where we'll be
publishing it.
If anything, we want to pull things out of Puppet (e.g., the Nagios
code), rather than push more in. I'd like to get the core smaller and
just make it eaiser to extend it.
--
The cure for writer's cramp is writer's block.
-- Inigo DeLeon
Marcin Owsiany
Yes, that is why I think the approach of using an attribute for the
volume resource is safer for filesystem creation.
However I imagine that for lossless filesystem conversion it should be
reasonably safe. Something like:
filesystem { "/dev/sda3":
from => ext2,
to => ext3
}
that would only call "tune2fs -j /dev/sda3" if the current type is
detected to be ext2. Even if that is an artifact of a corruption, then
tune2fs should be able to detect that and abort, I assume.
OTOH maybe having a full filesystem type for this purpose is a little
too much - there are not that many filesystems that can be upgraded on
the fly.
Marcin Owsiany
> When a FS really is that corrupt you may be out of luck anyway.
This is very true, but if puppet decides for you, there isn't even a way
to find out.
> In my oppinion Puppet should have this feature and the user can decide
> if he wants to use it.
I think it would be good to make people aware of the risk - something
like require that a "i_have_read_the_documentation" parameter to be
specified on each filesystem resource.
Trevor Vaughan
Hash: SHA1
On 02/26/2010 02:32 PM, Luke Kanies wrote:
> On Feb 26, 2010, at 11:28 AM, Marcin Owsiany wrote:
<snip/>
>> I'm not sure it matters. There could always be a situation where your
>> filesystem is broken enough for file/mount/kernel to not be able to
>> recognize it, but for the data to still be in a recoverable state (at
>> least partially). The point is that we don't want puppet to do anything
>> harmful (like recreate the filesystem) in that case. Then again I don't
>> know if there is anything that you can do to a filesystem without
>> potentially harming it..
>
> This sounds basically intractable - there's no dependable way to know if
> there's a filesystem in place, which to me means there's no way to know
> if we should create a filesystem. Right?
>
> Even recording the fact that we've created an FS before isn't foolproof
> - if someone removes the state.yaml file, we lose that recorded
> information.
>
Could you use the information in fstab/mtab?
In theory (heh), there wouldn't be an entry in fstab or mtab for the
partition that you're trying to create if it hasn't yet been created.
If there is an entry, a tag like 'force => true' would let you tell
puppet to go ahead and blow away the fs if it can't determine that there
is one there. Default, of course, would be 'force => false'.
On a side note, I have found that the included LVM fs resizing for ext3
will not work if the partition requires an fsck. If it fails, you then
have to use resize2fs. It might be best to just use resize2fs in all cases.
Also, you may want to require users to set a flag if they want to
*shrink* a partition. It's doable, but you can easily screw up your
data. (Apologies if you did this and I missed it in the thread).
Thanks,
Trevor
- --
Trevor Vaughan
Vice President, Onyx Point, Inc.
email: tvau...@onyxpoint.com
phone: 410-541-ONYX (6699)
- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkuM7UQACgkQyWMIJmxwHpRRywCfQg9nmrErtxaFqZNckJwYjSO1
Z+4AoL9FsxEnKYsXodYFnDslmlPiHBic
=mT9+
-----END PGP SIGNATURE-----
Daniel Kerwin
could lead to data loss and should be handled manually.
Mike Pountney
>>
>> Even recording the fact that we've created an FS before isn't foolproof
>> - if someone removes the state.yaml file, we lose that recorded
>> information.
>>
>
> Could you use the information in fstab/mtab?
>
> In theory (heh), there wouldn't be an entry in fstab or mtab for the
> partition that you're trying to create if it hasn't yet been created.
That's not always the case, for example with DRBD mounts handled by heartbeat. I guess you could put a 'noauto' entry in though, for reference.
Trevor Vaughan
Hash: SHA1
That would work.
Didn't think about the DRBD case, good catch!
Trevor
- --
Trevor Vaughan
Vice President, Onyx Point, Inc.
email: tvau...@onyxpoint.com
phone: 410-541-ONYX (6699)
- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAkuRsRQACgkQyWMIJmxwHpQVfgCgqv9zgHf+H6BAEcz0h9CRvBDd
vioAoNbGLALDHMuGaGxh+aee9nzoincQ
=rvex
-----END PGP SIGNATURE-----