Composite primary keys as namevars
David Schmitt
a recurring problem with the namevars are resources that do not have a
natural unique single-property key. For example, mysql users are defined
by their username and the host(-pattern) they are connection from.
Traditionally, this was solved by ugly hackery in the type. In the
windows session at the puppetcamp, Markus and I developed the idea of
associating a set of patterns/syntaxes with the namevar to automatically
parse such multi-key titles and automatically put them into proper
parameters, which can then be validated, munged and used without needing
to know about the actual syntax elsewhere in the type/provider.
I'll start with an example, of how this could look like:
Puppet::Type.newtype(:mysql_user) do
@doc = "Manage a mysql database user."
ensurable
newtitle do
pattern "([^@]+)@([^@]+)", :user, :host
pattern "([^@]+)", :user
end
newparam(:user) do
desc "The username."
end
newparam(:host) do
desc "The host, the users connects from. This may be a pattern
using '%' and '_'."
defaultto "%"
end
Patterns would be tried in order and match groups assigned to the
specified parameters. Having no pattern match woul
This could be used in the following ways:
mysql_user {
"foo": ;
"foo@%": ; # both 'foo@%'
"bar": host => "localhost";
"bar@localhost": ; # both 'bar@localhost'
}
Mysql_user { host => "webserver" }
mysql_user { [ "frob", "frub" ]: }
# expands to 'frob@webserver' and 'frub@webserver'
mysql_user {
"frob": host => "web1";
"frob": host => "web2";
"frob": host => "web3";
}
The last stanza shows how this can be used to confusing effects. This
could be mitigated by allowing using arrays in parameters. For example
creating four users:
mysql_user {
[ "frob", "frub" ]:
host => [ "web1", "web2" ]
}
Here are some illegal constructs:
# Setting a property twice
mysql_user {
"foo": user => "bar";
"foo@%": host => "localhost";
}
# redefine resources
mysql_user {
"frob": ;
"frob@%": ;
"foo": host => "localhost";
"frob@localhost": ;
}
# pattern match fails:
mysql_user {
[ "", "@", "foo@", "@localhost", "foo@bar@localhost" ]: ;
}
For more flexibility when parsing, the list-of-paramnames could be
replaced by blocks:
pattern "complicated_regexp" do
# insert disgusting hack here
end
The block would have to return true or false to accept or reject the title.
By substituting the param names into the patterns, puppet could
automatically create documentation like this:
Patterns:
1) user@host:
user must be [^@]+
host must be [^@]+
2) user:
user must be [^@]+
user: The username.
host: The host, the users connects from. This may be a pattern using
'%' and '_'.
This could be enhanced by specifying additional docstrings with the
patterns.
Of course, this is currently vapour-ware, as it'll need some changes in
the puppet core to handle the new uniqueness, but I hope to accelerate
the development by providing a good spec/discussion of how the
"user-facing" side should look like.
Looking forward to your feedback,
Best Regards, David
--
dasz.at OG Tel: +43 (0)664 2602670 Web: http://dasz.at
Klosterneuburg UID: ATU64260999
FB-Nr.: FN 309285 g FB-Gericht: LG Korneuburg
Trevor Vaughan
Hash: SHA1
Wow...this is only slightly terrifying ;-).
Since you're effectively creating a database serialization, why not
simplify the problem by using the concept of primary keys.
So, currently, namevar is built from <name>, could you not instead build
it from multiple values?
In the following example, I'm using the '@' symbol to indicate that
$name/$namevar should be built from an array of values inside the 'foo'
stanza.
foo { @:
+bar => 'baz',
+bar2 => 'baz2',
notakey => 'value'
}
So, $name would now be "baz_baz2" built from the names prepended with a
'+'. I suppose you would probably want to sort the values alphabetically
so that you achieve consistency in your code. Arrays could be handled
through a concatenation of the values or by spawning additional 'foo'
resources.
Does this do what you want? To me it seems to be able to be applied more
generically across all types (including the current ones).
Trevor
- --
Trevor Vaughan
Vice President, Onyx Point, Inc.
email: tvau...@onyxpoint.com
phone: 410-541-ONYX (6699)
pgp: 0x6C701E94
- -- This account not approved for unencrypted sensitive information --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEARECAAYFAkwI364ACgkQyWMIJmxwHpS9eACeMraJMOvRFao9SjbRlGDWfANR
4gkAoMOQf/jj9p9qRQFNlG220kfT1S5x
=rQuK
-----END PGP SIGNATURE-----
David Schmitt
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Wow...this is only slightly terrifying ;-).
>
> Since you're effectively creating a database serialization, why not
> simplify the problem by using the concept of primary keys.
No and exactly. The unparsed title would only exist in the manifest and
would be converted to parameters at parse time.
> So, currently, namevar is built from<name>, could you not instead build
> it from multiple values?
That's what happening. In the example, :user and :host form the combined
primary key/namevar.
> In the following example, I'm using the '@' symbol to indicate that
> $name/$namevar should be built from an array of values inside the 'foo'
> stanza.
>
> foo { @:
> +bar => 'baz',
> +bar2 => 'baz2',
> notakey => 'value'
> }
>
> So, $name would now be "baz_baz2" built from the names prepended with a
> '+'. I suppose you would probably want to sort the values alphabetically
> so that you achieve consistency in your code. Arrays could be handled
> through a concatenation of the values or by spawning additional 'foo'
> resources.
Are you referring to resources.title in the storedconfigs database? I
don't see this as a primary problem. Queries should use the constituent
parameters (:user, :host) and the title in the db should not be
user-visible at all.
> Does this do what you want? To me it seems to be able to be applied more
> generically across all types (including the current ones).
It is my intention to have this as a core feature, that also can be used
by current types.
Best Regards, DavidS
Markus Roberts
> mysql_user {
> "frob": ;
> "frob@%": ;
>
> "foo": host => "localhost";
> "frob@localhost": ;
> }
In the following example, I'm using the '@' symbol to indicate that
$name/$namevar should be built from an array of values inside the 'foo'
stanza.
foo { @:
+bar => 'baz',
+bar2 => 'baz2',
notakey => 'value'
}
So, $name would now be "baz_baz2" built from the names prepended with a
'+'. I suppose you would probably want to sort the values alphabetically
so that you achieve consistency in your code. Arrays could be handled
through a concatenation of the values or by spawning additional 'foo'
resources.
The fundamental deference is that David and I are working on a system where, rather than the namevar being a specific predetermined attribute it would be be a specific predetermined set of attributes; that's the extent of the change.
What you're describing appears to be having it be an arbitrary, user determined set of attributes which is way beyond what we're proposing. Going that far opens a whole case of cans of worms, which we'd rather not.
-- Markus
The power of accurate observation is
commonly called cynicism by those
who have not got it. ~George Bernard Shaw
------------------------------------------------------------
David Schmitt
> > # redefine resources
> > mysql_user {
> > "frob": ;
> > "frob@%": ;
> >
> > "foo": host => "localhost";
> > "frob@localhost": ;
> > }
>
>
> The last pair isn't really a duplicate though, unless you change foo to
> frob or frob to foo.
Exactly. I'm glad if that's the only notable thing on your side :-)
Luke Kanies
> Hi all,
>
> a recurring problem with the namevars are resources that do not have
> a natural unique single-property key. For example, mysql users are
> defined by their username and the host(-pattern) they are connection
> from.
>
> Traditionally, this was solved by ugly hackery in the type. In the
> windows session at the puppetcamp, Markus and I developed the idea
> of associating a set of patterns/syntaxes with the namevar to
> automatically parse such multi-key titles and automatically put them
> into proper parameters, which can then be validated, munged and used
> without needing to know about the actual syntax elsewhere in the
> type/provider.
It's definitely long-past time to solve this, and I'm glad it's
finally being worked on.
What's the reason for not just requiring full parameter specification,
rather than supporting the title short-hand?
It seems to me we kind of have two problems: How to uniquely specify
the resource with parameters, and how to specify it with a resource
reference. Your solution (the title becomes a template filled in by
the different parameters) does kind of neatly solve it, but I'm afraid
it might be too complicated for normal humans to use.
It'd probably be a good idea to run through three or four other
resource types with this to see how it works. E.g., ports (/etc/
services), packages (optionally specify a version number or
architecture, where two packages with the same name but different
version/arch are considered different packages depending on the
provider, I think), and probably something else.
--
Trying to determine what is going on in the world by reading
newspapers is like trying to tell the time by watching the second
hand of a clock. --Ben Hecht
---------------------------------------------------------------------
Luke Kanies -|- http://puppetlabs.com -|- +1(615)594-8199
Markus Roberts
It seems to me we kind of have two problems: How to uniquely specify the resource with parameters, and how to specify it with a resource reference. Your solution (the title becomes a template filled in by the different parameters) does kind of neatly solve it, but I'm afraid it might be too complicated for normal humans to use.
Playing around with it at puppet came we came to the conclusion that, while it could in principle get too complicated to use, in the actual use cases it was dead obvious what was intended--in part because the formats we are wanting to model are themselves extremely well known.
It'd probably be a good idea to run through three or four other resource types with this to see how it works. E.g., ports (/etc/services), packages (optionally specify a version number or architecture, where two packages with the same name but different version/arch are considered different packages depending on the provider, I think), and probably something else.
We did go through several (ports, mysql users, etc.) though we didn't do the package/arch case.
David Schmitt
> It seems to me we kind of have two problems: How to uniquely
> specify the resource with parameters, and how to specify it with a
> resource reference. Your solution (the title becomes a template
> filled in by the different parameters) does kind of neatly solve it,
> but I'm afraid it might be too complicated for normal humans to use.
>
>
> Playing around with it at puppet came we came to the conclusion that,
> while it could in principle get too complicated to use, in the actual
> use cases it was dead obvious what was intended--in part because the
> formats we are wanting to model are themselves extremely well known.
What I think you were referring to ("title becomes a template") was only
meant for documentation generation. (Ab-)Using this as a way to create a
canonical title, seems to magical to me.
I failed to include a little bit of code to actually make the title
generation explicit. There should be a separate, explicit code-block in
the newtitle that creates the canonical title from the resource:
newtitle do
# match incoming parameters
pattern "([^@]+)@([^@]+)", :user, :host
pattern "([^@]+)", :user
# produce canonical title form for referencing
canonical_form do
"%s@%s" % [ @resource[:user], @resource[:host] ]
end
end
Especially on the file type, which has very different syntaxes for the
same property combinations (eg: "host:/share" for NFS and "\\host\share"
for UNC) whould need more intelligence than simple substitution for
title generation, while for documentation it would still work.
Does this address your concerns?
Luke Kanies
On a completely side note, apparently one of the code smells in
Puppet's existing internal DSL around building resource types is the
use of 'new' in method names. So I recommend going with just 'title'
or something similar for the method name.
--
Dawkins's Law of Adversarial Debate:
When two incompatible beliefs are advocated with equal intensity,
the truth does not lie half way between them.
Markus Roberts
David --
If I'm understanding you correctly, this is completely opposite of what was discussed at puppet camp (which leads me to suspect that I'm not understanding you correctly).
My understanding is that the situation can be broken down into several steps:
- Instead of having a single namevar, we have a non-empty collection of them, and two resources are the same if and only if all of them match. Note that the present situation is a special case of this, where the collection always has exactly one member.
- As currently, namevar is determined by the type.
- Instead just of inferring the single namevar from the title we let types decompose the title into values for several (perhaps all) of the namevar components; note that the present situation is again a special case of this.
- As a natural extension of the above, we discussed permitting Brice's hash notation in the same role.
-- Markus
David Schmitt
>
> David --
>
> If I'm understanding you correctly, this is completely opposite of what
> was discussed at puppet camp (which leads me to suspect that I'm not
> understanding you correctly).
>
> My understanding is that the situation can be broken down into several
> steps:
>
> of them, and two resources are the same if and only if all of them
> match. Note that the present situation is a special case of this,
> where the collection always has exactly one member.
> * Instead just of inferring the single namevar from the title we let
> types decompose the title into values for several (perhaps all) of
> the namevar components; note that the present situation is again a
> special case of this.
> Brice's hash notation in the same role.
That's the internal/implementation side of it. Please see below for
explainations about the frills I added for the more "user"/developer
oriented spec in my mail.
> At no time that I recall were we talking about "title generation", using
> a generated title for uniqueness, or this just being a documentation
> convention.
We didn't talk about it. But how would puppet reference the following
resources in a log message?
mysql_user {
"frob": host => "web1";
"frob": host => "web2";
"frob": host => "web3";
}
Shouldn't that be called 'Mysql_user["frob@web1"]' and so on? Since the
manifest doesn't provide that "frob@web" string anywhere, the type must
generate it and the implementor should explicitly choose how.
Also the question arose around Trevor's mail how storedconfig's
resources.title is filled. Which, like the log message, is more of a
usability thing than anything else, because the user would expect a
"well-formed" title, that corresponds to the specified parameters,
independently of how they are specified.
The stuff about automatically generating documentation from patterns was
added by me, when drafting the spec. Substituting the parameter names
into the match groups was a seemingly easy way to generate passable
docstrings from the existing information.
Markus Roberts
> That's the internal/implementation side of it. Please see below for explainations about the frills
> I added for the more "user"/developer oriented spec in my mail.
> We didn't talk about it. But how would puppet reference the following resources in a log message?
Also the question arose around Trevor's mail how storedconfig's resources.title is filled. Which, like the log message, is more of a usability thing than anything else, because the user would expect a "well-formed" title, that corresponds to the specified parameters, independently of how they are specified.
Ditto with storeconfigs; I'd say use the title, as now. The idea is very simple when it's unidirectional and very complicated (or perhaps "simple but full of edge cases") when you try to make it bi-directional.
BTW, I'll be back in the office as of today & will kick this around with Jesse, who may convince me one way or the other, and I'll be trying to get the patches I promised you by the end of this coming week out ASAP .
-- Markus
David Schmitt
I'll leave the final decision to you. Just let me restate that I believe
that having the additional "render" or "to_s" block (what I called
"canonical_form" in <4C0C0E2A...@dasz.at>) for display purposes[1]
wouldn't hurt as much as not being able to use defaults for properties.
[1] Resource references (for example for dependencies) can parse the
specified string, using the patterns on the type.
Luke Kanies
> David --
>
> > That's the internal/implementation side of it. Please see below
> for explainations about the frills
> > I added for the more "user"/developer oriented spec in my mail.
>
> I saw it, I'm just not sold on it (yet). The internal side, as you
> call it, looks pretty clean and high payoff; the additions are
> (IMHO) lower payoff and more problematic. I'm not saying that I
> couldn't be convinced, but I (and I'd thought we) had stopped where
> we did for exactly that reason.
>
> > We didn't talk about it. But how would puppet reference the
> following resources in a log message?
>
> By their titles. And for this reason users shouldn't give resources
> titles that they won't be able to subsequently recognize, just as now.
This is basically where I'd planned on taking this - either lose the
requirement that titles be unique (e.g., dependencies are lazy binding
anyway, so you could just depend on all things that have that title),
or don't care about unique titles unless there's a relationship
involved. I prefer the former. Titles are for humans anyway, so they
have complete control over how it works.
> Also the question arose around Trevor's mail how storedconfig's
> resources.title is filled. Which, like the log message, is more of a
> usability thing than anything else, because the user would expect a
> "well-formed" title, that corresponds to the specified parameters,
> independently of how they are specified.
>
> Ditto with storeconfigs; I'd say use the title, as now. The idea is
> very simple when it's unidirectional and very complicated (or
> perhaps "simple but full of edge cases") when you try to make it bi-
> directional.
>
> BTW, I'll be back in the office as of today & will kick this around
> with Jesse, who may convince me one way or the other, and I'll be
> trying to get the patches I promised you by the end of this coming
> week out ASAP .
--
Don't hit at all if it is honorably possible to avoid hitting; but
never hit soft! -- Theodore Roosevelt
Nigel Kersten
On Jun 7, 2010, at 7:43 AM, Markus Roberts wrote:This is basically where I'd planned on taking this - either lose the requirement that titles be unique (e.g., dependencies are lazy binding anyway, so you could just depend on all things that have that title), or don't care about unique titles unless there's a relationship involved. I prefer the former. Titles are for humans anyway, so they have complete control over how it works.
David --
> That's the internal/implementation side of it. Please see below for explainations about the frills
> I added for the more "user"/developer oriented spec in my mail.
I saw it, I'm just not sold on it (yet). The internal side, as you call it, looks pretty clean and high payoff; the additions are (IMHO) lower payoff and more problematic. I'm not saying that I couldn't be convinced, but I (and I'd thought we) had stopped where we did for exactly that reason.
> We didn't talk about it. But how would puppet reference the following resources in a log message?
By their titles. And for this reason users shouldn't give resources titles that they won't be able to subsequently recognize, just as now.
Don't hit at all if it is honorably possible to avoid hitting; but
Also the question arose around Trevor's mail how storedconfig's resources.title is filled. Which, like the log message, is more of a usability thing than anything else, because the user would expect a "well-formed" title, that corresponds to the specified parameters, independently of how they are specified.
Ditto with storeconfigs; I'd say use the title, as now. The idea is very simple when it's unidirectional and very complicated (or perhaps "simple but full of edge cases") when you try to make it bi-directional.
BTW, I'll be back in the office as of today & will kick this around with Jesse, who may convince me one way or the other, and I'll be trying to get the patches I promised you by the end of this coming week out ASAP .
--
never hit soft! -- Theodore Roosevelt
---------------------------------------------------------------------
Luke Kanies -|- http://puppetlabs.com -|- +1(615)594-8199
--You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To post to this group, send email to puppe...@googlegroups.com.
To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
--
nigel
Markus Roberts
> I'll leave the final decision to you. Just let me restate that I believe that having the additional
> "render" or "to_s" block (what I called "canonical_form" in <4C0C0E2A...@dasz.at>)
> for display purposes[1] wouldn't hurt as much as not being able to use defaults for properties.
In the course of the discussions I realized that I don't understand your "not being able to use defaults for properties" comment. Could you elaborate?
Trevor Vaughan
How does this work in the case of multi-party development?
For instance, you now have the Module Forge (woo), but I'm sure you're
going to get two modules with a 'service { "mysql"' or somesuch.
I know that it's easy to say "just be careful", but throwing up the
overworked puny human SA defense, not conflicting on duplicate names
is going to cause a massive amount of headache in my opinion.
How else could this be solved so that people are not bitten by
conflicts? Warnings perhaps?
Thanks,
Trevor
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.
>
>
--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
Markus Roberts
> How does this work in the case of multi-party development?
>
> For instance, you now have the Module Forge (woo), but I'm sure you're
> going to get two modules with a 'service { "mysql"' or somesuch.
>
> I know that it's easy to say "just be careful", but throwing up the
> overworked puny human SA defense, not conflicting on duplicate names
> is going to cause a massive amount of headache in my opinion.
>
> How else could this be solved so that people are not bitten by
> conflicts? Warnings perhaps?
I'm not sure I'm seeing the problem you are worried about.
The feature we're planning on implementing is very targeted at a
specific class of use cases: places where the natural title of a
resource actually contains two or more pieces of data, such as an MS
Win file path "C:/foo" which contains both a traditional path and a
"drive letter." The idea is to let the user specify these in the
natural way and parse them to get the individual fields which
(collectively) would be the unique identifier for the resource.
Thus you could have "C:/foo" and "D:/foo" and "C:/bar", etc. so long
as there was only one occurrence of each {drive_letter, path}
combination.
I'm not seeing the multi-party development issue here.
David Schmitt
> David --
>
> > I'll leave the final decision to you. Just let me restate that I
> believe that having the additional
> > "render" or "to_s" block (what I called "canonical_form" in
> > for display purposes[1] wouldn't hurt as much as not being able to
> use defaults for properties.
>
> So after hashing it out in various groups the overwhelming consensus is
> to do the parsing part now and leave the generation part for later if
> the need materializes.
>
> In the course of the discussions I realized that I don't understand your
> "not being able to use defaults for properties" comment. Could you
> elaborate?
Sure. I was referring to this snippet in my original spec:
> Mysql_user { host => "webserver" }
> mysql_user { [ "frob", "frub" ]: }
> # expands to 'frob@webserver' and 'frub@webserver'
>
> mysql_user {
> "frob": host => "web1";
> "frob": host => "web2";
> "frob": host => "web3";
> }
>
> The last stanza shows how this can be used to confusing effects.
Specifically, I am worried that Mysql_user[frob] becomes illegal and/or
confusing. Thinking more about it, I come to the realisation that
Mysql_user[frob@web1] can use the patterns on the type to create a valid
reference to a single resource.
Luke Kanies
> All,
>
> How does this work in the case of multi-party development?
>
> For instance, you now have the Module Forge (woo), but I'm sure you're
> going to get two modules with a 'service { "mysql"' or somesuch.
>
> I know that it's easy to say "just be careful", but throwing up the
> overworked puny human SA defense, not conflicting on duplicate names
> is going to cause a massive amount of headache in my opinion.
>
> How else could this be solved so that people are not bitten by
> conflicts? Warnings perhaps?
I think there's a misunderstanding - we're not removing the concept of
resource uniqueness, just how it's implemented.
That is, currently the only way Puppet can know if a resource is
unique is to require that the type and title are a unique combination,
but this works poorly for some types - e.g., ports (in /etc/services)
are not uniquely identifiable without knowing the name, port, and
protocol (mostly through bad design, IMO, but it's too late now).
What David and Markus are talking about is enhancing the concept of
uniqueness to enable Puppet to uniquely identify resources by a
combination of multiple attributes.
Thus, you'll still get failures if two modules specify the same
resource - it's just the mechanism for validating this is being
enhanced a bit.
--
He attacked everything in life with a mix of extraordinary genius and
naive incompetence, and it was often difficult to tell which was
which. -- Douglas Adams
Luke Kanies
I think in this case we need a clear definition of how this should
behave, but the specific definition doesn't actually matter that much.
I'd tend toward Mysql_user[frob] in this case essentially translating
to the three resources; that is, requiring that reference is
functionally equivalent to requiring all three of the actual user
instances.
I'd be just as happy for this to fail if it referred to more than one
resource. That is, if you wanted to depend on Mysql_user[frob], you'd
have to change your code to this:
mysql_user {
frob_web1: name => frob, host => web1;
frob_web2: name => frob, host => web2;
...
}
Then the reference would have to be Mysql_user[frob_web1].
This is obviously the easiest short-term solution and is probably what
I would go with to start.
--
It is said that power corrupts, but actually it's more true that power
attracts the corruptible. The sane are usually attracted by other things
than power. -- David Brin
Trevor Vaughan
Thanks for the clarification, sounds great to me!
Trevor
David Schmitt
>> that Mysql_user[frob@web1] can use the patterns on the type to create
>> a valid reference to a single resource.
>
> I think in this case we need a clear definition of how this should
> behave, but the specific definition doesn't actually matter that much.
>
> I'd tend toward Mysql_user[frob] in this case essentially translating to
> the three resources; that is, requiring that reference is functionally
> equivalent to requiring all three of the actual user instances.
>
> I'd be just as happy for this to fail if it referred to more than one
> resource. That is, if you wanted to depend on Mysql_user[frob], you'd
> have to change your code to this:
>
> mysql_user {
> frob_web1: name => frob, host => web1;
> frob_web2: name => frob, host => web2;
> ...
> }
>
> Then the reference would have to be Mysql_user[frob_web1].
>
> This is obviously the easiest short-term solution and is probably what I
> would go with to start.
+1.
Later this can integrate the collection query syntax to reference more
resources like Mysql_user[|user == 'frob'|] or something.