Queuing some server-side operations
Luke Kanies
As mentioned in an email yesterday, I'm working with someone to add a
queueing service to Puppet so that some server-side operations are
queued and executed as cpu time is available - generally those that
aren't on the critical path but particularly the storeconfigs save
operation.
The goal here is that the client wouldn't have to wait for the server
to finish storing the catalog before it got its copy of the catalog;
instead, the server would just put the catalog into the queue, and a
queue reader would come behind and process as time was available. The
only other operation I can think of where this makes sense right now
is reporting, but I expect we'll have others over time.
The big benefit here, of course, is that you get configuration storage
out of the critical path for returning catalogs to clients, and this
has the benefit of making the client queries shorter and thus less
likely to overlap. It also means that if you get a big rush of client
connections, we can queue up the more expensive operations, and even
if storeconfigs eventually becomes cheap, it would still make sense to
queue anything that wasn't on the critical path.
So, I'm looking for input on what people think the right tools and
architecture are for this, and just generally how we should go about
it. I don't want to have a six week discussion on it, but I do want
to make sure we get as much feedback as possible during development,
and we'll be doing normal code review and publishing code as often as
possible.
This work is being funded by a client, but I'm trying to make the
development process as open as it would normally be. I haven't yet
found the right balance there, but certainly pushing to more openness
is the right choice.
Ethan Rowe, who is doing most of the queueing development, will be
following up on this email with more detail on the architecture and
tools he's picked.
--
Commit suicide. A hundred thousand lemmings can't be wrong.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Paul Nasrat
> queueing service to Puppet so that some server-side operations are
> queued and executed as cpu time is available - generally those that
> aren't on the critical path but particularly the storeconfigs save
> operation.
Sounds interesting.
> The goal here is that the client wouldn't have to wait for the server
> to finish storing the catalog before it got its copy of the catalog;
> instead, the server would just put the catalog into the queue, and a
> queue reader would come behind and process as time was available. The
> only other operation I can think of where this makes sense right now
> is reporting, but I expect we'll have others over time.
Do we have a summary of current performance and some perfomance tests
so we can objectively measure improvements? If so getting them visibly
graphing from hudson builds of that branch would be awesome.
> The big benefit here, of course, is that you get configuration storage
> out of the critical path for returning catalogs to clients, and this
> has the benefit of making the client queries shorter and thus less
> likely to overlap. It also means that if you get a big rush of client
> connections, we can queue up the more expensive operations, and even
> if storeconfigs eventually becomes cheap, it would still make sense to
> queue anything that wasn't on the critical path.
Do we have a set of requirements that we need both for functionality
and for ie do we need persistence, etc)
> So, I'm looking for input on what people think the right tools and
> architecture are for this, and just generally how we should go about
> it. I don't want to have a six week discussion on it, but I do want
> to make sure we get as much feedback as possible during development,
> and we'll be doing normal code review and publishing code as often as
> possible.
Without the requirements it's hard to say exactly, I'd probably look
at starling/workling, beanstalk, and nanite amongst others.
If we have good criteria we could do some time boxed evaluation to
choose best fit for puppet at the moment, but ensuring that we
implement it in a pluggable way (eg choose a pluggable transport or
standard such as amqp, thrift, protocol buffers )
Paul
Larry Ludwig
On Mar 18, 2009, at 2:01 PM, Paul Nasrat wrote:
>
>> I'm working with someone to add a
>> queueing service to Puppet so that some server-side operations are
>> queued and executed as cpu time is available - generally those that
>> aren't on the critical path but particularly the storeconfigs save
>> operation.
>
What will be the determination of when CPU time is available? A
configuration option, like what's in exim when load is above X only
queue? What's considered a 'busy' puppetmaster?
>
>
>> The goal here is that the client wouldn't have to wait for the server
>> to finish storing the catalog before it got its copy of the catalog;
>> instead, the server would just put the catalog into the queue, and a
>> queue reader would come behind and process as time was available.
>> The
>> only other operation I can think of where this makes sense right now
>> is reporting, but I expect we'll have others over time.
>
> Do we have a summary of current performance and some perfomance tests
> so we can objectively measure improvements? If so getting them visibly
> graphing from hudson builds of that branch would be awesome.
>
More specifically what's the most expensive CPU and/or memory
operations when a node connects and process? Is there a break down in
that process? Of that process what's needed immediately by the client
and what can wait?
In the my case queueing reporting wouldn't help me that much since
we use a centralized syslog to do reporting.
-L
Ethan Rowe
[snip]
> So, I'm looking for input on what people think the right tools and
> architecture are for this, and just generally how we should go about
> it. I don't want to have a six week discussion on it, but I do want
> to make sure we get as much feedback as possible during development,
> and we'll be doing normal code review and publishing code as often as
> possible.
>
> This work is being funded by a client, but I'm trying to make the
> development process as open as it would normally be. I haven't yet
> found the right balance there, but certainly pushing to more openness
> is the right choice.
>
> Ethan Rowe, who is doing most of the queueing development, will be
> following up on this email with more detail on the architecture and
> tools he's picked.
Howdy.
The idea at this point is:
* implement an abstract terminus for queuing, to be used as the caching
terminus for the catalog. So, terminus_class => :queue
* implement a simple queue client registry to back the abstract
terminus, such that in an executable's configuration, we can specify
what type of messaging queue client we want to use.
* provide some message queue client wrappers for use; at present, we're
looking at Stomp::Client for starters, and possibly a wrapper for the
memcached client as well; each client wrapper gets registered with a
simple symbolic name, much like the termini themselves (:stomp, for
instance)
* implement a simple new daemon (we're tentatively calling it puppetqd
for working discussion) that: subscribes to the relevant message queue
for the catalog class; receives catalog objects to store (as a message,
which it deserializes) from the queue; stores the catalog objects to the
database.
This workflow fits into the Puppet indirector idiom pretty well. We're
trying to design the interaction between the queue terminus and the
underlying message broker (the middleware that provides the queuing
services) to maximize extensibility, so we can over time support any
number of different messaging solutions.
The message queue topic is very well-covered in free and non-free
software ecosystems. It was not my intent to implement a custom queuing
solution for Puppet itself; there are too many simple, proven, effective
options out there, and we're collectively better-served if sysadmins in
charge of deployments can choose for themselves the messaging solution
that best meets their needs.
Our initial work, using Stomp::Client as the messaging client (and
therefore using the Stomp protocol), focuses on ActiveMQ for the
messaging piece.
Stomp::Client: http://rubyforge.org/projects/stomp/
The Stomp protocol: http://stomp.codehaus.org/Protocol
ActiveMQ: http://activemq.apache.org/
ActiveMQ is pretty popular and is quite easy to get running with a basic
configuration. These were both factors in the ActiveMQ choice for our
initial focus. Critically, you can configure it to be backed by any
number of standard RDBMSes for message persistence, which was a concern
for the client in question.
http://activemq.apache.org/jdbc-support.html
Stomp::Client and ActiveMQ is just a start. We can potentially use
Ruby's StompServer, which is pretty lightweight and has some persistence
options, but appears to be possibly stalled (not much activity since
late 2007).
Furthermore, Stomp should be usable by other traditional JMS solutions.
On the lighter-weight side: we can set up a client for the memcachced
protocol-based solutions like Starling and Sparrow. That should be
relatively easy and give sysadmins more options in their deployments.
Ultimately, the resulting workflow should be (Luke may correct me here):
* in the client, the catalog class uses :rest indirection, meaning it
performs a find() RESTfully through puppetmasterd
* puppetmasterd indirects the same class to the :compiler terminus, so
the internal find() call in puppetmasterd goes to the parser to do the
catalog compilation
* when the catalog object is compiled and is ready for returning to the
client, the caching step kicks in and will indirect to the :queue terminus.
* the :queue terminus serializes the object to YAML, gets the right
message queue client object, and sends the object YAML as a message to a
queue named for the indirection itself; at this point, puppetmasterd's
work is done and it returns to the client with no further delay
* puppetqd runs as a separate process, and is listening to the relevant
queue (though the degree to which it uses the indirection system
natively for queue subscription needs to be worked out a bit more); as
messages come in, it serializes them and stores them to the database,
using a new database indirection in development for a different thread.
I hope this makes sense and seems reasonable to folks. Fire away!
Thanks.
- Ethan
--
Ethan Rowe
End Point Corporation
et...@endpoint.com
James Turnbull
> * puppetqd runs as a separate process, and is listening to the relevant
> queue (though the degree to which it uses the indirection system
> natively for queue subscription needs to be worked out a bit more); as
> messages come in, it serializes them and stores them to the database,
> using a new database indirection in development for a different thread.
I think this is filled with awesome (to steal a Shafer-ism).
My only comments are mostly end-user operability ones - which I guess
people should realise now is my focus because I often write the doco :)
1. Configuration should all run out of puppet.conf - no new files for
this (I am assuming that was the intent).
2. I like options - making a variety of clients available would be good
and making sure adding new clients is easy and well documented is
crucial to this I think - people's opinions on messenging and message
buses will vary greatly.
3. This shouldn't be mandatory - it should be an optional extra rather
than the default to have queuing enabled.
4. What happens with the queue daemon breaks in the process?
5. It'd be good to see both architectures/workflows documented - I
think - other than talking to Luke - that's the first time I've seen the
new indirection workflow articulated in words on screen as it were. I
see a whole new world full of diagrams and workflows... :)
Cheers
James Turnbull
--
Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)
Ethan Rowe
> Ethan Rowe wrote:
>> * puppetqd runs as a separate process, and is listening to the relevant
>> queue (though the degree to which it uses the indirection system
>> natively for queue subscription needs to be worked out a bit more); as
>> messages come in, it serializes them and stores them to the database,
>> using a new database indirection in development for a different thread.
>
> I think this is filled with awesome (to steal a Shafer-ism).
>
> My only comments are mostly end-user operability ones - which I guess
> people should realise now is my focus because I often write the doco :)
>
> 1. Configuration should all run out of puppet.conf - no new files for
> this (I am assuming that was the intent).
Since the message service particulars will need to be shared by at least
daemons (puppetmaster, puppetqd), it makes sense to place those
particulars in a common location.
Our intent is to make this as simple as reasonably possible. Though to
be honest, our approach to the configuration arrangement is "get
something working now, improve it when we have the new logic working." :)
> 2. I like options - making a variety of clients available would be good
> and making sure adding new clients is easy and well documented is
> crucial to this I think - people's opinions on messenging and message
> buses will vary greatly.
The working design absolutely takes these considerations to heart.
Yesterday I did some fledgling implementation for the abstract terminus,
the client registry, etc., and I'm taking the trouble to document as I go.
Whether my documentation is comprehensible is another matter, but the
effort has been and will continue to be there.
> 3. This shouldn't be mandatory - it should be an optional extra rather
> than the default to have queuing enabled.
Right. My understanding is that the default behavior should be for
catalog storage to act as it acts right now: just do the storage to the
database right from puppetmasterd. Use of the queue option is something
to determine on a per-deployment basis.
> 4. What happens with the queue daemon breaks in the process?
To be sure I understand what you're asking, did you perhaps mean "when
the queue daemon breaks" rather than "with"?
> 5. It'd be good to see both architectures/workflows documented - I
> think - other than talking to Luke - that's the first time I've seen the
> new indirection workflow articulated in words on screen as it were. I
> see a whole new world full of diagrams and workflows... :)
A diagram would presumably help quite a lot. Words can express the
design, but the abstractions are of the sort that will tend to lose a
reader.
James Turnbull
>> 1. Configuration should all run out of puppet.conf - no new files for
>> this (I am assuming that was the intent).
>
> Since the message service particulars will need to be shared by at least
> daemons (puppetmaster, puppetqd), it makes sense to place those
> particulars in a common location.
Agreed - puppet.conf seems the logical choice.
> The working design absolutely takes these considerations to heart.
> Yesterday I did some fledgling implementation for the abstract terminus,
> the client registry, etc., and I'm taking the trouble to document as I go.
>
> Whether my documentation is comprehensible is another matter, but the
> effort has been and will continue to be there.
Let me know and I can work on it. It's probably what I can best contribute.
>> 4. What happens with the queue daemon breaks in the process?
>
> To be sure I understand what you're asking, did you perhaps mean "when
> the queue daemon breaks" rather than "with"?
Yes - sorry s/with/when/
> A diagram would presumably help quite a lot. Words can express the
> design, but the abstractions are of the sort that will tend to lose a
> reader.
Again - if someone does a rough one I can polish if required. It'd
probably be an excuse for me to do a bunch for various processes. I had
intended to do such for the new wiki.
Cheers
James
Steven Jenkins
...
> A diagram would presumably help quite a lot. Words can express the
> design, but the abstractions are of the sort that will tend to lose a
> reader.
Luke recently pointed out :
http://reductivelabs.com/trac/puppet/wiki/ParsingArchitecture, which
should probably be the starting point for a diagram.
Step 9 will be changed to be configurable at run time, and if queuing is
chosen, the diagram will be (apologies for bad ASCII art here):
Compiler -> Queue broker -> puppetqd -> Database
Where 'Queue broker', as mentioned already, could be one of several options.
Steven Jenkins
End Point Corporation
Luke Kanies
>
>> As mentioned in an email yesterday, I'm working with someone to add a
>> queueing service to Puppet so that some server-side operations are
>> queued and executed as cpu time is available - generally those that
>> aren't on the critical path but particularly the storeconfigs save
>> operation.
>
> Sounds interesting.
>
>> The goal here is that the client wouldn't have to wait for the server
>> to finish storing the catalog before it got its copy of the catalog;
>> instead, the server would just put the catalog into the queue, and a
>> queue reader would come behind and process as time was available.
>> The
>> only other operation I can think of where this makes sense right now
>> is reporting, but I expect we'll have others over time.
>
> Do we have a summary of current performance and some perfomance tests
> so we can objectively measure improvements? If so getting them visibly
> graphing from hudson builds of that branch would be awesome.
That would be great to have, but no, we have neither that nor even an
objective idea of performance.
We probably need to come up with a couple of standard configurations
with a given amount of complexity and then start tracking the times
those things take. We could stick entirely to 'notify' resources, so
that we're not hitting the disk at all.
Anyone interested in taking this up?
>
>> The big benefit here, of course, is that you get configuration
>> storage
>> out of the critical path for returning catalogs to clients, and this
>> has the benefit of making the client queries shorter and thus less
>> likely to overlap. It also means that if you get a big rush of
>> client
>> connections, we can queue up the more expensive operations, and even
>> if storeconfigs eventually becomes cheap, it would still make sense
>> to
>> queue anything that wasn't on the critical path.
>
> Do we have a set of requirements that we need both for functionality
> and for ie do we need persistence, etc)
Ethan, Steven, and I have developed largely complete requirements for
functionality, but, erm, no, we don't have a literal requirements
document or anything. Yes, persistence is required. Hopefully Ethan
can send out a summary of the requirements list he's developed.
>
>> So, I'm looking for input on what people think the right tools and
>> architecture are for this, and just generally how we should go about
>> it. I don't want to have a six week discussion on it, but I do want
>> to make sure we get as much feedback as possible during development,
>> and we'll be doing normal code review and publishing code as often as
>> possible.
>
> Without the requirements it's hard to say exactly, I'd probably look
> at starling/workling, beanstalk, and nanite amongst others.
>
> If we have good criteria we could do some time boxed evaluation to
> choose best fit for puppet at the moment, but ensuring that we
> implement it in a pluggable way (eg choose a pluggable transport or
> standard such as amqp, thrift, protocol buffers )
Yeah, the goal is to make it relatively pluggable, but it'll certainly
be a whole new category of functionality for me, so I know at least
I'll take some time to get up to speed.
--
Brand's Shortcut:
The only way to predict the future is to make sure it stays
exactly the same as the present.
Luke Kanies
>
>
> On Mar 18, 2009, at 2:01 PM, Paul Nasrat wrote:
>
>>
>>> I'm working with someone to add a
>>> queueing service to Puppet so that some server-side operations are
>>> queued and executed as cpu time is available - generally those that
>>> aren't on the critical path but particularly the storeconfigs save
>>> operation.
>>
>
> What will be the determination of when CPU time is available? A
> configuration option, like what's in exim when load is above X only
> queue? What's considered a 'busy' puppetmaster?
The queue processor will run either on a separate node or as a niced
process on the master, so the answer should be relatively
straightforward from there.
>>> The goal here is that the client wouldn't have to wait for the
>>> server
>>> to finish storing the catalog before it got its copy of the catalog;
>>> instead, the server would just put the catalog into the queue, and a
>>> queue reader would come behind and process as time was available.
>>> The
>>> only other operation I can think of where this makes sense right now
>>> is reporting, but I expect we'll have others over time.
>>
>> Do we have a summary of current performance and some perfomance tests
>> so we can objectively measure improvements? If so getting them
>> visibly
>> graphing from hudson builds of that branch would be awesome.
>>
>
> More specifically what's the most expensive CPU and/or memory
> operations when a node connects and process? Is there a break down in
> that process? Of that process what's needed immediately by the client
> and what can wait?
Compiling and storage are definitely the most expensive CPU-wise, and
file serving is probably the most expensive memory wise.
The compiled catalog is needed immediately by the client, but the
catalog storage isn't needed on a particular timeframe. Fileserving,
of course, is also needed immediately.
>
> In the my case queueing reporting wouldn't help me that much since
> we use a centralized syslog to do reporting.
Yeah, reporting is generally a pretty inexpensive operation and it's
probably not worth queueing in most cases; I threw it in there just to
make the point that this should be a general process.
--
There are three social classes in America: upper middle class, middle
class, and lower middle class. --Judith Martin
Sam Rowe
>
> Hi all,
>
> As mentioned in an email yesterday, I'm working with someone to add a
> queueing service to Puppet so that some server-side operations are
> queued and executed as cpu time is available - generally those that
> aren't on the critical path but particularly the storeconfigs save
> operation.
>
> The goal here is that the client wouldn't have to wait for the server
> to finish storing the catalog before it got its copy of the catalog;
> instead, the server would just put the catalog into the queue, and a
> queue reader would come behind and process as time was available. The
> only other operation I can think of where this makes sense right now
> is reporting, but I expect we'll have others over time.
Would this help with the splay problem we discussed recently?
Luke Kanies
The queueing would only help with reducing client contention when the
contention was caused by StoreConfigs, unfortunately.
Although...
I could see having a ping-back model, where a client sends a request
for a compilation but then just sleeps, and when the config is
compiled the server pushes the configuration to the client.
This is obviously a far cry from the current architecture, but it's
mostly a reconfiguration, once you have the queueing, rather than a
fundamental shift.
--
Getting caught is the mother of invention. --Robert Byrne
Sam Rowe
> The queueing would only help with reducing client contention when the
> contention was caused by StoreConfigs, unfortunately.
>
> Although...
>
> I could see having a ping-back model, where a client sends a request
> for a compilation but then just sleeps, and when the config is
> compiled the server pushes the configuration to the client.
>
> This is obviously a far cry from the current architecture, but it's
> mostly a reconfiguration, once you have the queueing, rather than a
> fundamental shift.
Seems like chocolate and peanut butter to me. :)
Ethan Rowe
I'm a newcomer, so feel free to ignore me. But, if RESTful design has
become an increasingly important consideration in interactions between
client and server (and it's my impression that it has), I think you
might find that it fits better in this case to:
* have the server issue a 202 Accepted response, with an response entity
containing: possibly a URL to periodically check for results; possibly
an estimate of time before the new resource is available
* have the client periodically consult the relevant URL until the data
is available
It's somewhat annoying to consider the possibility of having a client
repeatedly, possibly fruitlessly, checking in with a server. But it may
be easier to scale that approach and fit it within the current
architecture than having bidirectional data flow of the sort described.
Luke Kanies
You're right - that's much simpler, doesn't require the ability to
contact the client, and is low-cost.
Good point; thanks.
--
You don't learn anything the second time you're kicked by a mule.
-- Anonymous Texan
Luke Kanies
> Ethan Rowe wrote:
>> * puppetqd runs as a separate process, and is listening to the
>> relevant
>> queue (though the degree to which it uses the indirection system
>> natively for queue subscription needs to be worked out a bit more);
>> as
>> messages come in, it serializes them and stores them to the database,
>> using a new database indirection in development for a different
>> thread.
>
> I think this is filled with awesome (to steal a Shafer-ism).
>
> My only comments are mostly end-user operability ones - which I guess
> people should realise now is my focus because I often write the
> doco :)
>
> 1. Configuration should all run out of puppet.conf - no new files for
> this (I am assuming that was the intent).
>
> 2. I like options - making a variety of clients available would be
> good
> and making sure adding new clients is easy and well documented is
> crucial to this I think - people's opinions on messenging and message
> buses will vary greatly.
>
> 3. This shouldn't be mandatory - it should be an optional extra
> rather
> than the default to have queuing enabled.
Yep, these are all in the plan, although the pluggability might suffer
a bit in the first versions, not surprisingly.
>
> 4. What happens with the queue daemon breaks in the process?
"Breaks" would need to be defined a bit, but most likely things
couldn't be queued, and thus would be lost.
>
> 5. It'd be good to see both architectures/workflows documented - I
> think - other than talking to Luke - that's the first time I've seen
> the
> new indirection workflow articulated in words on screen as it were. I
> see a whole new world full of diagrams and workflows... :)
Steven posted this picture already:
http://reductivelabs.com/images/Puppet_Parsing_Architecture_0.25.x.png
And here's what the queueing does:
http://reductivelabs.com/images/Puppet_Parsing_Architecture_With_Queueing_0.25.x.png
Do they help?
--
He attacked everything in life with a mix of extraordinary genius and
naive incompetence, and it was often difficult to tell which was
which. -- Douglas Adams
Ohad Levy
Steven posted this picture already:
http://reductivelabs.com/images/Puppet_Parsing_Architecture_0.25.x.png
And here's what the queueing does:
http://reductivelabs.com/images/Puppet_Parsing_Architecture_With_Queueing_0.25.x.png
Do they help?
Yes, a few questions:
1. Can we assume that non queued request are RO requests?
2. Can the queued request use another database server?
The reason I'm asking, is that I'm looking for a way to use storeconfig in a large multi location environment, until now it hasn't been possible because of the dependency and latency to the database.
if I could use a local replica on the puppetmaster for exported resources, which would be a RO db replica, and store all new information using the queue to a db master, I think that would help a lot ;)
my 2 cents.
Ohad
Nigel Kersten
Storeconfigs just hasn't been practical for us, although 0.24.8 is
looking significantly better on a recent linux distro.
>
> my 2 cents.
> Ohad
>
>
> >
>
--
Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.
Luke Kanies
>
> Steven posted this picture already:
>
> http://reductivelabs.com/images/Puppet_Parsing_Architecture_0.25.x.png
>
> And here's what the queueing does:
>
> http://reductivelabs.com/images/Puppet_Parsing_Architecture_With_Queueing_0.25.x.png
>
> Do they help?
>
> Yes, a few questions:
>
> 1. Can we assume that non queued request are RO requests?
Yes.
>
> 2. Can the queued request use another database server?
I would assume so. One of the benefits of using a "real" queue system
is that there's already a wide range of expertise on it and you can
configure it however you want.
>
> The reason I'm asking, is that I'm looking for a way to use
> storeconfig in a large multi location environment, until now it
> hasn't been possible because of the dependency and latency to the
> database.
> if I could use a local replica on the puppetmaster for exported
> resources, which would be a RO db replica, and store all new
> information using the queue to a db master, I think that would help
> a lot ;)
Makes a lot of sense. I expect you'd be doing some experimentation
and ground-breaking, but it should be relatively straightforward once
you grok the queueing system.
--
However beautiful the strategy, you should occasionally look at the
results. -- Sir Winston Churchill
David Lutterkort
> I'm a newcomer, so feel free to ignore me. But, if RESTful design has
> become an increasingly important consideration in interactions between
> client and server (and it's my impression that it has), I think you
> might find that it fits better in this case to:
> * have the server issue a 202 Accepted response, with an response entity
> containing: possibly a URL to periodically check for results; possibly
> an estimate of time before the new resource is available
> * have the client periodically consult the relevant URL until the data
> is available
I've spent some time recently banging my head against the limitations of
REST for oVirt - the assumption that REST makes that everything is a
resource on which you execute a fixed number of actions is really nice
if you are dealing with something that looks like a resource fairly
naturally, but gets very annoying when that assumption is broken (like
VM's which can be started, stopped, shut down etc.)
We are in the process of implementing API's for oVirt now based on
QMF[1], a small layer on top of Qpid, an open source message bus; the
programming model of QMF is pretty nice since it abstracts all the
messaging etc. away nicely. From a coding point of view, you are dealing
with objects, call methods on them etc. and QMF/Qpid take care of
distributing changes throughout the messaing infrastructure.
Architecturally, everybody is a client connecting to a central broker,
even the server on which the objects really live; the logical client
gets a view of those objects, calling methods on them causes some method
on the server to be executed etc. We're using that to connect various
virtualized hosts together - each host is the 'server' for the VM's that
are running to it, and a client using the QMF API gets to see a
collection of all the VM's on all hosts, can start/stop etc. them
without the need to connect to individual hosts.
I am mostly mentioning this because it's an important new way of
connecting infrastructure together, and radically different from all the
other approaches I have seen. If anybody's interested in diving into
this a little more in the context of puppet, feel free to contact me. A
rough idea would be to have each client serve its facts and its current
catalog over QMF and have the puppetmaster expose appropriate methods to
retrieve an updated catalog.
Of course, this is not directly connected to the issue at hand (speeding
up storeconfigs), but it's got to do with messaging ;)
David
[1] http://qpid.apache.org/qpid-management-framework.html
Clay McClure
> I've spent some time recently banging my head against the limitations of
> REST for oVirt - the assumption that REST makes that everything is a
> resource on which you execute a fixed number of actions is really nice
> if you are dealing with something that looks like a resource fairly
> naturally, but gets very annoying when that assumption is broken (like
> VM's which can be started, stopped, shut down etc.)
I can relate. With this shiny REST hammer in our toolboxes, we want
every problem to look like a nail. But while you *can* drive a screw
with a hammer (ever find yourself without a screwdriver at just the
wrong time?), it's much easier if you use the right tool for the job.
REST is great for resource-oriented programming, but is awkward for
activity-oriented programming. This article breaks down the difference
pretty well:
http://www.ibm.com/developerworks/webservices/library/ws-restvsoap/
A hybrid REST-RPC approach seems most appropriate for your needs.
Model the VMs and their attributes as resources, but model the actions
you might want to perform on a VM (install, configure, start, stop,
suspend, deinstall) as methods (XML-RPC, SOAP, roll your own, etc).
> We are in the process of implementing API's for oVirt now based on
> QMF[1], a small layer on top of Qpid, an open source message bus; the
> programming model of QMF is pretty nice since it abstracts all the
> messaging etc. away nicely. From a coding point of view, you are dealing
> with objects, call methods on them etc. and QMF/Qpid take care of
> distributing changes throughout the messaing infrastructure.
I like how using a message bus drives the design toward an
asynchronous, event-based model. That feels right to me. I also like
the ability to snoop on the channels to see what the various
components are doing :) This is particularly easy when using Jabber
as the message transport, as in Vertebra.
I'm not familiar with QMF, but the XML schema definition seems like it
might make upgrades painful: will you have to contemporaneously
upgrade all components plugged into the bus whenever the schema
changes? A schema-free design (the messaging equivalent of duck
typing) might be easier to maintain.
> Architecturally, everybody is a client connecting to a central broker,
> even the server on which the objects really live; the logical client
> gets a view of those objects, calling methods on them causes some method
> on the server to be executed etc. We're using that to connect various
> virtualized hosts together - each host is the 'server' for the VM's that
> are running to it, and a client using the QMF API gets to see a
> collection of all the VM's on all hosts, can start/stop etc. them
> without the need to connect to individual hosts.
Some hypothetical questions for you -- no need to reply since this may
already be off-topic:
If you send one global start command to the bus, how do you know when
all VMs have in fact started? Even if you don't have to send commands
to individual hosts, you still need to monitor them individually.
What happens when a host is down or otherwise disconnected from the
message bus when the command to start/stop/delete/etc all VMs goes
out? Unless Qpid guarantees delivery, the VM state on that host may be
out of sync with the rest of the cluster when it comes back up.
I'm a newbie on the list -- apologies for butting in :)
Clay
Luke Kanies
>
> On Thu, 2009-03-19 at 23:06 -0400, Ethan Rowe wrote:
>> I'm a newcomer, so feel free to ignore me. But, if RESTful design
>> has
>> become an increasingly important consideration in interactions
>> between
>> client and server (and it's my impression that it has), I think you
>> might find that it fits better in this case to:
>> * have the server issue a 202 Accepted response, with an response
>> entity
>> containing: possibly a URL to periodically check for results;
>> possibly
>> an estimate of time before the new resource is available
>> * have the client periodically consult the relevant URL until the
>> data
>> is available
>
> I've spent some time recently banging my head against the
> limitations of
> REST for oVirt - the assumption that REST makes that everything is a
> resource on which you execute a fixed number of actions is really nice
> if you are dealing with something that looks like a resource fairly
> naturally, but gets very annoying when that assumption is broken (like
> VM's which can be started, stopped, shut down etc.)
Yeah, this is why the process that retrieves items from the queue, for
instance, isn't using a REST model for doing so.
It just so happens that *most* of Puppet's network services are easy
to model with resources, but clearly not all - 'status' doesn't make
much sense, for instance.
This messaging project is limited enough that I'm skeptical it's going
to have a big impact on Puppet pervavisely using messaging suddenly,
but I've always thought messaging was a much more appropriate means of
sending a lot of information, especially reports.
--
A gentleman is a man who can play the accordion but doesn't. --Unknown