As mentioned in an email yesterday, I'm working with someone to add a queueing service to Puppet so that some server-side operations are queued and executed as cpu time is available - generally those that aren't on the critical path but particularly the storeconfigs save operation.
The goal here is that the client wouldn't have to wait for the server to finish storing the catalog before it got its copy of the catalog; instead, the server would just put the catalog into the queue, and a queue reader would come behind and process as time was available. The only other operation I can think of where this makes sense right now is reporting, but I expect we'll have others over time.
The big benefit here, of course, is that you get configuration storage out of the critical path for returning catalogs to clients, and this has the benefit of making the client queries shorter and thus less likely to overlap. It also means that if you get a big rush of client connections, we can queue up the more expensive operations, and even if storeconfigs eventually becomes cheap, it would still make sense to queue anything that wasn't on the critical path.
So, I'm looking for input on what people think the right tools and architecture are for this, and just generally how we should go about it. I don't want to have a six week discussion on it, but I do want to make sure we get as much feedback as possible during development, and we'll be doing normal code review and publishing code as often as possible.
This work is being funded by a client, but I'm trying to make the development process as open as it would normally be. I haven't yet found the right balance there, but certainly pushing to more openness is the right choice.
Ethan Rowe, who is doing most of the queueing development, will be following up on this email with more detail on the architecture and tools he's picked.
-- Commit suicide. A hundred thousand lemmings can't be wrong. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> As mentioned in an email yesterday, I'm working with someone to add a > queueing service to Puppet so that some server-side operations are > queued and executed as cpu time is available - generally those that > aren't on the critical path but particularly the storeconfigs save > operation.
Sounds interesting.
> The goal here is that the client wouldn't have to wait for the server > to finish storing the catalog before it got its copy of the catalog; > instead, the server would just put the catalog into the queue, and a > queue reader would come behind and process as time was available. The > only other operation I can think of where this makes sense right now > is reporting, but I expect we'll have others over time.
Do we have a summary of current performance and some perfomance tests so we can objectively measure improvements? If so getting them visibly graphing from hudson builds of that branch would be awesome.
> The big benefit here, of course, is that you get configuration storage > out of the critical path for returning catalogs to clients, and this > has the benefit of making the client queries shorter and thus less > likely to overlap. It also means that if you get a big rush of client > connections, we can queue up the more expensive operations, and even > if storeconfigs eventually becomes cheap, it would still make sense to > queue anything that wasn't on the critical path.
Do we have a set of requirements that we need both for functionality and for ie do we need persistence, etc)
> So, I'm looking for input on what people think the right tools and > architecture are for this, and just generally how we should go about > it. I don't want to have a six week discussion on it, but I do want > to make sure we get as much feedback as possible during development, > and we'll be doing normal code review and publishing code as often as > possible.
Without the requirements it's hard to say exactly, I'd probably look at starling/workling, beanstalk, and nanite amongst others.
If we have good criteria we could do some time boxed evaluation to choose best fit for puppet at the moment, but ensuring that we implement it in a pluggable way (eg choose a pluggable transport or standard such as amqp, thrift, protocol buffers )
>> I'm working with someone to add a >> queueing service to Puppet so that some server-side operations are >> queued and executed as cpu time is available - generally those that >> aren't on the critical path but particularly the storeconfigs save >> operation.
What will be the determination of when CPU time is available? A configuration option, like what's in exim when load is above X only queue? What's considered a 'busy' puppetmaster?
>> The goal here is that the client wouldn't have to wait for the server >> to finish storing the catalog before it got its copy of the catalog; >> instead, the server would just put the catalog into the queue, and a >> queue reader would come behind and process as time was available. >> The >> only other operation I can think of where this makes sense right now >> is reporting, but I expect we'll have others over time.
> Do we have a summary of current performance and some perfomance tests > so we can objectively measure improvements? If so getting them visibly > graphing from hudson builds of that branch would be awesome.
More specifically what's the most expensive CPU and/or memory operations when a node connects and process? Is there a break down in that process? Of that process what's needed immediately by the client and what can wait?
In the my case queueing reporting wouldn't help me that much since we use a centralized syslog to do reporting.
> So, I'm looking for input on what people think the right tools and > architecture are for this, and just generally how we should go about > it. I don't want to have a six week discussion on it, but I do want > to make sure we get as much feedback as possible during development, > and we'll be doing normal code review and publishing code as often as > possible.
> This work is being funded by a client, but I'm trying to make the > development process as open as it would normally be. I haven't yet > found the right balance there, but certainly pushing to more openness > is the right choice.
> Ethan Rowe, who is doing most of the queueing development, will be > following up on this email with more detail on the architecture and > tools he's picked.
Howdy.
The idea at this point is: * implement an abstract terminus for queuing, to be used as the caching terminus for the catalog. So, terminus_class => :queue * implement a simple queue client registry to back the abstract terminus, such that in an executable's configuration, we can specify what type of messaging queue client we want to use. * provide some message queue client wrappers for use; at present, we're looking at Stomp::Client for starters, and possibly a wrapper for the memcached client as well; each client wrapper gets registered with a simple symbolic name, much like the termini themselves (:stomp, for instance) * implement a simple new daemon (we're tentatively calling it puppetqd for working discussion) that: subscribes to the relevant message queue for the catalog class; receives catalog objects to store (as a message, which it deserializes) from the queue; stores the catalog objects to the database.
This workflow fits into the Puppet indirector idiom pretty well. We're trying to design the interaction between the queue terminus and the underlying message broker (the middleware that provides the queuing services) to maximize extensibility, so we can over time support any number of different messaging solutions.
The message queue topic is very well-covered in free and non-free software ecosystems. It was not my intent to implement a custom queuing solution for Puppet itself; there are too many simple, proven, effective options out there, and we're collectively better-served if sysadmins in charge of deployments can choose for themselves the messaging solution that best meets their needs.
Our initial work, using Stomp::Client as the messaging client (and therefore using the Stomp protocol), focuses on ActiveMQ for the messaging piece.
ActiveMQ is pretty popular and is quite easy to get running with a basic configuration. These were both factors in the ActiveMQ choice for our initial focus. Critically, you can configure it to be backed by any number of standard RDBMSes for message persistence, which was a concern for the client in question.
Stomp::Client and ActiveMQ is just a start. We can potentially use Ruby's StompServer, which is pretty lightweight and has some persistence options, but appears to be possibly stalled (not much activity since late 2007).
Furthermore, Stomp should be usable by other traditional JMS solutions.
On the lighter-weight side: we can set up a client for the memcachced protocol-based solutions like Starling and Sparrow. That should be relatively easy and give sysadmins more options in their deployments.
Ultimately, the resulting workflow should be (Luke may correct me here): * in the client, the catalog class uses :rest indirection, meaning it performs a find() RESTfully through puppetmasterd * puppetmasterd indirects the same class to the :compiler terminus, so the internal find() call in puppetmasterd goes to the parser to do the catalog compilation * when the catalog object is compiled and is ready for returning to the client, the caching step kicks in and will indirect to the :queue terminus. * the :queue terminus serializes the object to YAML, gets the right message queue client object, and sends the object YAML as a message to a queue named for the indirection itself; at this point, puppetmasterd's work is done and it returns to the client with no further delay * puppetqd runs as a separate process, and is listening to the relevant queue (though the degree to which it uses the indirection system natively for queue subscription needs to be worked out a bit more); as messages come in, it serializes them and stores them to the database, using a new database indirection in development for a different thread.
I hope this makes sense and seems reasonable to folks. Fire away!
Thanks. - Ethan
-- Ethan Rowe End Point Corporation et...@endpoint.com
Ethan Rowe wrote: > * puppetqd runs as a separate process, and is listening to the relevant > queue (though the degree to which it uses the indirection system > natively for queue subscription needs to be worked out a bit more); as > messages come in, it serializes them and stores them to the database, > using a new database indirection in development for a different thread.
I think this is filled with awesome (to steal a Shafer-ism).
My only comments are mostly end-user operability ones - which I guess people should realise now is my focus because I often write the doco :)
1. Configuration should all run out of puppet.conf - no new files for this (I am assuming that was the intent).
2. I like options - making a variety of clients available would be good and making sure adding new clients is easy and well documented is crucial to this I think - people's opinions on messenging and message buses will vary greatly.
3. This shouldn't be mandatory - it should be an optional extra rather than the default to have queuing enabled.
4. What happens with the queue daemon breaks in the process?
5. It'd be good to see both architectures/workflows documented - I think - other than talking to Luke - that's the first time I've seen the new indirection workflow articulated in words on screen as it were. I see a whole new world full of diagrams and workflows... :)
James Turnbull wrote: > Ethan Rowe wrote: >> * puppetqd runs as a separate process, and is listening to the relevant >> queue (though the degree to which it uses the indirection system >> natively for queue subscription needs to be worked out a bit more); as >> messages come in, it serializes them and stores them to the database, >> using a new database indirection in development for a different thread.
> I think this is filled with awesome (to steal a Shafer-ism).
> My only comments are mostly end-user operability ones - which I guess > people should realise now is my focus because I often write the doco :)
> 1. Configuration should all run out of puppet.conf - no new files for > this (I am assuming that was the intent).
Since the message service particulars will need to be shared by at least daemons (puppetmaster, puppetqd), it makes sense to place those particulars in a common location.
Our intent is to make this as simple as reasonably possible. Though to be honest, our approach to the configuration arrangement is "get something working now, improve it when we have the new logic working." :)
> 2. I like options - making a variety of clients available would be good > and making sure adding new clients is easy and well documented is > crucial to this I think - people's opinions on messenging and message > buses will vary greatly.
The working design absolutely takes these considerations to heart. Yesterday I did some fledgling implementation for the abstract terminus, the client registry, etc., and I'm taking the trouble to document as I go.
Whether my documentation is comprehensible is another matter, but the effort has been and will continue to be there.
> 3. This shouldn't be mandatory - it should be an optional extra rather > than the default to have queuing enabled.
Right. My understanding is that the default behavior should be for catalog storage to act as it acts right now: just do the storage to the database right from puppetmasterd. Use of the queue option is something to determine on a per-deployment basis.
> 4. What happens with the queue daemon breaks in the process?
To be sure I understand what you're asking, did you perhaps mean "when the queue daemon breaks" rather than "with"?
> 5. It'd be good to see both architectures/workflows documented - I > think - other than talking to Luke - that's the first time I've seen the > new indirection workflow articulated in words on screen as it were. I > see a whole new world full of diagrams and workflows... :)
A diagram would presumably help quite a lot. Words can express the design, but the abstractions are of the sort that will tend to lose a reader.
Thanks. - Ethan
-- Ethan Rowe End Point Corporation et...@endpoint.com
Ethan Rowe wrote: >> 1. Configuration should all run out of puppet.conf - no new files for >> this (I am assuming that was the intent).
> Since the message service particulars will need to be shared by at least > daemons (puppetmaster, puppetqd), it makes sense to place those > particulars in a common location.
Agreed - puppet.conf seems the logical choice.
> The working design absolutely takes these considerations to heart. > Yesterday I did some fledgling implementation for the abstract terminus, > the client registry, etc., and I'm taking the trouble to document as I go.
> Whether my documentation is comprehensible is another matter, but the > effort has been and will continue to be there.
Let me know and I can work on it. It's probably what I can best contribute.
>> 4. What happens with the queue daemon breaks in the process?
> To be sure I understand what you're asking, did you perhaps mean "when > the queue daemon breaks" rather than "with"?
Yes - sorry s/with/when/
> A diagram would presumably help quite a lot. Words can express the > design, but the abstractions are of the sort that will tend to lose a > reader.
Again - if someone does a rough one I can polish if required. It'd probably be an excuse for me to do a bunch for various processes. I had intended to do such for the new wiki.
>> As mentioned in an email yesterday, I'm working with someone to add a >> queueing service to Puppet so that some server-side operations are >> queued and executed as cpu time is available - generally those that >> aren't on the critical path but particularly the storeconfigs save >> operation.
> Sounds interesting.
>> The goal here is that the client wouldn't have to wait for the server >> to finish storing the catalog before it got its copy of the catalog; >> instead, the server would just put the catalog into the queue, and a >> queue reader would come behind and process as time was available. >> The >> only other operation I can think of where this makes sense right now >> is reporting, but I expect we'll have others over time.
> Do we have a summary of current performance and some perfomance tests > so we can objectively measure improvements? If so getting them visibly > graphing from hudson builds of that branch would be awesome.
That would be great to have, but no, we have neither that nor even an objective idea of performance.
We probably need to come up with a couple of standard configurations with a given amount of complexity and then start tracking the times those things take. We could stick entirely to 'notify' resources, so that we're not hitting the disk at all.
Anyone interested in taking this up?
>> The big benefit here, of course, is that you get configuration >> storage >> out of the critical path for returning catalogs to clients, and this >> has the benefit of making the client queries shorter and thus less >> likely to overlap. It also means that if you get a big rush of >> client >> connections, we can queue up the more expensive operations, and even >> if storeconfigs eventually becomes cheap, it would still make sense >> to >> queue anything that wasn't on the critical path.
> Do we have a set of requirements that we need both for functionality > and for ie do we need persistence, etc)
Ethan, Steven, and I have developed largely complete requirements for functionality, but, erm, no, we don't have a literal requirements document or anything. Yes, persistence is required. Hopefully Ethan can send out a summary of the requirements list he's developed.
>> So, I'm looking for input on what people think the right tools and >> architecture are for this, and just generally how we should go about >> it. I don't want to have a six week discussion on it, but I do want >> to make sure we get as much feedback as possible during development, >> and we'll be doing normal code review and publishing code as often as >> possible.
> Without the requirements it's hard to say exactly, I'd probably look > at starling/workling, beanstalk, and nanite amongst others.
> If we have good criteria we could do some time boxed evaluation to > choose best fit for puppet at the moment, but ensuring that we > implement it in a pluggable way (eg choose a pluggable transport or > standard such as amqp, thrift, protocol buffers )
Yeah, the goal is to make it relatively pluggable, but it'll certainly be a whole new category of functionality for me, so I know at least I'll take some time to get up to speed.
-- Brand's Shortcut: The only way to predict the future is to make sure it stays exactly the same as the present. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
>>> I'm working with someone to add a >>> queueing service to Puppet so that some server-side operations are >>> queued and executed as cpu time is available - generally those that >>> aren't on the critical path but particularly the storeconfigs save >>> operation.
> What will be the determination of when CPU time is available? A > configuration option, like what's in exim when load is above X only > queue? What's considered a 'busy' puppetmaster?
The queue processor will run either on a separate node or as a niced process on the master, so the answer should be relatively straightforward from there.
>>> The goal here is that the client wouldn't have to wait for the >>> server >>> to finish storing the catalog before it got its copy of the catalog; >>> instead, the server would just put the catalog into the queue, and a >>> queue reader would come behind and process as time was available. >>> The >>> only other operation I can think of where this makes sense right now >>> is reporting, but I expect we'll have others over time.
>> Do we have a summary of current performance and some perfomance tests >> so we can objectively measure improvements? If so getting them >> visibly >> graphing from hudson builds of that branch would be awesome.
> More specifically what's the most expensive CPU and/or memory > operations when a node connects and process? Is there a break down in > that process? Of that process what's needed immediately by the client > and what can wait?
Compiling and storage are definitely the most expensive CPU-wise, and file serving is probably the most expensive memory wise.
The compiled catalog is needed immediately by the client, but the catalog storage isn't needed on a particular timeframe. Fileserving, of course, is also needed immediately.
> In the my case queueing reporting wouldn't help me that much since > we use a centralized syslog to do reporting.
Yeah, reporting is generally a pretty inexpensive operation and it's probably not worth queueing in most cases; I threw it in there just to make the point that this should be a general process.
-- There are three social classes in America: upper middle class, middle class, and lower middle class. --Judith Martin --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Wed, Mar 18, 2009 at 1:16 PM, Luke Kanies <l...@madstop.com> wrote:
> Hi all,
> As mentioned in an email yesterday, I'm working with someone to add a > queueing service to Puppet so that some server-side operations are > queued and executed as cpu time is available - generally those that > aren't on the critical path but particularly the storeconfigs save > operation.
> The goal here is that the client wouldn't have to wait for the server > to finish storing the catalog before it got its copy of the catalog; > instead, the server would just put the catalog into the queue, and a > queue reader would come behind and process as time was available. The > only other operation I can think of where this makes sense right now > is reporting, but I expect we'll have others over time.
Would this help with the splay problem we discussed recently?
> On Wed, Mar 18, 2009 at 1:16 PM, Luke Kanies <l...@madstop.com> wrote:
>> Hi all,
>> As mentioned in an email yesterday, I'm working with someone to add a >> queueing service to Puppet so that some server-side operations are >> queued and executed as cpu time is available - generally those that >> aren't on the critical path but particularly the storeconfigs save >> operation.
>> The goal here is that the client wouldn't have to wait for the server >> to finish storing the catalog before it got its copy of the catalog; >> instead, the server would just put the catalog into the queue, and a >> queue reader would come behind and process as time was available. >> The >> only other operation I can think of where this makes sense right now >> is reporting, but I expect we'll have others over time.
> Would this help with the splay problem we discussed recently?
The queueing would only help with reducing client contention when the contention was caused by StoreConfigs, unfortunately.
Although...
I could see having a ping-back model, where a client sends a request for a compilation but then just sleeps, and when the config is compiled the server pushes the configuration to the client.
This is obviously a far cry from the current architecture, but it's mostly a reconfiguration, once you have the queueing, rather than a fundamental shift.
-- Getting caught is the mother of invention. --Robert Byrne --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, Mar 19, 2009 at 10:57 PM, Luke Kanies <l...@madstop.com> wrote: > The queueing would only help with reducing client contention when the > contention was caused by StoreConfigs, unfortunately.
> Although...
> I could see having a ping-back model, where a client sends a request > for a compilation but then just sleeps, and when the config is > compiled the server pushes the configuration to the client.
> This is obviously a far cry from the current architecture, but it's > mostly a reconfiguration, once you have the queueing, rather than a > fundamental shift.
Luke Kanies wrote: > On Mar 19, 2009, at 9:52 PM, Sam Rowe wrote:
>> On Wed, Mar 18, 2009 at 1:16 PM, Luke Kanies <l...@madstop.com> wrote: >>> Hi all,
>>> As mentioned in an email yesterday, I'm working with someone to add a >>> queueing service to Puppet so that some server-side operations are >>> queued and executed as cpu time is available - generally those that >>> aren't on the critical path but particularly the storeconfigs save >>> operation.
>>> The goal here is that the client wouldn't have to wait for the server >>> to finish storing the catalog before it got its copy of the catalog; >>> instead, the server would just put the catalog into the queue, and a >>> queue reader would come behind and process as time was available. >>> The >>> only other operation I can think of where this makes sense right now >>> is reporting, but I expect we'll have others over time. >> Would this help with the splay problem we discussed recently?
> The queueing would only help with reducing client contention when the > contention was caused by StoreConfigs, unfortunately.
> Although...
> I could see having a ping-back model, where a client sends a request > for a compilation but then just sleeps, and when the config is > compiled the server pushes the configuration to the client.
> This is obviously a far cry from the current architecture, but it's > mostly a reconfiguration, once you have the queueing, rather than a > fundamental shift.
I'm a newcomer, so feel free to ignore me. But, if RESTful design has become an increasingly important consideration in interactions between client and server (and it's my impression that it has), I think you might find that it fits better in this case to: * have the server issue a 202 Accepted response, with an response entity containing: possibly a URL to periodically check for results; possibly an estimate of time before the new resource is available * have the client periodically consult the relevant URL until the data is available
It's somewhat annoying to consider the possibility of having a client repeatedly, possibly fruitlessly, checking in with a server. But it may be easier to scale that approach and fit it within the current architecture than having bidirectional data flow of the sort described.
Thanks. - Ethan -- Ethan Rowe End Point Corporation et...@endpoint.com
> Luke Kanies wrote: >> On Mar 19, 2009, at 9:52 PM, Sam Rowe wrote:
>>> On Wed, Mar 18, 2009 at 1:16 PM, Luke Kanies <l...@madstop.com> >>> wrote: >>>> Hi all,
>>>> As mentioned in an email yesterday, I'm working with someone to >>>> add a >>>> queueing service to Puppet so that some server-side operations are >>>> queued and executed as cpu time is available - generally those that >>>> aren't on the critical path but particularly the storeconfigs save >>>> operation.
>>>> The goal here is that the client wouldn't have to wait for the >>>> server >>>> to finish storing the catalog before it got its copy of the >>>> catalog; >>>> instead, the server would just put the catalog into the queue, >>>> and a >>>> queue reader would come behind and process as time was available. >>>> The >>>> only other operation I can think of where this makes sense right >>>> now >>>> is reporting, but I expect we'll have others over time. >>> Would this help with the splay problem we discussed recently?
>> The queueing would only help with reducing client contention when the >> contention was caused by StoreConfigs, unfortunately.
>> Although...
>> I could see having a ping-back model, where a client sends a request >> for a compilation but then just sleeps, and when the config is >> compiled the server pushes the configuration to the client.
>> This is obviously a far cry from the current architecture, but it's >> mostly a reconfiguration, once you have the queueing, rather than a >> fundamental shift.
> I'm a newcomer, so feel free to ignore me. But, if RESTful design has > become an increasingly important consideration in interactions between > client and server (and it's my impression that it has), I think you > might find that it fits better in this case to: > * have the server issue a 202 Accepted response, with an response > entity > containing: possibly a URL to periodically check for results; possibly > an estimate of time before the new resource is available > * have the client periodically consult the relevant URL until the data > is available
> It's somewhat annoying to consider the possibility of having a client > repeatedly, possibly fruitlessly, checking in with a server. But it > may > be easier to scale that approach and fit it within the current > architecture than having bidirectional data flow of the sort > described.
You're right - that's much simpler, doesn't require the ability to contact the client, and is low-cost.
Good point; thanks.
-- You don't learn anything the second time you're kicked by a mule. -- Anonymous Texan --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> Ethan Rowe wrote: >> * puppetqd runs as a separate process, and is listening to the >> relevant >> queue (though the degree to which it uses the indirection system >> natively for queue subscription needs to be worked out a bit more); >> as >> messages come in, it serializes them and stores them to the database, >> using a new database indirection in development for a different >> thread.
> I think this is filled with awesome (to steal a Shafer-ism).
> My only comments are mostly end-user operability ones - which I guess > people should realise now is my focus because I often write the > doco :)
> 1. Configuration should all run out of puppet.conf - no new files for > this (I am assuming that was the intent).
> 2. I like options - making a variety of clients available would be > good > and making sure adding new clients is easy and well documented is > crucial to this I think - people's opinions on messenging and message > buses will vary greatly.
> 3. This shouldn't be mandatory - it should be an optional extra > rather > than the default to have queuing enabled.
Yep, these are all in the plan, although the pluggability might suffer a bit in the first versions, not surprisingly.
> 4. What happens with the queue daemon breaks in the process?
"Breaks" would need to be defined a bit, but most likely things couldn't be queued, and thus would be lost.
> 5. It'd be good to see both architectures/workflows documented - I > think - other than talking to Luke - that's the first time I've seen > the > new indirection workflow articulated in words on screen as it were. I > see a whole new world full of diagrams and workflows... :)
-- He attacked everything in life with a mix of extraordinary genius and naive incompetence, and it was often difficult to tell which was which. -- Douglas Adams --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
1. Can we assume that non queued request are RO requests? 2. Can the queued request use another database server?
The reason I'm asking, is that I'm looking for a way to use storeconfig in a large multi location environment, until now it hasn't been possible because of the dependency and latency to the database. if I could use a local replica on the puppetmaster for exported resources, which would be a RO db replica, and store all new information using the queue to a db master, I think that would help a lot ;)
> 1. Can we assume that non queued request are RO requests?
> 2. Can the queued request use another database server?
> The reason I'm asking, is that I'm looking for a way to use storeconfig in a
> large multi location environment, until now it hasn't been possible because
> of the dependency and latency to the database.
> if I could use a local replica on the puppetmaster for exported resources,
> which would be a RO db replica, and store all new information using the
> queue to a db master, I think that would help a lot ;)
++ to everything Ohad just said.
Storeconfigs just hasn't been practical for us, although 0.24.8 is
looking significantly better on a recent linux distro.
> my 2 cents.
> Ohad
-- Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.
> 1. Can we assume that non queued request are RO requests?
Yes.
> 2. Can the queued request use another database server?
I would assume so. One of the benefits of using a "real" queue system is that there's already a wide range of expertise on it and you can configure it however you want.
> The reason I'm asking, is that I'm looking for a way to use > storeconfig in a large multi location environment, until now it > hasn't been possible because of the dependency and latency to the > database. > if I could use a local replica on the puppetmaster for exported > resources, which would be a RO db replica, and store all new > information using the queue to a db master, I think that would help > a lot ;)
Makes a lot of sense. I expect you'd be doing some experimentation and ground-breaking, but it should be relatively straightforward once you grok the queueing system.
-- However beautiful the strategy, you should occasionally look at the results. -- Sir Winston Churchill --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, 2009-03-19 at 23:06 -0400, Ethan Rowe wrote: > I'm a newcomer, so feel free to ignore me. But, if RESTful design has > become an increasingly important consideration in interactions between > client and server (and it's my impression that it has), I think you > might find that it fits better in this case to: > * have the server issue a 202 Accepted response, with an response entity > containing: possibly a URL to periodically check for results; possibly > an estimate of time before the new resource is available > * have the client periodically consult the relevant URL until the data > is available
I've spent some time recently banging my head against the limitations of REST for oVirt - the assumption that REST makes that everything is a resource on which you execute a fixed number of actions is really nice if you are dealing with something that looks like a resource fairly naturally, but gets very annoying when that assumption is broken (like VM's which can be started, stopped, shut down etc.)
We are in the process of implementing API's for oVirt now based on QMF[1], a small layer on top of Qpid, an open source message bus; the programming model of QMF is pretty nice since it abstracts all the messaging etc. away nicely. From a coding point of view, you are dealing with objects, call methods on them etc. and QMF/Qpid take care of distributing changes throughout the messaing infrastructure.
Architecturally, everybody is a client connecting to a central broker, even the server on which the objects really live; the logical client gets a view of those objects, calling methods on them causes some method on the server to be executed etc. We're using that to connect various virtualized hosts together - each host is the 'server' for the VM's that are running to it, and a client using the QMF API gets to see a collection of all the VM's on all hosts, can start/stop etc. them without the need to connect to individual hosts.
I am mostly mentioning this because it's an important new way of connecting infrastructure together, and radically different from all the other approaches I have seen. If anybody's interested in diving into this a little more in the context of puppet, feel free to contact me. A rough idea would be to have each client serve its facts and its current catalog over QMF and have the puppetmaster expose appropriate methods to retrieve an updated catalog.
Of course, this is not directly connected to the issue at hand (speeding up storeconfigs), but it's got to do with messaging ;)
On Tue, Mar 31, 2009 at 12:49 AM, David Lutterkort <lut...@redhat.com> wrote: > I've spent some time recently banging my head against the limitations of > REST for oVirt - the assumption that REST makes that everything is a > resource on which you execute a fixed number of actions is really nice > if you are dealing with something that looks like a resource fairly > naturally, but gets very annoying when that assumption is broken (like > VM's which can be started, stopped, shut down etc.)
I can relate. With this shiny REST hammer in our toolboxes, we want every problem to look like a nail. But while you *can* drive a screw with a hammer (ever find yourself without a screwdriver at just the wrong time?), it's much easier if you use the right tool for the job. REST is great for resource-oriented programming, but is awkward for activity-oriented programming. This article breaks down the difference pretty well:
A hybrid REST-RPC approach seems most appropriate for your needs. Model the VMs and their attributes as resources, but model the actions you might want to perform on a VM (install, configure, start, stop, suspend, deinstall) as methods (XML-RPC, SOAP, roll your own, etc).
> We are in the process of implementing API's for oVirt now based on > QMF[1], a small layer on top of Qpid, an open source message bus; the > programming model of QMF is pretty nice since it abstracts all the > messaging etc. away nicely. From a coding point of view, you are dealing > with objects, call methods on them etc. and QMF/Qpid take care of > distributing changes throughout the messaing infrastructure.
I like how using a message bus drives the design toward an asynchronous, event-based model. That feels right to me. I also like the ability to snoop on the channels to see what the various components are doing :) This is particularly easy when using Jabber as the message transport, as in Vertebra.
I'm not familiar with QMF, but the XML schema definition seems like it might make upgrades painful: will you have to contemporaneously upgrade all components plugged into the bus whenever the schema changes? A schema-free design (the messaging equivalent of duck typing) might be easier to maintain.
> Architecturally, everybody is a client connecting to a central broker, > even the server on which the objects really live; the logical client > gets a view of those objects, calling methods on them causes some method > on the server to be executed etc. We're using that to connect various > virtualized hosts together - each host is the 'server' for the VM's that > are running to it, and a client using the QMF API gets to see a > collection of all the VM's on all hosts, can start/stop etc. them > without the need to connect to individual hosts.
Some hypothetical questions for you -- no need to reply since this may already be off-topic:
If you send one global start command to the bus, how do you know when all VMs have in fact started? Even if you don't have to send commands to individual hosts, you still need to monitor them individually.
What happens when a host is down or otherwise disconnected from the message bus when the command to start/stop/delete/etc all VMs goes out? Unless Qpid guarantees delivery, the VM state on that host may be out of sync with the rest of the cluster when it comes back up.
I'm a newbie on the list -- apologies for butting in :)
> On Thu, 2009-03-19 at 23:06 -0400, Ethan Rowe wrote: >> I'm a newcomer, so feel free to ignore me. But, if RESTful design >> has >> become an increasingly important consideration in interactions >> between >> client and server (and it's my impression that it has), I think you >> might find that it fits better in this case to: >> * have the server issue a 202 Accepted response, with an response >> entity >> containing: possibly a URL to periodically check for results; >> possibly >> an estimate of time before the new resource is available >> * have the client periodically consult the relevant URL until the >> data >> is available
> I've spent some time recently banging my head against the > limitations of > REST for oVirt - the assumption that REST makes that everything is a > resource on which you execute a fixed number of actions is really nice > if you are dealing with something that looks like a resource fairly > naturally, but gets very annoying when that assumption is broken (like > VM's which can be started, stopped, shut down etc.)
Yeah, this is why the process that retrieves items from the queue, for instance, isn't using a REST model for doing so.
It just so happens that *most* of Puppet's network services are easy to model with resources, but clearly not all - 'status' doesn't make much sense, for instance.
> We are in the process of implementing API's for oVirt now based on > QMF[1], a small layer on top of Qpid, an open source message bus; the > programming model of QMF is pretty nice since it abstracts all the > messaging etc. away nicely. From a coding point of view, you are > dealing > with objects, call methods on them etc. and QMF/Qpid take care of > distributing changes throughout the messaing infrastructure.
> Architecturally, everybody is a client connecting to a central broker, > even the server on which the objects really live; the logical client > gets a view of those objects, calling methods on them causes some > method > on the server to be executed etc. We're using that to connect various > virtualized hosts together - each host is the 'server' for the VM's > that > are running to it, and a client using the QMF API gets to see a > collection of all the VM's on all hosts, can start/stop etc. them > without the need to connect to individual hosts.
> I am mostly mentioning this because it's an important new way of > connecting infrastructure together, and radically different from all > the > other approaches I have seen. If anybody's interested in diving into > this a little more in the context of puppet, feel free to contact > me. A > rough idea would be to have each client serve its facts and its > current > catalog over QMF and have the puppetmaster expose appropriate > methods to > retrieve an updated catalog.
> Of course, this is not directly connected to the issue at hand > (speeding > up storeconfigs), but it's got to do with messaging ;)
This messaging project is limited enough that I'm skeptical it's going to have a big impact on Puppet pervavisely using messaging suddenly, but I've always thought messaging was a much more appropriate means of sending a lot of information, especially reports.
-- A gentleman is a man who can play the accordion but doesn't. --Unknown --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
