Re: Digest for pallet-clj@googlegroups.com - 2 Messages in 1 Topic
Jeff Rose
>
>> thoughts out there regarding the API...
>
>> How about doing something like this in Pallet?
>
>> (def ec2 (cloud "ec2" id key))
>
> Is this different to pallet.compute/compute-service?
>
>> service) around as a "second class" argument to primary functions like
>> converge and lift. I would have expected it to be the first argument.
>> (i.e. If cloud were a deftype lift and converge would be its methods.)
>
> lift and converge aren't really methods on a cloud object, even
> conceptually.
>
> eg. lift can function without a cloud at all, and you can also pass in a
> blobstore service, as well as a compute service, etc.
>
>> was expecting functions like boot, shutdown, configure, execute, stuff
>> like that. (I've got zero experience managing machines in a cloud
>> though, so this could be very naive perspective.)
>
> We can probably add these type of methods, though many are supported
> directly by jclouds. Pallet doesn't really operate on single nodes, but
> on node groups.
Ok, although I would have thought that underlying any group
functionality would be functions to control individual nodes. My
previous experience in controlling many "nodes" is in the context of
network simulators, where within the course of a given workload it's
typical to do things like start nodes up smoothly or randomly over
time according to some distribution, kill off random nodes in the
middle of operation, sample a subset of nodes to gather a snapshot of
the current state, etc. I'm sure these aren't typical use cases for
people who are setting up web clusters, although for testing
distributed databases and other fault-tolerant software it would still
be useful.
> You can define a node type containing this script
>
> (defnode mynode {}
> :check (pallet.resource/phase
> (pallet.resource.exec-script/exec-script
> (cd somewhere)
> (cat some-log))))
Beautiful. Thanks.
> (pallet.core/lift mynode :phase :check :compute ec2)
>
> and the :results key of the return value will contain all script output.
How about grabbing a specific file from each node?
> You can also run ad hoc scripts:
>
> (pallet.core/lift mynode
> :compute ec2
> :phase (pallet.resource/phase
> (pallet.resource.exec-script/exec-script
> (cd somewhere)
> (cat some-log))))
Perfect. My next task was to figure out how to send a message to each
peer, for example to bootstrap the network and give them a starting
list of IP addresses, and this should do the trick. Should be a fun
weekend.
Thanks,
Jeff
Hugo Duncan
>> "Hugo Duncan" <dunca...@gmail.com> Mar 03 12:46PM -0500 ^
>> We can probably add these type of methods, though many are supported
>> directly by jclouds. Pallet doesn't really operate on single nodes, but
>> on node groups.
>
> Ok, although I would have thought that underlying any group
> functionality would be functions to control individual nodes.
It is certainly an idea worth exploring, and I think can be achieved
relatively easily with a change of interface in
pallet.compute.ComputeService.
> My
> previous experience in controlling many "nodes" is in the context of
> network simulators, where within the course of a given workload it's
> typical to do things like start nodes up smoothly or randomly over
> time according to some distribution, kill off random nodes in the
> middle of operation, sample a subset of nodes to gather a snapshot of
> the current state, etc. I'm sure these aren't typical use cases for
> people who are setting up web clusters, although for testing
> distributed databases and other fault-tolerant software it would still
> be useful.
It is possible to do all the above via the converge command.
>> You can define a node type containing this script
>>
>> (defnode mynode {}
>> :check (pallet.resource/phase
>> (pallet.resource.exec-script/exec-script
>> (cd somewhere)
>> (cat some-log))))
>
> Beautiful. Thanks.
>
>> (pallet.core/lift mynode :phase :check :compute ec2)
>>
>> and the :results key of the return value will contain all script output.
>
> How about grabbing a specific file from each node?
>
>> You can also run ad hoc scripts:
>>
>> (pallet.core/lift mynode
>> :compute ec2
>> :phase (pallet.resource/phase
>> (pallet.resource.exec-script/exec-script
>> (cd somewhere)
>> (cat some-log))))
>
> Perfect. My next task was to figure out how to send a message to each
> peer, for example to bootstrap the network and give them a starting
> list of IP addresses, and this should do the trick. Should be a fun
> weekend.
The nodes are available in the request map that is passed as the first
argument to a crate function. In a crate you can obtain a sequence of
nodes using pallet.request-map/nodes-in-tag, and query the ip-address with
pallet.compute/primary-ip
(defn write-ip-config
[request group-tag]
(->
request
(pallet.resource.remote-file/remote-file
"/path/to/config-file"
:content (->> (pallet.request-map/nodes-in-tag request group-tag)
(map pallet.compute/primary-ip)
(clojure.string/join \newline)))))
(pallet.core/defnode controller {}
:bootstrap automated-admin-user
:configure (pallet.resource/phase (write-ip-config :worker)))
(pallet.core/defnode worker {}
:bootstrap automated-admin-user)
(pallet.core/converge {controller 2 worker 10} :compute ec2)
The above would start two "controller" nodes and ten worker nodes, and the
list of worker ips would end up in "/path/to/config-file" on both
controller nodes.
If you then decide you want 12 workers instead of 10, then executing the
command below will start two new worker nodes and add them to the config
file on the controller:
(pallet.core/converge {controller 2 worker 12} :compute ec2)
Hope that helps. Look forward to hearing more on your experiments with
pallet.
--
Hugo Duncan