feature 7474 against master
Mo Morsi
representing web resources/requests which can
be included in a puppet scipt.
A default provider of this type is also included
using the curl interface for ruby as provided by
the 'curb' gem
The second patch implements a few spec cases for
the new type and provider
Mo Morsi
and implements a provider of that type using the
ruby curl interface provided by the 'curb' rubygem
Signed-off-by: Mo Morsi <mmo...@redhat.com>
---
Local-branch: feature/master/7474
lib/puppet/provider/web_request/curl.rb | 128 +++++++++++++++++++++++++++++++
lib/puppet/type/web_request.rb | 50 ++++++++++++
2 files changed, 178 insertions(+), 0 deletions(-)
create mode 100644 lib/puppet/provider/web_request/curl.rb
create mode 100644 lib/puppet/type/web_request.rb
diff --git a/lib/puppet/provider/web_request/curl.rb b/lib/puppet/provider/web_request/curl.rb
new file mode 100644
index 0000000..479bfd5
--- /dev/null
+++ b/lib/puppet/provider/web_request/curl.rb
@@ -0,0 +1,128 @@
+require 'curb'
+require 'uuid'
+require 'fileutils'
+
+# Helper to invoke the web request w/ curl
+def web_request(method, uri, request_params, params = {})
+ raise Puppet::Error, "Must specify valid http method and uri" if method.nil? || uri.nil? || method == "" || uri == ""
+
+ curl = Curl::Easy.new
+
+ if params.has_key?(:cookie)
+ curl.enable_cookies = true
+ curl.cookiefile = params[:cookie]
+ curl.cookiejar = params[:cookie]
+ end
+
+ curl.follow_location = (params.has_key?(:follow) && params[:follow])
+
+ case(method)
+ when 'get'
+ url = uri
+ url += ";" + request_params.collect { |k,v| "#{k}=#{v}" }.join("&") unless request_params.nil?
+ curl.url = url
+ curl.http_get
+ return curl
+
+ when 'post'
+ cparams = []
+ request_params.each_pair { |k,v| cparams << Curl::PostField.content(k,v) } unless request_params.nil?
+ curl.url = uri
+ curl.http_post(cparams)
+ return curl
+
+ #when 'put'
+ #when 'delete'
+ end
+end
+
+# Helper to verify the response
+def verify_result(result, verify = {})
+ returns = (verify.has_key?(:returns) && !verify[:returns].nil?) ? verify[:returns] : "200"
+ returns = [returns] unless returns.is_a? Array
+ unless returns.include?(result.response_code.to_s)
+ raise Puppet::Error, "Invalid HTTP Return Code: #{result.response_code},
+ was expecting one of #{returns.join(", ")}"
+ end
+
+ if verify.has_key?(:body) && !verify[:body].nil? && !(result.body_str =~ Regexp.new(verify[:body]))
+ raise Puppet::Error, "Expecting #{verify[:body]} in the result"
+ end
+end
+
+# Helper to process/parse web parameters
+def process_params(request_method, params, uri)
+ begin
+ # Set request method and generate a unique session key
+ session = "/tmp/#{UUID.new.generate}"
+
+ # Invoke a login request if necessary
+ if params[:login]
+ login_params = params[:login].reject { |k,v| ['http_method', 'uri'].include?(k) }
+ web_request(params[:login]['http_method'], params[:login]['uri'],
+ login_params, :cookie => session, :follow => params[:follow]).close
+ end
+
+ # Check to see if we should actually run the request
+ skip_request = !params[:unless].nil?
+ if params[:unless]
+ result = web_request(params[:unless]['http_method'], params[:unless]['uri'],
+ params[:unless]['parameters'],
+ :cookie => session, :follow => params[:follow])
+ begin
+ verify_result(result,
+ :returns => params[:unless]['returns'],
+ :body => params[:unless]['verify'])
+ rescue Puppet::Error => e
+ skip_request = false
+ end
+ result.close
+ end
+ return if skip_request
+
+ # Actually run the request and verify the result
+ uri = params[:name] if uri.nil?
+ result = web_request(request_method, uri, params[:parameters],
+ :cookie => session, :follow => params[:follow])
+ verify_result(result,
+ :returns => params[:returns],
+ :body => params[:verify])
+ result.close
+
+ # Invoke a logout request if necessary
+ if params[:logout]
+ logout_params = params[:login].reject { |k,v| ['http_method', 'uri'].include?(k) }
+ web_request(params[:logout]['http_method'], params[:logout]['uri'],
+ logout_params, :cookie => session, :follow => params[:follow]).close
+ end
+
+ rescue Exception => e
+ raise Puppet::Error, "An exception was raised when invoking web request: #{e}"
+
+ ensure
+ FileUtils.rm_f(session) if params[:logout]
+ end
+end
+
+# Puppet provider definition
+Puppet::Type.type(:web_request).provide :curl do
+ desc "Use curl to access web resources"
+
+ def get
+ @uri
+ end
+
+ def post
+ @uri
+ end
+
+ def get=(uri)
+ @uri = uri
+ process_params('get', @resource, uri)
+ end
+
+ def post=(uri)
+ @uri = uri
+ process_params('post', @resource, uri)
+ end
+end
diff --git a/lib/puppet/type/web_request.rb b/lib/puppet/type/web_request.rb
new file mode 100644
index 0000000..d95e286
--- /dev/null
+++ b/lib/puppet/type/web_request.rb
@@ -0,0 +1,50 @@
+Puppet::Type.newtype(:web_request) do
+ @doc = "Issue a request via the world wide web"
+
+ newparam :name
+
+ newproperty(:get) do
+ desc "Issue get request to the specified uri"
+ # TODO valid value to be a uri
+ end
+
+ newproperty(:post) do
+ desc "Issue get request to the specified uri"
+ # TODO valid value to be a uri
+ end
+
+ # TODO implement
+ #newproperty(:delete)
+ #newproperty(:put)
+
+ newparam(:parameters) do
+ desc "Hash of parameters to include in the web request"
+ end
+
+ newparam(:returns) do
+ desc "Expected http return codes of the request"
+ defaultto "200"
+ # TODO validate value(s) is among possible valid http return codes
+ end
+
+ newparam(:follow) do
+ desc "Boolean indicating if redirects should be followed"
+ newvalues(:true, :false)
+ end
+
+ newparam(:verify) do
+ desc "String to verify as being part of the result"
+ end
+
+ newparam(:login) do
+ desc "Login parameters to be used if a login is required before making the request"
+ end
+
+ newparam(:logout) do
+ desc "Logout parameters to be used if a logout is requred after making the request"
+ end
+
+ newparam(:unless) do
+ desc "Do not run request if the request specified here succeeds"
+ end
+end
--
1.7.2.3
Mo Morsi
spec/unit/type/web_request_spec.rb | 29 +++++++++++++++++++++++++++++
2 files changed, 61 insertions(+), 0 deletions(-)
create mode 100644 spec/unit/provider/web/curl_spec.rb
create mode 100644 spec/unit/type/web_request_spec.rb
diff --git a/spec/unit/provider/web/curl_spec.rb b/spec/unit/provider/web/curl_spec.rb
new file mode 100644
index 0000000..3b868ea
--- /dev/null
+++ b/spec/unit/provider/web/curl_spec.rb
@@ -0,0 +1,32 @@
+#!/usr/bin/env rspec
+require 'spec_helper'
+
+provider_class = Puppet::Type.type(:web_request).provider(:curl)
+
+describe provider_class do
+ before :each do
+ @resource = Puppet::Resource.new(:web_request, 'foo')
+ @provider = provider_class.new(@resource)
+ end
+
+ describe "#get" do
+ it "should fail if no valid uri is specified" do
+ lambda { @provider.get=("") }.should raise_error(Puppet::Error,"An exception was raised when invoking web request: Must specify valid http method and uri")
+ end
+
+ it "should issue get request to uri" do
+ @provider.get=("http://www.puppetlabs.com")
+ end
+ end
+
+ describe "#post" do
+ it "should fail if no valid uri is specified" do
+ lambda { @provider.post=("") }.should raise_error(Puppet::Error,"An exception was raised when invoking web request: Must specify valid http method and uri")
+ end
+
+ it "should issue post request to uri" do
+ @provider.post=("http://www.puppetlabs.com")
+ end
+ end
+
+end
diff --git a/spec/unit/type/web_request_spec.rb b/spec/unit/type/web_request_spec.rb
new file mode 100644
index 0000000..bcffd7f
--- /dev/null
+++ b/spec/unit/type/web_request_spec.rb
@@ -0,0 +1,29 @@
+#!/usr/bin/env rspec
+require 'spec_helper'
+
+host = Puppet::Type.type(:web_request)
+
+describe Puppet::Type.type(:web_request) do
+ before do
+ @class = host
+ end
+
+ it "should have :name be its namevar" do
+ @class.key_attributes.should == [:name]
+ end
+
+ describe "when validating attributes" do
+ [:parameters, :returns, :follow, :verify, :login, :logout, :unless].each do |param|
+ it "should have a #{param} parameter" do
+ @class.attrtype(param).should == :param
+ end
+ end
+
+ [:get, :post ].each do |property|
+ it "should have a #{property} property" do
+ @class.attrtype(property).should == :property
+ end
+ end
+ end
+
+end
--
1.7.2.3
Luke Kanies
How do you expect to use this type? Can you provide some example use cases?
I've got a comment below, too.
On May 16, 2011, at 8:22 PM, Mo Morsi wrote:
> Adds a new resource type to puppet for web requests
> and implements a provider of that type using the
> ruby curl interface provided by the 'curb' rubygem
>
> Signed-off-by: Mo Morsi <mmo...@redhat.com>
> ---
> Local-branch: feature/master/7474
> lib/puppet/provider/web_request/curl.rb | 128 +++++++++++++++++++++++++++++++
> lib/puppet/type/web_request.rb | 50 ++++++++++++
> 2 files changed, 178 insertions(+), 0 deletions(-)
> create mode 100644 lib/puppet/provider/web_request/curl.rb
> create mode 100644 lib/puppet/type/web_request.rb
>
> diff --git a/lib/puppet/provider/web_request/curl.rb b/lib/puppet/provider/web_request/curl.rb
> new file mode 100644
> index 0000000..479bfd5
> --- /dev/null
> +++ b/lib/puppet/provider/web_request/curl.rb
> @@ -0,0 +1,128 @@
> +require 'curb'
> +require 'uuid'
> +require 'fileutils'
> +
> +# Helper to invoke the web request w/ curl
> +def web_request(method, uri, request_params, params = {})
These methods should really all be in the provider, rather than at the global level.
> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
>
--
The trouble with the rat race is that even if you win, you're still a
rat. -- Lily Tomlin
---------------------------------------------------------------------
Luke Kanies -|- http://puppetlabs.com -|- http://about.me/lak
Nigel Kersten
> Thanks for the patches.
>
> How do you expect to use this type? Can you provide some example use cases?
I also wanted the community to give feedback on the name in
particular. I'm not convinced 'web' is the best name for this type,
but I can see a bunch of utility here.
I like the idea of being able to treat REST API calls as Puppet resources...
--
Nigel Kersten
Product, Puppet Labs
@nigelkersten
Jacob Helwig
>
> On Tue, May 17, 2011 at 5:40 AM, Luke Kanies <lu...@puppetlabs.com> wrote:
> > Thanks for the patches.
> >
> > How do you expect to use this type? Can you provide some example use cases?
>
> I also wanted the community to give feedback on the name in
> particular. I'm not convinced 'web' is the best name for this type,
> but I can see a bunch of utility here.
>
> I like the idea of being able to treat REST API calls as Puppet resources...
>
Sounds like the "http_request" resource that Chef has[1].
"http_request" seems like an appropriate enough name. I can't think of
a better one, anyway.
[1] http://wiki.opscode.com/display/chef/Resources#Resources-HTTPRequest
--
Jacob Helwig
Mo
cases. I also renamed the resource type and provider back to 'web' as
I thought that better represented the 'web resource' concept that is
being implemented here. If ya'll think http_request would be a better
name for the resource type and provider, I have no problem renaming it
to that for upstream inclusion.
http://groups.google.com/group/puppet-dev/browse_thread/thread/468646ba47fdd4e
On May 17, 1:40 am, Luke Kanies <l...@puppetlabs.com> wrote:
> Thanks for the patches.
>
> How do you expect to use this type? Can you provide some example use cases?
>
this would be to query google for example:
web{google:
get => "http://google.com",
follow => true
}
A more detailed example would be to request the aeolus conductor API
(http://aeolusproject.org)
Web{
login => { 'http_method' => 'post',
'uri' => 'https://localhost/conductor/
user_session',
'user_session[login]' => "$admin_user",
'user_session[password]' => "$admin_password",
'commit' => 'submit' },
logout => { 'http_method' => 'post',
'uri' => 'https://localhost/conductor/
logout' }
}
web{"provider-foobar":
post => "https://localhost/conductor/admin/providers",
parameters => { 'provider[name]' => 'foobar',
'provider[url]' => 'http://localhost:3003/api',
'provider[provider_type_codename]' => 'ec2' },
returns => '200',
verify => '.*Provider added.*',
follow => true,
unless => { 'http_method' => 'get',
'uri' => 'https://localhost/conductor/
admin/providers',
'verify' => '.*foobar.*' }
}
I elaborate on this example in my blog posting on this subject
http://mo.morsi.org/blog/node/336
belong. Feel free to send additional comments my way on anything that
you think could use improvement.
Michael Stahnke
The basic premise is that if you gather a resource externally from puppet's catalog, that resource can change, but the catalog might not.
Let me explain a bit more.
Given you have a a REST API or even a simply GET to get a configuration file for service foo.
When configuration file for service foo changes, the catalog might not change, and maybe it should.
If you have a newer version of a configuration file available, and thus it requires configuration changes, but for some reason the catalog won't compile (or you're on cached catalogs, or whatever), you now have a non congruent version of the RESTFUL resource paired with the catalog.
I ran into some sticky situations at my former job when I tried to do things like this.
You also run into issues of how puppet handles an external resource when it gives a 500 or a 404. The catalog will still compile, because it doesn't verify the external resource via http at catalog compilation time.
I could be the only person that has ever run into issues like this, or your implementation may be handling this better than my previous experiences were able to.
Mike
Mo
On May 27, 1:09 am, Michael Stahnke <stah...@puppetlabs.com> wrote:
> I have a concern with this type, but it's more generic and not your
> implementation.
>
> The basic premise is that if you gather a resource externally from puppet's
> catalog, that resource can change, but the catalog might not.
>
> Let me explain a bit more.
>
> Given you have a a REST API or even a simply GET to get a configuration file
> for service foo.
> When configuration file for service foo changes, the catalog might not
> change, and maybe it should.
>
> If you have a newer version of a configuration file available, and thus it
> requires configuration changes, but for some reason the catalog won't
> compile (or you're on cached catalogs, or whatever), you now have a non
> congruent version of the RESTFUL resource paired with the catalog.
>
from an underlying assumption that this web resource would be
primarily used to retrieve files.
I wasn't thinking about this in that regard, after all puppet already
has file synchronization mechanisms. Rather this new type would
provide the ability to invoke calls on external web based resources
and verify the results.
As a matter of fact the plugin as it stands does not save the http
result's body, but we could always add this as an optional feature if
we wanted.
> I ran into some sticky situations at my former job when I tried to do things
> like this.
>
> You also run into issues of how puppet handles an external resource when it
> gives a 500 or a 404. The catalog will still compile, because it doesn't
> verify the external resource via http at catalog compilation time.
that the puppet recipe author can specify the http return status code
they are expecting in the result (or an array of possible values)
>
> I could be the only person that has ever run into issues like this, or your
> implementation may be handling this better than my previous experiences were
> able to.
robust and comprehensive as possible.