feature 7474 against master
Mo Morsi
representing web resources/requests which can
be included in a puppet scipt.
A default provider of this type is also included
using the curl interface for ruby as provided by
the 'curb' gem
The second patch implements a few spec cases for
the new type and provider
Mo Morsi
and implements a provider of that type using the
ruby curl interface provided by the 'curb' rubygem
This is the first revision of the patch I had previous sent out,
containing improved validations, methods in the global namespace
being encapsulated in the type/provider, the re-renaming of 'web
request' back to 'web' so as to better represent the 'web resource'
Signed-off-by: Mo Morsi <mmo...@redhat.com>
---
Local-branch: feature/master/7474
lib/puppet/external/curl.rb | 47 ++++++++++++++
lib/puppet/provider/web/curl.rb | 131 +++++++++++++++++++++++++++++++++++++++
lib/puppet/type/web.rb | 96 ++++++++++++++++++++++++++++
3 files changed, 274 insertions(+), 0 deletions(-)
create mode 100644 lib/puppet/external/curl.rb
create mode 100644 lib/puppet/provider/web/curl.rb
create mode 100644 lib/puppet/type/web.rb
diff --git a/lib/puppet/external/curl.rb b/lib/puppet/external/curl.rb
new file mode 100644
index 0000000..ab189c8
--- /dev/null
+++ b/lib/puppet/external/curl.rb
@@ -0,0 +1,47 @@
+# Provides an interface to curl using the curb gem for puppet
+require 'curb'
+
+class Curl::Easy
+
+ def self.web_request(method, uri, request_params, params = {})
+ raise Puppet::Error, "Must specify http method (#{method}) and uri (#{uri})" if method.nil? || uri.nil?
+
+ curl = self.new
+
+ if params.has_key?(:cookie)
+ curl.enable_cookies = true
+ curl.cookiefile = params[:cookie]
+ curl.cookiejar = params[:cookie]
+ end
+
+ curl.follow_location = (params.has_key?(:follow) && params[:follow])
+
+ case(method)
+ when 'get'
+ url = uri
+ url += ";" + request_params.collect { |k,v| "#{k}=#{v}" }.join("&") unless request_params.nil?
+ curl.url = url
+ curl.http_get
+ return curl
+
+ when 'post'
+ cparams = []
+ request_params.each_pair { |k,v| cparams << Curl::PostField.content(k,v) } unless request_params.nil?
+ curl.url = uri
+ curl.http_post(cparams)
+ return curl
+
+ #when 'put'
+ #when 'delete'
+ end
+ end
+
+ def valid_status_code?(valid_values=[])
+ valid_values.include?(response_code.to_s)
+ end
+
+ def valid_response_body?(test=/.*/)
+ body_str =~ Regexp.new(test)
+ end
+
+end
diff --git a/lib/puppet/provider/web/curl.rb b/lib/puppet/provider/web/curl.rb
new file mode 100644
index 0000000..e84fe02
--- /dev/null
+++ b/lib/puppet/provider/web/curl.rb
@@ -0,0 +1,131 @@
+require 'uuid'
+require 'fileutils'
+require 'puppet/external/curl'
+
+# Puppet provider definition
+Puppet::Type.type(:web).provide :curl do
+ desc "Use curl to access web resources"
+
+ def get
+ @uri
+ end
+
+ def post
+ @uri
+ end
+
+ def delete
+ @uri
+ end
+
+ def put
+ @uri
+ end
+
+ def get=(uri)
+ @uri = uri
+ process_params('get', @resource, uri)
+ end
+
+ def post=(uri)
+ @uri = uri
+ process_params('post', @resource, uri)
+ end
+
+ def delete=(uri)
+ @uri = uri
+ process_params('delete', @resource, uri)
+ end
+
+ def put=(uri)
+ @uri = uri
+ process_params('put', @resource, uri)
+ end
+
+ private
+
+ # Helper to process/parse web parameters
+ def process_params(request_method, params, uri)
+ begin
+ # Set request method and generate a unique session key
+ session = "/tmp/#{UUID.new.generate}"
+
+ # Invoke a login request if necessary
+ session_request(params[:login]['http_method'], params[:login]['uri'], session,
+ params[:login].reject{ |k,v| k == 'http_method' || k == 'uri' },
+ params ) if params[:login]
+
+
+ # Check to see if we should actually run the request
+ return if params[:if] &&
+ skip_request?(params[:if]['http_method'], params[:if]['uri'], session,
+ params[:if]['parameters'], params[:if])
+
+ return if params[:unless] &&
+ !skip_request?(params[:unless]['http_method'], params[:unless]['uri'], session,
+ params[:unless]['parameters'], params[:unless])
+
+ # Actually run the request and verify the result
+ result = Curl::Easy::web_request(request_method, uri, params[:parameters],
+ :cookie => session, :follow => params[:follow])
+ verify_result(result,
+ :returns => params[:returns],
+ :body => params[:verify])
+ result.close
+
+ # Invoke a logout request if necessary
+ session_request(params[:logout]['http_method'], params[:logout]['uri'], session,
+ params[:logout].reject{ |k,v| k == 'http_method' || k == 'uri' },
+ params ) if params[:logout]
+
+ rescue Exception => e
+ raise Puppet::Error, "An exception was raised when invoking web request: #{e}"
+
+ ensure
+ FileUtils.rm_f(session) if params[:logout]
+ end
+ end
+
+ # Helper to issue login/logout requests
+ def session_request(http_method, uri, session, request_params, params = {})
+ Curl::Easy::web_request(http_method, uri, request_params,
+ :cookie => session, :follow => params[:follow]).close
+
+ end
+
+ # Helper to issue request to query if we should skip main request or not
+ def skip_request?(http_method, uri, session, request_params, params = {})
+ result = Curl::Easy::web_request(http_method, uri, request_params,
+ :cookie => session, :follow => params[:follow])
+
+ begin
+ verify_result(result,
+ :returns => params['returns'],
+ :body => params['verify'])
+
+ rescue Puppet::Error => e
+ return true
+
+ ensure
+ result.close
+ end
+
+ return false
+ end
+
+ # Helper to verify the response
+ def verify_result(result, verify = {})
+ if verify.has_key?(:returns) && !verify[:returns].nil? &&
+ !result.valid_status_code?(verify[:returns])
+ raise Puppet::Error, "Invalid HTTP Return Code: #{result.response_code},
+ was expecting one of #{verify[:returns].join(", ")}"
+ end
+
+ if verify.has_key?(:body) && !verify[:body].nil? &&
+ !result.valid_response_body?(verify[:body])
+ raise Puppet::Error, "Expecting #{verify[:body]} in the result"
+ end
+ end
+
+
+end
diff --git a/lib/puppet/type/web.rb b/lib/puppet/type/web.rb
new file mode 100644
index 0000000..a9e912c
--- /dev/null
+++ b/lib/puppet/type/web.rb
@@ -0,0 +1,96 @@
+require 'uri'
+
+Puppet::Type.newtype(:web) do
+ @doc = "Issue a request to a resource on the world wide web"
+
+ private
+
+ def self.validate_uri(url)
+ begin
+ uri = URI.parse(url)
+ raise ArgumentError, "Specified uri #{url} is not valid" if ![URI::HTTP, URI::HTTPS].include?(uri.class)
+ rescue URI::InvalidURIError
+ raise ArgumentError, "Specified uri #{url} is not valid"
+ end
+ end
+
+ def self.validate_http_status(status)
+ status = [status] unless status.is_a?(Array)
+ status.each { |stat|
+ stat = stat.to_s
+ unless ['100', '101', '102', '122',
+ '200', '201', '202', '203', '204', '205', '206', '207', '226',
+ '300', '301', '302', '303', '304', '305', '306', '307',
+ '400', '401', '402', '403', '404', '405', '406', '407', '408', '409',
+ '410', '411', '412', '413', '414', '415', '416', '417', '418',
+ '422', '423', '424', '425', '426', '444', '449', '450', '499',
+ '500', '501', '502', '503', '504', '505', '506', '507', '508', ' 509', '510'
+ ].include?(stat)
+ raise ArgumentError, "Invalid http status code #{stat} specified"
+ end
+ }
+ end
+
+ newparam :name
+
+ newproperty(:get) do
+ desc "Issue get request to the specified uri"
+ validate do |value| Puppet::Type::Web.validate_uri(value) end
+ end
+
+ newproperty(:post) do
+ desc "Issue post request to the specified uri"
+ validate do |value| Puppet::Type::Web.validate_uri(value) end
+ end
+
+ newproperty(:delete) do
+ desc "Issue delete request to the specified uri"
+ validate do |value| Puppet::Type::Web.validate_uri(value) end
+ end
+
+ newproperty(:put) do
+ desc "Issue put request to the specified uri"
+ validate do |value| Puppet::Type::Web.validate_uri(value) end
+ end
+
+ newparam(:parameters) do
+ desc "Hash of parameters to include in the web request"
+ validate do |value| Puppet::Type::Web.validate_uri(value) end
+ end
+
+ newparam(:returns) do
+ desc "Expected http return codes of the request"
+ defaultto ["200"]
+ validate do |value| Puppet::Type::Web.validate_http_status(value) end
+ munge do |value|
+ value = [value] unless value.is_a?(Array)
+ value = value.collect { |val| val.to_s }
+ value
+ end
+ end
+
+ newparam(:follow) do
+ desc "Boolean indicating if redirects should be followed"
+ newvalues(:true, :false)
+ end
+
+ newparam(:verify) do
+ desc "String to verify as being part of the result"
+ end
+
+ newparam(:login) do
+ desc "Login parameters to be used if a login is required before making the request"
+ end
+
+ newparam(:logout) do
+ desc "Logout parameters to be used if a logout is requred after making the request"
+ end
+
+ newparam(:unless) do
+ desc "Do not run request if the request specified here succeeds"
+ end
+
+ newparam(:if) do
+ desc "Only run request if the request specified here succeeds"
+ end
+end
--
1.7.2.3
Mo Morsi
spec/unit/type/web_spec.rb | 44 +++++++++++++++
2 files changed, 148 insertions(+), 0 deletions(-)
create mode 100644 spec/unit/provider/web/curl_spec.rb
create mode 100644 spec/unit/type/web_spec.rb
diff --git a/spec/unit/provider/web/curl_spec.rb b/spec/unit/provider/web/curl_spec.rb
new file mode 100644
index 0000000..1bb6f6c
--- /dev/null
+++ b/spec/unit/provider/web/curl_spec.rb
@@ -0,0 +1,104 @@
+#!/usr/bin/env rspec
+require 'spec_helper'
+
+provider_class = Puppet::Type.type(:web).provider(:curl)
+
+describe provider_class do
+ before :each do
+ @resource = Puppet::Resource.new(:web, 'foo')
+ @provider = provider_class.new(@resource)
+ end
+
+ def http_request(http_method, url)
+ @provider.method("#{http_method}=".to_sym).call url
+ end
+
+ ['get', 'post'].each do |http_method|
+ describe "##{http_method}" do
+ it "should issue #{http_method} request to uri" do
+ proc {
+ http_request http_method, "http://www.puppetlabs.com"
+ }.should_not raise_error(Puppet::Error)
+ end
+
+ it "should accept parameters for #{http_method} request to uri" do
+ proc {
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:parameters).returns({:q => 'puppet' })
+ @resource.stubs(:[]).with(:follow).returns(true)
+ http_request http_method, "http://www.google.com"
+ }.should_not raise_error(Puppet::Error)
+ end
+
+ it "should verify default success return http status code for #{http_method} request to uri" do
+ proc {
+ http_request http_method, "http://foobar"
+ }.should raise_error(Puppet::Error)
+ end
+
+ it "should verify return http status code for #{http_method} request to uri" do
+ proc {
+ expected_return_code = http_method == "get" ? '301' : '405'
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:follow).returns(false)
+ @resource.stubs(:[]).with(:returns).returns([expected_return_code])
+ http_request http_method, "http://google.com"
+ }.should_not raise_error(Puppet::Error)
+ end
+
+ it "should verify body contents for #{http_method} request to uri" do
+ proc {
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:verify).returns('.*http://docs.puppetlabs.com.*')
+ @resource.stubs(:[]).with(:follow).returns(true)
+ http_request http_method, "http://www.puppetlabs.com"
+ }.should_not raise_error(Puppet::Error)
+ end
+
+ # TODO figure out a good way to verify login/logout
+
+ it "should only issue #{http_method} request to uri if 'if' request is true" do
+ query_result = Curl::Easy.new
+ Curl::Easy.expects(:web_request).times(2).returns(query_result)
+ query_result.expects(:valid_status_code?).returns(true)
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:if).returns({ 'http_method' => 'get',
+ 'uri' => 'http://google.com', 'follow' => true, 'returns' => ['500'] })
+
+ http_request http_method, "http://www.puppetlabs.com"
+ end
+
+ it "should not issue #{http_method} request to uri if 'if' request is false" do
+ query_result = Curl::Easy.new
+ Curl::Easy.expects(:web_request).times(1).returns(query_result)
+ query_result.expects(:valid_status_code?).returns(false)
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:if).returns({ 'http_method' => 'get',
+ 'uri' => 'http://google.com', 'follow' => true, 'returns' => ['400'] })
+ http_request http_method, "http://www.puppetlabs.com"
+ end
+
+ it "should only issue #{http_method} request to uri if 'unless' request is false" do
+ query_result = Curl::Easy.new
+ Curl::Easy.expects(:web_request).times(2).returns(query_result)
+ query_result.expects(:valid_status_code?).returns(false)
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:unless).returns({ 'http_method' => 'get',
+ 'uri' => 'http://google.com', 'follow' => true, 'returns' => ['500'] })
+
+ http_request http_method, "http://www.puppetlabs.com"
+ end
+
+ it "should not issue #{http_method} request to uri if 'unless' request is true" do
+ query_result = Curl::Easy.new
+ Curl::Easy.expects(:web_request).times(1).returns(query_result)
+ query_result.expects(:valid_status_code?).returns(true)
+ @resource.stubs(:[]).returns(nil)
+ @resource.stubs(:[]).with(:unless).returns({ 'http_method' => 'get',
+ 'uri' => 'http://google.com', 'follow' => true, 'returns' => ['400'] })
+ http_request http_method, "http://www.puppetlabs.com"
+ end
+
+ end
+ end
+end
diff --git a/spec/unit/type/web_spec.rb b/spec/unit/type/web_spec.rb
new file mode 100644
index 0000000..c78ab83
--- /dev/null
+++ b/spec/unit/type/web_spec.rb
@@ -0,0 +1,44 @@
+#!/usr/bin/env rspec
+require 'spec_helper'
+
+host = Puppet::Type.type(:web)
+
+describe Puppet::Type.type(:web) do
+ before do
+ @class = host
+ end
+
+ it "should have :name be its namevar" do
+ @class.key_attributes.should == [:name]
+ end
+
+ describe "when validating attributes" do
+ [:parameters, :returns, :follow, :verify, :login, :logout, :unless].each do |param|
+ it "should have a #{param} parameter" do
+ @class.attrtype(param).should == :param
+ end
+ end
+
+ [:get, :post ].each do |property|
+ it "should have a #{property} property" do
+ @class.attrtype(property).should == :property
+ end
+ end
+ end
+
+ describe "when validating values" do
+ [:get, :post ].each do |property|
+ it "should require a valid uri for #{property} requests" do
+ proc { @class.new(:name => "#{property}_uri", property.to_sym => "") }.should raise_error(Puppet::Error,"Parameter #{property} failed: Specified uri is not valid")
+ end
+
+ it "should require a valid returns for #{property} requests if specified" do
+ proc { @class.new(:name => "#{property}_uri", property.to_sym => "http://www.puppetlabs.com", :returns => "invalid" ) }.should raise_error(Puppet::Error,"Parameter returns failed: Invalid http status code invalid specified")
+ end
+
+ it "should require a valid follow value for #{property} requests" do
+ proc { @class.new(:name => "#{property}_uri", property.to_sym => "http://www.puppetlabs.com", :follow => "invalid" ) }.should raise_error(Puppet::Error,'Parameter follow failed: Invalid value "invalid". Valid values are true, false. ')
+ end
+ end
+ end
+end
--
1.7.2.3
Jacob Helwig
Mo,
Thanks for taking the time to work on this.
I was going to ask why the login & logout parameters were necessary, but
then I took a peek in the provider, and saw the session handling code.
It seems like being able to specify something like
web { 'login_service':
post => 'http://...',
parameters => { ... },
}
web { 'logout_service':
post => 'http://...',
parameters => { ... },
}
web { 'actual_request':
get => 'http://...',
require => Web['login_service'],
before => Web['logout_service'],
}
would be nice. Especially if you're looking to do more than one
'authenticated' request, so you're not logging in and out for each
request. Though I haven't really explored the behavior around
refresh only resources and requiring them, and it seems like this could
get complex quickly. I also haven't looked into how to reasonably share
the session across the resources (which sounds like a royal pain to do
securely and sanely).
On a related note to the session handling code, it looks like the
session will only be removed if you specify both login & logout, but not
if you only specify login. Seems like you always expect login & logout
to either both be specified, or neither, but there aren't any explicit
checks in the type to enforce this.
The two commits should really be squashed together since we prefer that
tests go in the same commit as the code they're testing.
The hanging indent on the commit message is a bit odd, and there's a
bunch of trailing whitespace, but that can all be cleaned up when it's
merged in.
The only things I would really like to see before this gets merged in is
some additional testing on the type & provider, and the commits squashed
together.
* It doesn't look like anything in the tests check that something like
'blargle' won't be accepted for the returns parameter on the type.
* put and delete don't seem to be covered by the tests for the type or
provider.
* Now that I look more closely at the provider specs, I noticed that
you seem to be stubbing to set the properties on the resource, which
you don't need to do with a real resource. You should be able to
set them directly.
--
Jacob Helwig