Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Contemplating a URI Module

306 views
Skip to first unread message

Romeo Theriault

unread,
Feb 4, 2013, 3:48:32 PM2/4/13
to ansible...@googlegroups.com
So, I've been thinking about creating a general purpose http client URI/URL module for ansible; allowing ansible to interact with web services. Some example use cases I've thought of:

* Create a JIRA ticket on some condition using JIRA's RESTful API
* Delete/Update said JIRA ticket
* Add a host to 'your-favorite' monitoring/CMDB system via it's web API. (I use zabbix which has such an API.)
* Delete said host from the monitoring/CMDB system.
* Sending ansible 'setup' module json output to some yet created webservice for host-inventory cataloging.
* Generally, just allowing ansible to interact with web-services.

Still in the planning stages but the biggest problem I see right now is that the standard & included HTTP client libraries in python have some rather big holes.

urllib2 only supports GET and POST HTTP methods.
httplib doesn't appear to support authentication, proxies, compression, cookies or redirects.

That leaves either non-standard python http client libraries or curl. Sticking with python will certainly make the job easier. The two popular python http client libs I see are cover all/most features are:

* httplib2 (big bonus in that it runs on python 2.4)
* requests (doesn't support python 2.4)

I'm tempted to use httplib2 for this since it'll be sticking with python but also support python 2.4. But before I start working on this though I'd like to get feedback from others to see if I might be missing something glaringly obvious. While I realize this likely won't be support in core due to external deps (?) I'd like to make this module useful to others as well.

Thanks!

--
Romeo

Michael DeHaan

unread,
Feb 4, 2013, 3:57:07 PM2/4/13
to ansible...@googlegroups.com
Sounds fantastic for moving us closer to general purpose orchestration.

Anything as long as it is not requests -- *it* has too many holes in
various cases.

urllib2 with httplib DELETE for odd methods was, I think, how I did it
in the past ... can't remember?

(I know Seth has some opinions.)
> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Brian Coca

unread,
Feb 4, 2013, 5:17:22 PM2/4/13
to ansible...@googlegroups.com
I'm using pycurl for something similar but this requires itself and
curl as dependencies
--
Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001
Pedo mellon a minno

Michael DeHaan

unread,
Feb 4, 2013, 5:32:59 PM2/4/13
to ansible...@googlegroups.com
BTW, for POSTS and PUTS, do you see that going from a template in many
cases? with_template could be very good for that.

XML should not /usually/ care about whitespace.

I also want to make AnsibleModuleCommon transparently start using JSON
which will cleanup some parsing dependencies anyway... and enable what
some people have been wanting about sending complex arguments to
modules.

Romeo Theriault

unread,
Feb 4, 2013, 6:26:14 PM2/4/13
to ansible...@googlegroups.com
On Mon, Feb 4, 2013 at 12:32 PM, Michael DeHaan <michael...@gmail.com> wrote:
BTW, for POSTS and PUTS, do you see that going from a template in many
cases?  with_template could be very good for that.

Yes, I imagined templates would be quite useful in generating the appropriate payload to send. Wasn't aware of 'with_template', thanks for pointing that out.
 
XML should not /usually/ care about whitespace.

I also want to make AnsibleModuleCommon transparently start using JSON
which will cleanup some parsing dependencies anyway... and enable what
some people have been wanting about sending complex arguments to
modules. 

Ok, I'll keep this in mind as I proceed.

--
Romeo

Jan-Piet Mens

unread,
Feb 5, 2013, 1:35:17 AM2/5/13
to ansible...@googlegroups.com
> So, I've been thinking about creating a general purpose http client URI/URL
> module for ansible; allowing ansible to interact with web services.

Yes, please!

> While I realize this likely won't be support in core due
> to external deps (?) I'd like to make this module useful to others as well.

I certainly hope it gets into core... but it shouldn't have external
dependencies (like requests, pycurl, etc.). Or make those optional (i.e.
if they are available, use them, else fall back to whatever's there? A
lot of work, but maybe worthwhile? :)

-JP

Romeo Theriault

unread,
Feb 5, 2013, 7:44:34 PM2/5/13
to ansible...@googlegroups.com
On Mon, Feb 4, 2013 at 12:32 PM, Michael DeHaan <michael...@gmail.com> wrote:
BTW, for POSTS and PUTS, do you see that going from a template in many
cases?  with_template could be very good for that.

XML should not /usually/ care about whitespace.

I also want to make AnsibleModuleCommon transparently start using JSON
which will cleanup some parsing dependencies anyway... and enable what
some people have been wanting about sending complex arguments to
modules.

I've a very basic prototype of the 'uri' module going but am finally understanding what you mean by parsing issues with complex arguments.  

Right now I'm accepting the following arguments to the module: url, dest, user, password, body, method and headers.

I'm trying to figure out the best way to send headers, which the module should see as a dict, to the module. I was hoping to be able to specify like so:

# Example task
 - name: Testing
   uri: url=http://super.awesome.com user=joe password='secret' method=GET headers='{"Content-Type": "application/atom+xml", "Second-Header-name": "header-val"}' body='body in here'

But this doesn't work, since yaml doesn't like the ':' in the string. I've tried a bunch of quoting variations etc... but am thinking this might not be the best method to go at this.

Any advice on how I should approach sending this type of data to a module?

Thanks,
Romeo

Romeo Theriault

unread,
Feb 5, 2013, 7:50:16 PM2/5/13
to ansible...@googlegroups.com
On Mon, Feb 4, 2013 at 8:35 PM, Jan-Piet Mens <jpm...@gmail.com> wrote:
> So, I've been thinking about creating a general purpose http client URI/URL
> module for ansible; allowing ansible to interact with web services.

Yes, please!
 
Cool, glad you think it'll be useful.
 

> While I realize this likely won't be support in core due
> to external deps (?) I'd like to make this module useful to others as well.

I certainly hope it gets into core... but it shouldn't have external
dependencies (like requests, pycurl, etc.). Or make those optional (i.e.
if they are available, use them, else fall back to whatever's there? A
lot of work, but maybe worthwhile? :)

Unfortunately, At least in the short term, I'm going to use httplib2 for this since it covers everything in httplib and urllib2 in one library. While I'm wrapping my head around module development and programming http I don't want to have to worry about learning and debugging two different libraries :) Good news is that is works on python 2.4 and seems to be well supported in the different distros.

--
Romeo

Michael DeHaan

unread,
Feb 5, 2013, 10:00:11 PM2/5/13
to ansible...@googlegroups.com
We really need to implement complex args to modules to make this tenable.

That being said, we are doing that, but headers is a hash, it should
not be a eviley quoted string to make it accessible.

All of this will be in work, as I've discussed a few times in the
past, so stay tuned :)

--Michael

Romeo Theriault

unread,
Feb 5, 2013, 11:59:48 PM2/5/13
to ansible...@googlegroups.com
On Tue, Feb 5, 2013 at 5:00 PM, Michael DeHaan <michael...@gmail.com> wrote:
We really need to implement complex args to modules to make this tenable.

That being said, we are doing that, but headers is a hash, it should
not be a eviley quoted string to make it accessible.

All of this will be in work, as I've discussed a few times in the
past, so stay tuned :)

Alright, thanks. I'll stay tuned. In the mean time I'll keep working on other aspects of the module. 

--
Romeo

Jan-Piet Mens

unread,
Feb 6, 2013, 3:33:46 AM2/6/13
to ansible...@googlegroups.com
> Good news is that is works on python 2.4 and seems to be well
> supported in the different distros.

That's all that matters, at least to me. Good stuff.

-JP

Darren Chamberlain

unread,
Feb 6, 2013, 11:35:52 AM2/6/13
to ansible...@googlegroups.com
* Romeo Theriault <romeo.theriault at maine.edu> [2013/02/05 14:44]:
> But this doesn't work, since yaml doesn't like the ':' in the
> string. I've tried a bunch of quoting variations etc... but am
> thinking this might not be the best method to go at this.
>
> Any advice on how I should approach sending this type of data to a
> module?

URI encoding?

--
Darren Chamberlain <dar...@boston.com>

Romeo Theriault

unread,
Feb 6, 2013, 6:47:54 PM2/6/13
to ansible...@googlegroups.com
On Wed, Feb 6, 2013 at 6:35 AM, Darren Chamberlain <dar...@boston.com> wrote:
* Romeo Theriault <romeo.theriault at maine.edu> [2013/02/05 14:44]:
> But this doesn't work, since yaml doesn't like the ':' in the
> string. I've tried a bunch of quoting variations etc... but am
> thinking this might not be the best method to go at this.
>
> Any advice on how I should approach sending this type of data to a
> module?

URI encoding?

Thanks for the response. I'm no expert on this, so by all means prove me wrong :), but I believe URI encoding is a different thing than HTTP Header encoding. I've been googling a bit trying to see how/if it would be possible to encode the headers so they could be passed to the module in one go and am coming up somewhat empty handed. The best I'm finding is this RFC [Character Set and Language Encoding for Hypertext Transfer Protocol (HTTP) Header Field Parameters]

 

(https://tools.ietf.org/html/rfc5987)  which states:

By default, message header field parameters in Hypertext Transfer Protocol (HTTP) messages cannot carry characters outside the ISO- 8859-1 character set....

I'm thinking though, that even if this is possible, it's really not the way for users to be interacting with the module. They'd expect to pass an un-encoded header name and value and I believe that's what we should do. Also, that's what the library expects :)  


--
Romeo

Darren Chamberlain

unread,
Feb 6, 2013, 9:17:35 PM2/6/13
to ansible...@googlegroups.com
* Romeo Theriault <romeo.theriault at maine.edu> [2013/02/06 13:47]:
> On Wed, Feb 6, 2013 at 6:35 AM, Darren Chamberlain <dar...@boston.com>wrote:
> > * Romeo Theriault <romeo.theriault at maine.edu> [2013/02/05
> > 14:44]:
> > > Any advice on how I should approach sending this type of data
> > > to a module?
> >
> > URI encoding?
>
> Thanks for the response. I'm no expert on this, so by all means
> prove me wrong :), but I believe URI encoding is a different thing
> than HTTP Header encoding.

Yes, I just meant to protect the specical characters (like : and /)
from yaml. Then the uri module would decode them and treat them as
MIME, as TimBL intended.

--
Darren Chamberlain <dar...@boston.com>

Romeo Theriault

unread,
Feb 6, 2013, 10:36:47 PM2/6/13
to ansible...@googlegroups.com
Ahh, gotcha. Neat idea. If we're going to add the ability to pass more complex arguments to modules I think that would be clearer solution in this case.

--
Romeo

Michael DeHaan

unread,
Feb 7, 2013, 7:37:51 AM2/7/13
to ansible...@googlegroups.com
> Ahh, gotcha. Neat idea. If we're going to add the ability to pass more
> complex arguments to modules I think that would be clearer solution in this
> case.

ETA is probably 1-2 weeks and then this will be possible. Have some
other features I want to finish higher up on the list (fireball
upgrades, dry run completion).

Seth Vidal

unread,
Feb 7, 2013, 12:19:25 PM2/7/13
to ansible...@googlegroups.com
Imo - requests not working on 2.4 is a benefit as it makes 2.4 hosts go
away :)

However, I understand the limits people are under.

Of all the http-interfacing modules i've used in python - I like
python-requests the most.

urllib* make simple things difficult and difficult things only possible.

pycurl is fine - right up to the point where you need ssl support and then
it is a bag-of-cats based on which ssl stack you're built against (nss,
openssl or gnutls)


if 2.4 is not really an issue - I'd say use python-requests.

that's my 2c nothing more
-sv

Romeo Theriault

unread,
Feb 7, 2013, 2:58:21 PM2/7/13
to ansible...@googlegroups.com
Sounds good. Thanks. I continue to hack away at other aspects of the module.

--
Romeo

Brian Dunbar

unread,
Mar 18, 2014, 4:39:07 PM3/18/14
to ansible...@googlegroups.com
I came here looking exactly for this.

Is this uri module ready for prime time?  Or even a beta?

What I've facing is managing an elasticsearch (ES) cluster.  ES likes to be managed using http like so ..

- Take a snapshot
curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_1?

Which I can do with a lot of shell in playbooks but - as you know - that gets messy in a big hurry.  

Matt Martz

unread,
Mar 18, 2014, 4:48:35 PM3/18/14
to ansible...@googlegroups.com
Brian,

This thread is over a year old. There is a URI module and you can find the docs at 
http://docs.ansible.com/uri_module.html

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.


--
Matt Martz
ma...@sivel.net
http://sivel.net/

Michael DeHaan

unread,
Mar 18, 2014, 4:48:36 PM3/18/14
to ansible...@googlegroups.com
The uri module is in core and is therefore something we support.

Let us know if you are having specific problems.





--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages