Rack driver support for direct HTTP calls

157 views
Skip to first unread message

Rob Holland

unread,
Mar 15, 2010, 8:10:28 AM3/15/10
to ruby-c...@googlegroups.com
Hi all,

I did some work on allowing the rack driver to support direct HTTP
calls get/put/post/delete so that we could use capybara to test a
site's API alongside testing it's web UI interface.

Without this work we would have to have two test suites, one to test
the website, and one to test the API. The API suite would not be able
to use capybara because it makes use of HTTP calls that browsers would
not make (such as PUT/DELETE), which also aren't found via forms/links
(as they are an API).

In order to test this API with something other than capybara we would
basically need to copy the vast majority of the capybara code with
it's rack driver into something which did support direct HTTP calls.

I've asked Jonas to merge my work, but he's concerned that people
would make use of this with the rack driver and then be surprised when
it didn't work if they switched to a different driver (there is no way
to get culerity/selenium to make direct HTTP calls as they will only
do what a browser will). My feeling is that people should already be
used to drivers having different functionality, javascript being a
good case in point.

Jonas' other concern was that capybara shouldn't do what a browser
itself can't. Now, I have to say I do agree in general, but the
pragmatist in me feels that seeing as the change required to allow
capybara to be used for testing both APIs and websites themselves is
so small, and that not allowing it means entirely separate suites
(because even if we wrote our own code, most likely the step
definitions would conflict), I'd like to make this feature an
exception.

Does anyone else have an opinion either way?

The changes in question are on my headers branch
http://github.com/robholland/capybara/tree/headers

Changes versus master
http://github.com/robholland/capybara/compare/master...headers

Thanks,

Rob

Jonas Nicklas

unread,
Mar 15, 2010, 10:40:08 AM3/15/10
to ruby-c...@googlegroups.com
I'm not dogmatically opposed to this, the pragmatist in me does see
the value. I'd just like there to be a careful discussion of the
possible benefits/pitfalls before we merge something that might work
in an undesirable way.

/Jonas

Hedge Hog

unread,
Mar 15, 2010, 5:45:55 PM3/15/10
to ruby-c...@googlegroups.com
On Tue, Mar 16, 2010 at 1:40 AM, Jonas Nicklas <jonas....@gmail.com> wrote:
> I'm not dogmatically opposed to this, the pragmatist in me does see
> the value. I'd just like there to be a careful discussion of the
> possible benefits/pitfalls before we merge something that might work
> in an undesirable way.
>

On the whole I am in favor of what Rob suggests, I too will need to
exercise a server API in the days ahead.
Just to be sure I have not misunderstood the only issues are:
1.) The Rack driver will have different behavior than some other drivers
2.) "capybara shouldn't do what a browser itself can't"

On 1.) I agree with Rob's observation, and different driver behaviors
could be higlighted in the documentation.
On 2.) my reflex is that when using capybara we aren't exercising
browser behavior, rather we are exercising the server's behavior
(given it has to deal with client/browser X). Specifically we are
checking that when the 'user' does "some things" the server responds
in a way that ensures some expected result is visible/accessible to
the user. Often that result must be delivered in a way that some
intemediary (browser) can process in order for the user to see/get the
expected result. But it seems to me that libs like capybara are not
about describing a browser's behavior, and more about describing the
server's behavior given it has to deal with browser X.
If this is the case then I don't see a need to limit capybara to
browser behavior, since it is actually server behavior that is of
interest.
It might be worth stating whether capybara aims to support a cleint
that is fully HTTP 1.1 compliant?

Perhaps I've misunderstood?

My 2c.
H

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

Hedge Hog

unread,
Mar 15, 2010, 5:51:56 PM3/15/10
to ruby-c...@googlegroups.com
On Tue, Mar 16, 2010 at 8:45 AM, Hedge Hog <hedgeho...@gmail.com> wrote:

> It might be worth stating whether capybara aims to support a cleint
> that is fully HTTP 1.1 compliant?

As the Secretary of State would say, 'I misspoke':
It might be worth stating whether capybara aims to support servers
that expect a HTTP 1.1 compliant client to be present.

Regards

Dante Briones

unread,
Mar 16, 2010, 2:53:30 PM3/16/10
to ruby-c...@googlegroups.com
>
> On 2.) my reflex is that when using capybara we aren't exercising
> browser behavior, rather we are exercising the server's behavior
> (given it has to deal with client/browser X).

I have a different view: tools like Capybara allow the developer to test the application stack from the user's perspective. Sometimes (like when using the Selenium driver), you want to test (nearly) the complete stack. In other circumstances, (to reduce test execution time, for instance), you test a narrower slice of the stack by using C[u|e]lerity, etc. The server's behavior is an important part of that. But only part.

> But it seems to me that libs like capybara are not
> about describing a browser's behavior, and more about describing the
> server's behavior given it has to deal with browser X.

Huh. That doesn't make a lot of sense to me in the case of Capybara. If the focus of Capybara were about testing the server, I would expect its API to include methods for making arbitrary HTTP requests. Instead, the API is written in terms of user actions like click() and visit(). Can you expand on that?

I think this gets to the heart of an issue that's been hanging around for some time, which is that there is no clearly defined design philosophy for Capybara. For comparison's sake, take a look at how WebDriver defines itself:

"WebDriver is a tool for writing automated tests of websites. It *aims to mimic the behaviour of a real user*, and as such interacts with the HTML of the application."

http://code.google.com/p/selenium/wiki/FrequentlyAskedQuestions

I think there should first be some consensus about whether Capybara is a tool for interacting with a web application in the same way a user would, or a more general-purpose tool for making HTTP requests.

--
Dante Briones
http://dantebriones.com/
mobile: +1 (415) 283-5462


Rob Holland

unread,
Mar 16, 2010, 4:06:01 PM3/16/10
to ruby-c...@googlegroups.com
On Tue, Mar 16, 2010 at 6:53 PM, Dante Briones <dbri...@gmail.com> wrote:

> I think there should first be some consensus about whether Capybara is a tool for interacting with a web application in the same way a user would, or a more general-purpose tool for making HTTP requests.

I believe it is almost certainly the former, hence the uncertainty
about my proposed changes.

Hedge Hog

unread,
Mar 16, 2010, 5:12:18 PM3/16/10
to ruby-c...@googlegroups.com
On Wed, Mar 17, 2010 at 5:53 AM, Dante Briones <dbri...@gmail.com> wrote:
>>
>> On 2.) my reflex is that when using capybara we aren't exercising
>> browser behavior, rather we are exercising the server's behavior
>> (given it has to deal with client/browser X).
>
> I have a different view: tools like Capybara allow the developer to test the application stack from the user's perspective.  Sometimes (like when using the Selenium driver), you want to test (nearly) the complete stack.  In other circumstances, (to reduce test execution time, for instance), you test a narrower slice of the stack by using C[u|e]lerity, etc.  The server's behavior is an important part of that.  But only part.

Agreed. However, you'll find in an earlier post I related my
experience that when you are using Rack::Test you are likely not
exercising that stack, I posted some links to an alternative that had
been suggested, but don't have that link right at hand.
Also it does seem I'm close to the only one with that view so I
wouldn't put too much weight on my observations/experiences.

Anyway, even claiming to test pure-server behavior with capybara is
also likely incorrect when standing behind RT - I don't have a
complete example now so treat that claim as an ignorant user's innane
chatter ;)

>
>>   But it seems to me that libs like capybara are not
>> about describing a browser's behavior, and more about describing the
>> server's behavior given it has to deal with browser X.
>
> Huh.  That doesn't make a lot of sense to me in the case of Capybara. If the focus of Capybara were about testing the server, I would expect its API to include methods for making arbitrary HTTP requests.

I had tended to think of browser based interactions as a special case
of HTTP requests. It seems the question is whether capybara aims to
accomodate the more general case, testing RESTful API's, albeit with a
specialised driver if required. Or if the broswer case is its
speciality with no intentions beyond that.

> Instead, the API is written in terms of user actions like click() and visit().  Can you expand on that?
>

To my mind these were just, important, aliases/euphamisms for invoking
some HTTP, e.g. capybara dosn't handle click on a QT/Gnome dialog, or
visiting a FTP site. Don't get me wrong, I think the HTTP abstraction
is central, but needn't be exclusive (barring some implementation
gotcha).
However, if more and more people are going to be working on projects
that are a hybrid of serving some RESTful API and classical browser
pages, it would be convenient to be able to use one library for
automated tests of both sets if behaviours.

> I think this gets to the heart of an issue that's been hanging around for some time, which is that there is no clearly defined design philosophy for Capybara.  For comparison's sake, take a look at how WebDriver defines itself:
>
> "WebDriver is a tool for writing automated tests of websites. It *aims to mimic the behaviour of a real user*, and as such interacts with the HTML of the application."
>
> http://code.google.com/p/selenium/wiki/FrequentlyAskedQuestions
>
> I think there should first be some consensus about whether Capybara is a tool for interacting with a web application in the same way a user would, or a more general-purpose tool for making HTTP requests.
>

Agreed.

Regards

Denro

unread,
Mar 16, 2010, 9:21:28 PM3/16/10
to ruby-c...@googlegroups.com
"Capybara aims to simplify the process of integration testing Rack applications, ..." - from the readme. I think this simplifies the process of integration testing. If you want to do that with capybara on the other hand is up to the user.

Also my experience is that some simulators or drivers have different benefits and pitfalls just as Mr. Briones describes ( also in context of actual browser features ) and I believe that is why capybara have flourished because it provides a wide and "agile" arsenal for many intended purposes without the hassle.

hope i dont scare anyone away in my (ab)use of the english language. 

Stephen Eley

unread,
Mar 17, 2010, 1:00:24 AM3/17/10
to ruby-c...@googlegroups.com
On Mon, Mar 15, 2010 at 8:10 AM, Rob Holland <rob.h...@gmail.com> wrote:
>
> Jonas' other concern was that capybara shouldn't do what a browser
> itself can't. Now, I have to say I do agree in general, but the
> pragmatist in me feels that seeing as the change required to allow
> capybara to be used for testing both APIs and websites themselves is
> so small, and that not allowing it means entirely separate suites
> (because even if we wrote our own code, most likely the step
> definitions would conflict), I'd like to make this feature an
> exception.

I realize I'm getting into this fairly late, but on reading this
through, my first thought was that Capybara seems too _heavyweight_
for testing an API with REST calls. Almost everything about Capybara
is there to specify 'visual' or page-oriented interaction -- selecting
things in forms, clicking on links, etc.

I just don't see what you'd *gain* by trying to use Capybara to test
your non-visual API. Instead, why not use Rest-Client or HTTParty and
write Cucumber steps that drive them directly? You'd probably end up
with less code, and the literal responses you got back would be easier
to write matchers against.

I say -1, unless I'm missing something obvious.


--
Have Fun,
Steve Eley (sfe...@gmail.com)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org

Matt Wynne

unread,
Mar 17, 2010, 4:19:01 AM3/17/10
to ruby-c...@googlegroups.com

I think Stephen has a point here. Keeping Capybara specialised for
simulating user actions against a user interface will keep it simple.

This might be a stupid question as I haven't looked at the OP's code,
but can't this kind of thing already be done using rack-test?

cheers,
Matt

http://mattwynne.net
+447974 430184

Rob Holland

unread,
Mar 17, 2010, 5:26:39 AM3/17/10
to ruby-c...@googlegroups.com
> I think Stephen has a point here. Keeping Capybara specialised for
> simulating user actions against a user interface will keep it simple.
>
> This might be a stupid question as I haven't looked at the OP's code, but
> can't this kind of thing already be done using rack-test?

Yes, except then I have to use different language in my stories.

We currently do things like:

When I put json "{'name':'fred'}" to
"/api/v1/person/something.json" as "us...@example.com"
Then I should see "fred"

The should see would need to be worded differently simply to avoid the
existing capybara version.

I'm not going to argue about it though. This seemed like the pragmatic
solution. If people don't want it in capybara I'll turn it into a
tag-triggered patch al la the rails javascript emulation.

aslak hellesoy

unread,
Mar 17, 2010, 6:14:08 AM3/17/10
to ruby-c...@googlegroups.com
On Mon, Mar 15, 2010 at 1:10 PM, Rob Holland <rob.h...@gmail.com> wrote:

Although this allows programmers to do things you can't do as a browser user, the pragmatist in me says it's a neat addition.

One caveat - as far as I know, Capybara assumes all responses are HTML, and in rack mode tries to parses response with Nokogiri. -Or does it only do that if the MIME type is text/html? It's not clear to me how things would work if we #get a resource that returns something that is e.g. application/json.

Aslak

Thanks,

Rob

Jonas Nicklas

unread,
Mar 17, 2010, 8:32:28 AM3/17/10
to ruby-c...@googlegroups.com
I agree with Stephen that if you're testing a REST API, Capybara is
the wrong tool, and so is Cucumber, probably. RackTest offers you all
you need to write REST API tests. That being said, where I see it as
useful is in something like the following:

Given I go to the new ticket page
When I open a ticket
Then I should see "Status: open"
And the ticket is closed via the API
When I refresh the page
Then I should see "Status: closed"

Here, an API action becomes a part of the user's workflow, and I can
see how this is useful. BUT, imagine we change this scenario now to
something like this:

Given I go to the new ticket page
When I open a ticket via the fancy AJAX interface
Then I should see "Status: open"
And the ticket is closed via the API
When I refresh the page
Then I should see "Status: closed"

That's just not going to work the way we want it to. I dislike the
idea of encouraging something that's going to make people write
features in a way that simply will never work on the other drivers.

As for Aslak's concern about what to do with the response, if we do
this I don't think the REST calls should affect the "user session" at
all, it should be as though the REST calls were issued by a completely
different entity. That actually makes them substantially less useful
as well, since you'd be unable to test any aspect of the response.

/Jonas

Rob Holland

unread,
Mar 17, 2010, 9:01:37 AM3/17/10
to ruby-c...@googlegroups.com
On Wed, Mar 17, 2010 at 12:32 PM, Jonas Nicklas <jonas....@gmail.com> wrote:
> I agree with Stephen that if you're testing a REST API, Capybara is
> the wrong tool, and so is Cucumber, probably. RackTest offers you all
> you need to write REST API tests.

Ok, I'll turn it into a tag-triggered patch. I'll rework my headers
branch to remove the direct HTTP verb stuff but keep the delegation
tidy up and the response-building optimisation.

Rob Holland

unread,
Mar 17, 2010, 9:20:42 AM3/17/10
to ruby-c...@googlegroups.com
On Wed, Mar 17, 2010 at 1:01 PM, Rob Holland <rob.h...@gmail.com> wrote:

> Ok, I'll turn it into a tag-triggered patch. I'll rework my headers
> branch to remove the direct HTTP verb stuff but keep the delegation
> tidy up and the response-building optimisation.

Done, please consider merging the headers branch now.

Thanks,

Rob

Stephen Eley

unread,
Mar 17, 2010, 12:26:40 PM3/17/10
to ruby-c...@googlegroups.com
On Wed, Mar 17, 2010 at 5:26 AM, Rob Holland >

> Yes, except then I have to use different language in my stories.
>
> We currently do things like:
>
>    When I put json "{'name':'fred'}" to
> "/api/v1/person/something.json" as "us...@example.com"
>    Then I should see "fred"

I think you could make your life easier by generalizing that.
Cucumber's optimal for really high-level stuff:

Given I am authenticated as "us...@example.com"
When I submit a "name" of "fred" to "something"


Then I should see "fred"

Your step definitions can then actually set up the basic
authentication for us...@example.com and know that _submitting to
"something"_ really means converting to JSON and sending a PUT. And
writing a matcher against HTTP responses for "Then I should see" is
nearly trivial.

The beauty of doing it that way -- besides the "business people can
read it" argument that's often given -- is that when you need to test
the same interaction via Web forms (or batch loading from a file, or
punch cards, or whatever) your scenarios don't need to change. You
can just swap out the step definitions.

George Dinwiddie

unread,
Mar 17, 2010, 1:01:43 PM3/17/10
to ruby-c...@googlegroups.com

Well said, Stephen! That's exactly what I was thinking, but didn't have
the energy to put into words. I think working at that high level is the
only way to keep your tests maintainable. (See, also,
http://dhemery.com/pdf/writing_maintainable_automated_acceptance_tests.pdf
thought the examples use different technology.)

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

Matt Wynne

unread,
Mar 20, 2010, 9:26:40 PM3/20/10
to ruby-c...@googlegroups.com

On 17 Mar 2010, at 12:32, Jonas Nicklas wrote:

> I agree with Stephen that if you're testing a REST API, Capybara is
> the wrong tool, and so is Cucumber, probably.

FWIW, We used RSpec and Cucumber to write acceptance tests for
Songkick's API, and we ended up getting on a *lot* better with
Cucumber. It was really nice to have examples of the JSON / XML
responses in the features, even if they did seem a bit long-winded.

cheers,

Jonas Nicklas

unread,
Mar 22, 2010, 5:11:48 PM3/22/10
to ruby-c...@googlegroups.com
Just pulled in the changes from Rob's branch. This gives us very
unofficial support for HTTP calls, but I'm not going to document it
and caution people against using it at the moment.

/Jonas

> To unsubscribe from this group, send email to
> ruby-capybara+unsubscribegooglegroups.com or reply to this email with the
> words "REMOVE ME" as the subject.
>

JM

unread,
Apr 1, 2010, 6:57:51 PM4/1/10
to Capybara
Hi there,

This is my first post in capybara mailing list and I hope not the last
one ;-)

After I saw Jonas at scotruby conf last WE, I was very impressed and I
have decided to migrate from webrat to capybara.

I am having an hard time converting the API features :-(

This is what we have with webrat:
When I call the API "project search" with data:
| page | per-page |
| 1 | 2 |

When /^I call the API "([^\"]*)" with data:$/ do |path,table|
host! "api.betterplace.local"
basic_auth('user','pwd')
visit path_to("api #{path}"), :post, table.hashes.first.to_xml(:root
=> 'request')
end

I have successfully implemented a host! and basic_auth methods for
capybara:
def host!(hostname)
Capybara.default_host = hostname
end

def basic_auth(user, pass)
page.driver.basic_authorize(user, pass)
end

But I am blocked with the API call itself, how do you simulate a post
with Rack::Test and include some param in the request?

Rob, coud you send us some gist with simple examples pls?

It's getting late and I should really go to bed!

Cheers,

JM

Jonas Nicklas

unread,
Apr 9, 2010, 8:39:48 AM4/9/10
to ruby-c...@googlegroups.com
something like

page.driver.post('url', { :params => 'here' })

should do the trick

/Jonas

> --
> To unsubscribe, reply using "remove me" as the subject.
>

Reply all
Reply to author
Forward
0 new messages