http-client testing
Markus Pilman
I am implementing http-client for Zorba. The implementation is now
nearly finished and needs to be tested. I made some simple tests but
for more complicated stuff (like multipart), it is quite difficult to
write good test cases (without implementing something like a
HTTP-Server). So I have three questions:
1. Is there something like a test suite I could not find yet?
2. Has someone written some tests he can/want to share?
3. What is the best way to write tests (simple string comparison is
probably not powerful enough and relying on existing web-services is
probably also a bad idea)?
Thanks in advance for any input
Markus
Florent Georges
Hi,
> I am implementing http-client for Zorba.
> The implementation is now nearly finished and needs to be
> tested. I made some simple tests but for more complicated
> stuff (like multipart), it is quite difficult to write good
> test cases (without implementing something like a HTTP-Server).
we'd like to test is more the interaction with the external world
than the resut of an XPath expression. And I cannot see anything
else than writing some part of the tests on the server side to
simulate the external world in the case of the HTTP Client.
> So I have three questions:
> 1. Is there something like a test suite I could not find yet?
> 2. Has someone written some tests he can/want to share?
> 3. What is the best way to write tests (simple string
> comparison is probably not powerful enough and relying on
> existing web-services is probably also a bad idea)?
XSpec <http://code.google.com/p/xspec/> would be quite well
adapted. You write your tests as scenrios, associating a literal
result to compare or a test expression (or several of those) to
expressions to tests (here, calling the function with some
parameters).
Those are technical issues, and I am sure we could find
solutions to them quite easily if we look for practical ways to
perform function calls and test the result. But the point here,
I think, is to define test cases and write and maintain the tests
once we agreed on technical issues.
I guess a wiki would be here a nice tool to develop a test
suite, by sharing the test cases, either in a textual form or
with some runnable tests. I planned to make some maintenance on
the server this weekend, among which installing a wiki on the
EXPath website, so normally we should have one soon.
How did you write the tests you are using for now? (I mean,
technically: how are they run, and how are expressed the tests on
the result?)
Regards,
--
Florent Georges
http://www.fgeorges.org/
Florent Georges
Markus,
> And I cannot see anything else than writing some part of the
> tests on the server side to simulate the external world in the
> case of the HTTP Client.
I gave it more thoughts, and I think a server-side support is
needed here. I've written a simple servlet that loads test
definitions like the following:
<test xmlns="http://www.expath.org/ns/http-client/test">
<!-- method must be GET -->
<method>GET</method>
<!-- by default, unknown headers are rejected -->
<headers default="reject">
<!-- must be there, and value must match regex @value -->
<header name="X-EXPath-Test" value="The exact value"/>
<!-- just must be there -->
<header name="Host"/>
<!-- ignore them -->
<ignore name="User-Agent"/>
<ignore name="Connection"/>
</headers>
</test>
Based on the above definition, it will check the request is a
GET, has the header "X-EXPath-Test" with a specific value, has
the header "Host", and has no other headers than those and maybe
"User-Agent" and "Connection".
On success, it returns an entity application/xml with the
content "<ok>headers-001</ok>" (the name of the test). On
failure, it returns (for instance):
<fail test="headers-001">
<error>Header should be there: x-expath-test</error>
</fail>
With a bit of instrumentation, you can easily write test cases
as files with a http:request element (your test run query will
send the request and check the result).
This is just an idea, server-side tests should for instance be
able to check content of the request (for instance for a POST),
as well as other stuff. And this is only for checking request; I
should add the ability to return quite static pre-defined
responses also in order to test the analysis of the response by
the extension. But I like the idea.
You can find the servlet source code, the WAR file, a few
server tests as well as th corrsponding test cases (for my
implementation for XSLT for Saxon) at:
http://www.fgeorges.org/tmp/expath-test/
What do you think?
Florent Georges
> 2. Has someone written some tests he can/want to share?
By the way, maybe you will be interested by having look at
the XProc test suite: http://tests.xproc.org/testsuite/. In
particular, look for the tests http-request-*.
Markus Pilman
Thanks for your answer!
I think your idea for testing has a problem (or at least for our test
system): the tests needs to be configured on server and client side.
So either every developer needs to run his own server or every
developer needs access to the server. I think writing tests should be
as easy as possible.
I also thought about the problem and I would propose (and if you also
think it's a good idea also implement) a small server application
which just does the following:
- If the client makes a request to /test/request the server just reads
the request and generates an http:request-Node as defined in the
specification. It then sends this back to the client. The test system
has then just to check, that the http:request node from the anwser is
equivalent to the http:request node from the request. This could be
done easily with a query
-If the client makes a POST to /test/response the server excpects a
http:response node in the body of the request. The server will then
send the answer back to the client as requested by the http:response
node. Then again, two nodes can be tested for equivalience.
Of course we should define first, how these nodes have to be
equivalent (are additional headers allowed, has the order of the
headers to be the same etc.). But I think with a system like that
testing would be quite easy and there would be no need to change the
server for adding new tests.
What do you think about that?
Markus
> --
>
> You received this message because you are subscribed to the Google Groups "EXPath" group.
> To post to this group, send email to exp...@googlegroups.com.
> To unsubscribe from this group, send email to expath+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/expath?hl=en.
>
>
>
Markus Pilman
> How did you write the tests you are using for now? (I mean,
> technically: how are they run, and how are expressed the tests on
> the result?)
Our test suite simply executes queries and compares the result to some
expected result. So at the moment there are only very primitive tests
(like GET www.google.com). The rest of the functionality is nearly
untested (I just made some manual tests - but of course this is not a
solution also becuse http-client is a very important feature for us).
Markus
Florent Georges
Markus,
> I think your idea for testing has a problem (or at least for
> our test system): the tests needs to be configured on server
> and client side. So either every developer needs to run his
> own server or every developer needs access to the server. I
> think writing tests should be as easy as possible.
Yes, this "solution" is not as agile as tests like plain JUnit
or equivalent tests. However, the server can be configured to
look for tests into a repository divided by developers or
domains. Or each developers can run its own server (really,
these days, a servlet container like Tomcat or Jetty is really
light and fast, and can be started and stoped for every test
suite run).
I wonder though whether this is really worth the price to
develop such a flexible test framework. Normally, for
http:send-request, we would not end up with thousands of
different tests. But of course, that depends on your own coding
and testing standards and environments.
> - If the client makes a request to /test/request the server
> just reads the request and generates an http:request-Node as
> defined in the specification. It then sends this back to the
> client. The test system has then just to check, that the
> http:request node from the anwser is equivalent to the
> http:request node from the request. This could be done easily
> with a query
I think I see what you are trying to solve here: the server is
written once, then every developer can write his/her own tests
without requiring to add the equivalent test configuration on the
server side.
However, I think it will be rather hard to infer the original
request element back from the HTTP request (has this header be
set explicitely or by the imlpementation?, etc.)
> -If the client makes a POST to /test/response the server
> excpects a http:response node in the body of the request. The
> server will then send the answer back to the client as
> requested by the http:response node. Then again, two nodes can
> be tested for equivalience.
But then, how to test POSTing or PUTing a binary file? ;-)
> Of course we should define first, how these nodes have to be
> equivalent (are additional headers allowed, has the order of
> the headers to be the same etc.). But I think with a system
> like that testing would be quite easy and there would be no
> need to change the server for adding new tests.
Of course, not requiring to change the server side each time
you add a test would be interesting. But I am not convinced the
above description will allow you to test every aspects of the
extension.
Note that in the sample I wrote yesterday, the servlet looks
into the classpath for convenience, but it could instead look
into a plain directory. So a developer could start Jetty and
configure it with his/her own directory with test descriptions,
from the command line (that means, from your test suite run
scripts).
Of course, I maybe don't see some issues that could be related
to the way your developers have to define and run their tests.
But whatever technique you choose, I would be very interested if
you could share either (or ideally both of :-)) the technical
details and the tests themselves (in an "executable" or textual
form).
Anyway, thanks for your comemnts! Testing is a very important
part of the project, and this is really important to have other
points of view.
Michael Kay
The difficulty of testing has probably been the main reason I haven't added
HTTP functionality to Saxon a long time ago. I'm very conscious that any
area of the product with complex environment dependencies is a fruitful
source of bugs and support requests.
That's not to say it shouldn't be done.
Regards,
Michael Kay
http://www.saxonica.com/
http://twitter.com/michaelhkay