Unit testing

904 views
Skip to first unread message

Matthew Brunelle

unread,
Jun 23, 2011, 9:01:51 AM6/23/11
to scrapy-users
How does one go about implementing unit testing in a scrapy project?
I understand that scrapy uses twisted's trail for internal testing.
Should the same thing be used for testing a project? If so how?

Pablo Hoffman

unread,
Jun 23, 2011, 1:09:00 PM6/23/11
to scrapy...@googlegroups.com
What are you planning to test? spiders or other project common code?

Standard Python unit-testing should be enough, unless you need to test
asynchronous/deferred code.

> --
> You received this message because you are subscribed to the Google Groups "scrapy-users" group.
> To post to this group, send email to scrapy...@googlegroups.com.
> To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

Matthew Brunelle

unread,
Jun 23, 2011, 1:18:43 PM6/23/11
to scrapy-users
I'd like to test the spider.

Pablo Hoffman

unread,
Jun 23, 2011, 2:13:37 PM6/23/11
to scrapy...@googlegroups.com
There's an idea for automating some spider tests that never got implemented (so
far): http://dev.scrapy.org/wiki/SEP-017

But nothing prevents you from using standard Python unit-testing facilities to
test the Spider object methods (ie. the callbacks).

You build the Response in the test code, and then call the method with the
response and assert that you get the expected items/requests in the output.

A warning though: automating is the key. Otherwise, if you manage a lot of
spiders, you may find that testing may take even longer than writing the
spiders themselves - though this applies to other software testing as well, not
only spiders :)

Pablo.

Matthew Brunelle

unread,
Jun 23, 2011, 4:26:41 PM6/23/11
to scrapy-users
Thank you very much. Now how do I go about calling the spider from a
script?

Matthew Brunelle

unread,
Jun 24, 2011, 10:56:33 AM6/24/11
to scrapy-users
Here's my cross post to stack overflow http://stackoverflow.com/questions/6456304/scrapy-unit-testing

On Jun 23, 4:26 pm, Matthew Brunelle <matthew.brune...@gmail.com>
wrote:

Leonardo Lazzaro

unread,
Jun 24, 2011, 1:24:54 PM6/24/11
to scrapy...@googlegroups.com
check this snipplet

Matthew Brunelle

unread,
Jun 24, 2011, 2:16:10 PM6/24/11
to scrapy-users
Thank you very much! The spider runs perfectly with this, but it seems
the spider never returns and the script just hangs after executing the
firs spider.

On Jun 24, 1:24 pm, Leonardo Lazzaro <lazzaroleona...@gmail.com>
wrote:
Reply all
Reply to author
Forward
0 new messages