Unit testing

Matthew Brunelle

unread,

Jun 23, 2011, 9:01:51 AM6/23/11

to scrapy-users

How does one go about implementing unit testing in a scrapy project?
I understand that scrapy uses twisted's trail for internal testing.
Should the same thing be used for testing a project? If so how?

Pablo Hoffman

unread,

Jun 23, 2011, 1:09:00 PM6/23/11

to scrapy...@googlegroups.com

What are you planning to test? spiders or other project common code?

Standard Python unit-testing should be enough, unless you need to test
asynchronous/deferred code.

> --
> You received this message because you are subscribed to the Google Groups "scrapy-users" group.
> To post to this group, send email to scrapy...@googlegroups.com.
> To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

Matthew Brunelle

unread,

Jun 23, 2011, 1:18:43 PM6/23/11

to scrapy-users

I'd like to test the spider.

Pablo Hoffman

unread,

Jun 23, 2011, 2:13:37 PM6/23/11

to scrapy...@googlegroups.com

There's an idea for automating some spider tests that never got implemented (so
far): http://dev.scrapy.org/wiki/SEP-017

But nothing prevents you from using standard Python unit-testing facilities to
test the Spider object methods (ie. the callbacks).

You build the Response in the test code, and then call the method with the
response and assert that you get the expected items/requests in the output.

A warning though: automating is the key. Otherwise, if you manage a lot of
spiders, you may find that testing may take even longer than writing the
spiders themselves - though this applies to other software testing as well, not
only spiders :)

Pablo.

Matthew Brunelle

unread,

Jun 23, 2011, 4:26:41 PM6/23/11

to scrapy-users

Thank you very much. Now how do I go about calling the spider from a
script?

Matthew Brunelle

unread,

Jun 24, 2011, 10:56:33 AM6/24/11

to scrapy-users

Here's my cross post to stack overflow http://stackoverflow.com/questions/6456304/scrapy-unit-testing

On Jun 23, 4:26 pm, Matthew Brunelle <matthew.brune...@gmail.com>
wrote:

Leonardo Lazzaro

unread,

Jun 24, 2011, 1:24:54 PM6/24/11

to scrapy...@googlegroups.com

check this snipplet

http://snippets.scrapy.org/snippets/13/

Matthew Brunelle

unread,

Jun 24, 2011, 2:16:10 PM6/24/11

to scrapy-users

Thank you very much! The spider runs perfectly with this, but it seems
the spider never returns and the script just hangs after executing the
firs spider.

On Jun 24, 1:24 pm, Leonardo Lazzaro <lazzaroleona...@gmail.com>
wrote:

Reply all

Reply to author

Forward