Unit testing item pipeline that returns Twisted deferred

[Martin Siniawski]

Hey everyone!

I'm trying to write a unit test for a item pipeline class which has a process_item method that returns a Twisted deferred. This is the first time I need to test a method returning a deferred and I've read a bit online about Trial and when to use it, so I wanted to reach out an check which are the best practices.

The method goes something like this:

def process_item(self, item, spider):

        d = deferToThread(self._process_item, item)

        d.addCallback(self._process_item_success, item)

        d.addErrback(self._process_item_errback, item)

        return d

From where I'm standing it would seem that nothing fancy is in order. 

Just invoking the process_item method from the unit test, and making sure that a deferred with the proper arguments is returned would seem to be enough. I'd probably have to provide a mock of the deferred object and find some way to pluck the function that instantiates the mock (so that instead of using the deferToThread function the deferToMockedThread or something like that is used). 

Afterwards, all I'd need to do would be to execute the different callbacks and errbacks and check as well that they behave as would be expected.

Does this make sense? Any thoughts would be greatly appreciated!

Best,
Martin

[Pablo Hoffman]

I found the @inlineCallbacks decorator very convenient for writing
unit-tests of functions that return deferreds. You have some examples in
Scrapy code, look at the following files in scrapy/tests:

* test_utils_defer.py
* test_contrib_feedexport.py
* test_command_fetch.py

Or grep by "inlineCallbacks" in scrapy/tests.

> --
> You received this message because you are subscribed to the Google
> Groups "scrapy-users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/scrapy-users/-/Qfiy-GDP5sgJ.
> To post to this group, send email to scrapy...@googlegroups.com.
> To unsubscribe from this group, send email to
> scrapy-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/scrapy-users?hl=en.