Changing Tor identity in Scrapy over Polipo

[simus]

I am trying to build a CrawlSpider using Scrapy to crawl sites in Tor.

To do this I am using the Polipo proxy as described by this guide:

http://blog.privatenode.in/torifying-scrapy-project-on-ubuntu/

So right now my spider works and I am able to access .onion sites in Scrapy. However what I would like to do is, for each request, send a NEWNYM signal in order to change my Tor identity.

How can this be done in Scrapy?

[crawler]

You should write your own Downloader middleware. See http://doc.scrapy.org/en/latest/topics/architecture.html and http://doc.scrapy.org/en/latest/topics/downloader-middleware.html

Use stem: https://stem.torproject.org/api/control.html

from stem.control import Controller, EventType
from stem import Signal


def _change_tor_lock(self, event):
   self._tor_lock = False


self._tor_lock = True
controller = Controller.from_port(control_port=9051)
controller.authenticate('password')
controller.add_event_listener(self._change_tor_lock, EventType.SIGNAL)
controller.signal(Signal.NEWNYM)
while self._tor_lock:
   time.sleep(0.5)
else:
   controller.close()

And I guess it's better to use Privoxy instead of Polipo. I had a problem with Polipo when I was trying to change ip. 

Polipo seems to reuse old connections or does something like this.