Accessing Scrapy via a webservice

44 views
Skip to first unread message

Richard Simpson

unread,
Mar 26, 2014, 11:10:10 AM3/26/14
to scrapy...@googlegroups.com

I've had a search around SO and the web in general, and haven't been able to find a solution to my issue; so I thought I would raise a question here.

I'm currently playing around with Scrapy, and am successfully retrieving the data I need from my target website.

The creation of the spider is part of a larger project, and so I need to work out a way of getting the results from the spider into my website.

I don't want to access the database with the spider, because there is a variety of logic that would need to be applied to the data before it was stored, and I wouldn't want to have to repeat that logic elsewhere.

What I'm thinking is perhaps a webservice, that receives a JSON object as a request; and returns a JSON object with the result.

I'm fairly new to Scrapy and Python in a websetting, but I have read the Scrapy docs. Whilst there is a webservice for Scrapy, that seems to be more to do with the monitoring of spiders; and not returning results.

Any help and thoughts would be greatly appreciated.

Thanks,



(Originally posted on StackOverflow: http://stackoverflow.com/questions/22653821/accessing-scrapy-via-a-webservice?noredirect=1#comment34522584_22653821)

Pablo Hoffman

unread,
Apr 19, 2014, 10:48:12 AM4/19/14
to scrapy-users
One of the project ideas for GSoC 2014 is meant to address this topic:

It was one of the ideas that got closest to getting accepted, but it wasn't in the end. The idea is still pretty solid and useful, so maybe someone (you?) can pick it up and work on it, regardless of GSoC. If someone is already actively working on it, please chime in!



--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages