Spider not crawling url with query string

10 views
Skip to first unread message

Lee Adams

unread,
Jul 15, 2015, 8:18:16 AM7/15/15
to scrapy...@googlegroups.com
Hi all, 
I am a bit new to all this, so I am sure there is something simple I am overlooking:

I am trying to scrape a page in the format http://www.example.com/next/?year=2015&month=07&day=18
But the spider just keeps returning results from http://www.example.com/next/

The query string keeps getting dropped. 

If someone could point me to somewhere in the docs that talks about this, I would be more than grateful, as I am planning to do a lot of scraping in the future, so I would like to understand *why this is happening, and not just get a fix.


Thanks!

Lee
Reply all
Reply to author
Forward
0 new messages