Retrieve/Get URL paramenter value

9 views
Skip to first unread message

Raf Roger

unread,
Aug 25, 2016, 3:19:10 PM8/25/16
to scrapy-users
Hi,

i would like for each page (in pagination) crawled, to retrieve/get the value of page.
so if we are on page 1, i want to retrieve "1", if i'm on page 2, i want to get "2"

here is a simple example of my code:
def parse_start_url(self, response):
 return {"page number" : response.url.xxx}

where xxx should return the value of url argument/parameter.

However i do not know how to do that in scrapy :(

my url should look like that:

maybe there could be more parameters, so i can not search for string "p=" as i do not know how many digit the page number will be.

how can i do that ? Is there a way to get the parameter value ?
i search and i found something about getattr(response.url, the_missing_param) but i was not able to find more help or documentation about it.

thx a lot

Raf Roger

unread,
Aug 25, 2016, 3:50:51 PM8/25/16
to scrapy-users
ok i found i need to learn python :D :D
found hot to do it.
Reply all
Reply to author
Forward
0 new messages