Re: Issue with scraping AJAX site

[Travis Leleu]

It looks like it's updating the value via javascript.  You'll either need to mimic the ajax call (if it's retrieving data from the server based on current conversion rate), or recreate the logic in your crawler.

Scrapy doesn't execute javascript.

On Wed, Nov 11, 2015 at 10:30 AM, Mario <laki-paki...@hotmail.com> wrote:

Ok so a little update, it seems like I used form_data instead of formdata. Now the spider runs normally but it still scrapes prices that are in $, so it means that it didn't change at all :(

среда, 11. новембар 2015. 15.57.04 UTC+1, Mario је написао/ла:

Hi,

So I have issues with integrating forms into spider. The spider itself its really simple, here’s the code:

import scrapy
from scrapy.http import FormRequest, Request

class YellowkornerSpider(scrapy.Spider):
    name = "yellowkorner"
    allowed_domains = ["yellowkorner.com"]
    start_urls = ('http://www.yellowkorner.com/photos/index.aspx')

    def parse(self, response):
        yield FormRequest.from_response(response, formname='aspnetForm', form_data={'ctl00$languageSelection$d...': 'GB'}, callback=self.parse_photographs)

    def parse_photographs(self, response):
        prices = response.xpath('//*[@class="price"]/span/text()').extract()
        print prices

The error that I’m getting right now is this:

TypeError: __init__() got an unexpected keyword argument 'form_data'

Also what I’m trying to accomplish is to scrape prices from this page. The issue is that I want prices in UK not in US, so instead of getting prices in $ I will need prices in £.

I know issue is in yeilding form part but I don’t know how to tackle this. Here’s the part where countries are changed:

And here’s form name, I hope I got it right:

Here’s the page where I got form part implemented.

Any help is appreciated I’m trying to get this right by days now…

​

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

[Mario]

Hi Travis, thanks for the reply.

I messed something up with post so I deleted it and created new post.

I've used forms before on somewhat similiar issue that I was having but then there were only 3 items that were populated. So basically it looked like this:

```python

form_data = {"school_name": '', "school_type_id": school_type_id, "state_id": state_id}

request = scrapy.FormRequest(url, callback=self.parse_schools, formdata=form_data)

yield request

```
And that worked flawlessly. As for issue that I'm facing now I think this can be done via forms but I'm newbie at this so I don't know how to...

[Mario]

I've managed to get it working with adding additional form in Scrapy. Thanks once again for help.

среда, 11. новембар 2015. 19.56.15 UTC+1, Travis Leleu је написао/ла: