shell problems when scraping

[Jerry Wu]

Dear everyone,

I want to scrape forum information on http://www.douban.com/group/python/

It is OK for me to find xpath with xpath helper on chrome. "//div/table/tbody/tr" will show all item information on the page.

However, when I run scrapy shell for double check before coding, I find it return a blank list. "//div/table/tbody/tr" doesn't work under shell. 

It's strange. Could some one give me some thoughts?

Thanks.

[Mikhail Korobov]

Hi Jerry,

"tbody" is not in the source code, it is inserted by a browser during the rendering process. "//div/table//tr" should give you the same result. Check http://doc.scrapy.org/en/latest/topics/firefox.html#caveats-with-inspecting-the-live-browser-dom

четверг, 3 июля 2014 г., 17:36:59 UTC+6 пользователь Jerry Wu написал:

[Jerry Wu]

I fixed it. Thank you very much!

--
You received this message because you are subscribed to a topic in the Google Groups "scrapy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scrapy-users/dYDqr2pBig4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards.

Jerry Wu

Life is short. Change is possible : )