pyspider可以持续爬取一个页面么?

65 views
Skip to first unread message

www...@gmail.com

unread,
Sep 22, 2017, 6:34:54 AM9/22/17
to pyspider-users
我是要爬取直播网站的弹幕,但是每次spider爬取完页面就返回了,有办法让他持续爬取一个页面,直到我设置的时间(比如100分钟)之后再退出么?

Roy Binux

unread,
Sep 22, 2017, 6:36:05 AM9/22/17
to www...@gmail.com, pyspider-users

这不符合pyspider的设计意图,不建议用pyspider做


On Fri, 22 Sep 2017, 11:34 <www...@gmail.com> wrote:
我是要爬取直播网站的弹幕,但是每次spider爬取完页面就返回了,有办法让他持续爬取一个页面,直到我设置的时间(比如100分钟)之后再退出么?

--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/d24a16a2-a0e5-49e5-ab4d-c8cc73d5278b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

www...@gmail.com

unread,
Sep 22, 2017, 6:56:45 AM9/22/17
to pyspider-users
多谢提示,我会考虑直接用phantomjs来实现,或者您有任何建议么?
我也在看pyspider fetcher这部分内容,看能不能自己加一个功能上去,如果可以给我提供一下思路的话就更加感谢了


在 2017年9月22日星期五 UTC+8下午6:36:05,Roy Binux写道:
Reply all
Reply to author
Forward
0 new messages