--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/0a9028e5-75d1-42ef-a68c-74a0f847bc81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/84849d3c-3829-4e06-b921-528a972653b4%40googlegroups.com.
pyspider send_message MYPROJECTNAME "http://MYURL.com"
pyspider send_message MYPROJECTNAME '{"url": "http://www.MYURL.ch"}'
def on_message(self, project, msg):
for each in msg('a[href^="http"]').items():
self.crawl(each.attr.href, callback=self.detail_page)
def on_message(self, project, msg): self.crawl(msg, callback=self.detail_page)
[I 160327 18:44:58 processor:199] process MYPROJECTNAME:0a6b3271c7ff97a5a38728d7ede43d2c data:,on_message -> [200] len:10 -> result:None fol:1 msg:0 err:None
[E 160327 18:44:58 scheduler:170] unknown project: MYPROJECTNAME
#!/usr/bin/env python# -*- encoding: utf-8 -*-# Created on 2016-03-28 01:02:46# Project: TEST11
from pyspider.libs.base_handler import *
class Handler(BaseHandler): crawl_config = { }
@every(minutes=24 * 60) def on_start(self): sys.stderr.write("START\n") #self.crawl('127.0.0.1', callback=self.index_page)
@config(age=10 * 24 * 60 * 60) def index_page(self, response): for each in response.doc('a[href^="http"]').items(): self.crawl(each.attr.href, callback=self.detail_page)
@config(priority=2) def detail_page(self, response): return { "url": response.url, "title": response.doc('title').text(), }
def on_message(self, project, msg): self.crawl(msg, callback=self.index_page)
--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/3502736a-65fd-448d-b8bc-28ba88ff92e6%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspide...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/326de9c6-90f2-444e-9133-6afc312e3e1b%40googlegroups.com.