Interact with pyspider via other program / command line

[HD]

Hi,

is it possible to interact with pyspider via command line oder other python code (Without using the WebUI):

MyCode -------(URL2Spider)----------> pyspider ---------(Result_URLs)------->MyCode2

I want my code to send an url to pyspider which spiders the pages.
Then i want to feed the results of pyspider back to another/the same program.

Is that possible?
(If so, can u give me more info how e.g. a link maybee even sample code?)



Ps.: Sry for this complete beginner question to pyspider, which i just came across and didnt find an awswer in the www.

[Roy Binux]

Hi,

Actually, I hadn't designed such usage.

pyspider have 3 main components -- scheduler, fetcher and processor (plus result_worker). The scheduler would manage and track the tasks. The fetcher fetch and the processor execute the code. The scheduler is the most important part comparing with other scrape frameworks, and the fetcher and processor are almost similar with others and easy to implement (as mostly depend on your script)

In your usage, scheduler is not necessary and the tasks not managed by pyspider. I think you could use fetcher and processor separately and imitate the pack schema. Or just write a new fetcher and processor with the libs used in pyspider (tornado and pyquery)

Best,

Roy Binux

--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/0a9028e5-75d1-42ef-a68c-74a0f847bc81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jensvon...@googlemail.com]

Hi,

i would like to use the scheduler too, but i dont want to insert the sites to spider via WebUI but via command line or via my python code.

Is there no possibility for this?

And afterwards i would like to use the result for my own code. (I think this should be easy, because i can directly access the resultDb)

Regards

[Roy Binux]

Yes,  you can use `pyspider send_message project message` to send a message to a project via CLI then get the message via handler's on_message callback.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/84849d3c-3829-4e06-b921-528a972653b4%40googlegroups.com.

[jensvon...@googlemail.com]

What i try to achieve is send to simply add an URL that should completely be crawled to my pyspider-project. (And all links on the page should be crawled afterwards.)

from console:

pyspider send_message MYPROJECTNAME "http://MYURL.com"


or 

pyspider send_message MYPROJECTNAME '{"url": "http://www.MYURL.ch"}'

and on the pyspider side:

    def on_message(self, project, msg):
        for each in msg('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)

    def on_message(self, project, msg):

        self.crawl(msg, callback=self.detail_page)

This results in the Error:

[I 160327 18:44:58 processor:199] process MYPROJECTNAME:0a6b3271c7ff97a5a38728d7ede43d2c data:,on_message -> [200] len:10 -> result:None fol:1 msg:0 err:None

[E 160327 18:44:58 scheduler:170] unknown project: MYPROJECTNAME

[Roy Binux]

what's the error?

msg is a string when sent from command line.

[jensvon...@googlemail.com]

Hi, it seems that i messed up the database somehow (not pyspider, own db statements). I reinstalled pyspider and created a new project and it worked.

Thx for the help

This i how i did it. (Its very simple)

In pyspider project:

#!/usr/bin/env python

# -*- encoding: utf-8 -*-

# Created on 2016-03-28 01:02:46

# Project: TEST11

from pyspider.libs.base_handler import *

class Handler(BaseHandler):

    crawl_config = {

    }

    @every(minutes=24 * 60)

    def on_start(self):

        sys.stderr.write("START\n")

        #self.crawl('127.0.0.1', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)

    def index_page(self, response):

        for each in response.doc('a[href^="http"]').items():

            self.crawl(each.attr.href, callback=self.detail_page)

    @config(priority=2)

    def detail_page(self, response):

        return {

            "url": response.url,

            "title": response.doc('title').text(),

        }

    

    def on_message(self, project, msg):

        self.crawl(msg, callback=self.index_page)

and then simply from commandline/shell/terminal:

pyspider send_message project_name "url"

eg:

pyspider send_message TEST11 "http://www.google.com/"

[ecom4...@gmail.com]

Hello.

I was copy-pasted this logic from this issue but is not working.

In my understanding, I understand by command line you can send a message to PROJECTNAME with the URL to start crawling.

But... with these lines in on_start:

    def on_start(self):

        sys.stderr.write("START\n")

        #self.crawl('127.0.0.1', callback=self.index_page)

[...]

    def on_message(self, project, msg):

        self.crawl(msg, callback=self.index_page)

Is not working

How could I do?

[ecom4...@gmail.com]

Hey.

I found something interesting that could also help....

Is possible to do something like that?

https://stackoverflow.com/questions/49212897/parsing-a-list-of-urls-in-python

Create a list of URLs in CSV and then import them into a project?

[Roy Binux]

Why do you say that not working? If you are expecting START trigger. It wouldn't. Only on_message will called.

--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/3502736a-65fd-448d-b8bc-28ba88ff92e6%40googlegroups.com.

[Joseph]

So... What should I do?

Sorry, I do not understand how to send the message and make the project start using the URL sent.

To unsubscribe from this group and stop receiving emails from it, send an email to pyspide...@googlegroups.com.

[Roy Binux]

Send URL to on_message callback, self.crawl submit the task. That's it. You don't need to "start" anything.

To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/326de9c6-90f2-444e-9133-6afc312e3e1b%40googlegroups.com.