Scrapyd: How to change front end style?

[jiaw...@gmail.com]

I successfully start scrapy server and run a spider on it. I can see the logs, items and spider status on localhost:6800. 

Now I want to change the front end style and make it more beautiful.

But I have no idea how to start. Can I use Flask to achieve my goal? Or should I inherit any class and add extra functions?

Thanks in advance!

[Pablo Hoffman]

Thanks for offering to improve Scrapyd UI, that'll be gladly welcome!

Scrapy uses twisted.web, not flask, for the web server.

Here's a link to the relevant source code parts:

https://github.com/scrapy/scrapy/blob/master/scrapyd/website.py

https://github.com/scrapy/scrapy/blob/master/scrapy/webservice.py

And more information on twisted.web:

http://twistedmatrix.com/documents/current/web/howto/web-in-60/index.html

Looking forward to your pull request :)

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/8M19WibIBOwJ.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

[Jiawan Xu]

Thanks very much for your reply. I read the source code of scrapyd, and will try to use some library, such as flask, to improve Scrapyd UI. If I get any problem or improvement, I will post it to this group. XD

[Pablo Hoffman]

Replacing twisted.web by flask would require substantial architectural changes (since the rest of the scrapyd components (like project management) are also based on twisted asynchronous nature where flask won't work. What you could definitely adopt is a templating language such as jinja2 (used by flask).

[Jiawan Xu]

I was finding which template language should I use until receive your reply. Thanks for your advice ! 

[Jordi Llonch]

Have you considered using twitter bootstrap?

Although is not a templating system, it makes easy and beautiful front-ends.

+1 for Jinja2

Regards,

Jordi

[Pablo Hoffman]

+1 to bootstrap

To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.

To post to this group, send email to scrapy...@googlegroups.com.

Visit this group at http://groups.google.com/group/scrapy-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jordi Llonch]

I think on a Dashboard page with the spider data and some options to manage existing spiders.

Pablo, Jiawan; Have you got any ideas on how to proceed?

Regards,

[Jordi Llonch]

Have a look: https://github.com/llonchj/scrapy/

[John Accoun]

Hey all, I've been thinking about writting my own frontend to scrapd for a wile, but I have to take it one step at a time. Pushing scrapy usage at work was already a great victory.

I don't think any extra work on the server side is necessary. Scrapyd's JSON API should be enough. What we need is to serve only static html, js and css files and build a frontend using ajax calls only.

Could be usefull to see be able to upload spiders and to schedule job. The JSON API already supports this. It's just a matter of building a JS GUI around it.

Put together Twitter bootsrap and some JS frameowrk with data binding to minimize boiler plate work. There are quite many options these daus: Angular, backbone, spine, knockout, etc.

On a related note, the possibility to accept AJAX calls from other hosts would be desirable. This can be achieved using CORS headers, it is really just add a line to scrapyd's HTTP responses.

More info on CORS headers:

http://enable-cors.org/

https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

I have never used twisted.web, but serving static content appears to be trivial:

http://twistedmatrix.com/documents/current/web/howto/web-in-60/static-content.html

--

[Jordi Llonch]

Hi John,

Please, have a look at: https://github.com/llonchj/scrapy

It is an initial implementation with Bootstrap and jQuery.

Any ideas?

2013/1/31 John Accoun <jso...@gmail.com>

[Jiawan Xu]

Thanks for all advice from Pablo, Jordi, John. ^_^ 

I will try some of them.

[jiaw...@gmail.com]

Sorry, this sentence makes me a little confused. Do you mean that if using bootstrap, how to design the spider management page?

[jiaw...@gmail.com]

I have a look at https://github.com/llonchj/scrapy/. As there are too many files forked from "scrapy/scrapy", could you please list the files you add? Thanks very much.

[Jordi Llonch]

Scrapyd has his own repository now: https://github.com/llonchj/scrapyd

Feel free to check the commits from the repo.

Regards

2013/2/22 <jiaw...@gmail.com>

--