django-dynamic-scraper released!

[Holger Drewes]

Hi,

I worked on a Django app during the last 3 month called django-dynamic-scraper

and just published the first version on GitHub:

https://github.com/holgerd77/django-dynamic-scraper

The app is build on top of Scrapy and let's you create and managy Scrapy spiders

in the admin interface of Django, and the results of scraper runs are also saved

in the Django DB using Scrapys experimental DjangoItem class.

The app is still pre-alpha, but I spent a lot of time writing tests and documentation,

so it's already relatively stable, though I would recommend using it in production

already. 

If you are interested, check out the GitHub page, I would appreciate your feedback!

Greetings from Berlin

Holger Drewes

[Espen Klem]

This is great news! I'll test it and comment within a month.

Espen

Holger Drewes

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/SzpBvOMMXAYJ.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

[Holger Drewes]

Cool, happy to get your feedback! :-)

Holger

[Максим Горковский]

Sounds amazing, going to test as soon as possible

2011/12/19 Holger Drewes <Holger...@gmail.com>

Cool, happy to get your feedback! :-)

Holger

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/-zqAkyjWB6cJ.

To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

--
С уважением,
Максим Горковский

[Holger Drewes]

One addition (to limit expectations): it's not fitting for all kind of Scrapy scrapers, the app is tailored to the relatively common case where you have got an updated summary list of items on a website, with each item summary leading to a detail page, e.g. events from event locations, recipes from recipe sites, or the example project coming with the source code e.g. is scraping open news content from Wikinews. The data for the items (event, recipe, news, ...) is then scraped either from the item summary on the overview page or from the item detail page.

I hope, that this is serving already a lot of use cases, I had to find a balance between complexity and ease of use. You can read more about what you can and what you can't do with Django Dynamic Scraper in the associated ReadTheDocs documentation (see link in the first post).

At the moment, items HAVE to have a detail page, just yesterday, the idea came to me that this probably isn't necessary to make this mandatory, so probably this restriction will fall in the next minor release, so that you can use the library as well to scrape items which are not linking to an item page, e.g. if you just have a list with new sensor data measurements with all the data of each item on the main page.

If you find bugs or if you feel that concepts could be implemented in a better way, please feel free to post an issue on GitHub with a description or - if you already fixed it - propose it as a pull request.

Greetings

Holger

P.S.: Sorry, in the first post I forgot a "NOT", wanting to write that the app is probably NOT yet production-ready!! Hope, this didn't lead to a lot confusion... :-)

[Holger Drewes]

Hi Ghassen,

thanks! :-) Actually you've got a similar problem like me, django-dynamic-scraper ist still very young, and so I also don't have much feedback how people use the site apart from some occasional questions.

Though there is not yet much going on there, I have created a Google Group: https://groups.google.com/forum/?fromgroups#!forum/django-dynamic-scraper

If you have got questions, I would ask you to post them there, then everyone can benefit from the answers. It'll probable be just me for some time to answer, but I promise that I try to answer questions relatively quickly.

Greetings

Holger

P.S.: If you actually will have moved forward a bit in the future and have a production site with DDS it would also be great, If you post a short summary of what is the use case and what you are doing, would be great!

2012/7/2 Ghassen Telmoudi <ghassen....@gmail.com>

Hello Holger,

First of all congratulation on django-dynamic-scraper, I really love it, and I believe it's a promising project.
For this matter I want to ask you how to find people uses DDS in production, and how to get in touch with people how actually can help out dealing with some issues you face when you're a beginner.
Is there some google-group, or some kind of IRC channel?

In fact I'm willing to use it in production, but I get no help when I got some errors, cause googling doesn't help me much, so I try to figure it out by myself lost of the time.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/t0REBINy8NYJ.

[Tsouras]

Sounds interesting!!!