Contribution to crawley

[Ramana Venkata]

Hi all,

I want to contribute to your project. I have no prior work experience. I have a basic idea about python. I want to learn python by actually working on an open source project. Can I work with you guys??

Ramana Venkata.

[David Litvak]

Hey Ramana,

I'm David, one of the core developers. I'm a Python and Ruby developer from Argentina focused on web applications and web frameworks.

You can see some of my personal projects on www.github.com/dlitvakb

It is awesome that you want to help us!

What part of Crawley are you interested in?

Have you tried using Crawley before?

Tell us a bit more about you!

Cheers,

2012/6/1 Ramana Venkata <idlike...@gmail.com>

Hi all,

I want to contribute to your project. I have no prior work experience. I have a basic idea about python. I want to learn python by actually working on an open source project. Can I work with you guys??

Ramana Venkata.

--
नारायण हरि ओम
सत्य नारायण हरि ओम
गोविन्द कृष्ण हरि ओम
गोपाल कृष्ण हरि ओम

David Litvak

Bachiller Técnico Orientado en Producción Musical - ORT
Estudiante de Ingeniería en Sistemas - UTN

http://vizualize.me/david.litvak
http://about.me/david.litvak
(011)15-6686-6714

[Ramana Venkata]

Hi David,

I am undergraduate student in India. I am taking mathematics as my major. Besides mathematics I have programming interests. So, I have learnt basic C and Python but I couldn't them in for any use. I want to work with open source projects so that I'll join as newbie and improve myself at programming.

No, there are some connectivity issues with my linux operating systems. I'll be able to fix them in few days. Then I'll try Crawley. I have read through documentation of Crawley. I have a rough idea how it works but I have to still have to understand it properly.

I don't have enough technical experience I guess. I have never used any datbases like MySQL, Oracle, etc., I am basically very new to most of the things. But I'll try to learn them quick enough all I need is some basic instructions.

I don't know which part to work on. I want first understand the code then later I can decide on which part to work on.

Ramana Venkata.

[David Litvak]

Awesome!

Don't hesitate to ask about anything, we're all here to help and we're very excited that you want to join us!

Feel free to ask about any doubts about Crawley or Python, we are more than willing to help you!

Cheers

2012/6/2 Ramana Venkata <idlike...@gmail.com>

[Ramana Venkata]

I have read how crawlers work in general from wikipedia. Where shall I start from?

[David Litvak]

You should try to look at the examples,

Make a simple crawler (like the Pypi one https://github.com/jmg/crawley/tree/master/examples/pypi_crawler)

Understand the basics of how the scraper is working, then try to understand what the more complex scrapers do, like the SmarScraper

Once you've got the basics, there are some pending stuff we need to add.

Just pick one, or propose your own ideas:

- Parsing/following of robots.txt

- Turnaround for noscript tags

- WebDriver support for clientside rendered pages

- Better redirect (302) handling

- Metadata extraction from flv objects

- Support for any file format or db format you can think of that is not currently supported

I think that's about it... if you have any idea to improve Crawley that is not listed above, please feel free to comment it, I'm sure it will be accepted with open arms.

If you want to contact me more directly to ask about anything, this email address is also my Gtalk and MSN account... and you can find me on skype: kroma.harry