help in contributing for GSOC2017

[SHANKAR JHA]

Hi Scrapians,

I am studying computer science and newbie in GSoC. I always want to contribute to open source project.

I just found  "new HTTP/1.1 downloader handler" in scrapy project ideas and I am very excited to work on this.

I am good in python and HTTP but never worked on twisted.

Can anyone help me and show me the direction, So that I can start working on my project.

With Due Regards,

Shankar Jha

[Paul Tremberth]

Hello Shankar,

It's great to know you're looking forward to participating in GSoC 2017!
For the HTTP/1.1 downloader project, I suggest that you get familiar with how scrapy uses Twised in [1]

Scrapy uses Twisted Agent [2] and customizes it to handle proxies with the CONNECT method, TLS connections without verifying peer certificates or using a specific TLS method, etc.

Note that some of the description for the project [3] is not up-to-date as I re-read it.
Especially, scrapy does not ship with Twisted code anymore in scrapy.xlib.tx

Also, you can check the open issues around HTTP in GitHub.

For example, there's an old ticket about handling responses without a reason phrase [4]

Here is a recent Pull Request to Twisted [5] by one of the contributors to Scrapy, namely Rolango, to be able to customize the HTTP client parser.

This could be a pre-requisite for the GSoC project.

I would also say that Scrapy HTTP/1.1 download handler needs more thorough tests, with all the various good and bad practices from web servers, especially for HTTP proxies and TLS connections.

Just to name a few:

- servers that never respond

- servers that send less bytes than advertized [6]

- servers can be very slow, or throttling a lot

Some of these tests are already implemented, some of them are less robust or incomplete (see [7])

Finally, as bonus points, it would be great to see how far Scrapy is from supporting an HTTP 2 client (see [8])

Hope this helps,

Paul.

[1] https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/handlers/http11.py

[2] https://twistedmatrix.com/documents/current/web/howto/client.html

[3] http://gsoc2017.scrapinghub.com/ideas/#download-handler

[4] https://github.com/scrapy/scrapy/issues/345

[5] https://github.com/twisted/twisted/pull/712

[6] https://github.com/scrapy/scrapy/issues/2586

[7] https://github.com/scrapy/scrapy/issues/2545

[8] https://github.com/scrapy/scrapy/issues/1854

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscribe@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

[Anshul Malik]

Hey there!

I am Anshul studying Computer Engineering in India. I am also planning to work on "New HTTP/1.1 download handler" project.
I have already read the guidelines mentioned by Paul. So I am now working on outlining the plan.

Is there anything that I need to know or would be best for me If I do check that out?

Thanks
Anshul

[Paul Tremberth]

Hi Anshul,

hi Shankar

Twisted recently fixed one of the issues Scrapy had with the standard HTTP/1.1 downloader,

namely to handle HTTP responses without a reason phrase: https://github.com/twisted/twisted/pull/723

Rolando also fixed a couple of issues related to HTTP responses:

- https://github.com/scrapy/scrapy/pull/2590

- https://github.com/scrapy/scrapy/pull/2622

The tests have also been improved a bit for these cases.

There's still for example an issue with HTTP 100 responses (https://github.com/scrapy/scrapy/issues/544)

but it also looks like the new HTTP/1.1 downloader idea is loosing some of it substance.

Maybe the focus for GSoC should be on HTTP/2 support. I'm not sure.

Regards,

Paul.

[Anshul Malik]

Hi Paul,

So can we think of HTTP/2 support for scrapy as a GSoC project?

Thanks

--
You received this message because you are subscribed to a topic in the Google Groups "scrapy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scrapy-users/UN5P1AJMKxk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scrapy-users+unsubscribe@googlegroups.com.

[Paul Tremberth]

Hi Anshul,

Indeed you can present a GSoC application around adding HTTP/2 support to Scrapy.

The ideas page that is currently online is just a list of proposals,
and you can certainly come up with another idea altogether.

Don't hesitate to interact here or open an issue on GitHub to discuss design options.

Best,
Paul.

--

You received this message because you are subscribed to the Google Groups "scrapy-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscribe@googlegroups.com.

[Atul Krishna]

Hi Paul,

This is Atul Krishna, a undergraduate from computer science department, MAKAUT, India. I am also interested in "new HTTP/1.1 downloader handler" in scrapy project ideas. I would love to work on "adding HTTP/2 support to Scrapy". Since i am a newbie it would be great if you guide on how to start with HTTP/2 support to Scrapy.

Regards,

Atul Krishna

To unsubscribe from this group and all its topics, send an email to scrapy-users...@googlegroups.com.

To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.

[Paul Tremberth]

Hey Atul,

I'm afraid I haven't looked at all at HTTP/2 support in Twisted (nor am I familiar enough with HTTP/2 in general)

Here's the open issue on GitHub for scrapy: https://github.com/scrapy/scrapy/issues/1854

There's an example from hyper-h2 showing how to use the library with Twisted:

https://python-hyper.org/projects/h2/en/stable/twisted-post-example.html

You can have a look at how download handlers are implemented in Scrapy to see how that would fit in:

https://github.com/scrapy/scrapy/tree/master/scrapy/core/downloader/handlers

Hope this helps,

Paul.

To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscribe@googlegroups.com.