GSoC 2016 - GRPC Python (Develop and test Python 3.5 Support for gRPC)

[thunder...@gmail.com]

Hi everyone,

Greetings from Bremen, Germany. I am a computer science senior at Jacobs University Bremen and am currently working with grpc (C++ and Python) for my thesis i.e. developing a client library in Python that enables data scientists to fetch data from remote rasdaman database instances in a convenient manner and uses grpc and protocol buffers to accomplish the mission. 

The experience, so far, has been quite interesting in both good and bad ways. Protocol buffers are certainly quite an interesting concept and grpc is an amazing tool but since it is still in beta it's quite iffy at times. 

I actually had a hangout session with Omar Ayoub (PM at Google) sometime in January about the state of affairs in grpc python and issues that I was facing (from installation issues to examples not working as expected to unexpected behaviour such as grpc not respecting SIGINT, et cetera). 

Since, the success of this project directly impacts my thesis work and the state of RasdaPy (the upcoming python client of Rasdaman based on grpc), I have a vested interest in the future of the project. While browsing over the projects for GSoC with a colleague who was interested in the program I came across grpc and got interested in delving deeper into the program. I'm quite interested in the development of Python 3.x branch for grpc python as well as potential PyPy support and am willing to lend my summer break to the cause since it very nicely aligns with the GSoC timeline.

Some of the things that came about during my discussions with Omar, were Python 3.x support and possibility for a complete Python based grpc (like Go) as that really helps python developers dig deeper into the code and improve their understanding of the library. A quick follow up since I couldn't follow up with Omar due to busy school schedule would be, is that (a completely independent python client) still on plan? i.e. Are we looking at just porting  just the python dependencies or also the grpc core dependencies?

Cheers,

Sid

[Nathaniel Manista]

On Thu, Mar 17, 2016 at 5:25 PM, <thunder...@gmail.com> wrote:

Greetings from Bremen, Germany. I am a computer science senior at Jacobs University Bremen and am currently working with grpc (C++ and Python) for my thesis i.e. developing a client library in Python that enables data scientists to fetch data from remote rasdaman database instances in a convenient manner and uses grpc and protocol buffers to accomplish the mission.

Cool! Is your code in a publicly accessible repository where we can see it (and its development history)?

The experience, so far, has been quite interesting in both good and bad ways. Protocol buffers are certainly quite an interesting concept and grpc is an amazing tool but since it is still in beta it's quite iffy at times. 

I actually had a hangout session with Omar Ayoub (PM at Google) sometime in January about the state of affairs in grpc python and issues that I was facing (from installation issues to examples not working as expected to unexpected behaviour such as grpc not respecting SIGINT, et cetera).

This criticism is fair and I apologize for gRPC Python's share of the defects.

Since, the success of this project directly impacts my thesis work and the state of RasdaPy (the upcoming python client of Rasdaman based on grpc), I have a vested interest in the future of the project. While browsing over the projects for GSoC with a colleague who was interested in the program I came across grpc and got interested in delving deeper into the program. I'm quite interested in the development of Python 3.x branch for grpc python as well as potential PyPy support and am willing to lend my summer break to the cause since it very nicely aligns with the GSoC timeline.

Some of the things that came about during my discussions with Omar, were Python 3.x support and possibility for a complete Python based grpc (like Go) as that really helps python developers dig deeper into the code and improve their understanding of the library. A quick follow up since I couldn't follow up with Omar due to busy school schedule would be, is that (a completely independent python client) still on plan? i.e. Are we looking at just porting  just the python dependencies or also the grpc core dependencies?

Interest in Python3.x has really taken off just in the last four or five weeks. Several students are interested... and there are some open pull requests that might be taking us in the direction of having Python3.x before the summer even starts for GSoC students.

PyPy support would be a bit of a further reach, but might be reasonable for a capable student. Do you have a use case in mind for gRPC-on-PyPy? Personally, when I've thought of it, I've thought of it mostly in "well that would be a nice compatibility matrix cell to have checked and it would be nice to answer people's questions with 'yes' and not 'no'" terms but without any compelling reason of my own to do it.

A "complete Python based grpc (like Go)", more commonly referred to as "a pure-Python implementation of gRPC", is also a question that keeps coming up - developers are just excited about Python! It strikes me as something that would either be an inordinate amount of work, or that would be missing some features relative to the C-Core-wrapping gRPC Python of today, and so would need a really, really, ridiculously compelling use case as a reason to exist. And because we've done a lot of work toward making the C-Core-wrapping gRPC Python of today easily installable almost everwhere, such a use case might be hard to find. Do you have one or know of one?

-Nathaniel

[thunder...@gmail.com]

On Friday, 18 March 2016 08:46:02 UTC+1, Nathaniel Manista wrote:

On Thu, Mar 17, 2016 at 5:25 PM,  <thunder...@gmail.com> wrote:

Greetings from Bremen, Germany. I am a computer science senior at Jacobs University Bremen and am currently working with grpc (C++ and Python) for my thesis i.e. developing a client library in Python that enables data scientists to fetch data from remote rasdaman database instances in a convenient manner and uses grpc and protocol buffers to accomplish the mission.

Cool! Is your code in a publicly accessible repository where we can see it (and its development history)?

 

 Well, it's not publicly accessible at the moment, and wouldn't be for a few more weeks to come. If you're interested give me your github handle and I can add you to the private repo temporarily. :)

The experience, so far, has been quite interesting in both good and bad ways. Protocol buffers are certainly quite an interesting concept and grpc is an amazing tool but since it is still in beta it's quite iffy at times. 

I actually had a hangout session with Omar Ayoub (PM at Google) sometime in January about the state of affairs in grpc python and issues that I was facing (from installation issues to examples not working as expected to unexpected behaviour such as grpc not respecting SIGINT, et cetera).

This criticism is fair and I apologize for gRPC Python's share of the defects.

 

I think it's pretty normal for a software in it's early phases. There's a reason it's called bleeding edge. ;)

 

Since, the success of this project directly impacts my thesis work and the state of RasdaPy (the upcoming python client of Rasdaman based on grpc), I have a vested interest in the future of the project. While browsing over the projects for GSoC with a colleague who was interested in the program I came across grpc and got interested in delving deeper into the program. I'm quite interested in the development of Python 3.x branch for grpc python as well as potential PyPy support and am willing to lend my summer break to the cause since it very nicely aligns with the GSoC timeline.

Some of the things that came about during my discussions with Omar, were Python 3.x support and possibility for a complete Python based grpc (like Go) as that really helps python developers dig deeper into the code and improve their understanding of the library. A quick follow up since I couldn't follow up with Omar due to busy school schedule would be, is that (a completely independent python client) still on plan? i.e. Are we looking at just porting  just the python dependencies or also the grpc core dependencies?

Interest in Python3.x has really taken off just in the last four or five weeks. Several students are interested... and there are some open pull requests that might be taking us in the direction of having Python3.x before the summer even starts for GSoC students.

 

 Huh, that's quite interesting. It's great to see the momentum catching up.

PyPy support would be a bit of a further reach, but might be reasonable for a capable student. Do you have a use case in mind for gRPC-on-PyPy? Personally, when I've thought of it, I've thought of it mostly in "well that would be a nice compatibility matrix cell to have checked and it would be nice to answer people's questions with 'yes' and not 'no'" terms but without any compelling reason of my own to do it.

Well, PyPy in itself is a community that's growing at a rapid pace (they recently had a 5.0 release). It's still more of a novelty feature but i believe people using PyPy are not your usual python script writers but people using Python on a massive application level and really appreciate the JIT compiler. I don't have any stats to support my claim, but just by the sheer number of python devs I find at every hackathon I go to, makes me feel it's about time PyPy starts getting recognized beyond it's "hipster" position.

 

A "complete Python based grpc (like Go)", more commonly referred to as "a pure-Python implementation of gRPC", is also a question that keeps coming up - developers are just excited about Python! It strikes me as something that would either be an inordinate amount of work, or that would be missing some features relative to the C-Core-wrapping gRPC Python of today, and so would need a really, really, ridiculously compelling use case as a reason to exist. And because we've done a lot of work toward making the C-Core-wrapping gRPC Python of today easily installable almost everwhere, such a use case might be hard to find. Do you have one or know of one?

The only use case I have for a pure python implementation is the accessibility for python developers. Currently, the stack trace becomes moot after a point because sometimes exceptions happen within the C libraries and people using interactive debuggers such as werkzeug are unable to debug beyond a certain point in the stack trace. Honestly, it's something that'd be great to have but at the same time comes with significant amount of workload. Moving core dependencies like boringssl, transport layer, etc. would be insane amount of work and would definitely take more than one student to achieve that. :)

-Nathaniel

Cheers,

Sid

[Nathaniel Manista]

On Fri, Mar 18, 2016 at 5:31 AM, <thunder...@gmail.com> wrote:

Well, it's not publicly accessible at the moment, and wouldn't be for a few more weeks to come. If you're interested give me your github handle and I can add you to the private repo temporarily. :)

I hide behind an impenetrable wall of secrecy as nathanielmanistaatgoogle.

 

Well, PyPy in itself is a community that's growing at a rapid pace (they recently had a 5.0 release). It's still more of a novelty feature but i believe people using PyPy are not your usual python script writers but people using Python on a massive application level and really appreciate the JIT compiler. I don't have any stats to support my claim, but just by the sheer number of python devs I find at every hackathon I go to, makes me feel it's about time PyPy starts getting recognized beyond it's "hipster" position.

I'm intrigued.

The only use case I have for a pure python implementation is the accessibility for python developers. Currently, the stack trace becomes moot after a point because sometimes exceptions happen within the C libraries and people using interactive debuggers such as werkzeug are unable to debug beyond a certain point in the stack trace. Honestly, it's something that'd be great to have but at the same time comes with significant amount of workload. Moving core dependencies like boringssl, transport layer, etc. would be insane amount of work and would definitely take more than one student to achieve that. :)

True, but when you look at the gRPC specification, does it look less insane?

-N

[thunder...@gmail.com]

On Wednesday, 23 March 2016 07:18:12 UTC+1, Nathaniel Manista wrote:

On Fri, Mar 18, 2016 at 5:31 AM, <thunder...@gmail.com> wrote:

Well, it's not publicly accessible at the moment, and wouldn't be for a few more weeks to come. If you're interested give me your github handle and I can add you to the private repo temporarily. :)

I hide behind an impenetrable wall of secrecy as nathanielmanistaatgoogle.

 

Added you to the repo. :)

 

Well, PyPy in itself is a community that's growing at a rapid pace (they recently had a 5.0 release). It's still more of a novelty feature but i believe people using PyPy are not your usual python script writers but people using Python on a massive application level and really appreciate the JIT compiler. I don't have any stats to support my claim, but just by the sheer number of python devs I find at every hackathon I go to, makes me feel it's about time PyPy starts getting recognized beyond it's "hipster" position.

I'm intrigued.

Yeah, me too. They're moving quite fast. It's a matter of time before they become commonplace. 

The only use case I have for a pure python implementation is the accessibility for python developers. Currently, the stack trace becomes moot after a point because sometimes exceptions happen within the C libraries and people using interactive debuggers such as werkzeug are unable to debug beyond a certain point in the stack trace. Honestly, it's something that'd be great to have but at the same time comes with significant amount of workload. Moving core dependencies like boringssl, transport layer, etc. would be insane amount of work and would definitely take more than one student to achieve that. :)

True, but when you look at the gRPC specification, does it look less insane?

A bit less, yeah. :)

Any suggestions with regards to the proposal? 

-N

[Nathaniel Manista]

On Wed, Mar 23, 2016 at 9:00 PM, <thunder...@gmail.com> wrote:

Any suggestions with regards to the proposal?

Put it in a document and/or PDF before the deadline tomorrow? I'm not yet seeing a proposal from you on my dashboard - are you registered and whatnot? Did you see this message from Kailash earlier in the month?

-N

[t.34....@gmail.com]

Hello everyone,

We're building a FinTech product right now and are willing to move off REST over HTTP and instead to bring grpc and protobuf for communication between mobile clients and backend services and also for cross-service communication.

Though I found that grpc supports python 2.7 only. And seems is blocking. But we'd like to have python 3.5 support to be able to process RPC calls in asynchronous manner along with some cool aio-stuff such as aiohttp, aiopg and so on. It's all about non-blocking I/O and concurrency. I was really surprised that grpc doesn't support all this great stuff at all. On the one hand Protocol Buffer and grpc aim to speed-up and optimize a communication especially for mobile devices and on the other hand it's not compatible with all those great server-side technologies for concurrency and asynchronous I/O in Python 3.5. And what's the profit of such optimization then, if the end mobile client is blocked by the server's response?

How do you think when it will be possible to use grpc with python 3.5 and asyncio?

[thunder...@gmail.com]

Hey there, Glad to see grpc catching up. While the support for 3.x branch is still not very good, it's getting there. As for Async methods, that already exists for simple RPC calls. Check out http://www.grpc.io/docs/tutorials/basic/python.html

In essence, where you would normally do:

feature = stub.GetFeature(point, timeout_in_seconds)

you can use futures and write it like:

feature_future = stub.GetFeature.future(point, timeout_in_seconds)
feature = feature_future.result()

Hope that helps. :)

[t.34....@gmail.com]

Thanks for the tip.

Though I'm afraid it won't play well with our existing code in Python 3.5.1 with aiohttp, aiopg and other async/await stuff.