Proposal: Use grpc-web for GopherJS UI <-> backend interactions

131 views
Skip to first unread message

Johan Brandhorst

unread,
Dec 22, 2017, 11:17:13 AM12/22/17
to Camlistore
Hi!

I'm the author of the GopherJS gRPC-Web bindings. I was introduced to Perkeep by Paul Jolly, who is the author of the GopherJS React bindings used by Perkeep to render some of its frontend UI. I thought since Perkeep has already invested in GopherJS through this UI it might be interested in trying out my gRPC-Web bindings to abstract away the communications layer between the frontend and backend in a familiar gRPC/Protobuf fashion.

The bindings are built on top of the Improbable gRPC-Web client. It features support for the Fetch API in supported browsers and automatically downgrades to XHR where necessary. It implements the gRPC-Web spec. However, the gRPC-Web spec only mandates support for unary and server-side streaming at this time. My GopherJS bindings adds support for client-side and bidirectional streaming with a custom Websocket proxy on top. Please see my demo website for an example of this. It's only a bandaid until the gRPC-Web spec adds support for client-side and bidi-streaming.

I've submitted a small CL with a prototype of what this could look like on a small scale, to show how the bindings are used and hopefully give an idea of the impact on the codebase. At this time I only implemented it in one place, but ideally for something like this the whole communications layer between the frontend and backend would go through the generated interface.

https://camlistore-review.googlesource.com/c/camlistore/+/12406

Please let me know your thoughts!

Johan

Mathieu Lonjaret

unread,
Dec 22, 2017, 11:27:24 AM12/22/17
to camli...@googlegroups.com
Hello,

Yes, Paul had enthusiastically mentioned your work to me already. :-)

Thanks for taking the time to set up that CL, that will help a lot.

Now for the dumb/naive question: how would using gRPC-Web help us
exactly? What does it make better?
I mean, communication between the server and client has not really
been much of a difficulty so far, even when using GopherJS, so I'm not
sure what the obvious win is here.

Also, for ease of discussion, feel free to open a dedicated issue
about that in the tracker.

Regards,
Mathieu


On 22 December 2017 at 17:15, Johan Brandhorst
> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camlistore+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Johan Brandhorst

unread,
Dec 22, 2017, 12:12:41 PM12/22/17
to Camlistore
No dumb questions!

I opened https://github.com/camlistore/camlistore/issues/993 as you suggested. I wasn't sure where the best place to take this discussion would be, but the contribution guidelines suggested I sign up for this forum, so that's why I made this post originally instead of making an issue to discuss it.

As for your question, I think technically there might not be a great argument right now as much of the server communication in the frontend is (as far as I can tell) very fragmented. You get small wins like automatically marshalled and unmarshalled payloads to the backend, automatic HTTP/2 connection and Fetch API instead of XHR where available and other things that [Improbable make a much better case for](https://improbable.io/games/blog/grpc-web-moving-past-restjson-towards-type-safe-web-apis) than I ever could. I would like to think it would make the frontend/backend communications more "maintainable", and nicer to work with, but I understand your question is really about the trade off between the amount of work it would take to migrate all communications to something like this vs keeping it as it is today and spend the time elsewhere, and whether that trade off is worth it.

To me, I think there's also a sense of "why not", and "to show that we can", presumably that was the reason for adopting GopherJS in the first place? I'm only tinkering on the bindings because it's fun and I imagine that's true for all contributions to Perkeep, and maybe it'd be fun to implement this.

I completely understand if this discussion ends on a "cool idea, but not worth the time investment". After digging around the internals of perkeep for the CL I can tell it's probably not going to be a trivial transition, though if we're moving in the direction of replacing other JS parts with GopherJS, this might make things easier as well.

Johan

Mathieu Lonjaret

unread,
Dec 22, 2017, 12:31:07 PM12/22/17
to camli...@googlegroups.com
On 22 December 2017 at 18:12, Johan Brandhorst
<johan.br...@gmail.com> wrote:
> No dumb questions!
>
> I opened https://github.com/camlistore/camlistore/issues/993 as you
> suggested. I wasn't sure where the best place to take this discussion would
> be, but the contribution guidelines suggested I sign up for this forum, so
> that's why I made this post originally instead of making an issue to discuss
> it.

Yes, opening a discussion here is fine, but follow-up progress is
easier to keep track of in an issue.

> As for your question, I think technically there might not be a great
> argument right now as much of the server communication in the frontend is
> (as far as I can tell) very fragmented.

Yes, originally most of it was is server-connection.js, but I think
we've been moving away from it with first React, then GopherJS
(because we can write, and hopefully reuse, client code in Go with
it).

> You get small wins like
> automatically marshalled and unmarshalled payloads to the backend, automatic
> HTTP/2 connection and Fetch API instead of XHR where available and other
> things that [Improbable make a much better case
> for](https://improbable.io/games/blog/grpc-web-moving-past-restjson-towards-type-safe-web-apis)
> than I ever could. I would like to think it would make the frontend/backend
> communications more "maintainable", and nicer to work with

If you want, I might have a communications use case that I (nor Paul
afaik) wasn't able to solve in a very satisfying way, and for which
maybe you could come up with a better solution with gRPC-Web? Let me
know and I'll send you the details.

> but I understand
> your question is really about the trade off between the amount of work it
> would take to migrate all communications to something like this vs keeping
> it as it is today and spend the time elsewhere, and whether that trade off
> is worth it.

Yes, I'm glad you perfectly understand.

> To me, I think there's also a sense of "why not", and "to show that we can",
> presumably that was the reason for adopting GopherJS in the first place? I'm
> only tinkering on the bindings because it's fun and I imagine that's true
> for all contributions to Perkeep, and maybe it'd be fun to implement this.

Jsyk, there's a much more pragmatic reason for GopherJS: neither Brad
nor I like to work in javascript and keep in touch with the madness of
its ecosystem. I haven't moved much of the existing codebase yet to
GopherJS, but the features/fixes I was was able to add to the web UI
were much more pleasant to do with the help of GopherJS than they
would otherwise have been.

> I completely understand if this discussion ends on a "cool idea, but not
> worth the time investment". After digging around the internals of perkeep
> for the CL I can tell it's probably not going to be a trivial transition,
> though if we're moving in the direction of replacing other JS parts with
> GopherJS, this might make things easier as well.

Yep, regardless of whether we adopt gRPC-Web, I'm still very
interested to hear of how it would integrate with Perkeep, and how it
would help us. Any time you want to spend on this and report about it
is very welcome.

thanks,
Mathieu

> On Friday, 22 December 2017 16:17:13 UTC, Johan Brandhorst wrote:
>>
>> Hi!
>>
>> I'm the author of the GopherJS gRPC-Web bindings. I was introduced to
>> Perkeep by Paul Jolly, who is the author of the GopherJS React bindings used
>> by Perkeep to render some of its frontend UI. I thought since Perkeep has
>> already invested in GopherJS through this UI it might be interested in
>> trying out my gRPC-Web bindings to abstract away the communications layer
>> between the frontend and backend in a familiar gRPC/Protobuf fashion.
>>
>> The bindings are built on top of the Improbable gRPC-Web client. It
>> features support for the Fetch API in supported browsers and automatically
>> downgrades to XHR where necessary. It implements the gRPC-Web spec. However,
>> the gRPC-Web spec only mandates support for unary and server-side streaming
>> at this time. My GopherJS bindings adds support for client-side and
>> bidirectional streaming with a custom Websocket proxy on top. Please see my
>> demo website for an example of this. It's only a bandaid until the gRPC-Web
>> spec adds support for client-side and bidi-streaming.
>>
>> I've submitted a small CL with a prototype of what this could look like on
>> a small scale, to show how the bindings are used and hopefully give an idea
>> of the impact on the codebase. At this time I only implemented it in one
>> place, but ideally for something like this the whole communications layer
>> between the frontend and backend would go through the generated interface.
>>
>> https://camlistore-review.googlesource.com/c/camlistore/+/12406
>>
>> Please let me know your thoughts!
>>
>> Johan
>

Johan Brandhorst

unread,
Dec 22, 2017, 12:52:27 PM12/22/17
to Camlistore

If you want, I might have a communications use case that I (nor Paul
afaik) wasn't able to solve in a very satisfying way, and for which
maybe you could come up with a better solution with gRPC-Web? Let me
know and I'll send you the details.

Sure I can take a look :)

Yep, regardless of whether we adopt gRPC-Web, I'm still very
interested to hear of how it would integrate with Perkeep, and how it
would help us. Any time you want to spend on this and report about it
is very welcome.

Another thing that I didn't mention is that once all the frontend <-> server communications are done over gRPC(-Web), you can implement a client in any language that supports writing gRPC clients. That's probably never going to be useful in this case but fun to consider!

On integrating it, I'm afraid I don't have a great amount of free time to dedicate to such a project, and I opened this discussion and created the CL mostly to see how much work it would be and what, if any, interest there would be from others. So while I'm happy to be part of any discussions, I probably can't dedicate my own time to putting more work into this right now.

Johan

Brad Fitzpatrick

unread,
Dec 28, 2017, 9:16:53 PM12/28/17
to camli...@googlegroups.com
On Fri, Dec 22, 2017 at 8:15 AM, Johan Brandhorst <johan.br...@gmail.com> wrote:
Hi!

I'm the author of the GopherJS gRPC-Web bindings. I was introduced to Perkeep by Paul Jolly, who is the author of the GopherJS React bindings used by Perkeep to render some of its frontend UI. I thought since Perkeep has already invested in GopherJS through this UI it might be interested in trying out my gRPC-Web bindings to abstract away the communications layer between the frontend and backend in a familiar gRPC/Protobuf fashion.

The bindings are built on top of the Improbable gRPC-Web client. It features support for the Fetch API in supported browsers and automatically downgrades to XHR where necessary. It implements the gRPC-Web spec. However, the gRPC-Web spec only mandates support for unary and server-side streaming at this time. My GopherJS bindings adds support for client-side and bidirectional streaming with a custom Websocket proxy on top. Please see my demo website for an example of this.

Cool!
 
It's only a bandaid until the gRPC-Web spec adds support for client-side and bidi-streaming.

Well, first Javascript itself (the Fetch API) needs to support JavaScript streaming an HTTP request body to a server. :) I've been waiting for that for years. 
 
I've submitted a small CL with a prototype of what this could look like on a small scale, to show how the bindings are used and hopefully give an idea of the impact on the codebase. At this time I only implemented it in one place, but ideally for something like this the whole communications layer between the frontend and backend would go through the generated interface.

https://camlistore-review.googlesource.com/c/camlistore/+/12406

Please let me know your thoughts!

This could be nice.

But if we get addicted to this, does it also speak JSON for people who want to speak JSON? I'm afraid we'd either let our JSON rot, or we'd have twice the maintenance cost. Ideally we'd get both for free, with one sufficiently-annotated proto file.




Eric Drechsel

unread,
Dec 28, 2017, 11:40:31 PM12/28/17
to camli...@googlegroups.com
I have experience with grpc-go and have thought about what camlistore would look like ported to protobuf/grpc, but ultimately feel neutral about it. It would be nice for client developers, since bindings could be generated for native iOS, Java, etc. It would also make implementing alternative servers easier. But it would be a giant compatibility break. I will at least try to find some time to look at your POC this weekend.

--
You received this message because you are subscribed to the Google Groups "Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
best, Eric
eric.pdxhub.org

Eric Drechsel

unread,
Dec 28, 2017, 11:47:13 PM12/28/17
to camli...@googlegroups.com
I guess "giant" is too big of a word. Most clients currently are first-party and using the same Go implementation. Maybe now is the time to make any breaking changes to get ready for ecosystem growth.
--
best, Eric
eric.pdxhub.org

Johan Brandhorst

unread,
Dec 29, 2017, 1:55:06 PM12/29/17
to Camlistore
This could be nice.

But if we get addicted to this, does it also speak JSON for people who want to speak JSON? I'm afraid we'd either let our JSON rot, or we'd have twice the maintenance cost. Ideally we'd get both for free, with one sufficiently-annotated proto file.

While the gRPC-Web spec mentions JSON content types, I don't think this is currently possible in the gRPC-Web client. However, we could deploy a gRPC-gateway on a handler and split traffic based on headers and HTTP version. The gRPC-gateway uses the Google annotations.proto annotations to map gRPC service methods to endpoints. So we could bake in a gRPC-gateway and a gRPC server and expose both JSON and gRPC and gRPC-web on the same port, with the interfaces all using the same protofile as the source of truth.

I could potentially bake this part into the current POC if it would be of interest.

Johan

Eric Drechsel

unread,
Dec 29, 2017, 3:19:00 PM12/29/17
to camli...@googlegroups.com
I can vouch for grpc-gateway being a solid solution for JSON-based access. I hope this functionality gets rolled into grpc-web.

If you want bidi streaming support for JSON you can add https://github.com/tmc/grpc-websocket-proxy. IDT grpc-gateway supports fetch-based streaming currently, but it likely will as soon as browsers do. Long term, I really hope grpc-web grows JSON support.



--
You received this message because you are subscribed to the Google Groups "Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
best, Eric
eric.pdxhub.org

Johan Brandhorst

unread,
Dec 30, 2017, 4:36:42 PM12/30/17
to camli...@googlegroups.com
Indeed, the gRPC-gateway supports server side streaming through XHR (newline delimited). However, I'm not sure we can use that bidi proxy with my gRPC-web proxy as they both assume they own all websocket upgrades sent. Might be possible to split traffic with headers but I haven't tested these running together. Travis's proxy was what inspired me to write the websocket functionality into gRPC-web.

As for building this into gRPC-web, it'd be a matter of implementing the translation in the proxy. Long term, as gRPC-web approaches the official gRPC protocol this translation will become someone else's job, as the need for the proxy disappears. So I don't think this functionality will be built into existing proxies, and we should just redirect traffic based on headers.

If I find the time I can try to modify my example to respond in JSON to requests providing the application/grpc-web+json content type header.

Johan


To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
best, Eric
eric.pdxhub.org

--
You received this message because you are subscribed to the Google Groups "Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+...@googlegroups.com.

Paul Jolly

unread,
Jan 3, 2018, 11:58:27 AM1/3/18
to camli...@googlegroups.com
> You get small wins like
> automatically marshalled and unmarshalled payloads to the backend, automatic
> HTTP/2 connection and Fetch API instead of XHR where available and other
> things that [Improbable make a much better case
> for](https://improbable.io/games/blog/grpc-web-moving-past-restjson-towards-type-safe-web-apis)
> than I ever could. I would like to think it would make the frontend/backend
> communications more "maintainable", and nicer to work with

If you want, I might have a communications use case that I (nor Paul
afaik) wasn't able to solve in a very satisfying way, and for which
maybe you could come up with a better solution with gRPC-Web? Let me
know and I'll send you the details.

Mathieu - apologies, I didn't find time to give that "problem" we discussed much time.

There were two parts to it as I recall (my memory is now somewhat vague):
  1. a better solution to the encoding workaround to pass values back from Go -> Javascript
  2. the control over concurrency/order of requests/responses
I've not given any thought to point 1, although I suspect that Johan's work _might_ help here if raw values on the wire can be simply passed back to Javascript world (hence avoiding the double encode/decode). (The ultimate solution, as we discussed, is to lose the requirement to return values from Go -> Javascript at all by converting more components to Go but that's another thread entirely :).

Regarding point 2, I did give that some thought (indeed there was another GopherJS thread talking about exactly the same issue, can't recall when/where hence don't have a link). I'm looking to put together an example that safely handles ordering of requests and responses in a non-blocking way that satisfies the GopherJS constraints of things being non-blocking.

Happy New Year all!


Paul
 

Mathieu Lonjaret

unread,
Jan 3, 2018, 12:24:53 PM1/3/18
to camli...@googlegroups.com
On 3 January 2018 at 17:58, Paul Jolly <pa...@myitcv.io> wrote:
>> > You get small wins like
>> > automatically marshalled and unmarshalled payloads to the backend,
>> > automatic
>> > HTTP/2 connection and Fetch API instead of XHR where available and other
>> > things that [Improbable make a much better case
>> >
>> > for](https://improbable.io/games/blog/grpc-web-moving-past-restjson-towards-type-safe-web-apis)
>> > than I ever could. I would like to think it would make the
>> > frontend/backend
>> > communications more "maintainable", and nicer to work with
>>
>> If you want, I might have a communications use case that I (nor Paul
>> afaik) wasn't able to solve in a very satisfying way, and for which
>> maybe you could come up with a better solution with gRPC-Web? Let me
>> know and I'll send you the details.
>
>
> Mathieu - apologies, I didn't find time to give that "problem" we discussed
> much time.

np, I don't think the map aspect suffers too much from it. yet.

> There were two parts to it as I recall (my memory is now somewhat vague):

> a better solution to the encoding workaround to pass values back from Go ->
> Javascript
> the control over concurrency/order of requests/responses

yes, these were roughly the 2 problems.

> I've not given any thought to point 1, although I suspect that Johan's work
> _might_ help here if raw values on the wire can be simply passed back to
> Javascript world (hence avoiding the double encode/decode). (The ultimate
> solution, as we discussed, is to lose the requirement to return values from
> Go -> Javascript at all by converting more components to Go but that's
> another thread entirely :).

well, as long as we can take the json encoding/decoding cost, it's not
terribly urgent to solve. But it is a recurring problem in the UI
components.

> Regarding point 2, I did give that some thought (indeed there was another
> GopherJS thread talking about exactly the same issue, can't recall
> when/where hence don't have a link). I'm looking to put together an example
> that safely handles ordering of requests and responses in a non-blocking way
> that satisfies the GopherJS constraints of things being non-blocking.

The particular case of the map aspect was not too hard because there
was only one kind of query exchanged between the server and client,
and it was acceptable to ensure that only one is in flight at a given
moment, which makes the ordering of responses moot. So once I got the
"concurrency" right (which you helped me reason about), this was more
or less fixed.

> Happy New Year all!

Happy new year to you!

> Paul
Reply all
Reply to author
Forward
0 new messages