gRPC-JSON proxy

[Yugui Sonoda]

Hello,

I guess you love gRPC but you sometimes want to provide RESTful JSON APIs over HTTP 1.x for backward compatibility or so.

So I have released a prototype of grpc-gateway, which is a generator of gRPC to JSON proxy.

https://github.com/gengo/grpc-gateway

Actually the marshaling into JSON is not the key since marshaling mechanism in gRPC is pluggable.

The key of the generator is that it can map an arbitrary HTTP request path and HTTP request method to a gRPC method call.

You just need to a small amount of custom options to your .proto files. Then the generator wraps your gRPC service with classic RESTful API over HTTP 1.x.

Regards,

-- 

Yuki Yugui Sonoda

[Nicolas Noble]

That's... actually pretty cool! Thanks for sharing!

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/82b1372f-0f2f-41f8-9c67-5e3ae21f3165%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Louis Ryan]

Hey Yugui,

Very cool! I don't think this will come as much of a shock but Google uses a similar mechanism for converting between REST & GRPC internally. I'll ask around and see what level of interest there is in sharing more details about it. Stay tuned

- Louis

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAOWnRi8o_iR7X36PueKgX9HdfXRTP00f%2B-CY1d6c27YKaT2k5Q%40mail.gmail.com.

[Walter Schulze]

Really cool :)

[s...@sourcegraph.com]

grpc-gateway looks fantastic and is exactly what we needed. Thank you!

To contribute some of the "want to support" items, should we just submit PRs? Or discuss on GitHub Issues beforehand? (E.g., the method params in the querystring, caching, etc.)

[Jayant Kolhe]

grpc-gateway indeed is very cool. As Louis mentioned earlier, we also have similar mechanism used internally at Google to convert between REST and gRPC.

We have now put those annotations/spec in github at https://github.com/google/googleapis/tree/master/google/api  It would be great if we can consolidate and 

use same mechanism for consistency in different projects that need gRPC-REST conversion. Can we update grpc-gateway to use same annotations?

Thanks,

 - Jayant

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/2f1ae4e4-8db0-4077-977e-28efccea510b%40googlegroups.com.

[Yugui]

Hi Jayant,

Thanks for sharing the annotation.

On Wed, Apr 22, 2015 at 5:03 AM 'Jayant Kolhe' via grpc.io <grp...@googlegroups.com> wrote:

grpc-gateway indeed is very cool. As Louis mentioned earlier, we also have similar mechanism used internally at Google to convert between REST and gRPC.

We have now put those annotations/spec in github at https://github.com/google/googleapis/tree/master/google/api  It would be great if we can consolidate and 

use same mechanism for consistency in different projects that need gRPC-REST conversion. Can we update grpc-gateway to use same annotations?

I definitely update grpc-gateway with the annotation.

I'll try implementing it this weekend.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALLfn8C1vJf4%3DogqicxZHPE8ZR%3DmW385o8DCJ17O%2Bzjp%3DL5%2BYQ%40mail.gmail.com.

[Yugui]

Hi Jayant,

I have several questions about the spec.  Let me clarify.

https://github.com/google/googleapis/blob/master/google/api/http.proto#L55

 1. The `body` field specifies either `*` or a field path, or is

      omitted. If omitted, it assumes there is no HTTP body.

I guess "a field path" is a path to a field in the message or one in a nested message.

What happens if there is a repeated field on the path?

Does the path select all items in the repeated field or only the first item?

https://github.com/google/googleapis/blob/master/google/api/http.proto#L73

//     Verb     = ":" LITERAL ;

How the Verb part of the template is to be used?

https://github.com/google/googleapis/blob/master/google/api/http.proto#L115

  // Additional HTTP bindings for the selector. Nested bindings must not

  // specify a selector and must not contain additional bindings.

  repeated HttpRule additional_bindings = 11;

What is "selector" in this context?

Would you mind showing me an example usage of additional_bindings?

Regards,

On Wed, Apr 22, 2015 at 5:03 AM 'Jayant Kolhe' via grpc.io <grp...@googlegroups.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALLfn8C1vJf4%3DogqicxZHPE8ZR%3DmW385o8DCJ17O%2Bzjp%3DL5%2BYQ%40mail.gmail.com.

[Louis Ryan]

+cc Wolfgang (wgg@google) who is the owner of the spec

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAB%3Drkjz7%3DEBtvp6_OfjAjE%3DPh%2Bti0_ipjHM181wnoEvHxNG%2BAQ%40mail.gmail.com.

[Yugui]

This is a quick update.

I had some time to work on the custom options in googleapis this weekend.
Pattern matcher for paths, path template parser were added in a branch. Next I'll integrate it with the code generator.

https://github.com/gengo/grpc-gateway/tree/feature/google-http-rule

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CADQ0XY2yt4dgRUS%3D2fq4XU%3D63-AwmZUTr8CoT_GvW5A150U7SQ%40mail.gmail.com.

[Jayant Kolhe]

Thanks Yugui...

[Yugui]

Update

I have just implemented a new code generator for the new custom option.
https://github.com/gengo/grpc-gateway/pull/12

The branch will be merged into master as soon as we complete the code review.

But I am still waiting for an answer from wgg@, and I might need to revise the implementation according to the answer.

Question

Also I have another question.  I'd like to add some fields in the custom option to support the following things.

How can I discuss this change in google/api/http.proto?  Can I just send a pull request to github.com/google/googleapis?

The features I want to add are:

• Content-Type of request/response body -- I'd like to support application/x-www-form-urlencoded
• optional request parameters (fields) in query strings.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALLfn8BbR%2BLjP0c0CHtMjMjujTcNgUkpFwOxLBfx8OAfo9fY8w%40mail.gmail.com.

[Wolfgang Grieskamp]

Yugul, I'm sorry but it appears my answer from 4/22 didn't went to the list, probably because I haven't joint at the point I hit reply. Here it is again.

Hi Yugui,

On Wednesday, April 22, 2015 at 6:28:34 AM UTC-7, Yugui Sonoda wrote:

Hi Jayant,

I have several questions about the spec.  Let me clarify.

https://github.com/google/googleapis/blob/master/google/api/http.proto#L55

 1. The `body` field specifies either `*` or a field path, or is

      omitted. If omitted, it assumes there is no HTTP body.

I guess "a field path" is a path to a field in the message or one in a nested message.

Yes.

 

What happens if there is a repeated field on the path?

Does the path select all items in the repeated field or only the first item?

Repeated fields are not allowed in the body field path or in the URL pattern. Thanks for pointing this ambiguity out, we are fixing the documentation. 

 

https://github.com/google/googleapis/blob/master/google/api/http.proto#L73

//     Verb     = ":" LITERAL ;

How the Verb part of the template is to be used?

This is an (optional) convention to express custom verbs in REST APIs. For example:

    POST /v1/collection/{name}:move

... would be a custom REST operation -- one which does not fit into the CRUD pattern. Sometimes this is expressed as

    POST  /v1/collection/{name}/move

... but we find the ':' notation more readable and it creates less problems with path clash in REST APIs.

 

https://github.com/google/googleapis/blob/master/google/api/http.proto#L115

  // Additional HTTP bindings for the selector. Nested bindings must not

  // specify a selector and must not contain additional bindings.

  repeated HttpRule additional_bindings = 11;

What is "selector" in this context?

It doesn't belong here. Doc bug ;-)

 

Would you mind showing me an example usage of additional_bindings?

The below RPC has two HTTP entry points, where the second is a shortcut which assumes defaults:

 

 rpc GetObject(GetObjectRequest) returns (Object) {

    option (google.api.http) = {

      get: "/v1/projects/{project_id}/objects/{object_id}"

      additional_bindings {

        get: "/v1/objects/{object_id}"

      }

    };

  }

            message GetObjectRequest {

              string project_id = 1; // if empty, assume default

              string object_id = 2;  

            }

 

Best

On Monday, May 4, 2015 at 4:15:41 AM UTC-7, Yugui Sonoda wrote:

Update

I have just implemented a new code generator for the new custom option.
https://github.com/gengo/grpc-gateway/pull/12

The branch will be merged into master as soon as we complete the code review.

[Wolfgang Grieskamp]

On Monday, May 4, 2015 at 4:15:41 AM UTC-7, Yugui Sonoda wrote:

Question

Also I have another question.  I'd like to add some fields in the custom option to support the following things.

How can I discuss this change in google/api/http.proto?  Can I just send a pull request to github.com/google/googleapis?

Yes, in general this would be the way to go, but we should discuss first whether an extension is needed.

 

The features I want to add are:

• Content-Type of request/response body -- I'd like to support application/x-www-form-urlencoded

This would not need an extension of the custom option. Rather, I think this should be always supported (and in our internal implementation, we do this.) I think we should move this paragraph from out internal doc into the http.proto:

"(Note) URL has a length limitation (go/longer-urls). It's enfoced by some browsers and proxies. If your GET request exceeds the limitation, browser may reject to send them. You may change to use a POST request with content type application/x-www-form-urlencoded instead. If the POST + URL is already been mapped to another backend function, you shall add "X-HTTP-Method-Override: GET" header to override the HTTP method and obtain the correct mapping.

 

• optional request parameters (fields) in query strings.

If I understand correctly, this is already covered. Every field not mapped via URL path or body automatically becomes an HTTP parameter.

[Yugui]

Oops, +grpc-io again

----

Wolfgang, thank you for your answers.

For the questions in the first mail, they are now clear for me.

Let me ask more questions about your reply to application/x-www-form-urlencoded and query sting parameters.

On Tue, May 5, 2015 at 2:11 AM Wolfgang Grieskamp <w...@google.com> wrote:

On Monday, May 4, 2015 at 4:15:41 AM UTC-7, Yugui Sonoda wrote:

The features I want to add are:

• Content-Type of request/response body -- I'd like to support application/x-www-form-urlencoded

This would not need an extension of the custom option. Rather, I think this should be always supported (and in our internal implementation, we do this.) I think we should move this paragraph from out internal doc into the http.proto:

Ah, that makes sense.  But it brings another question.

Originally I was thinking of supporting x-www-form-urlencoded only if the request type was flat and non-repeated.

But if we always support x-www-form-urlencoded, how do you represent repeated message fields?  For example, what would the urlencoded form of RequestBody message below look like?

message RequestBody {

  message Inner {

    repeated string value = 1;

  }

  repeated Inner nest = 1;

}

One possible solution is to extend application/x-www-form-urlencoded as Ruby on Rails or PHP does. In this extension, the serialization of RequestBody would look like below.  But I'd like to keep it consistent to the implementation in Google.

nest[0].value[0]=foo&nest[0].value[1]=bar&nest[1].value[0]=baz&nest[2].value[0]=qux

Also for query strings,

• optional request parameters (fields) in query strings.

If I understand correctly, this is already covered. Every field not mapped via URL path or body automatically becomes an HTTP parameter.

I sounds like that query parameters are a kind of "catch-all" things.  But how do you represent an API like the object insert API of Google Cloud Storage with that approach?

I thought it would be something like:

option (google.api.http) = {

  post: "upload/storage/v1/b/{bucket}/o"

  query_string: "upload_type"

  query_string: "content_encoding"

  query_string: ...

  body: "*"

};

Another example is the Disk insert API in Google Compute Engine.

https://github.com/google/googleapis/blob/3ee9064570fef8838792dda7da9bd0ef973597c8/google/api/http.proto#L184

// The name of the request field whose value is mapped to the HTTP body, or
// `*` for mapping all fields not captured by the path pattern to the HTTP
// body. NOTE: the referred field must not be a repeated field.

IIUC, what "body" field in google.api.http option can do is only to (1) map a single field to request body; or (2) catch all.

So it cannot coexist with "catch-all" query parameters.  Thus I need to explicitly declare a list of query parameters so that they are always provided in query strings but not in request body.

  

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/66aa1ca2-0038-4eac-9c6b-e3f4fb4efc92%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Yuki Sonoda (Yugui)
yu...@yugui.jp
http://yugui.jp

[Wolfgang Grieskamp]

On Mon, May 4, 2015 at 7:15 PM, Yugui <yu...@yugui.jp> wrote:

Wolfgang, thank you for your answers.

For the questions in the first mail, they are now clear for me.

Let me ask more questions about your reply to application/x-www-form-urlencoded and query sting parameters.

On Tue, May 5, 2015 at 2:11 AM Wolfgang Grieskamp <w...@google.com> wrote:

On Monday, May 4, 2015 at 4:15:41 AM UTC-7, Yugui Sonoda wrote:

The features I want to add are:

• Content-Type of request/response body -- I'd like to support application/x-www-form-urlencoded

This would not need an extension of the custom option. Rather, I think this should be always supported (and in our internal implementation, we do this.) I think we should move this paragraph from out internal doc into the http.proto:

Ah, that makes sense.  But it brings another question.

Originally I was thinking of supporting x-www-form-urlencoded only if the request type was flat and non-repeated.

But if we always support x-www-form-urlencoded, how do you represent repeated message fields?  For example, what would the urlencoded form of RequestBody message below look like?

message RequestBody {

  message Inner {

    repeated string value = 1;

  }

  repeated Inner nest = 1;

}

One possible solution is to extend application/x-www-form-urlencoded as Ruby on Rails or PHP does. In this extension, the serialization of RequestBody would look like below.  But I'd like to keep it consistent to the implementation in Google.

nest[0].value[0]=foo&nest[0].value[1]=bar&nest[1].value[0]=baz&nest[2].value[0]=qux

I should have been clearer. The idea is that urlencoded would only be used for methods declared as GET to bypass URL length restrictions. So only what is already a URL would appear here, and in URLs we only allow repeated fields of primitive types.

So this restriction helps us here ;-)

BTW, the restriction has the following motivation. Many existing REST API frameworks don't support complex value bindings in the URLs (e.g. Swagger). We felt it's better to not offer this feature to encourage API design with interoperability in mind. Its clear there are technical ways to do this (my preferred one would probably be oriented towards jQuery).

Also for query strings,

• optional request parameters (fields) in query strings.

If I understand correctly, this is already covered. Every field not mapped via URL path or body automatically becomes an HTTP parameter.

I sounds like that query parameters are a kind of "catch-all" things.  But how do you represent an API like the object insert API of Google Cloud Storage with that approach?

I thought it would be something like:

option (google.api.http) = {

  post: "upload/storage/v1/b/{bucket}/o"

  query_string: "upload_type"

  query_string: "content_encoding"

  query_string: ...

  body: "*"

};

The insert method would be represented as follows:

       rpc InsertObject(InsertObjectRequest) returns (Object) {

         option (google.api.http) = { 
           post: "/v1/b/{bucket}/o"

           body: "object"

       }

       message InsertObjectRequest {
         // path parameters
         string bucket = 1;

         // query parameters

         string upload_type = 2;

         string content_encoding = 3;

         ...

         // body

         Object object = 4;

       }

       ...

The fields upload_type and content_encoding automatically become HTTP parameters.

We figured the design of having this implicit rather than listing HTTP parameters has less boilerplate, and an implicit enforcement that all fields in the proto are reached by the HTTP request. We don't want 'dead' fields to maintain parity between the HTTP request and the GRPC request.

 

Another example is the Disk insert API in Google Compute Engine.

https://github.com/google/googleapis/blob/3ee9064570fef8838792dda7da9bd0ef973597c8/google/api/http.proto#L184

// The name of the request field whose value is mapped to the HTTP body, or
// `*` for mapping all fields not captured by the path pattern to the HTTP
// body. NOTE: the referred field must not be a repeated field.

IIUC, what "body" field in google.api.http option can do is only to (1) map a single field to request body; or (2) catch all.

So it cannot coexist with "catch-all" query parameters.  Thus I need to explicitly declare a list of query parameters so that they are always provided in query strings but not in request body.

The mechanics is as follows: all fields which are not reached either via a path parameter or via the body mapping become HTTP parameters. If we have 'body: "*"', there are no fields left to be mapped as HTTP parameters. We haven't found yet that this imposes restrictions for RESTful APIs at least.

Let me know what you think.

[Yugui]

On Tue, May 5, 2015 at 2:39 PM 'Wolfgang Grieskamp' via grpc.io <grp...@googlegroups.com> wrote:

On Mon, May 4, 2015 at 7:15 PM, Yugui <yu...@yugui.jp> wrote:

Wolfgang, thank you for your answers.

For the questions in the first mail, they are now clear for me.

Let me ask more questions about your reply to application/x-www-form-urlencoded and query sting parameters.

On Tue, May 5, 2015 at 2:11 AM Wolfgang Grieskamp <w...@google.com> wrote:

On Monday, May 4, 2015 at 4:15:41 AM UTC-7, Yugui Sonoda wrote:

The features I want to add are:

• Content-Type of request/response body -- I'd like to support application/x-www-form-urlencoded

This would not need an extension of the custom option. Rather, I think this should be always supported (and in our internal implementation, we do this.) I think we should move this paragraph from out internal doc into the http.proto:

Ah, that makes sense.  But it brings another question.

Originally I was thinking of supporting x-www-form-urlencoded only if the request type was flat and non-repeated.

But if we always support x-www-form-urlencoded, how do you represent repeated message fields?  For example, what would the urlencoded form of RequestBody message below look like?

message RequestBody {

  message Inner {

    repeated string value = 1;

  }

  repeated Inner nest = 1;

}

One possible solution is to extend application/x-www-form-urlencoded as Ruby on Rails or PHP does. In this extension, the serialization of RequestBody would look like below.  But I'd like to keep it consistent to the implementation in Google.

nest[0].value[0]=foo&nest[0].value[1]=bar&nest[1].value[0]=baz&nest[2].value[0]=qux

I should have been clearer. The idea is that urlencoded would only be used for methods declared as GET to bypass URL length restrictions. So only what is already a URL would appear here, and in URLs we only allow repeated fields of primitive types.

So this restriction helps us here ;-)

I understand. 

Actually I am thinking of supporting application/x-www-form-urlencoded in POST or PUT requests too for our requirement.  But I'd like to keep urlencoded parameters as simple as you explained.  Urlencoding of Body is supported only if body parameters are simple.

It won't break compatibility to Google's spec.

Let me ask some more questions.

• How the request path will look like when its path parameters are placed in the request body to bypass the length restriction?
  
  
• How does it look like in the request path when repeated fields of primitive types are placed in path parameters?
  
  
• I still wonder if it would be better to explicitly declare that the POST/PUT method supports application/x-www-form-urlencoded in a field like
  
    repeated string body_encoding = n;
  
  of the custom option.
  If we don't have such a option field, a method can silently lose support of application/x-www-form-urlencoded when an user adds a repeated message field to the request message type.
  On the other hand, protoc-gen-grpc-gateway can display an error if the method has both repeated message field and "body_encoding = 'application/x-www-form-urlencoded'".
  
  What do you think?

 

BTW, the restriction has the following motivation. Many existing REST API frameworks don't support complex value bindings in the URLs (e.g. Swagger). We felt it's better to not offer this feature to encourage API design with interoperability in mind. Its clear there are technical ways to do this (my preferred one would probably be oriented towards jQuery).

It sounds good to me.

I understand.  Thank you for your explanation.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAFuU7PacS2LGWJjFGjamAdnVGrrbNSV25GoABGjaOkrjwvo_Kg%40mail.gmail.com.

[Yugui]

Ah, the second question might be wrong.

When you said "So only what is already a URL would appear here, and in URLs we only allow repeated fields of primitive types.", I thought the repeated fields were coming from path of the URL.

But did you mean the repeated fields were coming from query string?

Let me confirm. Is the following understanding correct?

• Request Body (JSON)
• fields of aggregate types are not allowed
• repeated fields are allowed
• Request Path
• fields of aggregate types are not allowed
• nested fields (fields in a message field) are allowed if primitive
• repeated fields are not allowed
• Query string
• fields of aggregate types are not allowed
• nested fields are allowed if primitive?
• repeated fields are allowed if primitive

[Yugui]

I have prototyped additional_bindings, query strings and showing error if a field path has a repeated field.

The prototype is available in a branch.

But I'd like to wait for reply from Wolfgang about the three remaining questions before implementing application/x-www-form-urlencoded support and merging the branch into master.

[Wolfgang Grieskamp]

I'm wondering what's the use case for this. Is this for humans making those calls, and the urlencoded form would be more convenient for them?

Currently, I see this feature as a workaround to URL length only.

 

But I'd like to keep urlencoded parameters as simple as you explained.  Urlencoding of Body is supported only if body parameters are simple.

It won't break compatibility to Google's spec.

Actually it would. Suppose someone would call into an API using the grpc gateway using this feature. He couldn't switch e.g. to an implementation of the spec by Google or anybody else not supporting it. If we add this feature, every implementation will need to support it.

Let me ask some more questions.

• How the request path will look like when its path parameters are placed in the request body to bypass the length restriction?
  
  
• How does it look like in the request path when repeated fields of primitive types are placed in path parameters?
  

Ah, the second question might be wrong.

When you said "So only what is already a URL would appear here, and in URLs we only allow repeated fields of primitive types.", I thought the repeated fields were coming from path of the URL.

But did you mean the repeated fields were coming from query string?

Yes, I was referring to query parameters. To summarize: path parameters can only have primitive type, and query parameters can have primitive or repeated primitive type. 

Let me confirm. Is the following understanding correct?

• Request Body (JSON)
• fields of aggregate types are not allowed
• repeated fields are allowed

Request body is unrestricted, aggregate types are allowed.

 

• Request Path
• fields of aggregate types are not allowed
• nested fields (fields in a message field) are allowed if primitive
• repeated fields are not allowed

Yes 

• Query string
• fields of aggregate types are not allowed
• nested fields are allowed if primitive?
• repeated fields are allowed if primitive

Yes.

Nested fields are allowed if of primitive or repeated primitive type. While the implicit query parameter mapping is constructed, fields of message type are unfolded into paths to primitive fields. (A cyclic message would produce an error, as well as a repeated message.)

• I still wonder if it would be better to explicitly declare that the POST/PUT method supports application/x-www-form-urlencoded in a field like
  
    repeated string body_encoding = n;
  
  of the custom option.
  If we don't have such a option field, a method can silently lose support of application/x-www-form-urlencoded when an user adds a repeated message field to the request message type.
  On the other hand, protoc-gen-grpc-gateway can display an error if the method has both repeated message field and "body_encoding = 'application/x-www-form-urlencoded'".
  
  What do you think?

I agree this would need to be explicitly enabled, not at least because we would need to produce an error if the body is not urlencodable. However,  I think we should first add this option to a specific configuration for grpc gateway. We can move it over to the standard annotation once we understand better that it's widely needed.

For example:

    (google.api.http).post = "/v1/some/path";

    (grpc_gateway).body_url_encoded = true;

[Yugui]

Wolfgang, thank you for your answers.

Now it is clear enough to finalize the implementation.

For application/x-www-form-urlencoded not as a url length workaround, let me rethink it.

I needed it for compatibility to our old APIs to be reimplemented with gRPC, but I might be able to deprecate the APIs instead.  Even if I actually need the feature, certainly I can still implement it as another custom option as you suggested for explicit declaration of "application/x-www-form-urlencoded as a workaround".

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAFuU7PYdeuO8%3DAmZO52gtAuU3Ust_fWMkYEmrnDiBWCnSie6yQ%40mail.gmail.com.

[Yugui]

Implementation done.
https://github.com/gengo/grpc-gateway/pull/14

Now it is under code review.

[Yugui]

I have merged the pull request.  I believe grpc-gateway now satisfies all the spec shown.

Thank you for sharing the annotation, Googlers.