API Authentication

[dan....@stonebridgeprojects.com]

I am trying to create a RESTish API in Java that will run on App Engine Standard.  I am using Endpoints Framework and I have a sample POST method up and running both locally and on App Engine.  Now I want to add user authentication.  I thought I could do that with Endpoints Management using OpenAPI, but I am confused by the documentation.  

I am relying primarily on this page for instructions for integrating API Managent:

https://cloud.google.com/endpoints/docs/frameworks/java/adding-api-management

As a first step toward using Endpoint Management authentication I have generated the OpenAPI.json file using the Maven plugin  and deployed it.  I see my API calls being logged and I see my OpenAPI configuration in the GCP Endpints Services Deployment , but I don't see any API calls in the Endpoints Services graphs.  I am thinking maybe that is because I don't have an ESP because I am running on App Engine standard, an idea I got from this page:

https://cloud.google.com/endpoints/docs/frameworks/frameworks-extensible-service-proxy#endpoints_frameworks

I don't understand how these pieces:

App Engine Standard

Cloud Endpoints

Endpoints Framework

Open API

API Management

Extensible Service Proxy

fit together, although I think I am already using all of them except perhaps ESP.

I would be grateful for a quick explanation, along the lines of: This is the old name for this is obsolete, this can be used with this, but if you use this you cannot use this.

Thanks.

-Dan

[Andrew Gunsch]

Thanks for writing!

At a high-level, most of the Endpoints functionality works by having the proxy (ESP) in front of your API's implementation, passing just enough information to Google to do API key checks + report metrics that show up in the "Endpoints" > "Services" UI.

However, due to how App Engine Standard works, ESP can't run in front of your API's implementation in that environment. So in that case, we have the Endpoints Frameworks library to do a few things for your app, including:

* Providing functionality to easily generate an OpenAPI spec representing your API (which, once you deploy with `gcloud`, enables GCP to know enough about the surface of your API)

* Doing auth checks + reporting metrics to GCP, similar to what ESP would do otherwise.

+Daniel Tang, is there an easy way to confirm whether an app has correctly enabled the management functionality of the frameworks?

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dan....@stonebridgeprojects.com]

Hi, Andrew.  Thank you very much for your reply.  It is very helpful.

-Dan

On Tuesday, August 7, 2018 at 12:23:54 AM UTC+1, Andrew Gunsch wrote:

Thanks for writing!

At a high-level, most of the Endpoints functionality works by having the proxy (ESP) in front of your API's implementation, passing just enough information to Google to do API key checks + report metrics that show up in the "Endpoints" > "Services" UI.

However, due to how App Engine Standard works, ESP can't run in front of your API's implementation in that environment. So in that case, we have the Endpoints Frameworks library to do a few things for your app, including:

* Providing functionality to easily generate an OpenAPI spec representing your API (which, once you deploy with `gcloud`, enables GCP to know enough about the surface of your API)

* Doing auth checks + reporting metrics to GCP, similar to what ESP would do otherwise.

+Daniel Tang, is there an easy way to confirm whether an app has correctly enabled the management functionality of the frameworks?

On Sun, Aug 5, 2018 at 9:54 AM <dan.stone@stonebridgeprojects.com> wrote:

I am trying to create a RESTish API in Java that will run on App Engine Standard.  I am using Endpoints Framework and I have a sample POST method up and running both locally and on App Engine.  Now I want to add user authentication.  I thought I could do that with Endpoints Management using OpenAPI, but I am confused by the documentation.  

I am relying primarily on this page for instructions for integrating API Managent:

https://cloud.google.com/endpoints/docs/frameworks/java/adding-api-management

As a first step toward using Endpoint Management authentication I have generated the OpenAPI.json file using the Maven plugin  and deployed it.  I see my API calls being logged and I see my OpenAPI configuration in the GCP Endpints Services Deployment , but I don't see any API calls in the Endpoints Services graphs.  I am thinking maybe that is because I don't have an ESP because I am running on App Engine standard, an idea I got from this page:

https://cloud.google.com/endpoints/docs/frameworks/frameworks-extensible-service-proxy#endpoints_frameworks

I don't understand how these pieces:

App Engine Standard

Cloud Endpoints

Endpoints Framework

Open API

API Management

Extensible Service Proxy

fit together, although I think I am already using all of them except perhaps ESP.

I would be grateful for a quick explanation, along the lines of: This is the old name for this is obsolete, this can be used with this, but if you use this you cannot use this.

Thanks.

-Dan

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

[D. T.]

API Management is the key piece which is in charge of reporting requests so that they show up in the graphs. If your web.xml is not configured correctly, they won't show up. Can you show your web.xml configuration? In particular, do you have GoogleAppEngineControlFilter installed?

Daniel Tang |

Software Engineer |

ta...@google.com |

Google LLC

--

You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

[dan....@stonebridgeprojects.com]

Hi Daniel,

Thanks very much for offering to help.  Here is my web.xml.

-Dan

<?xml version="1.0" encoding="utf-8" standalone="no"?>

<!-- <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" 

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="2.5" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 

http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> -->

<web-app xmlns="http://java.sun.com/xml/ns/javaee" version="2.5">

<servlet>

<servlet-name>EndpointsServlet</servlet-name>

<servlet-class>com.google.api.server.spi.EndpointsServlet

</servlet-class>

<multipart-config>

<max-file-size>20848820</max-file-size>

<max-request-size>418018841</max-request-size>

<file-size-threshold>1048576</file-size-threshold>

</multipart-config>

<init-param>

<param-name>services</param-name>

<param-value>uk.co.scapps.tryagain.YourFirstAPI</param-value>

</init-param>

<init-param>

<param-name>restricted</param-name>

<param-value>false</param-value>

</init-param>

<filter>

<filter-name>ObjectifyFilter</filter-name>

<filter-class>com.googlecode.objectify.ObjectifyFilter</filter-class>

</filter>

<filter-mapping>

<filter-name>ObjectifyFilter</filter-name>

<url-pattern>/*</url-pattern>

</filter-mapping>

<!-- Add a filter that fetches the service config from service management. -->

<filter>

<filter-name>endpoints-api-configuration</filter-name>

<filter-class>com.google.api.control.ServiceManagementConfigFilter</filter-class>

</filter>

<!-- Add a filter that performs Endpoints logging and monitoring. -->

<filter>

<filter-name>endpoints-api-controller</filter-name>

<filter-class>com.google.api.control.extensions.appengine.GoogleAppEngineControlFilter</filter-class>

<init-param>

<param-name>endpoints.projectId</param-name>

<param-value>trydatastore-208114</param-value>

</init-param>

<init-param>

<param-name>endpoints.serviceName</param-name>

<param-value>trydatastore-208114.appspot.com</param-value>

</init-param>

</filter>

  <filter-mapping>

<filter-name>endpoints-api-configuration</filter-name>

<servlet-name>EndpointsServlet</servlet-name>

    <dispatcher>REQUEST</dispatcher>

    <dispatcher>INCLUDE</dispatcher>

    <dispatcher>FORWARD</dispatcher>

</filter-mapping>

<filter-mapping>

<filter-name>endpoints-api-controller</filter-name>

<servlet-name>EndpointsServlet</servlet-name>

</filter-mapping>

</servlet>

<listener>

<listener-class>uk.co.scapps.tryagain.Bootstrapper</listener-class>

</listener>

<servlet-mapping>

<!-- Endpoints Frameworks v2.0 -->

<servlet-name>EndpointsServlet</servlet-name>

<url-pattern>/_ah/api/*</url-pattern>

<!-- End of Endpoints Frameworks v2.0 -->

</servlet-mapping>

</web-app>

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

[D. T.]

I'm taking this off the group. Can you also provide your appengine-web.xml?

Daniel Tang |

Software Engineer |

ta...@google.com |

Google LLC

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cdff93ac-15f9-45b9-8ed5-16d302c45305%40googlegroups.com.

[dan....@stonebridgeprojects.com]

Hi Daniel,

Thanks for the reply.  Here it is.

-Dan

<?xml version="1.0" encoding="utf-8"?>

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">

<threadsafe>true</threadsafe>

<runtime>java8</runtime>

<system-properties>

<property name="java.util.logging.config.file" value="WEB-INF/logging.properties" />

</system-properties>

<application>trydatastore-208114</application>

<env-variables>

    <env-var name="ENDPOINTS_SERVICE_NAME" value="trydatastore-208114.appspot.com" />

</env-variables>

</appengine-web-app>

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

[dan....@stonebridgeprojects.com]

Hi Daniel,

I am not sure how to reply off the group.  I hope you received my appengine-web.xml, and I am looking forward to hearing your thoughts on the issue I am facing.

Thanks.

-Dan

[D. T.]

It looks like you are using autoscaling instead of basic scaling. So API management runs a bit differently in this mode, because background threads are disallowed. Therefore, request reporting is sent after some requests, but not all. If you are only sending very low amounts of traffic, sometimes the request stats become stale before they are sent. One possible symptom of this would be in your request log. Are you seeing any warnings you wouldn't normally see? If so, can you post a sample?

Daniel Tang |

Software Engineer |

ta...@google.com |

Google LLC

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cdff93ac-15f9-45b9-8ed5-16d302c45305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/0dabef34-d69b-4f7d-ab01-1e63f7c5ba91%40googlegroups.com.

[Dan Stone]

Thanks, Daniel. I guess when you mention the request log you are referring to the log in Stackdriver. Is that correct? I will check that later in the day and get back to you. I will also try to activate basic scaling.

There is, however, an additional piece of information that may be significant. I have tried to add authentication to my method and consistently receive a "method info is not set in the request" error. The combination of this behaviour and the graphing issue makes me think that perhaps API Management is not engaged for my API.

-Dan

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[D. T.]

After adding auth, did you run gcloud endpoints services deploy?

Daniel Tang |

Software Engineer |

ta...@google.com |

Google LLC

[Dan Stone]

I think so. When I get home I will repeat the process.

[dan....@stonebridgeprojects.com]

Hi Daniel,

Thanks again for taking the time to help me.

I doubt it matters but I thought I should mention that I am deploying the API from Eclipse.

It appears that I had regenerated and deployed openapi.json, but I did it again with the same apparent result.  If my method includes a User parameter then the API request fails with a 503.  When I run on the local emulator I can see  the "method info is not set in the request" error.  When I remove the User parameter and redeploy, the API works correctly and the request log entries look good:

86.136.239.240 - - [21/Aug/2018:21:37:15 +0100] "POST /_ah/api/skeletonapi/v1/sayHi/derf?id=3 HTTP/1.1" 200 140 - "curl/7.47.0" "trydatastore-208114.appspot.com" ms=NaN cpu_ms=92 cpm_usd=1.5646e-8 loading_request=0 instance=00c61b117c0b1b8433b67957289d3f588383e81e93dc0b7716bcf6b1a972ded44ee2f2 app_engine_release=1.9.54 trace_id=1ca390a2eb0819701e764f922d1de394

Expand all | Collapse all

 

 

 

 

 

 

 

 

 

 

}

and I can see the invocations in the GAE Dashboard graph.  Still no signs of life in the Endpoints graphs.

I switched the scaling to basic.  No change that I can see:  Still graphing in GAE Dashboard.  Still logging clean requests.  Still no graph entries in Endpoints.

-Dan

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cdff93ac-15f9-45b9-8ed5-16d302c45305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/0dabef34-d69b-4f7d-ab01-1e63f7c5ba91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[D. T.]

If you're getting an HTTP 503 then your Stackdriver logs should contain some interesting information. Also interesting would be any logs that might be written when an instance starts up. You also need to remember that essentially every time you change your annotations, you need to run gcloud endpoints services deploy.

Daniel Tang |

Software Engineer |

ta...@google.com |

Google LLC

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cdff93ac-15f9-45b9-8ed5-16d302c45305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/0dabef34-d69b-4f7d-ab01-1e63f7c5ba91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--

You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/94fc404e-e6a3-442b-b559-95ba2d46c0d1%40googlegroups.com.

[dan....@stonebridgeprojects.com]

Hi Daniel,

Thanks for the suggestions.

The logs from the 503 requests look like the stack trace I saw in my Eclipse console when I deployed the API with the User parameter on the Java method locally:

W  POST 503 280 B 81 ms curl/7.47.0 /_ah/api/skeletonapi/v1/sayHi/derf?id=3 POST 503 280 B 81 ms curl/7.47.0 5b7ce89e00ff063fae1e60231e0001687e7472796461746173746f72652d3230383131340001323031383038323274303533353336000100
  86.136.239.240 - - [22/Aug/2018:05:37:50 +0100] "POST /_ah/api/skeletonapi/v1/sayHi/derf?id=3 HTTP/1.1" 503 280 - "curl/7.47.0" "trydatastore-208114.appspot.com" ms=NaN cpu_ms=86 cpm_usd=3.1292e-8 loading_request=0 instance=00c61b117c933c3776b3bd9fc4976e46f46c2d6e51bfa84977bb3986c4ea5884ab3d0fc5 app_engine_release=1.9.54 trace_id=a7746df5cd51c49cff7df1cf16646177

2018-08-22 05:37:50.487 BSTcom.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler handle: exception occurred while invoking backend method (EndpointsMethodHandler.java:124) java.lang.IllegalStateException: method_info is not set in the request at com.google.api.server.spi.auth.EspAuthenticator.authenticate(EspAuthenticator.java:67) at com.google.api.server.spi.request.Auth.authenticate(Auth.java:101) at com.google.api.server.spi.request.ServletRequestParamReader.getUser(ServletRequestParamReader.java:214) at com.google.api.server.spi.request.ServletRequestParamReader.deserializeParams(ServletRequestParamReader.java:142) at com.google.api.server.spi.request.RestServletRequestParamReader.read(RestServletRequestParamReader.java:135) at com.google.api.server.spi.SystemService.invokeServiceMethod(SystemService.java:349) at com.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler.handle(EndpointsMethodHandler.java:119) at com.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler.handle(EndpointsMethodHandler.java:102) at com.google.api.server.spi.dispatcher.PathDispatcher.dispatch(PathDispatcher.java:50) at com.google.api.server.spi.EndpointsServlet.service(EndpointsServlet.java:72) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:120) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182) at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:171) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:297) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:202) at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81) at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:693) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:655) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:625) at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:819) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:274) at java.lang.Thread.run(Thread.java:745)

I have not been changing my annotations and the generated JSON file has not been changing, although I have regenerated it frequently for comparison. My understanding is that my HTTP method signature does not change when I use User in my Java method, but that User should be part of my Java method signature. Kindly correct me if I am mistaken.

I am not sure where to find the startup log you mentioned. Please tell me where to find it.

Thanks.

-Dan

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/da39cf07-5166-4bc6-84a6-52ee273b6022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/cdff93ac-15f9-45b9-8ed5-16d302c45305%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/0dabef34-d69b-4f7d-ab01-1e63f7c5ba91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.