Endpoints v2 InconsistentApiConfigurationException

[Thomas Wiradikusuma]

Hi guys,

I'm a longtime user of Cloud Endpoints (v1), and decided to give v2 a try. So I followed https://cloud.google.com/appengine/docs/java/endpoints/migrating and here's my diff:

- compile 'com.google.appengine:appengine-endpoints:1.9.42'

+ compile 'com.google.endpoints:endpoints-framework:2.0.0-beta.8'

- <servlet-class>com.google.api.server.spi.SystemServiceServlet</servlet-class>

+ <servlet-class>com.google.api.server.spi.EndpointsServlet</servlet-class>

- <url-pattern>/_ah/spi/*</url-pattern>

+ <url-pattern>/_ah/api/*</url-pattern>

I did *not* run the "generating client lib" as documented in https://cloud.google.com/endpoints/docs/frameworks/java/generate-client-libraries-android as I thought, "from the client perspective it should be the same".

I hit this error in the server:

SEVERE: javax.servlet.ServletContext log: unavailable

com.google.api.server.spi.config.validation.InconsistentApiConfigurationException: api: API-wide configuration does not match between the classes com.foobar.ApiV2Public and com.foobar.ApiV2. All API classes with the same API name and version must have the exact same API-wide configuration. Differing property: issuers (com.google.api.server.spi.config.model.ApiIssuerConfigs@0 vs com.google.api.server.spi.config.model.ApiIssuerConfigs@d03e1481).

        at com.google.api.server.spi.config.validation.ApiConfigValidator.validate(ApiConfigValidator.java:89)

        at com.google.api.server.spi.SystemService.registerLoadedService(SystemService.java:218)

        at com.google.api.server.spi.SystemService.registerService(SystemService.java:193)

How do I fix this?

[Nick (Cloud Platform Support)]

Hey Thomas,

This is just a guess, but it could be that it's necessary to update the client as well. Does that error occur when you deploy, or when you receive a request? That would be the way to determine if the issue is in the client needing to be updated or another issue (not clarified as yet) in the server code. 

As a quick side-note, this forum is meant for general high level discussion of the platform and services, design patterns, etc. So a thread like this should actually be a Stack Overflow post. Not to worry as we watch that forum as well. You are more likely to find more users who can help, though, with a technical question in that forum.

Cheers,

Nick
Cloud Platform Community Support

[Thomas Wiradikusuma]

Hi Nick,

The error happens when receiving requests. I don't think it's necessary to update the client, as the update doesn't change the server.

I'm posting here since I think it's either a bug of, or lack of documentation for, the V2 implementation @ApiClass. Let me explain.

I have 2 classes that share @Api (https://cloud.google.com/endpoints/docs/frameworks/java/multiclass): 

@Api(name = "api", version = "v2", namespace = @ApiNamespace(ownerDomain = "bla.com", ownerName = "bla.com"))

@ApiClass(authenticators = {AccountManager.class}, scopes = {EMAIL_SCOPE}, clientIds = {API_V2_ANDROID_CLIENT_ID, API_V2_WEB_CLIENT_ID, API_EXPLORER_CLIENT_ID}, audiences = {API_V2_ANDROID_AUDIENCE})

public class ApiV2 { .. }

@Api(name = "api", version = "v2", namespace = @ApiNamespace(ownerDomain = "bla.com", ownerName = "bla.com"))

public class ApiV2Public { .. }

This is how they're defined in web.xml:

<servlet>

<servlet-name>api</servlet-name>

<servlet-class>com.google.api.server.spi.EndpointsServlet</servlet-class>

<init-param>

<param-name>services</param-name>

<param-value>

com.bla.ApiV2,com.bla.ApiV2Public

</param-value>

</init-param>

</servlet>

If I take out one from web.xml, it works. 

This used to work in V1. It could be due to this new property "issuers" that's poorly documented (compare https://cloud.google.com/appengine/docs/java/endpoints/annotations vs https://cloud.google.com/endpoints/docs/frameworks/java/annotations, Ctrl+F for "issuers").

Looking forward for a solution!

Regards,

Thomas Wiradikusuma

[Nick (Cloud Platform Support)]

Hey Thomas,

Could you show what the web.xml file looks like before you take one out? I'm not sure how you've included both of them and it appears you uploaded a section from web.xml with only one Api class present.

Also, just for testing's sake, have you managed to check whether recompiling the client against the new API version resolves the issue?

Cheers,

Nick
Cloud Platform Community Support

[Thomas Wiradikusuma]

Hi Nick,

The web.xml snippet I gave earlier is the one before I take one out:

<param-value>com.bla.ApiV2,com.bla.ApiV2Public</param-value>

Notice class ApiV2 and ApiV2Public.

Also, I'm testing this from API explorer, which I think doesn't require the generation of client libraries. But, for curiosity sake, I did try it (https://cloud.google.com/endpoints/docs/frameworks/java/generate-client-libraries-android):

endpoints-framework-tools get-client-lib --war=target/war -bs gradle com.bla.ApiV2

endpoints-framework-tools get-client-lib --war=target/war -bs gradle com.bla.ApiV2Public

No error in the generation (but doesn't fix my issue).

Regards,

Thomas Wiradikusuma

[Nick (Cloud Platform Support)]

Hey Thomas,

The issue wouldn't be that generation would fail - the idea is that due to having recompiled the API against different classes for V2, the old client might not be sending requests in the way expected. Could you try to load the updated client library into a client machine and see if the same error persists with the new version of the library? I'm continuing to work on reproducing this issue on my own side.

Cheers,

Nick
Cloud Platform Community Support

[Thomas Wiradikusuma]

Hi Nick,

Attached is my Gradle-based project stripped to the simplest to reproduce the issue. I'm unclear with what you meant, but hopefully my project can help on your side!

Regards,

Thomas Wiradikusuma

[Nick (Cloud Platform Support)]

Hey Thomas,

Thanks for uploading. I'll be sure to take a look at this project and see if I can manage to deploy it. However, in the mean time, I think there's a possible solution on your end that hasn't been investigated so far, as far as I can tell. The issue appears to me to be caused by compiling and deploying a new version of the API, while still using the old generated client library in the clients. If you could not only generate the new client library (which should not have errors, as you observed), but also put it into the clients and have them send requests, to see whether the errors persist, that could short-circuit the process of testing on our end.

Cheers,

Nick
Cloud Platform Community Support 

[Thomas Wiradikusuma]

Hi Nick,

I've been using API Explorer (built-in tool from Cloud Endpoints), which does not require client library. Or should I use other tools?

Regards,

Thomas Wiradikusuma

[Nick (Cloud Platform Support)]

Hey Thomas,

Could you try to add the same @ApiClass annotation that you have on ApiV2.java to ApiV2Public.java and see if that prevents the error message?

I'm unable to replicate the issue using your code and keeping all the API names in web.xml, even regardless of whether I add the @ApiClass annotation to ApiV2Public.

I've tested using appcfg.sh from SDK version 1.9.38 and the latest SDK (1.9.42).

I don't see any "issuers" field defined in your API. Is this correct?

Cheers,

Nick
Cloud Platform Community Support

[Thomas Wiradikusuma]

Hi Nick,

> Could you try to add the same @ApiClass annotation that you have on ApiV2.java to ApiV2Public.java and see if that prevents the error message? 

Yes, making the annotations identical in both classes prevents the error message. But, it means both must be authenticated, which defeats my purpose of separating it.

> I'm unable to replicate the issue using your code and keeping all the API names in web.xml, even regardless of whether I add the @ApiClass annotation to ApiV2Public. 

I'm sorry for not making it clear. The sample code is _working_, you need to migrate it to Endpoints v2 as mentioned in the README.md to make it not working. Anyway, for your convenience, I've attached the migrated version (which is not working).

> I don't see any "issuers" field defined in your API. Is this correct?

Yes, that field is a new field, which I think the documentation doesn't clearly explain what it is.

Regards,

Thomas Wiradikusuma

[Thomas Wiradikusuma]

Sorry I forgot to attach the project.

I'm sorry for not making it clear. The sample code is _working_, you need to migrate it to Endpoints v2 as mentioned in the README.md to make it not working. Anyway, for your convenience, I've attached the migrated version (which is not working).

Regards,

Thomas Wiradikusuma 

[Nick (Cloud Platform Support)]

Hey Thomas,

Thanks for your persistent reporting here. I've now managed to reproduce the behaviour and I've made a Public Issue Tracker thread to follow this for anybody along with you who is also interested in its progress.

As a workaround for now, simply renaming one of the APIs will remove the issue. Of course your pattern of using ApiClass for differentiated properties in a multiclass API should be defined as the docs say, for now this will allow you to circumvent the issue.

In future, don't hesitate to make a post like this to the Public Issue Tracker if you suspect the error is not in your code but the platform. We'll be happy to assist!

Cheers,

Nick
Cloud Platform Community Support

[Nick (Cloud Platform Support)]

Hey Thomas,

Just an update, in the Public Issue Tracker thread, we appear to have found another workaround as well.

Cheers,

Nick
Cloud Platform Community Support

[Thomas Wiradikusuma]

Hi Nick,

Thanks! I have completed the migration, including the stuff needed so I can see fancy graphs in https://console.cloud.google.com/endpoints.

Some feedback:

* The Errors graph is showing some 4xx, but drilling down just gives me log in json, and it doesn't show what the error is.

* The Total Latency graph is not too useful nor actionable. I see some spikes, great. Are they caused by my code? How do I fix them? Can't even click to drill down.

I'm sorry I complain a lot. Just to be clear, I love what you guys doing. Please keep up the good work!

Regards,

Thomas Wiradikusuma

[Nick (Cloud Platform Support)]

Hey Thomas,

The reason you might see a 4xx on a request is best explained by checking whether an endpoints method actually exists on that path, whether your request had authorization, whether the request payload was malformed, or whether the correct request method (POST / GET) was used. The specific 4xx error will enable you to differentiate between these possibilities.

As far as debugging latency, you can check either the request logs for the affected time or use the "Current load" section of the App Engine dashboard in the Cloud Console (if the latency was recent) to view which request paths are serving with latency. You should then take a look at the code used to serve that request path and see which aspects of the code could be causing latency.

Cheers,

Nick
Cloud Platform Community Support