NDB urlsafe keys and REST api requests

[Jason Yoon]

I was wondering what others are doing to expose REST api endpoints with the datastore. I want to use urlsafe keys but 1 - I'd rather not pass this data directly as it poses a security risk, and 2 - the keys that are generated are very long and would not be great when multiple need to be passed as a query parameter to form a get request (and would probably exceed browser character limits).

I was thinking maybe using compression of some sort to compress the urlsafe keys which would solve both 1 and 2, but want to see if there is a better way to create REST endpoints. Or if some type of compression method is already baked into ndb?

[Kenworth (Google Cloud Platform)]

To help you find the answers you are looking for, can you please explain further what are you trying to achieve? Including the following:

1- Do you want to expose Datastore REST api to clients?

2- Do you want to design your own API?

[Jason Yoon]

Sure, here is the context for my question.

I have a frontend angular client that is deployed in a separate front-end service and calls another back-end service which returns appropriate JSON data back to the angular client. As part of this process, the backend (built off python) uses the App Engine Python NDB Standard library to access the datastore and process the data before returning it back to the client.

Currently, I construct my REST endpoints such that it is <resource>/<id>/<resource>/<id> etc. etc. where <id> represents the 12-digit unique ID of the entity so that the respective backend endpoint can identify the appropriate resource and return it back to the client.

The issue I am facing now is that many of the datastore models have ancestor relationships associated with them and it turns out that if an entity is a part of a different ancestor hierarchy, it can generate the same 12-digit ID as another entity with a different hierarchy. Though the chance of this is low, I've encountered it with some stress tests in the development environment, which then causes collisions as the datastore is just returning the first entity with that ID even though it may not be the one required by the call.

Therefore, to resolve this, I decided to pass urlsafe keys between client and backend services as keys are always unique, regardless of ancestor relationships because ancestors are built into the keys that are generated. This leads to the question above.

[Kenworth (Google Cloud Platform)]

Hi Jason. I can see that you also posted this on StackOverflow which we also monitor, and are already getting the right responses from the community. I encourage users having similar issues to continue discussions on that thread.