Hi Ben,
I am not familiar with OpenKM. But from the official Download page, I see that OpenKM could be installed in Linux / Windows environments as a jar file, which makes it possible in App Engine environments since Google App Engine is based on container/Docker and Java is also supported.
I recommend you to explore the App Engine Documentation for more information. Feel free to reply if you have any further questions.
Hope it helps!