Appscale instance gets killed and restarts almost every day

[Sampath Kuve]

Hi,

In our customer Appscale deployment, we see that the appscale instance gets killed and then restarts 1 or 2 times almost every day. 

Each time this happens, we see the following message in the log:

Executing [/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java, -Dsocket.permit_connect=true, -Xmx150m, -Djava.security.egd=file:/dev/./urandom, -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2, -javaagent:/root/appscale/AppServer_Java/appengine-java-sdk-repacked/lib/agent/appengine-agent.jar, -Xbootclasspath/p:/root/appscale/AppServer_Java/appengine-java-sdk-repacked/lib/override/appengine-dev-jdk-overrides.jar, -classpath, /root/appscale/AppServer_Java/appengine-java-sdk-repacked/bin/../lib/appengine-tools-api.jar, com.google.appengine.tools.development.DevAppServerMain, --property=kickstart.user.dir=/, --port=20001, --disable_update_check, --address=192.168.0.110, --datastore_path=192.168.0.110, --login_server=192.168.0.110, --appscale_version=1, --APP_NAME=localapp, --NGINX_ADDRESS=192.168.0.110, --TQ_PROXY=192.168.0.110, --pidfile=/var/run/appscale/app___localapp_default_v1_1569307654232-20001.pid, --external_api_port=19998, --api_using_python_stub=app_identity_service, /var/apps/localapp_default_v1_1569307654232/app/war/appscale-1.0-SNAPSHOT]

Nov 12, 2019 5:38:15 PM com.google.appengine.tools.development.agent.AppEngineDevAgent premain

SEVERE: Unable to load the App Engine dev agent. Security restrictions will not be completely emulated.

java.lang.RuntimeException: Unexpected exception during cast.

        at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:385)

        at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:252)

        at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:263)

        at com.google.appengine.tools.development.agent.AppEngineDevAgent.premain(AppEngineDevAgent.java:61)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)

        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

Caused by: java.lang.IllegalAccessException: Class com.google.apphosting.utils.clearcast.ClearCast$CasterImpl can not access a member of class com.google.appengine.tools.development.agent.$Proxy0 with modifiers "public"

        at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)

        at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)

        at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:413)

        at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:383)

        ... 9 more 

Such abrupt kills sometimes causes functionality issues in our application(in addition to customer being logged out abruptly). I appreciate if anybody can help us with how we can avoid this from happening.

Regards,

Sampath K