502 bad gateway - flexible environment n1-standard-1

[Deepak Singh]

Dear Team,

We just upgraded to flexible environment with n1-standard-1 and started getting 502 / 503 most frequently.

App is configured with Cloudflare so we raised an incident with cloudflare and here is the result that they provided

https://support.cloudflare.com/hc/en-us/requests/904469?flash_digest=d8eeedee4ea25b0d37f24b01ca4c0c89fc29b4a8

Origin server is responding to 502 /503 for around 70% requests.

Pls let us know. 

--

Tech

[Nicholas (Google Cloud Support)]

Thanks for posting your issue here. Unfortunately, the link you provided requires a login so we cannot access it.

However, given that flexible environment uses Compute Engine instances and the time frame of your post, this outage may very well be relevant to this issue. Can you confirm the time frame during which your application was serving 502 and 503 errors? Does it align with the Compute Engine outage between 2016-04-11 18:25 PST and 2016-04-11 19:27 PST?

[Deepak Singh]

The issue is still continued and does not belong to a particular time rather than for a specific URL and it is always 502 / 503 for that url.

The url does a simple work - 

Gets some data by using URLFetch. Saves it to session. Converts it to JSON and sends the json as response.

Url is publically accessible. If you want, we will send you that url seperately as it requires our app id.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/fc4b8a02-e1e2-40fe-8a56-12072afffdff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Deepak Singh

[Nicholas (Google Cloud Support)]

After much private discussion, it seems still uncertain if this is an issue with your code or with the platform. Please file a new issue on the App Engine public issue tracker so that we may investigate further. That is a better forum for this sort of issue. In addition, please include a reference to this post in said issue.

Thanks,

Nicholas

On Tuesday, April 12, 2016 at 12:25:54 AM UTC-4, Deepak Singh wrote:

[Harris Robin Kalash]

Hi,

I am getting the same issue. However I did notice that it is specifically with a POST endpoint that returns JSON. I don't have this problem with GET. 

Thanks,

Harris

On Tuesday, April 12, 2016 at 12:25:54 AM UTC-4, Deepak Singh wrote:

[Nikolaos Volakis]

Hi all,

Actually it is happening to me as well . A post request/returning json that was working fine in the non-flexible gae is giving me 502 bad gateway . Any progress on this?

Thank you in advance
Nikos

[Stefano Ciccarelli]

As already stated on another thread, we have the same issue every time an instance starts or stops.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/94d136ce-73c5-462b-a705-bb3c89061f00%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Stefano Ciccarelli 
GAE Application Division 
/ Director 
stefano.c...@mmbsoftware.it 

M.M.B. s.r.l. 
via Granarolo, 177/7 - 48018 Faenza (RA) - Italy 
tel. +39.0546.637711 - fax +39.0546.46077 
www.mmbsoftware.it - in...@mmbsoftware.it

Le informazioni contenute in questa comunicazione sono riservate e destinate esclusivamente alla/e persona/e o all'ente sopra indicati. E' vietato ai soggetti diversi dai destinatari qualsiasi uso, copia, diffusione di quanto in esso contenuto sia ai sensi dell'art. 616 c.p., sia ai sensi del DL n. 196/03. Se questa comunicazione Vi e' pervenuta per errore, Vi preghiamo di rispondere a questa e-mail e successivamente cancellarla dal Vostro sistema.

[Nicholas (Google Cloud Support)]

There are many differences between the standard and flexible runtimes.  Some of these differences require that changes be made to the underlying code and may otherwise generate errors.  As mentioned in a previous post, if you believe this to be an issue with the platform, please file a new issue on the App Engine public issue tracker.  Please be sure to include the following:

• Output of running tracert to your application.  The 502/503 error suggests this might be an issue with the nginx proxy before your application instance.  tracert may reveal this.
  
• Handler entries in the app.yaml or appengine-web.xml
  
• Dispatch rules in dispatch.yaml or dispatch.xml
  
• Any log entries for HTTP requests to the URLs returning 502/503
  
• Code sample for the error-producing handlers so we may deploy and test this for ourselves
  

With the above information, we should be in a better position to determine if this in fact a platform issue.  If not, we may be able to determine the cause of said errors in user code as well.

On Tuesday, April 12, 2016 at 12:25:54 AM UTC-4, Deepak Singh wrote:

[Suresh V]

Any update on this? My instance, https://vplan-147418.appspot.com is doing this since last night.

Any tips to troubleshoot? This also uses the flexible environment,

[Jeff Payne]

I had this same problem using a memory intensive Flask based flex env app.  I finally investigated this locally at the system level and realized that a function, which I had not yet thought to optimize, was loading a huge file (500+ Gb) into memory.  I couldn't find anything in the logs to support this (I probably wasn't looking in the right logs), but I figured that the process/thread handling the request was choking behind the scenes.

My solution was:

1) Move the code that loaded the file into memory outside of the function so it was only loaded once, as the resulting object is thread safe.

2) Increase the memory of the VM used by the flex env. app using the resource settings explained here: https://cloud.google.com/appengine/docs/flexible/python/configuring-your-app-with-app-yaml#resource-settings

Hope that helps!

[Nicholas (Google Cloud Support)]

For advice on troubleshooting, I would suggest posting to a new thread as the original underlying issue for this thread may not be related to issues you may be having.  In general, 500s and 502s are very generic error messages.  When using the flexible environment, your application is built into docker image using a preset Dockerfile and then deployed to App Engine with an added docker entrypoint so the application container can be run.  If using a custom runtime, the image is built using your own custom Dockerfile.  In front of your resulting application container sits a separate container with an nginx process for serving static content and observing the handlers specified in your app.yaml.

• When posting to a new thread, I recommend the following:
  
• Where are you testing?  Is this your App Engine application in production or in development?
  
• Are you using the flexible environment or standard environment?
  
• If using the flexible environment, are you using an App Engine-provided runtime or a custom runtime? (This and the previous question help us identify where the issue may be originating and how you can investigate the source for yourself)
  
• Is the error served from every handler or only a specific subset?
  
• If you only get errors with a subset of handlers, are those static assets or handlers served by your application?
  
• Is your application successful in responding to requests for a given handler?
  

The above are good starting points for investigating 500s served from your application.  When posting new threads about such errors, please try to include as much of this information as possible so that others in the community can have a better defined picture of your the architecture at play and know what you've already looked at.

Should you decide to post a new thread, feel free to include a link to it here so that others in the community can follow through.

[Kishuk Goyal]

Accidentally I removed $JETTY_ARGS from JAVA_OPTS (by specifying JAVA_OPTS: "-Djava.util.logging.config.file=WEB-INF/logging.properties" in app.yaml env_variables), And my pipeline start working just fine.

So after removing $JETTY_ARGS my final command line is looking like this:

java -Djava.util.logging.config.file=WEB-INF/logging.properties -Djetty.base=/var/lib/jetty -jar /opt/jetty-home/start.jar

Which was: 

java -showversion -Djava.io.tmpdir=/tmp/jetty -agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/var/log/app_engine,--logtostderr=false,--cdbg_extra_class_path=/var/lib/jetty/webapps/root/WEB-INF/classes:/var/lib/jetty/webapps/root/WEB-INF/lib -Xms491M -Xmx491M -XX:+UseG1GC -XX:+ParallelRefProcEn

abled -XX:+PrintCommandLineFlags -Djetty.base=/var/lib/jetty -jar /opt/jetty-home/start.jar 

Can somebody please explain what might be wrong with $JETTY_ARGS ?