high latency in urlfetch when calling an external API with google app engine

[ssmcooki...@gmail.com]

We have a Java application that make use of urlfetch calling an external api. We are having issues with the request time. When we do a request in a local environment we have a mean time of 300ms, however in production we have times of 1s or 1.4s.

This is the code we are using:

        Calendar start = Calendar.getInstance();

    try {

          URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();

          URL url = new URL( String.format( API_URL, productsIds ) );
          Future<HTTPResponse> future = fetcher.fetchAsync(url);


          HTTPResponse responseHttp = future.get();
          byte[] content = responseHttp.getContent();

          response = new String(content);

          System.out.println("request time :" + (Calendar.getInstance().getTimeInMillis() - start.getTimeInMillis() ) );


    } catch ( Exception e ) {
        LOGGER.log( Level.SEVERE, "Error calling api : " + e.getMessage() );
    }

Is Google doing any kind of validation that produces this time increase?

Is there a way to reduce the request time?

Thank you very much

[Nick (Cloud Platform Support)]

Hey ssmcookiemonster,

When debugging latency of a request, it's best to only surround the request action itself with timing code, so you should profile the time to execute the following two lines only:

Future<HTTPResponse> future = fetcher.fetchAsync(url);
HTTPResponse responseHttp = future.get();

This will minimize the impact of memory-management, object-creation, string-buffering, etc., and tend to show a more accurate measure of network latency. 

Another way to get insight into the problem is to ping / traceroute / mtr the target of your request from various locations. You might want to run these diagnostic tools from:

* your local development box
* a compute-engine instance in the same region as your app (US or EU)

* other locations around the world

This will allow you get a handle on how traffic will be routed to the endpoint you're attempting to reach, and the relative latencies from different locations.

You may also want to run Url Fetch requests to locations that are closer to your app and/or development box and see how latency is affected.

Finally, you can install AppStats (Java | Python) on your app to see the statistics around UrlFetch calls your app makes.

I hope this information has provided a helpful guide to debugging latency with Url Fetch. 

[Cookie Monster Samsamia]

Dear Nick,

Thank you for your response. We measured the latency as you suggested and the time is almost the same. The instance is in Europe and we are doing tests from Madrid and from London. 

Here is the problem we see. When we call to the API from a normal computer it takes around 300ms, however when we call it from Google's compute engine the trace is 3 or 4 times that. We don't understand why. 

Is Google doing some internal operation that increases the request time? are we doing something wrong?

Thanks

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/ryYsvyEily4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/3447916e-079b-4186-be6a-7a84fa99f1ea%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nick (Cloud Platform Support)]

Some further advice I can give you:

If you check your App Engine logs, you can find that the two fields CPU milliseconds and Wall-clock milliseconds represent different metrics.

CPU milliseconds is a measure of how long your instance itself spent calculating the response, and Wall-clock milliseconds is the total amount of wall-clock time spent by App Engine serving the request. 

Keeping in mind that CPU milliseconds are measured against a baseline of 1.2 GHz intel x86 CPU (so if your instance is equipped with a processor that's a multiple of this, perform the appropriate scaling to get the true CPU milliseconds), you can find the difference between wall-clock time and CPU time to be the amount of time your instance was waiting on network requests to return (assuming you use synchronous HTTP fetches, which it appears you do since you immediately call .get(), which is a blocking call, on the HTTP Future).

However, your timing code itself is also sufficient for this purpose, generally. It would be interesting to see if there's any difference between these two measures, since it could expose that there is other network latency other than the HTTP request you profiled directly with timing code, which could be affecting overall request latency.

Overall, the infrastructure responsible for routing Url Fetch requests from your app is very, very different from the devserver SDK implementation of Url Fetch, which is basically just using your computer to send the request. Some discrepancies in timing are to be expected since Url Fetch in production is a much more robust system. However, if you're interested in further analyzing the timing further, realize that where your app is located, where the endpoint you're trying to reach is located, where your dev-box is located, can all influence the various timings. 

You say that you've run a test from a Compute Engine instance in the same region as your application (Europe). If you didn't already, I'd suggest running MTR to see the exact path of your connection and the latency of the various hops. If the request takes 300ms to complete from your own computer, you should also run MTR to determine how much of that is network latency vs. processing on the actual endpoint you connect to. The latency on requests from the Compute Engine instance seems odd, and I'd be very interested to see the MTR output from that machine to the host you want to reach.

So, these are some of my further thoughts on the analysis of network latency with regard to Url Fetch, Compute Engine, and global IP routing. Let me know if you have any further questions!

To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.