Images served very slowly this week.

[Jay Kyburz]

Hey all,

It might be my imagination but I think static resources like images are loading very slowly for my app this week.  It doesn't seem like a latency issue so much as a bandwidth issue. Is that completely crazy?

This image for example

https://blight.ironhelmet.com/images/blight_menu_bg.jpg

It's 382K but it can take up to 8 seconds to download. 

I don't think the problem is on my side as I've had users complaining as well.

I'm not sure where to begin look at where my app might have a problem.

The app-id is blightsrage

[Nicholas (Google Cloud Support)]

Thanks for posting your concerns here. As network throughput diagnostics can have many variables to measure, please provide as much of the following information as possible:

• Where are these images hosted?
• Are they served by App Engine as a static resource?
• Are they served from Google Cloud Storage?
• Time frame.
• When did you notice this slowness?
• Is it still going on?
• If not ongoing, when did this end?
• What is the scope of affected resources?
• Is it all images?
• Are other resources also transferred slowly?
• Is this slowness present for all users?
• How consistently are you getting this slowness? Every time the resource is downloaded?

If served from a GCS bucket, you can use gsutil perfdiag to run some performance diagnostics. If served by App Engine as a static resource, testing essentially requires consistent results. Resources on App Engine may be served from different locations frequently depending on scalability. Could you test a ~1 MB-sized resource multiple times and get the same results? This may not be conclusive but might at least narrow down the scope of investigation.

[Nicholas (Google Cloud Support)]

I forgot to add that I have not been able to reliably reproduce the slowness you describe as either a static App Engine resource or publicly shared Google Cloud Storage object. I also am not aware of any currently known outage or network congestion for either App Engine or Google Cloud Storage.

On Wednesday, March 2, 2016 at 12:00:10 AM UTC-5, Jay Kyburz wrote:

[Sam]

took about a half second for me

[Jay Kyburz]

Thanks very much Nicholas,

Can I first say how great it is to have Google Cloud Support here helping us out! I know it can be a tough job dealing with people problems all day. I really appreciate it and I'm sure there are many others that do as well! 

1. The images are all hosted on App Engine and are is a static directory specified in the app.yaml.

I also server stylesheets, fonts, sounds, and scripts the same way and they all seem to be slow downloading.

2. I have noticed some slowness for a few weeks, but last Sunday the game was virtually unplayable because of the slow images. Since then, it seems to be intermittent.  The 400k Images above is now loaded on my system in less than 4 seconds. (still seems kind of slow to me - but better than 8 seconds of yesterday) - To get these times I'm using Chrome Devtools Timeline.

I might see if I can write a script to monitor how fast it takes and get some actual data rather than relying on anecdotal memories.

3. Yes it appears to be all images and all static resources. Yes I believe it's all users affected.

4. when you say "testing requires consistent results", what might the problem be if we did have consistent results? 

[Jay Kyburz]

Did you try the image I linked? How fast did you receive it?

[Barry Hunter]

http://www.webpagetest.org/ is great for gathering some relatively impartical timing data, has the benefit of keeping a detailed record at a permalink

I just run a test... 

http://www.webpagetest.org/result/160302_ME_a8240b5e57f8c08455b8b96b5740e5d8/

Not that familar with your site, to help interpret it. BUt there does seem to be a bunch of images (26-31) with relative long Time to first byte. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f2befadd-82e0-4ba1-8515-85888fdfa442%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Christian F. Howes]

Any chance that you changed the caching rules in your app.yaml file for your static files?  i suspect that can have an effect on load times.

On Tuesday, March 1, 2016 at 9:00:10 PM UTC-8, Jay Kyburz wrote:

[Jay Kyburz]

Thats a great resource thanks Barry. There might be some initial slowness because it's a fairly large single page app so a bunch of scripts need to load before it starts building ui.

Anyhow, to further the investigation I've written a script that downloads a static image from my site over and over every 10s.

window.setInterval(function() {

    var timeStart = Date.now();

    var img = new Image();

    img.onload = function() {

        var timeEnd = Date.now();

        console.log("Duration", (timeEnd-timeStart)/1000);

        var output = document.getElementById("output");

        output.innerHTML += "Duration " + (timeEnd-timeStart)/1000 + "<br>";

    };

    // note: adding a variable to the url forces the cache not to be used.

    img.src = "https://blight.ironhelmet.com/images/blight_menu_bg.jpg?" + timeStart ;

}, 10*1000);

You can run it here:

https://blight.ironhelmet.com/help/test

I have asked some of my users to run it as well to see if they see similar number to me. 

What I found was the 380k image can take between 2 and 10 seconds to download. 

It seems to be a lot more random that I would expect!

If I look at my logs on the app engine console it reports that every request was served in under 1ms so I don't there there is a problem there. 

Here is the raw times for each download 

Duration 3.744

Duration 5.161

Duration 5.436

Duration 7.482

Duration 9.698

Duration 7.768

Duration 7.702

Duration 9.913

Duration 10.746

Duration 10.119

Duration 5.755

Duration 4.574

Duration 3.134

Duration 2.926

Duration 2.717

Duration 4.633

Duration 6.743

Duration 5.758

Duration 4.313

Duration 5.568

Duration 6.018

Duration 1.981

Duration 3.031

Duration 5.413

Duration 3.09

Duration 2.779

Duration 6.429

Duration 5.047

Duration 5.931

Duration 4.737

Duration 4.589

Duration 2.926

Duration 2.063

Duration 4.08

Duration 4.428

Duration 4.138

Duration 3.162

Duration 1.724

Duration 1.794

Duration 2.518

Duration 3.166

Duration 4.374

Duration 5.447

Duration 8.417

Duration 8.957

Duration 5.172

Duration 5.029

Duration 4.768

Duration 2.66

[Jay Kyburz]

Hi Christian, I have not changed the rules since setting up the project more than a year ago. They are 7d. Thanks for the tip though.

[Jay Kyburz]

>Not that familiar with your site, to help interpret it. BUt there does seem to be a bunch of images (26-31) with relative long Time to first byte. 

Just look at the data and I thought you mean time till its requested but now I see time to first byte is after the request. I'm not sure what that means for my App Engine app though. Might I have something configured incorrectly?

[Jay Kyburz]

Quick update on this issue. 

A number of my US / Canadian players have reported the images takes less than 300ms to download the image. 

Another Australian is reporting > 2 seconds like us. (We are also in Aus)

[Nicholas (Google Cloud Support)]

I've tested average bandwidth requesting your application's main page from various locations and observing image download times/size:

• London, EU - 220 to 603 KB per second
  
• California, NA - 291 to 503 KB per second
  
• Hong Kong, ASIA - 118 to 401 KB per second
  
• Sydney, AUS - 144 to 366 KB per second
  

The sample was 5 attempts from each location considering all images over 100KB. From the stats above, it's entirely conceivable that 500+KB images take several seconds to acquire. In addition, these durations do not include waiting time (Time To First Byte) which can add up to times experienced by you and your users.

These appear to be expected serving times. I've also tested a few alternatives myself including serving from a Google Cloud Storage bucket located in North America and serving the same content from a bucket in Asia. The results did not suggest that either of those bucket regions would serve content considerably faster than App Engine with static resources. The TTFBs were almost identical during most tests and the download times ever so slightly longer.

My only suggestions to potentially improve the user experience would be as follows:

• Test serving from a storage bucket to see if this is effective for your use case in the event my tests are not representative of your experience
  
• Preload images on the client side in Javascript to be done concurrently with the user's interactions
  
• Consider different image or content compression for serving said images
  
• Consider combing image assets to a single image to effectively reduce the number of HTTP requests and as a result, reducing how often you incur the TTFB cost. This guide though a little old can be quite helpful for such things.
  

I hope this has been helpful, if not at least informative.

On Wednesday, March 2, 2016 at 12:00:10 AM UTC-5, Jay Kyburz wrote: