How to alleviate 200gb/day - 20$+ daily outgoing bandwidth image serving costs

[Kaan Soral]

Hi everyone

I recently noticed a bump in costs after improving the static image quality of my app, this bumped the daily bandwidth costs from ~7$'s to 20$+'s

Overall there are ~200 statically served images, I tried reducing the quality of png's and turning them into jpg's, yet it degrades the image quality, at the same level of quality, the size of the jpg's are identical to the png's, so, as it is, paying the extra costs seem to be one option

I'm also wondering whether there is an easier solution out there, maybe using pagespeed as a CDN/cache with an easy config change, but as far as I've researched, it didn't seem feasible/simple

Setting up cloudflare for a subset of the traffic is another option, but I'm not sure whether cloudflare can reliably serve all those images

I checked some other CDN's / cloud solutions, they all seemed close to AppEngine's pricing, CloudFront for example: http://aws.amazon.com/cloudfront/pricing/ ~30% lower

As it is, App Engine serves images well, yet the bandwidth costs are higher than the instance hour costs, so I would like to reduce the bandwidth cost if there is a simple solution out there that I'm missing

(My default_expiration is "1h", I've also verified the "expires" header, yet probably the consumer of the images doesn't cache them, I really wish the issue was caching, but it doesn't seem the issue can be solved by modifying the http caching)

[Alexander Trakhimenok]

Use AWS. 30% is a quite a difference. Also you would have more control over headers and images can be closer to end customers - would not help with costs but it's nice.

Increasing cache time if you can. We set caching for over a month. Theoretically they can be cached by proxies as well (but probably not much gain here unless majority of you customer is from the same corporate network).

Make sure the consumer of the image caches it if possible. Test in all major browsers.

[Jesse Scherer (Google Cloud Support)]

Some thoughts:

You could serve images from Cloud Datastore instead of from App Engine. You mention that bandwidth costs dominate instance time, but it should help a bit. If my math is correct, you're serving at least 250GB per day, so provided you're serving mostly to the US, the you'd get some volume discount now and then a 33% discount once you top 10TB per month. If nothing else this would help you scale better.

You mentioned the images are served statically. If that's the case, can you crank up the expiration? What are you seeing which makes you suspect that your clients aren't cacheing properly? If your images periodically change can you just generate new filenames in conjunction with effectively infinite expiration time?

[Qian Qiao]

Maybe consider dumping the images into Cloud Storage and serve from there? If the images are static, using Cloud Storage could save you money on multiple fronts:

1. Cheaper bandwidth

2. Cheaper storage cost

-- Joe

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2d625c60-6b09-4edf-92e5-6951fd7cf0ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kaan Soral]

Thanks a lot for all your input

I've cranked up the default_expiration to 60d, yet I doubt it will have any effect, unless there is a cut-off structure and 1h is below that cut-off, the majority of these images are consumed by an API which is pretty much a black-box

My monthly usage is around 6TB's, so cloud storage isn't worth the effort, app engine is 0.12/gb, while cloud storage becomes 0.11/gb as far as I see

If all else fails, I will probably move the serving to AWS, as the 33% saves stack up to $2k+ in a year, or just degrade the images and reduce the user satisfaction, but the AWS option is obviously better

-----

On the other hand, Cloudflare might reduce this cost to 0, but I'm not sure about the reliability, they don't advertise limits for their free CDN, there were also some App Engine incompatibilities in the past, their system might also block the API whose access is crucial

[Jason Collins]

Hi Kaan,

Out of curiosity, have you tried resampling the PNG (while leaving them as PNG) with something like https://imageoptim.com/? I've seen huge amounts of size come out of images, without any noticeable degredation to the image quality.

j

[Jeff Schnitzer]

Try CF. I screamed bloody murder years ago when CF-inbound traffic tripped Google's threat system and all CF users were borked, but that was a long time ago. CF got whitelisted by Google and I haven't heard of any similar problems since. I would be more scared of other CDNs that might not have gone through this process, might not be on the whitelist... and therefore are ticking time bombs.

That said, even though I've been using CF for years, I don't really know how effective their cache is.

Jeff

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/20043daf-f915-42ed-bc69-52be58cb0340%40googlegroups.com.

[Kaan Soral]

The images are pre-processed with https://github.com/aheckmann/gm combining different components, As a short term solid solution, I managed to get the same quality by pre-resizing important image components and using JPG's with 90% quality

As a result the images look very similar, yet the size reduced 60%+ compared to full quality png's, 80% reduces the size a lot more, yet the quality is distinctively reduced (the main gain is from reducing the resolution to a minimally acceptable state)

I'm 90% satisfied with this solution, I will look imageoptim/other sampling/compression methods, it's a great idea, thanks

As it is, the daily usage dropped well below 100gb's

I'm meaning to try CF, but it's a lot like a blackbox, if they block the sources that requests the images, the damage would be more than the gain, I guess configuring the policy settings to be very permissive could prevent that from happening, I also couldn't find the time yet to figure out how to configure CF to only proxy the images, I will likely use the existing custom domain, which isn't used anymore, so the risk is low on that front

[Kaan Soral]

After the traffic increased and costs jumped to $80's, I ended up using an existing CloudFlare integration I had, just for the images, saved 90gb's just in the first hour + didn't notice any issues at all

So for now, cloudflare saves the day, don't know why I didn't think of using the existing integration before ... :)

[Rafael]

We had the same problem very early on and no money in the bank.

The cost per GB on appengine image serving is too high. Maybe it tries to include the image resizing service fee or be a balance for many apps that don't have social traffic. Social networks are expensive.

We migrated to maxcdn which I thought wasn't top tier. Then we migrated to akamai and their admin tools were super outdated, for example no support for spdy or http2. So, we came back to maxcdn and are happy with the decision.

We paid $3k for 150TB of maxcdn traffic. You can see that's probably 10x cheaper than blobstore serving.

On May 15, 2015 4:34 AM, "Kaan Soral" <kaan...@gmail.com> wrote:

After the traffic increased and costs jumped to $80's, I ended up using an existing CloudFlare integration I had, just for the images, saved 90gb's just in the first hour + didn't notice any issues at all

So for now, cloudflare saves the day, don't know why I didn't think of using the existing integration before ... :)

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ae373a09-987e-40fa-8180-a7828002f099%40googlegroups.com.