Static files in newly deployed version not available in the apps domain. (Major problem)

[Tim Hoffman]

Hi

We have been trying to deploy a major revision of one of our apps.

Under the specific version 2-0-0.latest... all the new css/js and
static images are available.

When we make the new version default, the css, js and static files are
accessible via <appid>.appspot.com

However when accessing the same site via the google apps domain
mapping we are still
getting the old versions css, js and static images.

I have tried removing the apps domain mapping and re-adding it with no
affect.

We have had to revert to the earlier version as all access to the site
is via the apps domain and
its not working with the latest version missing the new css/js.

has anyone got any ideas on how we can address this.

I know absolutely that the problem is not a browser cache, as I have
been checking the css files via wget.

Thanks for any help

regards

Tim

[François Masurel]

I've the same problem here.

I just added an extra parameter at the end of my urls (ex : "?v=1") to
be able to load them correctly.

François

[J]

Thanks, Tim, for letting me know I'm not going insane.

To the Google guys, this problem seems to have started yesterday
afternoon (1:30-ish PM, EDT) in case it helps you track down the
cause.

[Ikai L (Google)]

Thanks for bringing this up, Tim. Anyone else seeing this problem? If so, please post details. Are you guys setting any kind of cache headers?

I'm going to try to reproduce these issues, so any information will be helpful.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan 
Developer Programs Engineer, Google App Engine

Blog: http://googleappengine.blogspot.com 

Twitter: http://twitter.com/app_engine

Reddit: http://www.reddit.com/r/appengine

[J]

To reproduce the problem, go to http://qa.connectscholar.com/stylesheets/caSkin.css
and also to http://charityaxis-qa.appspot.com/stylesheets/caSkin.css.
Both URLs point to the same file but one returns the old content and
appspot.com returns the new content.

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

[Rafael Sierra]

On Wed, Jun 2, 2010 at 3:37 PM, J <j.s...@earlystageit.com> wrote:
> To reproduce the problem, go to http://qa.connectscholar.com/stylesheets/caSkin.css
> and also to http://charityaxis-qa.appspot.com/stylesheets/caSkin.css.
> Both URLs point to the same file but one returns the old content and
> appspot.com returns the new content.

Same file here:

Hidan:~ sdm$ curl http://charityaxis-qa.appspot.com/stylesheets/caSkin.css > 1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10941 0 10941 0 0 6146 0 --:--:-- 0:00:01 --:--:-- 31918
Hidan:~ sdm$ curl http://qa.connectscholar.com/stylesheets/caSkin.css > 2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10941 0 10941 0 0 2777 0 --:--:-- 0:00:03 --:--:-- 3854
Hidan:~ sdm$ md5 1
MD5 (1) = 8f5ef511be1a03fd722223337c334933
Hidan:~ sdm$ md5 2
MD5 (2) = 8f5ef511be1a03fd722223337c334933

> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Ikai L (Google)]

I wonder if there is some layer of the infrastructure that is performing caching without you guys having opted-in, hence the reason why a cache-buster like ?v=something works. Can you guys confirm? Also - are you guys setting any headers? Which headers get returned?

I've personally always used cache busters for the simple reason that sometimes users using web-accelerators or with aggressive ISP or caching settings tend to key off the URL. Can you guys use this workaround in the meantime?

[J]

That's interesting, Rafael. I wonder why it serves correctly for you.

They are still different from my PoV.

The bad side headers:
Etag "74EQOA"
Date Tue, 01 Jun 2010 18:12:57 GMT
Expires Sat, 05 Jun 2010 02:12:57 GMT
Content-Type text/css
Content-Encoding gzip
Server Google Frontend
Content-Length 2820
Cache-Control public, max-age=288000
Age 90134
X-XSS-Protection 0

The good side headers:
Etag "7xGL5w"
Date Wed, 02 Jun 2010 19:13:00 GMT
Expires Wed, 02 Jun 2010 19:12:04 GMT
Content-Type text/css
Content-Encoding gzip
Server Google Frontend
Content-Length 2834
Age 41
Cache-Control public, max-age=600

Short of using a cache-buster, is there a setting I can use on GAE or
in my app.yaml file?

On Jun 2, 3:01 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> I wonder if there is some layer of the infrastructure that is performing
> caching without you guys having opted-in, hence the reason why a
> cache-buster like ?v=something works. Can you guys confirm? Also - are you
> guys setting any headers? Which headers get returned?
>
> I've personally always used cache busters for the simple reason that
> sometimes users using web-accelerators or with aggressive ISP or caching
> settings tend to key off the URL. Can you guys use this workaround in the
> meantime?
>

> On Wed, Jun 2, 2010 at 11:57 AM, Rafael Sierra <rafaeljs...@gmail.com>wrote:

>
>
>
>
>
> > On Wed, Jun 2, 2010 at 3:37 PM, J <j.si...@earlystageit.com> wrote:
> > > To reproduce the problem, go to
> >http://qa.connectscholar.com/stylesheets/caSkin.css

> > > and also tohttp://charityaxis-qa.appspot.com/stylesheets/caSkin.css.

> > > Both URLs point to the same file but one returns the old content and
> > > appspot.com returns the new content.
>
> > Same file here:
>

> > Hidan:~ sdm$ curlhttp://charityaxis-qa.appspot.com/stylesheets/caSkin.css> 1

> >  % Total    % Received % Xferd  Average Speed   Time    Time     Time
> >  Current
> >                                 Dload  Upload   Total   Spent    Left
> >  Speed
> > 100 10941    0 10941    0     0   6146      0 --:--:--  0:00:01 --:--:--
> > 31918

> > Hidan:~ sdm$ curlhttp://qa.connectscholar.com/stylesheets/caSkin.css> 2

> > >> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

[Ikai L (Google)]

Okay, looks like the Google Front-End is kicking in to cache your stuff

Server  Google Frontend
Content-Length  2834
Age     41
Cache-Control   public, max-age=600

Are you guys setting this header anywhere? Unfortunately, there's no way to invalidate items in the frontend cache, so you'll have to use a cache buster or wait for the TTL to expire.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ikai L (Google)]

Okay, looks like the Google Front-End is kicking in to cache your stuff

Server  Google Frontend
Content-Length  2834
Age     41
Cache-Control   public, max-age=600

Are you guys setting this header anywhere? Unfortunately, there's no way to invalidate items in the frontend cache, so you'll have to use a cache buster or wait for the TTL to expire.

On Wed, Jun 2, 2010 at 12:22 PM, J <j.s...@earlystageit.com> wrote:

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[J]

Thanks, Ikai, for your help. It is much appreciated.

We'll use a cache buster for now.

> > > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

[Ikai L (Google)]

In the meantime, I'll investigate whether these headers are being implicitly or incorrectly set.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ikai L (Google)]

I can't reproduce this. Here's my YAML file:

application: ikailan-com

version: 1 

runtime: python

api_version: 1

default_expiration: "1d"

handlers:

- url: /

  script: main.py

- url: /assets

  static_dir: assets

Can you guys post your app.yaml?

[J]

Sorry for the delay, Ikai. I had to make a pervasive change in our
code base to put in cache-busting snippet.

Our app.yaml is below.

You will note two commented-out line for css. The 4d expiration was
removed yesterday in one of my attempts to get things to work. By that
time, the data was already in the cache, I guess!

---------------x-------------

application: charityaxis-qa

version: 1
runtime: python
api_version: 1

handlers:
- url: /stylesheets
static_dir: stylesheets
# mime_type: text/css
# expiration: "4d"

- url: /zipEditorCK/.*
script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /zipTreeTable/.*
script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /zipSyntaxHighlighter/.*
script: $PYTHON_LIB/google/appengine/ext/zipserve

- url: /scripts
static_dir: scripts
mime_type: text/javascript
expiration: "4h"

- url: /images
static_dir: images
expiration: "3d 8h"

- url: /favicon.ico
static_files: images/favicon.ico
upload: images/favicon.ico
mime_type: image/vnd.microsoft.icon

- url: /robots\.txt
script: caSEO.py

- url: /sitemap
script: caSEO.py

- url: /stats.*
script: appstats/ui.py

- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin

- url: /.*
script: ca.py

On Jun 2, 4:38 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> I can't reproduce this. Here's my YAML file:
>
> application: ikailan-com
> version: 1
> runtime: python
> api_version: 1
> default_expiration: "1d"
>
> handlers:
> - url: /
>   script: main.py
>
> - url: /assets
>   static_dir: assets
>
> Can you guys post your app.yaml?
>
> On Wed, Jun 2, 2010 at 1:04 PM, Ikai L (Google) <ika...@google.com> wrote:
>
>
>
> > In the meantime, I'll investigate whether these headers are being
> > implicitly or incorrectly set.
>

> ...
>
> read more »

[Tim Hoffman]

Hi Ikai

Here are bits of our app.yaml

application: psc-dev1
version: 1-0-2
runtime: python
api_version: 1

default_expiration: "4d 5h"

- url: /images
static_dir: images

expiration: "30d"

- url: /css
static_dir: css
expiration: "30d"

- url: /js
static_dir: js
expiration: "30d"

The main difference is we have explicit expiration set on the static
handlers.

Ikae one thing you didn't show in the your headers was the Etag. As
I understand
it, the Etag value is being set as a side affect of the deploying new
versions.
It seems the reverse proxy for app domain maps isn't honoring the Etag
for cache control.

This must be a fairly recent phenomenon as we have never experienced
this issue
with www.polytechnic.wa.edu.au before. (I can't show you it in action
at the moment as
we have serious site problems trying to run the new version).

We can't very well wait for the cache's to expire. (I think we will
reduce the time
somewhat, but even waiting a few days for the cache to expire won't
work.
The site is broken without the new js/css, so I can't leave the new
version active whilst
waiting for things to expire)

As to the notion of using a "cache-buster" that will mean effectively
changing
css/js paths in our templates and re-deploying the app. I suppose we
will have to
go with path at the moment. I am in Australia and I had gone to bed
(midnight)
before you guys cam online, and I need to get this new version
deployed. However
I don't believe this is a good solution. It effectively means the
developers (me and a few
other people) need to manage a changing version string on the end of
all css/js static urls
for each deployment, (if these change) each time we want to deploy/
update our code.

Thanks

Tim

On Jun 3, 4:38 am, "Ikai L (Google)" <ika...@google.com> wrote:
> I can't reproduce this. Here's my YAML file:
>
> application: ikailan-com
> version: 1
> runtime: python
> api_version: 1
> default_expiration: "1d"
>
> handlers:
> - url: /
>   script: main.py
>
> - url: /assets
>   static_dir: assets
>
> Can you guys post your app.yaml?
>
> On Wed, Jun 2, 2010 at 1:04 PM, Ikai L (Google) <ika...@google.com> wrote:
>
>
>
> > In the meantime, I'll investigate whether these headers are being
> > implicitly or incorrectly set.
>

> ...
>
> read more »

[Tim Hoffman]

Hi

I have raised an issue for this

http://code.google.com/p/googleappengine/issues/detail?id=3297

Rgds

T

[J]

Tim, what we ended up doing was a bit different. We have qa.abc.com
pointing to xyz-qa.appspot.com.

We did not change file names as was recommended -- that would be a
pain in the rear to manage. But we took advantage of the fact that xyz-
qa.appspot.com delivers content correctly. We changed references to /
scripts/pqr.js to http://xyz-qa/appspot.com/scripts/pqr.js.

It was faster to put in place and it buys us some time until we can
come up with something better.

Hope that helps.

[Tim Hoffman]

Hi

Not to bad an idea.

At the moment all of our templates use '/css/some_file.css'
rather than including hosts. So we don't need to make changes between
code bases
or have to lookup what the host should be, (in light of the issue we
may well have to change that strategy)

For the moment we just add ?<some value> to the end of the css/js
files that are changing, and will probably set up a script to be run
prior to appcfg, to increment
these values. All the css/js is referenced in just a couple of main
templates (zpt)

Rgds

T

On Jun 3, 10:17 pm, J <j.si...@earlystageit.com> wrote:
> Tim, what we ended up doing was a bit different. We have qa.abc.com
> pointing to xyz-qa.appspot.com.
>
> We did not change file names as was recommended -- that would be a
> pain in the rear to manage. But we took advantage of the fact that xyz-
> qa.appspot.com delivers content correctly. We changed references to /

> scripts/pqr.js tohttp://xyz-qa/appspot.com/scripts/pqr.js.

[Ikai L (Google)]

Tim, can you provide 2 URLs that are doing this?

I'm still trying to reproduce the problem. Could an ISP be caching these files?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ikai L (Google)]

Tim,

As a general rule, web developers should *always* tag assets with a version string. Besides our caching infrastructure (which I am still investigating), it's the single most reliable way to invalidate browser, web accelerator, ISP and proxy caches, which may or may not respect the headers you set. Headers aren't a guarantee anybody downstream will respect your invalidation strategy - headers are a way to, in the best case scenario, reduce HTTP calls to your assets server. This is why most web frameworks will append at least a timestamp or hash to asset tags. The simplest way is to do this:

"/js/somefile.js?v=123"

You won't have to rename the file - most caching mechanisms key on the unmodified URL string.

I know this might be painful in the short term, but trust me, this will save you a lot of headaches in the long term when users' computers start doing crazy things with cached assets. In the meantime, I'll try to figure out what's going on, but I haven't yet been able to reproduce the issue in my own applications no matter what I do. At the very least, I hope to provide an explanation of expected behavior.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Donovan]

Another technique for cache busting is URL rewriting within the path
to the resources, which has some benefits over the query string usage
in certain cases.

The Python AE runtime supports URL mapping for static resources which
can make this even less painful and allows long expire times - so for
example you could generate all static URLs like this:

/static/[app version from environment]/css/foo.css and have it map
to the static/your/foo.css withint your appengine upload This has an
advantage in the case of css because then all your resources
referenced from the CSS (as long as they're using relative URLs under
the same static directory) are also affected by the path versionion.
So a

background-image: url(../images/bar.jpeg) ;

declaration in the foo.css above would cause the browser to request
the resource with the injected app version /static/[app version]/
images/bar.jpeg. This would not happen with the query string cache
busting technique unless you willing to parse your css dynamically
(and then you're not using the static file appengine servers).

Using this technique you can feel free to set your static file caching
headers to large values (months to a year) to get a better client
experience without sacrificing the ability to update your app at will
(your app version id will change everytime you upload, which creates a
new URL and new set of cacheable objects to the browser)

Unfortunately, the Java AE runtime does not have a similar support for
static file URL rewriting - I've done a feature request here (PLEASE
STAR! :) ) http://code.google.com/p/googleappengine/issues/detail?id=2070

So, to get the same behavior and peformance benefit of hitting the AE
static file servers, you'd have to physically create a versioned
directory during the build process before uploading to appengine . You
could do the same thing for the python as well, but the URL mapping
feature is so easy to use I've never wanted to do so.

- Donovan

> > withwww.polytechnic.wa.edu.aubefore.  (I can't show you it in action

> ...
>
> read more »

[Ross M Karchner]

Partially off-topic---  if GFE serves a cached resource, do we get billed for the bandwidth?

[Ikai L (Google)]

You should still be billed for bandwidth. The advantage of using edge caching is that if the resource is being served from the datastore or dynamically generated via some other means, you will not incur CPU costs.

[Tim Hoffman]

Hi Ikae

We couldn't leave the production environment in place with the broken
urls.
So have already deployed a version with args on the end of all css/js
urls.

I don't believe it was an ISP issue. We tried accessing the site
from 5 different ISP's in Australia when we first discovered the
problem.

Maybe you behind googles infrastructure can't see the issue. More than
a few people have posted
links exhibiting the problem in the list (and they aren't in
Australia).

We have a full test instance in gae as well but we haven't been
mapping it do a domain to date.
As we have never experienced this issue before. (Site went live mid
last year www.polytechnic.wa.edu.au)

Rgds

T

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

[J]

A couple of closing-out notes from our vantage point in Massachusetts.

First, even though the cache for one of our files was not supposed to
expire until June 5, it was serving the correct content as of late
June 3. Whether it was an ISP issue or a GFE issue, we will never
know.

Second, we have deployed the technique recommended by Donovan and hope
to never see this problem again.

Finally, a bouquet of virtual flowers for Ikai. Thanks for jumping on
the problem and helping us put a solution in place.

[Ikai L (Google)]

Can you guys run a traceroute on your domains vs. the appspot domain?

E.g:

traceroute qa.connectscholar.com

traceroute charityaxis-qa.appspot.com

If you're on Windows, the equivalent command is "tracert".

I'm curious if there's an ISP or specific Google Frontend that is acting up.

[Ikai L (Google)]

This probably is still occurring. Just out of curiosity - is your ISP Comcast?

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Tim Hoffman]

Hi Ikai

So here is the route to the domain mapped app

traceroute to www.polytechnic.wa.edu.au (72.14.203.121), 30 hops max,
60 byte packets
1 dlink.dir451 (192.168.1.254) 1.390 ms 2.382 ms 9.842 ms
2 * * *
3 * * *
4 * * *
5 vlan511.o6ssc76fe.optus.net.au (59.154.14.125) 957.339 ms
958.177 ms *
6 * gi9-6.pstmadist02.aapt.net.au (202.10.13.102) 950.008 ms
951.868 ms
7 66.249.95.234 (66.249.95.234) 968.417 ms 329.715 ms 348.896 ms
8 209.85.249.52 (209.85.249.52) 447.684 ms
te0-3-1-0.pgroscore01.aapt.net.au (202.10.12.33) 358.795 ms 343.266
ms
9 209.85.250.90 (209.85.250.90) 300.117 ms 299.892 ms
gi0-0-0-1.mflincore01.aapt.net.au (202.10.10.84) 156.907 ms
10 209.85.250.103 (209.85.250.103) 315.736 ms 289.746 ms 300.058
ms
11 te2-2.sclardist02.aapt.net.au (202.10.12.4) 149.830 ms 149.851
ms 209.85.241.154 (209.85.241.154) 289.833 ms
12 te3-1.sclarbrdr01.aapt.net.au (202.10.14.5) 159.967 ms tx-in-
f121.1e100.net (72.14.203.121) 319.701 ms 300.186 ms
timh@chrome:~/qtrack$

Directly to the appspot app

timh@chrome:~/qtrack$ traceroute psc-prod1.appspot.com
traceroute to psc-prod1.appspot.com (66.102.11.141), 30 hops max, 60
byte packets
1 dlink.dir451 (192.168.1.254) 1.181 ms 2.085 ms 2.573 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * google.sd13.optus.net.au (119.225.36.2) 410.318 ms 409.141 ms
7 * 66.249.95.232 (66.249.95.232) 760.075 ms *
8 te0-3-1-0.pgroscore01.aapt.net.au (202.10.12.33) 636.765 ms
133.826 ms 158.857 ms
9 gi0-0-0-1.mflincore01.aapt.net.au (202.10.10.84) 199.897 ms
syd01s01-in-f141.1e100.net (66.102.11.141) 169.789 ms 159.778 ms


And to ghs.google.com looks the same as the first as I would expect.

Rgds

T

On Jun 5, 6:23 am, "Ikai L (Google)" <ika...@google.com> wrote:
> Can you guys run a traceroute on your domains vs. the appspot domain?
>
> E.g:
>
> traceroute qa.connectscholar.com
> traceroute charityaxis-qa.appspot.com
>
> If you're on Windows, the equivalent command is "tracert".
>
> I'm curious if there's an ISP or specific Google Frontend that is acting up.
>
> On Thu, Jun 3, 2010 at 2:02 PM, Ikai L (Google) <ika...@google.com> wrote:
>
>
>
> > You should still be billed for bandwidth. The advantage of using edge
> > caching is that if the resource is being served from the datastore or
> > dynamically generated via some other means, you will not incur CPU costs.
>

> > On Thu, Jun 3, 2010 at 12:55 PM, Ross M Karchner <rosskarch...@gmail.com>wrote:
>
> >> Partially off-topic---  if GFE serves a cached resource, do we get billed
> >> for the bandwidth?
>
> >> On Wed, Jun 2, 2010 at 3:27 PM, Ikai L (Google) <ika...@google.com>wrote:
>
> >>> Okay, looks like the Google Front-End is kicking in to cache your stuff
>
> >>> Server  Google Frontend
> >>> Content-Length  2834
> >>> Age     41
> >>> Cache-Control   public, max-age=600
>
> >>> Are you guys setting this header anywhere? Unfortunately, there's no way
> >>> to invalidate items in the frontend cache, so you'll have to use a cache
> >>> buster or wait for the TTL to expire.
>

> >>>> > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

> ...
>
> read more »

[Ikai L (Google)]

Hey guys,

I've tracked this down, and it is working as intended. Our infrastructure may or may not respect your cache headers and just cache assets for your application. It's clear that's what's happening here, with some front-ends caching and some not caching. The worst case scenario is that cache headers are not respected; the best case scenario is that all requests are served from an edge cache. Most developers will fall somewhere in between. If a cache header is set, you should not assume that the edge cache will expire it. On the flip side - you should also not assume that a deploy will invalid the cache, as the edge cache has no knowledge of your code being updated. 

In general, use of a cache-buster for static assets is an absolute must in web development for all the reasons I mentioned in a previous post: ISP, reverse proxy caching, desktop accelerators, weird browser caching, etc. I'll make it a point to make the documentation about our edge caching infrastructure more clear about this and include examples on how to correctly include media, stylesheets and Javascript includes.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Tim Hoffman]

Hi Ikai

So does that mean that the google front proxies do not honour Etags in
the headers as produced
by the google supplied static handler.?

You said

"
> On the flip side - you should also not assume that a deploy will invalid
> the cache, as the edge cache has no knowledge of your code being updated.
>
"

I was under the impression that the purpose of Etag's, was so that
upstream/edge caches could be invalidated.

Whilst I see cache busting working for js and css files, it would seem
to be a lot more difficult
to manage cache busting for images. (Thankfully they tend to change
less frequently)

Thanks for looking at this

Rgds

Tim

On Jun 5, 8:31 am, "Ikai L (Google)" <ika...@google.com> wrote:
> Hey guys,
>
> I've tracked this down, and it is working as intended. Our infrastructure
> may or may not respect your cache headers and just cache assets for your
> application. It's clear that's what's happening here, with some front-ends
> caching and some not caching. The worst case scenario is that cache headers
> are not respected; the best case scenario is that all requests are served
> from an edge cache. Most developers will fall somewhere in between. If a
> cache header is set, you should not assume that the edge cache will expire
> it. On the flip side - you should also not assume that a deploy will invalid
> the cache, as the edge cache has no knowledge of your code being updated.
>
> In general, use of a cache-buster for static assets is an absolute must in
> web development for all the reasons I mentioned in a previous post: ISP,
> reverse proxy caching, desktop accelerators, weird browser caching, etc.
> I'll make it a point to make the documentation about our edge caching
> infrastructure more clear about this and include examples on how to
> correctly include media, stylesheets and Javascript includes.
>
>
>
> On Fri, Jun 4, 2010 at 4:47 PM, Tim Hoffman <zutes...@gmail.com> wrote:
> > Hi Ikai
>
> > So here is the route to the domain mapped app
>

> > traceroute towww.polytechnic.wa.edu.au(72.14.203.121), 30 hops max,

> ...
>
> read more »

[J]

Our ISP is Verizon.

Here are the TraceRoutes:

C:\Users\Jitendra>tracert qa.connectscholar.com

Tracing route to ghs.l.google.com [74.125.113.121]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home
[192.168.1.1]
2 7 ms 6 ms 7 ms L100.BSTNMA-VFTTP-86.verizon-gni.net
[98.110.150.1]
3 9 ms 9 ms 7 ms G1-0-786.BSTNMA-LCR-07.verizon-gni.net
[130.81.110.244]
4 12 ms 10 ms 11 ms so-0-3-0-0.BOS-BB-RTR1.verizon-gni.net
[130.81.29.252]
5 22 ms 20 ms 21 ms so-9-1-0-0.NY325-BB-RTR1.verizon-
gni.net [130.81.19.70]
6 89 ms 19 ms 18 ms 0.so-0-0-0.XL3.NYC4.ALTER.NET
[152.63.1.41]
7 17 ms 19 ms 18 ms 0.xe-5-0-0.BR3.NYC4.ALTER.NET
[152.63.16.181]
8 22 ms 19 ms 19 ms te-7-0-0.edge2.NewYork2.level3.net
[4.68.110.69]
9 19 ms 18 ms 19 ms vlan51.ebr1.NewYork2.Level3.net
[4.69.138.222]
10 31 ms 28 ms 29 ms ae-3-3.ebr2.Washington1.Level3.net
[4.69.132.89]
11 38 ms 34 ms 36 ms ae-92-92.csw4.Washington1.Level3.net
[4.69.134.158]
12 27 ms 25 ms 26 ms ae-4-99.edge1.Washington1.Level3.net
[4.68.17.208]
13 28 ms 29 ms 83 ms GOOGLE-
INC.edge1.Washington1.Level3.net [4.79.231.6]
14 28 ms 26 ms 26 ms 209.85.241.50
15 39 ms 39 ms 59 ms 209.85.248.75
16 55 ms 56 ms 55 ms 209.85.243.25
17 55 ms 54 ms 53 ms 209.85.251.228
18 55 ms * * 72.14.236.193
19 56 ms 55 ms 54 ms vw-in-f121.1e100.net [74.125.113.121]

Trace complete.

C:\Users\Jitendra>tracert charityaxis-qa.appspot.com

Tracing route to appspot.l.google.com [72.14.204.141]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms Wireless_Broadband_Router.home
[192.168.1.1]
2 8 ms 6 ms 6 ms L100.BSTNMA-VFTTP-86.verizon-gni.net
[98.110.150.1]
3 11 ms 10 ms 10 ms G1-0-786.BSTNMA-LCR-07.verizon-gni.net
[130.81.110.244]
4 10 ms 11 ms 11 ms so-0-3-0-0.BOS-BB-RTR1.verizon-gni.net
[130.81.29.252]
5 20 ms 19 ms 19 ms so-9-1-0-0.NY325-BB-RTR1.verizon-
gni.net [130.81.19.70]
6 17 ms 18 ms 19 ms 0.so-0-0-0.XL3.NYC4.ALTER.NET
[152.63.1.41]
7 20 ms 69 ms 21 ms 0.xe-5-0-0.BR2.NYC4.ALTER.NET
[152.63.18.5]
8 17 ms 18 ms 18 ms xe-10-2-0.edge2.NewYork2.level3.net
[4.68.110.233]
9 19 ms 18 ms 18 ms vlan51.ebr1.NewYork2.Level3.net
[4.69.138.222]
10 29 ms 27 ms 28 ms ae-3-3.ebr2.Washington1.Level3.net
[4.69.132.89]
11 36 ms 36 ms 26 ms ae-92-92.csw4.Washington1.Level3.net
[4.69.134.158]
12 30 ms 32 ms 26 ms ae-3-89.edge1.Washington1.Level3.net
[4.68.17.144]
13 28 ms 29 ms 28 ms GOOGLE-
INC.edge1.Washington1.Level3.net [4.79.22.38]
14 30 ms 30 ms 31 ms 209.85.240.136
15 36 ms 31 ms 35 ms 66.249.94.54
16 29 ms 28 ms 26 ms iad04s01-in-f141.1e100.net
[72.14.204.141]

Trace complete.

C:\Users\Jitendra>

On Jun 4, 6:23 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> Can you guys run a traceroute on your domains vs. the appspot domain?
>
> E.g:
>
> traceroute qa.connectscholar.com
> traceroute charityaxis-qa.appspot.com
>
> If you're on Windows, the equivalent command is "tracert".
>
> I'm curious if there's an ISP or specific Google Frontend that is acting up.
>
> On Thu, Jun 3, 2010 at 2:02 PM, Ikai L (Google) <ika...@google.com> wrote:
>
>
>
> > You should still be billed for bandwidth. The advantage of using edge
> > caching is that if the resource is being served from the datastore or
> > dynamically generated via some other means, you will not incur CPU costs.
>

> > On Thu, Jun 3, 2010 at 12:55 PM, Ross M Karchner <rosskarch...@gmail.com>wrote:
>
> >> Partially off-topic---  if GFE serves a cached resource, do we get billed
> >> for the bandwidth?
>
> >> On Wed, Jun 2, 2010 at 3:27 PM, Ikai L (Google) <ika...@google.com>wrote:
>
> >>> Okay, looks like the Google Front-End is kicking in to cache your stuff
>
> >>> Server  Google Frontend
> >>> Content-Length  2834
> >>> Age     41
> >>> Cache-Control   public, max-age=600
>
> >>> Are you guys setting this header anywhere? Unfortunately, there's no way
> >>> to invalidate items in the frontend cache, so you'll have to use a cache
> >>> buster or wait for the TTL to expire.
>

> >>>> > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

> ...
>
> read more »

[TL]

Hi Ikai,
I had this question which I think you answered. Just to confirm:

Static file caching works for dynamically generated files as well
(such as JSPs). This means that the cached resource does not have to
physically exist in the application directories as a static file.

In other words, the static file caching works as reverse proxy (pull)
and not like amazon cloud front which requires all the cached files to
physically exists and be pushed.

E

[TL]

That is really the only solid way to do it. Using request parameter
has too many problems to list.

I did this in a Java app and it does require you to write some code,
since as you mention Java does not have a built in rewrite capability.

I may add google code project with a Java implementation for app
engine (using the app engine version as a directory). Will keep you
posted.

E

> STAR! :) )http://code.google.com/p/googleappengine/issues/detail?id=2070

[Ikai L (Google)]

The tradeoff with request parameters is that you need to output your image and asset tags in your HTML templates to append a parameter and be version aware. The benefit, however, is that it is an almost 100% foolproof way to bust caches. You don't only have the Google edge cache to worry about - you've always got to worry about ISP proxies, browser caches + other caching schemes that users introduce between their computers and your application. I highly recommend doing this, as you will spend countless hours debugging user issues (I have) that are cache related that could be easily solved by appending a URL parameter. This is especially true if you use any kind of JavaScript code that changes.

What are the other problems?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[TL]

On Jun 15, 4:36 pm, "Ikai L (Google)" <ika...@google.com> wrote:
>
> What are the other problems?

The major problem is that browsers and proxy servers may not cache any
URL that has request parameter, regardless of caching headers returned
by the response (such as max-age). This is up to the proxies and user
agents which have varying policies. On the other hand, using simple
resource name with no parameters, does not exhibit such behaviors.

[Ikai L (Google)]

Caching is a "best effort" attempt to reduce load and should never be thought of as a foolproof mechanism to prevent requests from hitting your server. In the case you are describing, URL parameters still work since they prevent the proxies and browsers from storing things they shouldn't store. Most proxies and browsers key off URL, including any request parameters. I have personally never seen browsers or proxies that will not cache a URL with parameters, but I believe they may exist. It's not the worst thing in the world if your assets are not cached - in some cases it is 100x worse to serve a stale asset, especially if your app is JavaScript heavy.

In any case, URL rewriting is a pretty good technique as well, I just don't think it's fair to call appending a query parameter problematic, as I feel it is is an overall easier approach than URL rewriting. In either case, they both encourage the best practice of creating methods in templating language to rewrite image, media and JS tags. For instance, in JSPs, you could have something like the following to output tags:

<%= AssetTag.imageUrl("someImage.jpg") %>

(I should write a blog post or something about this subject.)

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[TL]

Here is another problem with query parameters:

You have a CSS file with background URL
background:url("group/google-appengine/icon?hl=en");

You call the CSS file with cache buster parameter:
http://example.com/static/base.css?timestamp=12345678

The CSS file will call the image /group/google-appengine/icon?hl=en
without the cache buster. Unless you want to change your CSS files
manually every time you replace an image or do it in a build using
some script, you cannot version an image that you use from CSS.

On the other hand, if you name the CSS file:
http://example.com/static/123456/css/base.css

and make the URLs relative in CSS:
background:url("../group/google-appengine/icon?hl=en");

Then the browser will call the URL:
http://example.com/static/123456/ group/google-appengine/icon?hl=en

Every time you change a version number, it affects ALL your static
files in one shot, with zero work for you. Including images from CSS,
and many other things that can be done relative.

I agree that caching is not something that works accurately and
predictably, but you can still make it more effective by avoiding
known problems.

If you are into rapid development, where you release almost every day,
then the advantage of zero work system for versioning static files is
invaluable. You never have to mess with it, just increase the version
number and all static files get upgraded in one shot.

[Ikai L (Google)]

Ah, interesting argument. I've never personally been in a situation where we've had to replace CSS referenced images enough times that manually changing their URIs was a problem, even in environments where I was deploying 5+ times a day. In practice, every time I've used CSS referenced images, they've been because of backgrounds (rare changes), rounded corner hacks or other browser hacks that don't change enough. I think you may have sold me, and at the very least I shouldn't knock it until I try it. Anything to kill those nasty cache issues that pop up.

Moral of the story: always use a cache buster.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.