Investigating browser offline app cache behavior with Meteor

[Andrew Wilcox]

I've started to investigate the behavior of browsers' offline app caching in conjunction with Meteor's package loading / livedata / auto reload / cache busting URL's / etc.

For the interested or curious, this is a hacked branch of Meteor I'm using for the experiments:

https://github.com/awwx/meteor/tree/appcache-experiment#readme

Note that this is not code for delivering a Meteor application that uses an offline app cache, that would be a later project (if I'm successful with these experiments).

This is based on jonathanKingston's initial work on app cache support (though I'm generating the app manifest dynamically instead of statically to ensure that changes to Meteor's runtime config are seen by clients):

https://github.com/meteor/meteor/commit/0c5f8d14e6f373f3f17e0a73750095739201fe70

[Andrew Wilcox]

First success: preserving Meteor's dev code reload behavior when an application cache is used.

The normal browser behavior for applications that have been cached is:

1. The app is loaded from the assets (the .js and .css files) in the cache, without connecting to the Internet at all. (This provides for faster startup).  The app begins to run.  (With standard Meteor, Meteor.startup callbacks are called on dom ready, templates are rendered, livedata connects to the server, etc.)

2. The browser fetches the manifest file from the server. If it's changed, the app's assets are reloaded from the server in the background while the app continues to run. When the new assets are all loaded the app gets an "updateready" event, which it can use to for example give the user an "update is ready" message.

3. The next time the page is (re)loaded, the app is loaded from the new assets from the cache.

But this breaks Meteor's auto reload feature, because on the first reload after a code change, the new files are simply loaded into the cache -- the app is still running the old code.  The app doesn't pick up the new code until the next page load.

Meteor normally starts up on the client when the DOM is ready.  What I did was delay Meteor's startup until it gets the app cache "noupdate" event (which means the cache is fully up to date); and if it gets an "updateready" event instead (which means there's new code in the cache that hasn't been loaded yet) to reload the page.

updateready -> reload page -> noupdate -> domReady -> Meteor startup

[Andrew Wilcox]

Wow, this app cache stuff is insane! :)

Oh, by the way, if you're curious, here are some good resources to check out:

* http://www.alistapart.com/articles/application-cache-is-a-douchebag/

* http://appcachefacts.info/

* http://www.whatwg.org/specs/web-apps/current-work/multipage/offline.html

* https://developer.mozilla.org/en-US/docs/HTML/Using_the_application_cache

* http://diveintohtml5.info/offline.html

The usual problem people have with the offline application cache is that they don't change the manifest when a referenced resource changes, or they're specifying more caching than they realize through HTTP cache control headers, and the browser continues to use old versions of files when they change them on the server.  So I'm paying particular attention to update issues.

Here's a crazy one.

In my Meteor app I have an image in the public/ directory:  public/cat.jpg, and in my .html I display the image via <img src="/cat.jpg">.  As my test, I change contents of cat.jpg (to the image to a different cat), so that it still has the same name but is a different image.  The manifest file contains a hash of all the resources, so it gets changed when the image file changes, and I've changed the cache-control header from Meteor's one week to a max-age: 0 so I'm not getting any unexpected double caching.

Which works in Chrome, in iOS Safari on my iPad, with Android on my phone, and even in Internet Explorer: livedata auto-reloads the page in the usual Meteor way, the browser fetches the new manifest, fetches the new version of the image, the page re-renders... and I see a new cat!  ("What did you do today?"  "Look, I got my browser to display a different cat!")

but in Firefox

- From the server logs I see Firefox downloading the new manifest file

- and from the server logs I see Firefox downloading the new image file

- and in Firefox, I go to about:cache, and I see that the new contents are in the offline cache

But Firefox still displays the old image!

?

?

Oh, it's a bug in Firefox!  https://bugzilla.mozilla.org/show_bug.cgi?id=734062

^_^

[Andrew Wilcox]

The crazy continues...

Chrome has a 5MB limit for the size of the app cache.  If we go over the limit (say we're including public/ in the app cache so those resources will be available offline, but we put too much stuff in there), loading the app cache fails and the application gets sent an app cache "error" event.

Now what happens?  Assuming the app is already cached (this isn't the very first time we've loaded the app), the browser continues to use the old cached code.

Which may or may not be OK.  The default assumption in Meteor is that it's not OK to use old code on the client with new code on the server.  Which the programmer can change by setting the SERVER_ID (to say when a code change is a "minor" code change that doesn't need to force a reload).

Suppose we've made a "major" code change (there's no point in having old client code talk to the new server code; methods have been renamed or collections have a different structure or whatever).  Now we've got an app which doesn't work offline because we violated a browser limitation, but it also now doesn't work online either because it's using old cached client code.

So...  ideas... suppose the client were to detect this condition and set a "nocache" cookie which causes the server to deliver a plain web app (like standard Meteor does today) without an app cache. so then the app would at least be able to work online while the offline problem was being fixed.  (I.e., it's nice to be able to work offline when we can, but don't cause on other working online application to break).

Ah, but the application will get an app cache "error" event for any problem updating the cache.  If the browser doesn't have an Internet connection to the server when it checks the app cache manifest file and can't download the manifest, the application will also get an "error" event.  So getting an "error" event can simply mean that the browser is offline, and we don't happen to have an Internet connection at the moment.

And the "error" event has no error code or error message associated with it.  The only piece of information that the application receives is that there was a problem updating the cache.  We don't know if it was because our Internet connection happened to flake out for a moment, or if the manifest file is incorrectly formatted, or if we've violated some browser imposed limitation.

Ermmm, so the solution is to be very very very very careful and never violate a browser limitation?  Because if you do, it will look the same as if your app is always offline.

* * *

In other news I don't actually have the timing / interaction figured out between code updates, the app cache, and livedata figured out yet.  The strategy of delaying Meteor startup until the cache has been updated meant that no changes needed to be made to the livedata client code, but it also means the app won't actually work offline (because when it's offline the cache won't get updated).  Which is blindingly obvious once I thought of it, but didn't occur to me when I was focused on the code reload issue.

Maybe the trick is to delay the reloading of the web page in reload.js until the app cache has been updated.  Because at that point there's no point in reloading the web page if the browser will just reload the old code from the cache.