net/http: Serving static files with hash part (cache and proxy-friendly)

[Johann Höchtl]

This is more of a what do I need to do question - but very Go-specific.

Assume a html-template which contains this function:

img src="{{statichash file.png}}"

statichash will return for the parameter file.png --> file.HASH.png, so the (Go) server will eventually hit a request for file.HASH.png. At the server side, for performance reasons, the HASH will either be precomputed and fed to the cache or calculated upon first request (by an external "file watcher" receiving ioctls) and then added to the cache. A very long expiration date will be added for browser caching.

Now I wonder what to do when a request like file.HASH.png this hits the static handler.

If dived into the source of net/http/fs.go. Am I right that in order to implement this I have to mimic servefile? Like

    http.Handle("/static/", hashStaticHandler(statictimeout, http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

hashStaticHandler will perform:

* parse the request of file.HASH.png and extract the hash-part (if any)
 - validate the hash against the cached value of the hash of file.png (this is an implementation detail, I could calculate the hash upon every request but that gets computationally heavy)
 - IFequal: rewrite the request to file.png and continue serving with
        w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public, must-revalidate, proxy-revalidate", seconds)) // Update the time-out
        h.ServeHTTP(w, r)
 - else response with moved permanently and return file.NEWHASH.png

Is this feasible this way?

[Patrick Mylund Nielsen]

I would use a version number instead -- no real reason to have a complete digest for every file. When the file changes, increment foo.n.png, and rewrite if n is less than the current. Or use seconds since epoch.

I usually have a short max-age on the page linking to whatever images, then change the hrefs on the page if the file changes, so my static server doesn't need to do much/any rewriting.

--
 
 

[Rodrigo Kochenburger]

Why not pass the the last modification timestamp as hash and as a parameter? I.e. file.png?2012215183500

It will work out of the box ;)

[Johann Höchtl]

Parameters are not proxy-server nor CDN-cache friendly. Will work for the browser cache, but thats just one "cache-point"

[Johann Höchtl]

Now that was to fast for me. I understand the concept behind version numbers (and can't remember their disadvantages right now) but what would you keep for what reason on every page? Pls elaborate at bit.

(Sent from smart phone, can't properly cite)

[Rodrigo Kochenburger]

You're right. It does not work with all caches.

On your proposal, if you precompute the file and actually store it on the filesystem, you can just server the file with a straight FileServer.

[Johann Höchtl]

Am Mittwoch, 26. Dezember 2012 16:12:56 UTC+1 schrieb Hraban Luyat:

Hi,

It might help to have a close look at your requirements and see if you can cut corners, there. Two easier solutions that are not as functionally complete but that will do the job:

- Pre-compute the hashes, create symlinks with the new names (e.g. make fingerprints)

 

That sounds like a very good idea, I will investigate in that direction.

- Put your app behind a webserver and create a URL matching rule for hashes that strips the hash and serves the bare file

If you wrap all references to static files in a function (as you do now) it will make it easier to also change the host they are served from: {{cdn "myfile.png"}} -> //blabla.cloudfront.net/myfile-somehash.png", for example. Even if you do not have a CDN, this will make it easy to use a webserver only for your static files, which, in turn, makes dealing with this specific case much easier.

The nice thing about using hashes is that it relies solely on the contents of the file. Timestamps or versioning requires extra bookkeeping and can get confusing if you have multiple locations you serve your static content from.

If possible, try to add an extra hint in the filename, not just the hash. E.g.: myfile.png -> myfile-md5-d41d8cd98f00b204e9800998ecf8427e.png. This helps identifying the hash when you have just the requested resource. Useful when using regular expressions for URL rewrite rules, or find rules.

That are all very good advices, thank you!

With the symlink solution I can leave the StaticFileServer as is and wrap the logic into the template static handler which will do a filename --> hasedfilename handle.
 

Greetings,

Hraban

[Hraban Luyat]

Hi,

Here's a script I use to generate md5 fingerprinted names, md5name.sh:

#!/bin/bash

if [[ ! -f "$1" ]]
then
echo "Usage: $0 <filename>" >&2
echo >&2
echo "File must exist and be a standard file" >&2
exit 1
fi

h=`md5sum "$1" | awk '{ print $1 }'`
d=`dirname "$1"`
b=`basename "$1"`
e=""
if [[ "$b" == *.* ]]
then
e=".${b##*.}"
b="${b%.*}"
fi
echo "$d/$b-md5-$h$e"

I am not sure what would happen with filenames with spaces.

Here are the relevant parts of a Makefile (make fingerprints):

# Watch out for filenames with spaces
STATICS := $(shell find static/ -type f -and \! \( -name '.*' -o -name '*~' \))
# Use GNU Make friendly files to annotate when the latest fingerprint was made
FINGERHINTS := $(patsubst %,makegarbage/fingerprints/%,$(STATICS))

$(FINGERHINTS): makegarbage/fingerprints/%: %
mkdir -p $(shell dirname $@)
ln -s -f $(shell basename $<) `./md5name.sh $<`
touch $@

fingerprints: $(FINGERHINTS)

clean-fingerprints:
rm -rf makegarbage/fingerprints
find static -name '*-md5-*' -delete
CLEANERS += clean-fingerprints

clean: $(CLEANERS)
rm -rf makegarbage

I succombed to using a bookkeeping directory, "makegarbage", where I
keep track of when a fingerprint was created for each file. It is easy
to use, easy to clean (rm -rf makegarbage) and it leaves no noise in
the directory of the static files themselves.

To clear all old fingerprints, make clean does the job (or make
clean-fingerprints). To update the newest fingerprints, make
fingerprints (or add it as a dependency for the rest of the program).

The caveat here is that old fingerprints will point to new files,
until purged. There is a benefit to this: you can test changes by
refreshing the page without having to regenerate the fingerprints and
restart the server every time.

If somebody knows of a nice way to improve this workflow I would be
happy to hear about it.

Greetings,

Hraban

> --
>
>

[Martin Angers]

Unless the goal is to experiment with go to build this type of mechanism, I would recommend using Varnish in front of your Go web server to handle the caching. That's what it's for, and although I'm no Varnsih expert, I believe it handles your use-case quite easily.

[Johann Höchtl]

Thank you for this tips ... I always enjoy seeing some shell scripting
which is (almost) all wizardry to me.

Johann