Summary: What if there was a backwards compatible way to transfer all of the resources that are used on every single page in your site CSS, JS, images, anything else in a single HTTP request at the start of the first visit to the page? This is what Resource Package support in browsers will let you do.
> Summary: > What if there was a backwards compatible way to transfer all of the > resources that are used on every single page in your site CSS, JS, > images, anything else in a single HTTP request at the start of the > first visit to the page? This is what Resource Package support in > browsers will let you do.
It's a very good idea and it will help to speed up the loading time. You must take in consideration that the .zip file can contain a virus or other vulnerabilities so please consider this.
Overall I really like the idea. We're pretty much heading in this direction anyway (1 large CSS file, 1 large JS file + sprites).
An addition I'd add would be a way to have multiple manifest files. Currently a way to speed up websites/pages is to only load what is required for the current page and post-load the rest. An example would be a homepage would only load the manifest file it needs to make it load faster, but post-load another manifest file. Or another page can load a different manifest file later on. Considering some sites are huge, their manifest files could get far too large for most internet connections.
And making sure browsers can pull resources out of the manifest file *before* it has completed downloading should be a requirement. Currently browsers deal with missing/incomplete files as gracefully as they can, I'd like that graceful degradation to continue.
The ordering of assets should be CSS, JS then images inside the manifest file.
Might even want a way to notate if the JS should be parsed after the page is ready or not (defer). (Equivalent of putting it at the bottom of the page)
(I posted this on Bugzilla first, realized the discussion might belong here instead.)
I like this proposal and I hope that more browsers implement it soon. But, let's say that you have a partial ZIP file, where a CSS file is referenced on the page itself through a <link> tag, but does not exist in the ZIP. Will this be read and applied separately? Will this happen after everything from the ZIP file has been applied?
Or, let's say a file inside the ZIP was corrupt (or would that make the whole ZIP corrupt? What happens then?). If the browser can notice this, will it try to read the file referenced in the page instead?
And, a quite silly question, but to be ready for all situations, are multiple resource packages allowed on the same page? (While it would be quite dumb to do it, I don't see why it shouldn't be allowed)
Interesting spec. Sorry if some of these questions look familiar; I recall discussion but not any of the answers :( Here's a collection of mostly unorganized questions:
How would this deal with CDNs / files hosted on a completely different domain? Just have the href= be an absolute URL instead? (Zip files don't allow directory entries like ../../../file.html do they? Appnote.txt doesn't seem to say anything about that...) Would this lead to more GIFAR-like exploits?
Appnote.txt specifies that file names all come with separate size counts (i.e. they _can_ contain null bytes); it'll be fun to make sure that doesn't break things. But the jar: protocol handler probably does it right already :)
Would the content-type magic guessing do interesting things? I guess it would finally allow XHTML 1.1 transitional to be parsed with an XML parser (since I'd imagine IE to not understand it, and parse it with a HTML parser - especially if it ignores the resource package completely).
I'm having trouble understanding the third bullet about charsets in "Additonal notes". Zip files (with extra field 0x7075, UPath) can contain UTF8 file names; what are the charsets here referring to? The textual content of plain text files?
Are multiple <link rel="resource-package"> elements allowed per page (say, to have different caching policies)? If those elements end up containing the same resource, who wins?
How would that affect caching policy, for both requests from the current host, and for requests from actual documents hosted on the CDN host? What happens if the zip file has a different cache header than the normal, unpackaged resource? Possibly the first one to be used wins, I guess.
Is this straight up cache injection? If I include a HTML file, then have a link to where the HTML file maps to, does the browser not need to make a request?
I wish something other than .zip was as ubiquitous, since having a format with a central directory at the end of the file sucks, especially given that normal zip archives don't expose the order of files within an archive to the user at all. They're quite free to tack the new file at the end instead. That and the fact that pathnames are separated with a DOS-style backslash; I vaguely recall some problems with reading zip files with forward slashes for jar: in Firefox at some point.
(Clarification request: who is "we" in the context of the blog post?)
I also have a suggestion. We know that some files are more important than others on some sites, this is especially true on large sites loaded on slow connections. How about a way to set priority to files inside manifest.txt?
This would make sure the style sheets finish loading before the javascript, and after that comes the images. This would be completely optional and wouldn't matter on fast connections unless the site is huge. Still, it could be a large (perceived) performance win on slow connections as what is required for the site to function is loaded first.
those Resource Packages are a very interesting idea! I like your priority idea Magne, it's quite simple and might get rid of the sorting problems within the zip files.
But there's on issue, which Darkyndy already adressed in the first comment: Viruses. How do you think this should be adressed? With such a ZIP File to be downloaded almost anything could come along with it. I guess if you would first have to perform a virus check with the antivirus tool of each users choice, it would make up all the speed advantage formerly introduced. This client side virus check should be considered anyway. Wouldn't that ZIP file be adressed by any of the popular Antivirus Programs just like any download would be, and therefore be checked? May that become a problem? Especially on larger websites, where the ZIP File might be considerably larger than in most cases. Maybe the "mutliple manifest idea" mentioned above could help here?!
Of course there could be a server-side check of the ZIP File...
looking forward to see this come to life! greetz Christian
The priority idea by Magne (but maybe not the syntax) is what your proposal really lacks. However it is possible to put files in zip archive in arbitrary order, which can be considered as a priority by a browser, i don't think it's a transparent and easy operation with all zipping solutions.
As for viruses, I don't quite get how it can be more dangerous than downloading separate js files. Is it real to have a virus in this zip that would auto-execute after downloading?
On 17 Nov, 10:13, "semper.cras" <semper.c...@gmail.com> wrote:
> The priority idea by Magne (but maybe not the syntax) is what your > proposal really lacks. However it is possible to put files in zip > archive in arbitrary order, which can be considered as a priority by a > browser, i don't think it's a transparent and easy operation with all > zipping solutions.
> As for viruses, I don't quite get how it can be more dangerous than > downloading separate js files. Is it real to have a virus in this zip > that would auto-execute after downloading?
I agree with you that the syntax is a bit odd, and I'd like some feedback/suggestions for a better syntax.
> An addition I'd add would be a way to have multiple manifest files.
You mean multiple resource packages? The spec already supports that.
> load a different manifest file later on. Considering some sites are > huge, their manifest files could get far too large for most internet > connections.
The manifest file only contains a list of items in the zip file. If the manifest file is enormous, then the zip file will be 20x as big. I think you might be getting confused between manifest files and resource packages.
> And making sure browsers can pull resources out of the manifest file > *before* it has completed downloading should be a requirement.
This is also in the spec.
> The ordering of assets should be CSS, JS then images inside the > manifest file.
The spec doesn't mandate an order; that's up to the site designer.
> Might even want a way to notate if the JS should be parsed after the > page is ready or not (defer). (Equivalent of putting it at the bottom > of the page)
That notation will be on the <script> tag referring to the resource. Resource packages just give an alternate way to obtain the resource.
> (I posted this on Bugzilla first, realized the discussion might belong > here instead.)
> I like this proposal and I hope that more browsers implement it soon. > But, > let's say that you have a partial ZIP file, where a CSS file is > referenced on > the page itself through a<link> tag, but does not exist in the ZIP. > Will this > be read and applied separately?
Yes; the purpose of manifest files is to let the browser know quickly which files are in the ZIP and which are not. Files not in the ZIP will be downloaded as normal, in parallel, just like any other request.
> Or, let's say a file inside the ZIP was corrupt (or would that make > the whole > ZIP corrupt? What happens then?). If the browser can notice this, will > it try > to read the file referenced in the page instead?
Probably not, no. How would you detect "corrupt"? You might be able to in some cases, but then surely the fix is to fix the zip file.
> And, a quite silly question, but to be ready for all situations, are > multiple resource packages allowed on the same page? (While it would > be quite dumb to do it, I don't see why it shouldn't be allowed)
It wouldn't be at all dumb, and it's certainly allowed.
> I also have a suggestion. We know that some files are more important > than others on some sites, this is especially true on large sites > loaded on slow connections. How about a way to set priority to files > inside manifest.txt?
ZIP files contain files in a specific order; put the most important files first, and they will be downloaded and used first.
> But there's on issue, which Darkyndy already adressed in the first > comment: Viruses.
Resource packages don't introduce any new virus issues that I can see. The browser is not going to be executing any native code downloaded without asking the user.
> How do you think this should be adressed? With such a ZIP File to be > downloaded almost anything could come along with it.
> How would this deal with CDNs / files hosted on a completely different > domain?
The CDN could distribute the ZIP file.
> I'm having trouble understanding the third bullet about charsets in > "Additonal notes". Zip files (with extra field 0x7075, UPath) can > contain UTF8 file names; what are the charsets here referring to? The > textual content of plain text files?
Yes.
> Are multiple <link rel="resource-package"> elements allowed per page > (say, to have different caching policies)? If those elements end up > containing the same resource, who wins?
Yes. But it's a good question as to which wins if they both contain the same resource name but different resource data. I'd say the behaviour is undefined (i.e. "don't do that").
> How would that affect caching policy, for both requests from the current > host, and for requests from actual documents hosted on the CDN host? > What happens if the zip file has a different cache header than the > normal, unpackaged resource? Possibly the first one to be used wins, I > guess.
The normal unpackaged resource isn't downloaded if the resource is in the ZIP file. So the ZIP file's caching headers govern all the resources in it.
On 17 Nov, 10:37, Gervase Markham <g...@mozilla.org> wrote:
> On 17/11/09 08:40, Magne Andersson wrote:
> > I also have a suggestion. We know that some files are more important > > than others on some sites, this is especially true on large sites > > loaded on slow connections. How about a way to set priority to files > > inside manifest.txt?
> ZIP files contain files in a specific order; put the most important > files first, and they will be downloaded and used first.
> Gerv
Will all "proper" ZIP-programs contain them in that order? I mean, files in a classic folder can easily be sorted after name, or date of creation, or type.
> Summary: > What if there was a backwards compatible way to transfer all of the > resources that are used on every single page in your site CSS, JS, > images, anything else in a single HTTP request at the start of the > first visit to the page? This is what Resource Package support in > browsers will let you do.
I think this is the real solution and CSS Sprites are just workaround that was developed because this is one was missing. But (!) if there could be solution to that problem at HTTP level, I'd rather focus on that, let's cut it at the source.
> It wouldn't be at all dumb, and it's certainly allowed.
> Gerv
What I meant by that was that the goal of this proposal was to reduce the number of requests. By having multiple packages, you add at least one more request without any reason.
I think I'd rather not have to unzip/edit/re-zip every time I modify a resource (oops someone forgot to zip!) Especially when the resources in question are not maintained in a file system. eg. IBM/Lotus Domino holds the resources within another file structure (.nsf) I would much prefer to have the server dish up a single (cached) gzipped, minified resource to the browser. See www.dominoexperts. com/dapInfo
On 17 Nov, 10:55, Ictinus <icti...@gmail.com> wrote:
> I think I'd rather not have to unzip/edit/re-zip every time I modify a > resource (oops someone forgot to zip!) Especially when the resources > in question are not maintained in a file system. > eg. IBM/Lotus Domino holds the resources within another file structure > (.nsf) > I would much prefer to have the server dish up a single (cached) > gzipped, minified resource to the browser. > Seewww.dominoexperts. com/dapInfo
Well, obviously, you wouldn't include a frequently edited resource in the ZIP, but most sites have their resources static for quite a long time.
Gervase Markham wrote: > On 17/11/09 08:40, Mook wrote: >> Are multiple <link rel="resource-package"> elements allowed per page >> (say, to have different caching policies)? If those elements end up >> containing the same resource, who wins?
> Yes. But it's a good question as to which wins if they both contain the > same resource name but different resource data. I'd say the behaviour is > undefined (i.e. "don't do that").
I'd say this really _should_ be defined clearly before people start to rely on any particular (perceived) behaviour. I thought of several strategies where I'd have a large static "main" resource package, but would occasionally want to be able to override a file from it on a case-by-case basis with a different version of that file specified in a second resource package. It'd be hacky, but it'd also be really useful.
But then I realized that this'd mean that content from later resource packages could potentially override content from earlier resource packages, which would mean that the browser would have to wait for all resource packages to be downloaded before it'd be capable of doing certain things, which'd be a bad thing. (The same situation exists in reverse if a second small resource package finishes loading before a first large resource package, containing the same resource, though. If only speccing "whichever file gets seen first by the browser" wouldn't be so indeterminate...)
In general, the entire capability of defining multiple resource package probably deserves a more explicit mention in the spec than being tucked away inside an "additional note" about headers. Splitting up available resources into multiple packages was the first thing I (working web developer) wanted to ask about after reading the spec. On any non-trivial site I'll want to spit up my resources into two or three resource packages per page (one global site-wide, cached forever, one high priority (maybe reusable per section), and one with whatever is left over and specific for the page).
On Nov 17, 9:00 pm, Magne Andersson <zirroz...@gmail.com> wrote:
> On 17 Nov, 10:55, Ictinus <icti...@gmail.com> wrote:
> > I think I'd rather not have to unzip/edit/re-zip every time I modify a > > resource (oops someone forgot to zip!) Especially when the resources > > in question are not maintained in a file system. > > eg. IBM/Lotus Domino holds the resources within another file structure > > (.nsf) > > I would much prefer to have the server dish up a single (cached) > > gzipped, minified resource to the browser. > > Seewww.dominoexperts. com/dapInfo
> Well, obviously, you wouldn't include a frequently edited resource in > the ZIP, but most sites have their resources static for quite a long > time.
I agree, perhaps a ZIP file would be useful in the short term, but a server managed solution long term would allow all resources to get the performance benefit. Not having to manage the zip file makes a developers life easier. Having it server side means more benefit. After all we already do gzip server side.
On Nov 17, 12:37 pm, Gervase Markham <g...@mozilla.org> wrote:
> On 17/11/09 08:40, Magne Andersson wrote:
> > I also have a suggestion. We know that some files are more important > > than others on some sites, this is especially true on large sites > > loaded on slow connections. How about a way to set priority to files > > inside manifest.txt?
> ZIP files contain files in a specific order; put the most important > files first, and they will be downloaded and used first.
> Gerv
Basic ZIP programs do NOT let the user to (re-)sort files in archive manually (tried Gnome's File Roller and basic WinXP archivator). So the only way that'll probably work is to create archive and than add files there one by one, which is by no means elegant and transparent. Moreover, even after this one cannot be sure that the order is right, because there's no obvious way to check it.
And now imagine that I want to change the order of two images in the zip file with say 20 files. Do I have to unzip it and then zip all the files one by one to make them go in the need order?
So, there's a problem. If we set priority in the manifest file and the real priority (in the zip file) is different, we can lose the advantage of not waiting for the zip to download completely. Maybe the partial solution would be promoting/creating handy tools for managing file order in the archive?
> On Nov 17, 9:00 pm, Magne Andersson <zirroz...@gmail.com> wrote:
> > On 17 Nov, 10:55, Ictinus <icti...@gmail.com> wrote:
> > > I think I'd rather not have to unzip/edit/re-zip every time I modify a > > > resource (oops someone forgot to zip!) Especially when the resources > > > in question are not maintained in a file system. > > > eg. IBM/Lotus Domino holds the resources within another file structure > > > (.nsf) > > > I would much prefer to have the server dish up a single (cached) > > > gzipped, minified resource to the browser. > > > Seewww.dominoexperts. com/dapInfo
> > Well, obviously, you wouldn't include a frequently edited resource in > > the ZIP, but most sites have their resources static for quite a long > > time.
> I agree, perhaps a ZIP file would be useful in the short term, but a > server managed solution long term would allow all resources to get the > performance benefit. > Not having to manage the zip file makes a developers life easier. > Having it server side means more benefit. After all we already do gzip > server side.
I don't think you understood what I meant, I said that this solution is great for resources that doesn't update frequently. This is also made to eliminate requests, GZIP doesn't do that. It just makes the files smaller.
> On Nov 17, 12:37 pm, Gervase Markham <g...@mozilla.org> wrote:
> > On 17/11/09 08:40, Magne Andersson wrote:
> > > I also have a suggestion. We know that some files are more important > > > than others on some sites, this is especially true on large sites > > > loaded on slow connections. How about a way to set priority to files > > > inside manifest.txt?
> > ZIP files contain files in a specific order; put the most important > > files first, and they will be downloaded and used first.
> > Gerv
> Basic ZIP programs do NOT let the user to (re-)sort files in archive > manually (tried Gnome's File Roller and basic WinXP archivator). So > the only way that'll probably work is to create archive and than add > files there one by one, which is by no means elegant and transparent. > Moreover, even after this one cannot be sure that the order is right, > because there's no obvious way to check it.
> And now imagine that I want to change the order of two images in the > zip file with say 20 files. Do I have to unzip it and then zip all the > files one by one to make them go in the need order?
> So, there's a problem. If we set priority in the manifest file and the > real priority (in the zip file) is different, we can lose the > advantage of not waiting for the zip to download completely. Maybe the > partial solution would be promoting/creating handy tools for managing > file order in the archive?
Or ignore the order of the files in the ZIP if priority is already specified in the manifest file.
On Nov 17, 10:22 am, "semper.cras" <semper.c...@gmail.com> wrote:
> Basic ZIP programs do NOT let the user to (re-)sort files in archive > manually (tried Gnome's File Roller and basic WinXP archivator). So > the only way that'll probably work is to create archive and than add > files there one by one, which is by no means elegant and transparent. > Moreover, even after this one cannot be sure that the order is right, > because there's no obvious way to check it.
Most command line zip programs (e.g. Info-zip) will read a list of files and zip them in the order specified. So something like:
zip resource.zip manifest.txt -@ <manifest.txt
should be all you need to create you a resource file with manifext.txt first and then all the files listed in manifest.txt in the order they appear. That's both elegant and transparent.
On another note, what I like about this proposal is that you can potentially speed up browsing even for old broken browsers by adding support for resource packages to a proxy server. So everyone wins.
