Making browsers faster: Resource Packages
Alexander Limi
I have published the first public draft on Resource Packages, and
would like to hear your feedback:
http://limi.net/articles/resource-packages/
Summary:
What if there was a backwards compatible way to transfer all of the
resources that are used on every single page in your site — CSS, JS,
images, anything else — in a single HTTP request at the start of the
first visit to the page? This is what Resource Package support in
browsers will let you do.
darkyndy
It's a very good idea and it will help to speed up the loading time.
You must take in consideration that the .zip file can contain a virus
or other vulnerabilities so please consider this.
Hope that I will see it in FireFox 3.7
Ryan Doherty
direction anyway (1 large CSS file, 1 large JS file + sprites).
An addition I'd add would be a way to have multiple manifest files.
Currently a way to speed up websites/pages is to only load what is
required for the current page and post-load the rest. An example would
be a homepage would only load the manifest file it needs to make it
load faster, but post-load another manifest file. Or another page can
load a different manifest file later on. Considering some sites are
huge, their manifest files could get far too large for most internet
connections.
And making sure browsers can pull resources out of the manifest file
*before* it has completed downloading should be a requirement.
Currently browsers deal with missing/incomplete files as gracefully as
they can, I'd like that graceful degradation to continue.
The ordering of assets should be CSS, JS then images inside the
manifest file.
Might even want a way to notate if the JS should be parsed after the
page is ready or not (defer). (Equivalent of putting it at the bottom
of the page)
Good stuff!
Magne Andersson
here instead.)
I like this proposal and I hope that more browsers implement it soon.
But,
let's say that you have a partial ZIP file, where a CSS file is
referenced on
the page itself through a <link> tag, but does not exist in the ZIP.
Will this
be read and applied separately? Will this happen after everything from
the ZIP
file has been applied?
Or, let's say a file inside the ZIP was corrupt (or would that make
the whole
ZIP corrupt? What happens then?). If the browser can notice this, will
it try
to read the file referenced in the page instead?
And, a quite silly question, but to be ready for all situations, are
multiple resource packages allowed on the same page? (While it would
be quite dumb to do it, I don't see why it shouldn't be allowed)
Mook
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
Interesting spec. Sorry if some of these questions look familiar; I
recall discussion but not any of the answers :( Here's a collection of
mostly unorganized questions:
How would this deal with CDNs / files hosted on a completely different
domain? Just have the href= be an absolute URL instead? (Zip files don't
allow directory entries like ../../../file.html do they? Appnote.txt
doesn't seem to say anything about that...) Would this lead to more
GIFAR-like exploits?
Appnote.txt specifies that file names all come with separate size counts
(i.e. they _can_ contain null bytes); it'll be fun to make sure that
doesn't break things. But the jar: protocol handler probably does it
right already :)
Would the content-type magic guessing do interesting things? I guess it
would finally allow XHTML 1.1 transitional to be parsed with an XML
parser (since I'd imagine IE to not understand it, and parse it with a
HTML parser - especially if it ignores the resource package completely).
I'm having trouble understanding the third bullet about charsets in
"Additonal notes". Zip files (with extra field 0x7075, UPath) can
contain UTF8 file names; what are the charsets here referring to? The
textual content of plain text files?
Are multiple <link rel="resource-package"> elements allowed per page
(say, to have different caching policies)? If those elements end up
containing the same resource, who wins?
How would that affect caching policy, for both requests from the current
host, and for requests from actual documents hosted on the CDN host?
What happens if the zip file has a different cache header than the
normal, unpackaged resource? Possibly the first one to be used wins, I
guess.
Is this straight up cache injection? If I include a HTML file, then
have a link to where the HTML file maps to, does the browser not need to
make a request?
I wish something other than .zip was as ubiquitous, since having a
format with a central directory at the end of the file sucks, especially
given that normal zip archives don't expose the order of files within an
archive to the user at all. They're quite free to tack the new file at
the end instead. That and the fact that pathnames are separated with a
DOS-style backslash; I vaguely recall some problems with reading zip
files with forward slashes for jar: in Firefox at some point.
(Clarification request: who is "we" in the context of the blog post?)
--
Mook
Magne Andersson
than others on some sites, this is especially true on large sites
loaded on slow connections. How about a way to set priority to files
inside manifest.txt?
From your example:
PRIORITY: 1
styles/reset.css
styles/grid.css
styles/main.css
PRIORITY: 2
javascript/jquery.js
PRIORITY: 3
images/save.png
images/info.png
This would make sure the style sheets finish loading before the
javascript, and after that comes the images. This would be completely
optional and wouldn't matter on fast connections unless the site is
huge. Still, it could be a large (perceived) performance win on slow
connections as what is required for the site to function is loaded
first.
Christian
those Resource Packages are a very interesting idea!
I like your priority idea Magne, it's quite simple and might get rid
of the sorting problems within the zip files.
But there's on issue, which Darkyndy already adressed in the first
comment: Viruses.
How do you think this should be adressed? With such a ZIP File to be
downloaded almost anything could come along with it. I guess if you
would first have to perform a virus check with the antivirus tool of
each users choice, it would make up all the speed advantage formerly
introduced.
This client side virus check should be considered anyway. Wouldn't
that ZIP file be adressed by any of the popular Antivirus Programs
just like any download would be, and therefore be checked? May that
become a problem? Especially on larger websites, where the ZIP File
might be considerably larger than in most cases.
Maybe the "mutliple manifest idea" mentioned above could help here?!
Of course there could be a server-side check of the ZIP File...
looking forward to see this come to life!
greetz
Christian
semper.cras
proposal really lacks. However it is possible to put files in zip
archive in arbitrary order, which can be considered as a priority by a
browser, i don't think it's a transparent and easy operation with all
zipping solutions.
As for viruses, I don't quite get how it can be more dangerous than
downloading separate js files. Is it real to have a virus in this zip
that would auto-execute after downloading?
Magne Andersson
I agree with you that the syntax is a bit odd, and I'd like some
feedback/suggestions for a better syntax.
Gervase Markham
> An addition I'd add would be a way to have multiple manifest files.
You mean multiple resource packages? The spec already supports that.
> load a different manifest file later on. Considering some sites are
> huge, their manifest files could get far too large for most internet
> connections.
The manifest file only contains a list of items in the zip file. If the
manifest file is enormous, then the zip file will be 20x as big. I think
you might be getting confused between manifest files and resource packages.
> And making sure browsers can pull resources out of the manifest file
> *before* it has completed downloading should be a requirement.
This is also in the spec.
> The ordering of assets should be CSS, JS then images inside the
> manifest file.
The spec doesn't mandate an order; that's up to the site designer.
> Might even want a way to notate if the JS should be parsed after the
> page is ready or not (defer). (Equivalent of putting it at the bottom
> of the page)
That notation will be on the <script> tag referring to the resource.
Resource packages just give an alternate way to obtain the resource.
Gerv
Gervase Markham
> (I posted this on Bugzilla first, realized the discussion might belong
> here instead.)
>
> I like this proposal and I hope that more browsers implement it soon.
> But,
> let's say that you have a partial ZIP file, where a CSS file is
> referenced on
> the page itself through a<link> tag, but does not exist in the ZIP.
> Will this
> be read and applied separately?
Yes; the purpose of manifest files is to let the browser know quickly
which files are in the ZIP and which are not. Files not in the ZIP will
be downloaded as normal, in parallel, just like any other request.
> Or, let's say a file inside the ZIP was corrupt (or would that make
> the whole
> ZIP corrupt? What happens then?). If the browser can notice this, will
> it try
> to read the file referenced in the page instead?
Probably not, no. How would you detect "corrupt"? You might be able to
in some cases, but then surely the fix is to fix the zip file.
> And, a quite silly question, but to be ready for all situations, are
> multiple resource packages allowed on the same page? (While it would
> be quite dumb to do it, I don't see why it shouldn't be allowed)
It wouldn't be at all dumb, and it's certainly allowed.
Gerv
Gervase Markham
> I also have a suggestion. We know that some files are more important
> than others on some sites, this is especially true on large sites
> loaded on slow connections. How about a way to set priority to files
> inside manifest.txt?
ZIP files contain files in a specific order; put the most important
files first, and they will be downloaded and used first.
Gerv
Gervase Markham
> But there's on issue, which Darkyndy already adressed in the first
> comment: Viruses.
Resource packages don't introduce any new virus issues that I can see.
The browser is not going to be executing any native code downloaded
without asking the user.
> How do you think this should be adressed? With such a ZIP File to be
> downloaded almost anything could come along with it.
But that is also true of an <a href="...">.
Gerv
Gervase Markham
> How would this deal with CDNs / files hosted on a completely different
> domain?
The CDN could distribute the ZIP file.
> I'm having trouble understanding the third bullet about charsets in
> "Additonal notes". Zip files (with extra field 0x7075, UPath) can
> contain UTF8 file names; what are the charsets here referring to? The
> textual content of plain text files?
Yes.
> Are multiple <link rel="resource-package"> elements allowed per page
> (say, to have different caching policies)? If those elements end up
> containing the same resource, who wins?
Yes. But it's a good question as to which wins if they both contain the
same resource name but different resource data. I'd say the behaviour is
undefined (i.e. "don't do that").
> How would that affect caching policy, for both requests from the current
> host, and for requests from actual documents hosted on the CDN host?
> What happens if the zip file has a different cache header than the
> normal, unpackaged resource? Possibly the first one to be used wins, I
> guess.
The normal unpackaged resource isn't downloaded if the resource is in
the ZIP file. So the ZIP file's caching headers govern all the resources
in it.
Gerv
Magne Andersson
Will all "proper" ZIP-programs contain them in that order? I mean,
files in a classic folder can easily be sorted after name, or date of
creation, or type.
Mariusz Nowak
I think this is the real solution and CSS Sprites are just workaround
that was developed because this is one was missing.
But (!) if there could be solution to that problem at HTTP level, I'd
rather focus on that, let's cut it at the source.
Magne Andersson
>
> Gerv
What I meant by that was that the goal of this proposal was to reduce
the number of requests. By having multiple packages, you add at least
one more request without any reason.
Ictinus
resource (oops someone forgot to zip!) Especially when the resources
in question are not maintained in a file system.
eg. IBM/Lotus Domino holds the resources within another file structure
(.nsf)
I would much prefer to have the server dish up a single (cached)
gzipped, minified resource to the browser.
See www.dominoexperts. com/dapInfo
Magne Andersson
Well, obviously, you wouldn't include a frequently edited resource in
the ZIP, but most sites have their resources static for quite a long
time.
Sander
> On 17/11/09 08:40, Mook wrote:
>> Are multiple <link rel="resource-package"> elements allowed per page
>> (say, to have different caching policies)? If those elements end up
>> containing the same resource, who wins?
>
> Yes. But it's a good question as to which wins if they both contain the
> same resource name but different resource data. I'd say the behaviour is
> undefined (i.e. "don't do that").
I'd say this really _should_ be defined clearly before people start to
rely on any particular (perceived) behaviour. I thought of several
strategies where I'd have a large static "main" resource package, but
would occasionally want to be able to override a file from it on a
case-by-case basis with a different version of that file specified in a
second resource package. It'd be hacky, but it'd also be really useful.
But then I realized that this'd mean that content from later resource
packages could potentially override content from earlier resource
packages, which would mean that the browser would have to wait for all
resource packages to be downloaded before it'd be capable of doing
certain things, which'd be a bad thing. (The same situation exists in
reverse if a second small resource package finishes loading before a
first large resource package, containing the same resource, though. If
only speccing "whichever file gets seen first by the browser" wouldn't
be so indeterminate...)
In general, the entire capability of defining multiple resource package
probably deserves a more explicit mention in the spec than being tucked
away inside an "additional note" about headers. Splitting up available
resources into multiple packages was the first thing I (working web
developer) wanted to ask about after reading the spec. On any
non-trivial site I'll want to spit up my resources into two or three
resource packages per page (one global site-wide, cached forever, one
high priority (maybe reusable per section), and one with whatever is
left over and specific for the page).
Sander
Ictinus
I agree, perhaps a ZIP file would be useful in the short term, but a
server managed solution long term would allow all resources to get the
performance benefit.
Not having to manage the zip file makes a developers life easier.
Having it server side means more benefit. After all we already do gzip
server side.
semper.cras
Basic ZIP programs do NOT let the user to (re-)sort files in archive
manually (tried Gnome's File Roller and basic WinXP archivator). So
the only way that'll probably work is to create archive and than add
files there one by one, which is by no means elegant and transparent.
Moreover, even after this one cannot be sure that the order is right,
because there's no obvious way to check it.
And now imagine that I want to change the order of two images in the
zip file with say 20 files. Do I have to unzip it and then zip all the
files one by one to make them go in the need order?
So, there's a problem. If we set priority in the manifest file and the
real priority (in the zip file) is different, we can lose the
advantage of not waiting for the zip to download completely. Maybe the
partial solution would be promoting/creating handy tools for managing
file order in the archive?
Magne Andersson
I don't think you understood what I meant, I said that this solution
is great for resources that doesn't update frequently. This is also
made to eliminate requests, GZIP doesn't do that. It just makes the
files smaller.
Magne Andersson
Or ignore the order of the files in the ZIP if priority is already
specified in the manifest file.
Duncan
> Basic ZIP programs do NOT let the user to (re-)sort files in archive
> manually (tried Gnome's File Roller and basic WinXP archivator). So
> the only way that'll probably work is to create archive and than add
> files there one by one, which is by no means elegant and transparent.
> Moreover, even after this one cannot be sure that the order is right,
> because there's no obvious way to check it.
Most command line zip programs (e.g. Info-zip) will read a list of
files and zip them in the order specified. So something like:
zip resource.zip manifest.txt -@ <manifest.txt
should be all you need to create you a resource file with manifext.txt
first and then all the files listed in manifest.txt in the order they
appear. That's both elegant and transparent.
On another note, what I like about this proposal is that you can
potentially speed up browsing even for old broken browsers by adding
support for resource packages to a proxy server. So everyone wins.
Ictinus
No, I understand, and I agree (short term), but I believe a (long
term) better solution is one where we don't need to do the extra work
(maintain separate zip files) not matter how infrequently.
A server generated/managed single resource (combined js/css/image)
minified (where appropriate), gzipped, and with appropriate headers to
allow caching (based on similar decisions described in the above ZIP
suggestion), would provide the same performance benefits, but without
the developer needing to do the extra work (ie. let the server do the
work).
Perhaps the 'manifest' idea could be the actual request and the server
would read the manifest and build the single combined resource. If the
manifest is requested frequently, the resulting single resource can be
cached and not even re-built.
Joris
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
> Summary:
> What if there was a backwards compatible way to transfer all of the
> resources that are used on every single page in your site — CSS, JS,
> images, anything else — in a single HTTP request at the start of the
> first visit to the page? This is what Resource Package support in
> browsers will let you do.
This could be taken one step further: Allow direct linking to a file
in such a package (alongside the current proposal).
So I propose support is added to format urls like these examples:
http://www.mysite.com/resource.zip/contact.html ,
file:///home/joris/test/resource.zip/contact.html . If I recall
correctly, Firefox already uses something similar internally, but I
could be wrong.
index.html or a file defined in the manifest.txt would be the default
file, similar to DirectoryIndex in apache.
The above, in combination with a new file extension (.har?), would be
an excellent way to distribute documentation, all you need is one zip
file on your harddisk.
Henri Sivonen
jar and ODF?
The benefit of reuse would be the ability to use the jar tool to create
archive files that put the manifest first in the zip file.
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
Magne Andersson
> Is there a reason not to use the same manifest location already used by
> jar and ODF?
>
> The benefit of reuse would be the ability to use the jar tool to create
> archive files that put the manifest first in the zip file.
>
> --
> Henri Sivonen
For the lost ones, which is the location of the manifest file in jar
and ODF?
Jesper Kristensen
>> Are multiple <link rel="resource-package"> elements allowed per page
>> (say, to have different caching policies)? If those elements end up
>> containing the same resource, who wins?
>
> Yes. But it's a good question as to which wins if they both contain the
> same resource name but different resource data. I'd say the behaviour is
> undefined (i.e. "don't do that").
Please don't. It is hard to make 4 browser engine makers, who all know
what they are doing, follow a spec. It is impossible to make thousands
of web developers, some of whom don't have a clue about what they are
doing, follow a spec. Web developers should be able to expect that if
something works in one browser, it should also work in another.
Jesper Kristensen
> I also have a suggestion. We know that some files are more important
> than others on some sites, this is especially true on large sites
> loaded on slow connections. How about a way to set priority to files
> inside manifest.txt?
>
> From your example:
>
> PRIORITY: 1
> styles/reset.css
> styles/grid.css
> styles/main.css
> PRIORITY: 2
> javascript/jquery.js
> PRIORITY: 3
> images/save.png
> images/info.png
How would you implement such thing? The files are all located in one zip
file, and the zip file is downloaded from beginning to end, so the
priorities can be used for nothing. Except the browser may do multiple
HTTP Range requests for individual files in the resource to download the
most important first, but doesn't that defeat the whole purpose of this?
Jesper Kristensen
> I agree, perhaps a ZIP file would be useful in the short term, but a
> server managed solution long term would allow all resources to get the
> performance benefit.
> Not having to manage the zip file makes a developers life easier.
> Having it server side means more benefit. After all we already do gzip
> server side.
But a zip file is exactly a server managed solution that you request,
isn't it? No one says the zip file has to be created by the developer
manually. It can be generated by the server. You could use a server
configuration file or you could even do:
<link rel="resource-package"
href="site-resources.php?files=a.css,b.js,c.css,d.jpg" />
Jesper Kristensen
> This could be taken one step further: Allow direct linking to a file
> in such a package (alongside the current proposal).
> So I propose support is added to format urls like these examples:
> http://www.mysite.com/resource.zip/contact.html ,
> file:///home/joris/test/resource.zip/contact.html . If I recall
> correctly, Firefox already uses something similar internally, but I
> could be wrong.
Yes, you can just use
jar:http://www.mysite.com/resource.zip!contact.html but that is not
backwards compatible.
Magne Andersson
I thought one advantage with using ZIP in this case was that it didn't
have to be downloaded from beginning to end, but instead could
download styles and apply the previously downloaded at the same time?
"Can be unpacked even in partial state — which means that we can
stream the file, and put CSS and JavaScript first in the archive, and
they will unpacked and made available before the entire file has been
downloaded."
Magne Andersson
software, always having an up to date ZIP-file when uploading.
Neil
>I have published the first public draft on Resource Packages, and would like to hear your feedback
>
So, which resources does this apply to? I have only seen Images, style
sheets and scripts mentioned, but someone might want to supply audio
clips, or embedded SVG documents, or arbitrary frames?
--
Warning: May contain traces of nuts.
Magne Andersson
Don't forget fonts, those takes an enormous time to load ;)
I don't see why this can't apply to every form of file that can be
stored inside a ZIP-file/folder?
Jesper Kristensen
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
>
> http://limi.net/articles/resource-packages/
First of all, I think this is great and should be done sooner than later.
Is something like this allowed?
<link rel="resource-package"
href="http://some.other.site.com/site-resources.zip" />
It seems to be since you mention CDNs. What is the security implications
of this? A site cannot download info from another site unless in special
cases for historical reasons or if HTTP Access Control is used. E.g. a
site can include JS, CSS and JavaScript from another site and by
analyzing how the included code influenced the site, it can partly
detect whatever might have been in the file. Sites protect these things
by not displaying authenticated information in files, which conforms to
the syntax of these formats. Does current sites also have this
protection for the zip format. If not, they would have to add it or this
feature would make them vulnerable to information theft.
You say that the mainfest must be first in the zip. What happens if it
comes in the middle? Will it be ignored or used anyway? What happens if
a manifest does not list a file but it is still in the zip? I think it
is important to document browser behavior on these things, as we cannot
predict web developer behavior.
The purpose of the manifest file seems to be to make it faster to answer
"no" to the question of whether a file is in the zip or not. I see the
benefit of being able to answer "no" quickly to start downloading the
resource quicker. But do we need a manifest for that, or does the
manifest just adds unneeded complexity to the spec? You just have to
arrange the file structure on the server such that files packed in the
zip are located in one directory and all other files not. The browser
does not need a manifest to know that http://example.com/foo.css cannot
possibly be in the resource package http://example.com/static/resources.zip
Regarding the notes on MIME Type: While true that browsers must use
heuristics today to make this work, it is seen time and again on live
sites that this fails, and pages are not displayed correctly. We have to
live with that for those sites, who does not declare these informations
correctly, but making this the only option seems to spell trouble to me.
The spec should define standard mappings from file names to MIME types,
or maybe even allow mime types to be declared in the manifset.
Assuming that files inside the zip are UTF-8 seems really bad. Most text
based formats have set rules for defining character encoding inside the
files, like CSS with its @charset declaration and fallback to the
document encoding. Changing how this works is incompatible with existing
standards and practice and I could imagine it would break things.
Jesper Kristensen
>> How would you implement such thing? The files are all located in one zip
>> file, and the zip file is downloaded from beginning to end, so the
>> priorities can be used for nothing. Except the browser may do multiple
>> HTTP Range requests for individual files in the resource to download the
>> most important first, but doesn't that defeat the whole purpose of this?
>
> I thought one advantage with using ZIP in this case was that it didn't
> have to be downloaded from beginning to end, but instead could
> download styles and apply the previously downloaded at the same time?
>
> stream the file, and put CSS and JavaScript first in the archive, and
> they will unpacked and made available before the entire file has been
> downloaded."
Yes, files are available as they are downloaded. That is a feature of
the zip format. But that does not allow files to be downloaded in
another order than the order they are placed in the zip file.
Paweł Gawroński
manifest.txt.
It could be like mime-headers (but in utf-8) and one of those headers
should be "Version: 1.0".
Example file:
Version: 1.0
css/style.css
...etc.
Of course, if no headers are provided, specs fall back to 1.0.
This should simplify extending this protocol and provide space for
ideas like prioritization, http-headers overriding, language specific
blocks,
external resources (like "Import: [URL]"), etc. I hope you like this
idea.
PG
Magne Andersson
<moznewsgro...@something.to.remove.jesperkristensen.dk> wrote:
> Magne Andersson skrev:
>
> >> How would you implement such thing? The files are all located in one zip
> >> file, and the zip file is downloaded from beginning to end, so the
> >> priorities can be used for nothing. Except the browser may do multiple
> >> HTTP Range requests for individual files in the resource to download the
> >> most important first, but doesn't that defeat the whole purpose of this?
>
> > I thought one advantage with using ZIP in this case was that it didn't
> > have to be downloaded from beginning to end, but instead could
> > download styles and apply the previously downloaded at the same time?
>
> > stream the file, and put CSS and JavaScript first in the archive, and
> > they will unpacked and made available before the entire file has been
> > downloaded."
>
> Yes, files are available as they are downloaded. That is a feature of
> the zip format. But that does not allow files to be downloaded in
> another order than the order they are placed in the zip file.
But with a manifest.txt file which defines which files the ZIP
contains, it can skip ahead and download and apply the prioritized
files first.
Eddy Nigg
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
> What if there was a backwards compatible way to transfer all of the
> resources that are used on every single page in your site — CSS, JS,
> images, anything else — in a single HTTP request at the start of the
> first visit to the page? This is what Resource Package support in
> browsers will let you do.
>
Seems like maintenance headache to me, of course scripting to your help
eventually. Except if the package could include all resources, download
it once and reuse it for a complete site for whatever request from that
site comes along, e.g per site package, reusing the same package over
and over again for every page, caching the content of the zip obviously.
--
Regards
Signer: Eddy Nigg, StartCom Ltd.
XMPP: star...@startcom.org
Blog: http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg
Jesper Kristensen
> On 17 Nov, 14:08, Jesper Kristensen
> <moznewsgro...@something.to.remove.jesperkristensen.dk> wrote:
>> Magne Andersson skrev:
>>
>>>> How would you implement such thing? The files are all located in one zip
>>>> file, and the zip file is downloaded from beginning to end, so the
>>>> priorities can be used for nothing. Except the browser may do multiple
>>>> HTTP Range requests for individual files in the resource to download the
>>>> most important first, but doesn't that defeat the whole purpose of this?
>>> I thought one advantage with using ZIP in this case was that it didn't
>>> have to be downloaded from beginning to end, but instead could
>>> download styles and apply the previously downloaded at the same time?
>>> stream the file, and put CSS and JavaScript first in the archive, and
>>> they will unpacked and made available before the entire file has been
>>> downloaded."
>> Yes, files are available as they are downloaded. That is a feature of
>> the zip format. But that does not allow files to be downloaded in
>> another order than the order they are placed in the zip file.
>
> But with a manifest.txt file which defines which files the ZIP
> contains, it can skip ahead and download and apply the prioritized
> files first.
But skipping ahead means multiple HTTP requests. And the whole point is
to avoid those. So the simplest way of implementing priority would be to
do it the old way, as the performance gain would probably be lost anyway.
Bernhard H.
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
> Summary:
> What if there was a backwards compatible way to transfer all of the
> resources that are used on every single page in your site — CSS, JS,
> images, anything else — in a single HTTP request at the start of the
> first visit to the page? This is what Resource Package support in
> browsers will let you do.
Why not make pipelining work? (Or provide a fallback)
If you have a page with 20 images (like thumbnails) + css+ sprite +
js, you might save 2 requests with your .zip package. All other image
request are hardly predictable, dynamically creating .zips on the
server is not really an option (too much CPU and bad maintainability,
would slow down everything again)
On the other hand for pipelining you don't have to do anything and it
would save the time of 22 reqests.
Magne Andersson
<moznewsgro...@something.to.remove.jesperkristensen.dk> wrote:
> Magne Andersson skrev:
>
>
>
> > On 17 Nov, 14:08, Jesper Kristensen
> > <moznewsgro...@something.to.remove.jesperkristensen.dk> wrote:
> >> Magne Andersson skrev:
>
> >>>> How would you implement such thing? The files are all located in one zip
> >>>> file, and the zip file is downloaded from beginning to end, so the
> >>>> priorities can be used for nothing. Except the browser may do multiple
> >>>> HTTP Range requests for individual files in the resource to download the
> >>>> most important first, but doesn't that defeat the whole purpose of this?
> >>> I thought one advantage with using ZIP in this case was that it didn't
> >>> have to be downloaded from beginning to end, but instead could
> >>> download styles and apply the previously downloaded at the same time?
> >>> stream the file, and put CSS and JavaScript first in the archive, and
> >>> they will unpacked and made available before the entire file has been
> >>> downloaded."
> >> Yes, files are available as they are downloaded. That is a feature of
> >> the zip format. But that does not allow files to be downloaded in
> >> another order than the order they are placed in the zip file.
>
> > But with a manifest.txt file which defines which files the ZIP
> > contains, it can skip ahead and download and apply the prioritized
> > files first.
>
> But skipping ahead means multiple HTTP requests. And the whole point is
> to avoid those. So the simplest way of implementing priority would be to
> do it the old way, as the performance gain would probably be lost anyway.
You got me there. Good, you made me learn something. (I know that the
goal is to reduce them, but I didn't know that it would generate more
of them inside the same file.)
Ted Mielczarek
bernhard....@googlemail.com> wrote:
> Why not make pipelining work? (Or provide a fallback)
>
We've tried this, there are still issues with the real web:
https://bugzilla.mozilla.org/show_bug.cgi?id=414477 - https should run with
pipelining
https://bugzilla.mozilla.org/show_bug.cgi?id=422978 - pipelining breaks
secure Internet banking websites
https://bugzilla.mozilla.org/show_bug.cgi?id=264354 - Enable HTTP
pipelining by default
-Ted
Ant Bryan
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
> Summary:
> What if there was a backwards compatible way to transfer all of the
> resources that are used on every single page in your site — CSS, JS,
> images, anything else — in a single HTTP request at the start of the
> first visit to the page? This is what Resource Package support in
> browsers will let you do.
A couple comments.
Connection limits were removed from HTTPbis recently.
"You are probably familiar with the issue; a well-known optimization
technique is to reduce the number of HTTP requests that are done for a
given web site, since browsers only do 2–6 requests in parallel."
Missing "be"?
"Can be unpacked even in partial state — which means that we can
stream the file, and put CSS and JavaScript first in the archive, and
they will BE unpacked and made available before the entire file has
been downloaded."
Do you mean "...omit it in documents where it is INvalid..."?
"The default MIME type for a resource package is application/zip, and
you can omit it in documents where it is valid, like in HTML5, where
an equivalent would be:"
--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads
Boris Zbarsky
> Why not make pipelining work? (Or provide a fallback)
Because the pipelining breakage tends to be on the server side (or
rather transparent proxy side), not on our side. And we don't control
that code. And neither does the person authoring the website,
typically. And it's been broken for close to a decade now, with no
improvement in sight....
> On the other hand for pipelining you don't have to do anything and it
> would save the time of 22 reqests.
Not quite. It would save the cost of spinning up 22 connections; you
still have to make 22 separate requests on the one connection. Still a
big win, of course.
Note that you're assuming that there will be a sprite. One of the
points of the package idea is that you no longer have to sprite and take
the resulting memory and performance hits during rendering.
-Boris
Boris Zbarsky
> This could be taken one step further: Allow direct linking to a file
> in such a package (alongside the current proposal).
What's the benefit? It wouldn't be backwards compatible, so not usable
on the open web easily.
> The above, in combination with a new file extension (.har?), would be
> an excellent way to distribute documentation, all you need is one zip
> file on your harddisk.
It seems like a better use of time to implement support for existing
standards for this sort of thing (e.g. MHTML).
-Boris
Boris Zbarsky
> But (!) if there could be solution to that problem at HTTP level
Then you'd need cooperation from HTTP servers, HTTP proxies, etc, no?
Much bigger barriers to uptake, seems to me. Or did you have a
particular solution in mind that avoids that?
-Boris
sep332
wrote:
The amount of CPU is configurable, all the way down to ~0. The .jpg's
can be added without any compression, basically concatenated with the
rest of the file. This means you can configure the server to
practically just add a .zip header, 20 random .jpg's, and a .zip
footer, which will still benefit from avoiding multiple HTTP requests.
Magne Andersson
be saved. So, if anyone is good at predicting that through this
method, let's take the http://www.cnn.com homepage which is heavy on
both JS and SS. How much in percent, can be saved, do you think?
Laurence Rowe
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
> Summary:
> What if there was a backwards compatible way to transfer all of the
> resources that are used on every single page in your site — CSS, JS,
> images, anything else — in a single HTTP request at the start of the
> first visit to the page? This is what Resource Package support in
> browsers will let you do.
Great work Alex! I'm really glad to see this moving forward, both for
general browsing performance and because it could make offline mode
much more interesting.
* On caching, I imagine for static resource use case ETags should
normally be avoided in favour of unique resource-package ids (similar
to Plone's resource registry). An Etag would require a conditional get
to revalidate, a roundtrip that can be avoided if the page uses
references such as:
<link rel="resource-package" href="/static/site-
resources-54642.zip" />
* For offline mode especially, I think it would be helpful to define
the behaviour when multiple resource-pacakge links are defined for the
same path. Over at http://code.google.com/p/gears/issues/detail?id=36#c3
I made a suggestion for having layered resource stores. This would
enable an application with offline mode to synchronise changes without
downloading the entire resource store again. For example, on the first
sync, the browser sees a linked resource package as so:
<link rel="resource-package" href="/documents/offline-0.zip" />
---
manifest.txt
doc1.html
doc2.html
---
Offline the application would be able to access documents at /
documents/doc1.html and /documents/doc2.html. The user then performs
another sync and the browser now sees:
<link rel="resource-package" href="/documents/offline-1.zip" />
---
manifest.txt
doc1.html
doc3.html
---
<link rel="resource-package" href="/documents/offline-0.zip" />
doc1.html and doc3.html would now be read from the offline-1.zip,
doc2.html would continue to be read from offline-0.zip.
Laurence
Gervase Markham
> Please don't. It is hard to make 4 browser engine makers, who all know
> what they are doing, follow a spec. It is impossible to make thousands
> of web developers, some of whom don't have a clue about what they are
> doing, follow a spec. Web developers should be able to expect that if
> something works in one browser, it should also work in another.
Lots of specs say "the behaviour in this case is undefined" in order to
let makers implement whatever algorithm is best for them.
Perhaps it's not the best thing in this case, but it's not always a bad
idea.
Gerv
Robert Accettura
> Overall I really like the idea. We're pretty much heading in this
> direction anyway (1 large CSS file, 1 large JS file + sprites).
>
> An addition I'd add would be a way to have multiple manifest files.
> Currently a way to speed up websites/pages is to only load what is
> required for the current page and post-load the rest. An example would
> be a homepage would only load the manifest file it needs to make it
> load faster, but post-load another manifest file. Or another page can
> load a different manifest file later on. Considering some sites are
> huge, their manifest files could get far too large for most internet
> connections.
>
I'd like to echo this suggestion.
It would also make it much easier to integrate into build systems.
For example in one site I work on, the css, images, and js are
effectively separate projects. This works when everything is deployed
separately, however it would make the process of building 1 manifest
rather hacky or a major effort. Consolidating down to 3 requests
would be a balance of easy implementation and performance gain.
> And making sure browsers can pull resources out of the manifest file
> *before* it has completed downloading should be a requirement.
> Currently browsers deal with missing/incomplete files as gracefully as
> they can, I'd like that graceful degradation to continue.
>
> The ordering of assets should be CSS, JS then images inside the
> manifest file.
>
> Might even want a way to notate if the JS should be parsed after the
> page is ready or not (defer). (Equivalent of putting it at the bottom
> of the page)
>
My understanding is that they would be rendered/parsed when referenced
on the page, not when downloaded. So essentially it would be like
getting out of disk cache vs. a network fetch. Should be clarified in
any spec though.
Mithgol the Webmaster
Jo> This could be taken one step further:
Jo> Allow direct linking to a file in such a package (alongside the current
Jo> proposal).
Jo> So I propose support is added to format urls like these examples:
Jo> http://www.mysite.com/resource.zip/contact.html ,
Jo> file:///home/joris/test/resource.zip/contact.html
Jo> If I recall correctly, Firefox already uses something similar
Jo> internally, but I could be wrong.
Jo> index.html or a file defined in the manifest.txt would be the default
Jo> file, similar to DirectoryIndex in apache.
Jo> The above, in combination with a new file extension (.har?), would be
Jo> an excellent way to distribute documentation, all you need is one zip
Jo> file on your harddisk.
If made as you proposed, that'll break the backwards compatibility of the URLs.
Instead, use a server-side solution like this:
<link rel="resource-package"
type="application/zip"
href="/zip-storage/site-resources.zip" />
<img src="/zip-storage/unpacker.php/filename.png" />
If the browser supports resource packages, it gets "unpacker.php/filename.png"
from the archive (where "unpacker.php" is the subdirectory name).
If the browser does not support resource packages, then the PHP script
"unpacker.php" is started on the server, and gets "/filename.png"
as its PATH_INFO, and the script should unpack the file from the archive.
Writing the "unpacker.php" script is relatively simple task, it doesn't require
years of PHP experience:
<?php
if ( $zip = zip_open("site-resources.zip") ) {
while ( $zip_entry = zip_read($zip) ) {
if ( '/'.zip_entry_name($zip_entry) === $_SERVER["PATH_INFO"] ) {
if ( zip_entry_open($zip, $zip_entry, 'rb') ) {
$buffer = zip_entry_read(
$zip_entry,
zip_entry_filesize($zip_entry)
);
echo $buffer;
zip_entry_close($zip_entry);
break;
}
}
}
}
?>
Note: the above PHP code is untested and may contain bugs.
With best Fidonet 2.0 regards,
Mithgol the Webmaster.
.. 4. Shooting is *not* too good for my enemies.
tg...@mozilla.com
>> Or, let's say a file inside the ZIP was corrupt (or would that make
>> the whole
>> ZIP corrupt? What happens then?). If the browser can notice this, will
>> it try
>> to read the file referenced in the page instead?
Zip files support CRC checksums. It may make sense to enable CRC checks
for files read off network.
Taras
chris hofmann
It seems like there might need to be some research to define some real
world performance wins.
There appear to be two parts to the proposal. One is a compression part
and the other is a connection overhead part.
Standards have been created and adopted by the browsers on the
compression part, and some sites even have content systems that try to
use that optimization
see: .gz work from the past?
http://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/
the trade off has always been the overhead of building such management
systems.
It sounds to me like what the archive+compression proposal needs to
demostrate is what the incremental performance win is over just the
compression capabilities that already exist, and if it really makes
anything easlier on the management side. if the management challenges
stay the same, or get worse, I don't think we will see any greater
adoption of this as a standard or something that gets used on sites.
-chofmann
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
Justin Dolske
That depends entirely on the user's connection.
For a CNN webdev sitting next to the server on gigabit ethernet, it
would be hard to tell the difference. For a user in the middle of
nowhere using a high-latency / high-bandwidth satellite data connection,
the difference should be enormous. Everyone else falls somewhere in
between. :)
Justin
skierpage
> ...Allow direct linking to a file ...
> The above, in combination with a new file extension (.har?), would be
I believe that's what Sun developed JAR files for, and the jar:
protocol is a way to refer to files within ZIP files using a URL. The
result is a browsable single file that just works (with some bugs and
security implications ;-) ) for a collection of web pages and
resources such as documentation or a Wikipedia slice. jar: wraps both
http and file protocols, so users can browse the compressed content on
the web or browse a local saved copy. Alas Firefox won't offer to
view ZIP or jar files using its jar protocol implementation (I filed
bug 502528), but it's easy enough to do, simply surround the URL to a
remote or local ZIP file with jar: and !/ . For remote files you must
enable network.jar.open-unsafe-types in about:config. Some examples:
jar:https://bugzilla.mozilla.org/attachment.cgi?id=267646!/
index.html
jar:http://wiki.laptop.org/images/f/fe/WikisliceChemistry-en-8.xol!/
WikisliceChemistry-en/files/articles/Carbon.htm
jar:http://newton.codecauldron.org/site/community/opendocument-
writer.odt!/
The first URL shows the main use of jar files I've come across,
putting a complex HTML test case in a bugzilla attachment. See
http://weblogs.mozillazine.org/stephend/archives/2007/06/how_to_create_a.html
The last URL shows Firefox is a credible ZIP file viewer, which was
news to me!
I think the Mozilla codebase is the only major browser that implements
the jar: protocol :-( , one reason why Mr. Limi dismissed it in his
spec for making browsers faster.
Bernhard H.
I do mostly agree.
Compressing .jpeg seems considerable (deflate nearly 60%), depending
on connection speed and the actual JPEGs.
Not trying to compress the .png's makes zipping nearly as fast as
creating a tar archive, while keeping the size small.
You can use it like that:
zip -n .png static.zip img/* js/*
real 0m0.016s
user 0m0.008s
sys 0m0.000s
336K static.zip
seems way better than these:
zip static_png.zip img/* js/*
real 0m0.035s
user 0m0.020s
sys 0m0.008s
332K static_png.zip
tar -cvf static.tar img/* js/*
real 0m0.005s
user 0m0.000s
sys 0m0.004s
416K static.tar
You save (80KB-overhead)*KB/ms + time for the requests + ((20-overhead)
x~30ms) - 12 ms - time to uncrompress
which should be >0 most times.
(This is of course only sample data, but it would clearly work with my
site, so if you can't do pipelining as Opera please support zip files)
Gervase Markham
> I'd like to echo this suggestion.
The spec already supports multiple manifest files.
Gerv
Stephan Wehner
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
>
> http://limi.net/articles/resource-packages/
>
> Summary:
> What if there was a backwards compatible way to transfer all of the
> resources that are used on every single page in your site — CSS, JS,
> images, anything else — in a single HTTP request at the start of the
> first visit to the page? This is what Resource Package support in
> browsers will let you do.
Looks like a nice idea. It will keep some programmers busy for
sure :-) At the same time, craigslist has implemented a solution
already (minimal design overhead).
Three suggestions:
* It may also be nice to allow placing several HTML files in the zip
file (Contact Us, About Us)
* The HTTP request sent from the browser may be zipped as well ("I
already have all these files")
* Your plan opens up supporting encryption, which the ZIP format
supports, over http (although it would need some key exchange of some
kind).
But I am mainly writing to mention an issue with your choice of the
ZIP format. At http://en.wikipedia.org/wiki/ZIP_%28file_format%29#The_structure_of_a_ZIP_file
it says,
"The location of a file is indicated in the so called central
directory which is located at the end of the ZIP file. "
If you look at the graphic below ( http://en.wikipedia.org/wiki/File:ZIPformat.jpg
), the browser can't do much until the whole ZIP file is downloaded
over the network,
which looks like an unnecessary delay. Especially in the long run. Am
I overlooking something? I guess you can say you start with ZIP, and
add support for other formats later.
Stephan
Steven Roussey
are going to end up with lots of manifest files: manifests for the
various zip files, and manifests for offline (and manifests for google
gears offline).
Personally, I'd rather see the manifest file as telling the browser
the preferred order to start downloading resources at which point it
could start immediately, regardless of any parsing of the document.
Then use HTTP/1.1 standard pipelining, and I think most of benefit of
the zip file would be achieved with no additional maintenance work for
the web developer (in zip files). And to deal with being scared of
turning pipelining on by default, it could be enabled just on this
manifest request.
In addition, I don't think having pipelining on by default is a bad
thing. Some sites don't work? So what? There is a big database of
attack sites and web forgeries that Firefox checks anyhow. Why not
augment that with 'poor-performance' sites? Publicly list them.
Disable pipelining ("performance mode!") on them.
-steve--
Boris Zbarsky
> In addition, I don't think having pipelining on by default is a bad
> thing. Some sites don't work? So what? There is a big database of
> attack sites and web forgeries that Firefox checks anyhow. Why not
> augment that with 'poor-performance' sites? Publicly list them.
> Disable pipelining ("performance mode!") on them.
We don't maintain the attack site and web foregery databases and don't
have the infrastructure to do so, really.
If someone wants to maintain a database of sites that break with
pipelining, including the various intranet sites, etc, then I think we'd
be pretty interested in making use of such a database....
-Boris
Henri Sivonen
<dc105301-dea3-446e...@s15g2000yqs.googlegroups.com>,
Magne Andersson <zirr...@gmail.com> wrote:
> On 17 Nov, 13:11, Henri Sivonen <hsivo...@iki.fi> wrote:
> > Is there a reason not to use the same manifest location already used by
> > jar and ODF?
> >
> > The benefit of reuse would be the ability to use the jar tool to create
> > archive files that put the manifest first in the zip file.
> For the lost ones, which is the location of the manifest file in jar
For jar, it is META-INF/MANIFEST.MF
> and ODF?
My recollection was wrong. ODF and jar have different manifest
locations. ODF has its manifest in META-INF/manifest.xml
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
Henri Sivonen
<5c4c6161-a5e3-4e8b...@z41g2000yqz.googlegroups.com>,
Pawe�� Gawro��ski <pawel.g...@gmail.com> wrote:
> It could be like mime-headers (but in utf-8) and one of those headers
> should be "Version: 1.0".
What should the browser do when the version header is absent? What
should the browser do when there is a version header but its value isn't
"1.0"?
HTML5 offline manifests don't have version identifiers by design.
Henri Sivonen
Mook
<mook.moz+nntp.n...@gmail.com.please-avoid-direct-mail>
wrote:
> I'm having trouble understanding the third bullet about charsets in
> "Additonal notes". Zip files (with extra field 0x7075, UPath) can
> contain UTF8 file names; what are the charsets here referring to? The
> textual content of plain text files?
File name encoding in zip files is a huge can of worms. Basically, only
ASCII works interoperably.
http://www.google.com/search?q=jar+utf-8+marcos+site%3Alists.w3.org
http://www.google.com/search?q=zip+utf-8+marcos+site%3Alists.w3.org
Summary:
* UTF-8 in the .zip spec is a very recent addition.
* jar uses a legacy Java sort-of-UTF-8 that predates the UTF-8 .zip
spec addition.
* Mac OS X uses decomposed characters (basically, HFS+ leaks all over
on Mac OS X).
* Additional badness with implementations using their rogue encoding
choices (neither the ancient DOS code page nor any variant of UTF-8)
iGEL
The intention is good, but I believe, we should wait for Google's SPDY
instead. This approach has lots of disadvantages, the resource file
might contain more files than actually needed. The files might get out
of sync, if someone forgets to repack it. These things might resolved
by a server side logic, but if you change that, you could implement
Google SPDY instead. SPDY has some advantages over this approach
anyway: It requires only one TCP connection (this one will require ad
least two, unless you load the resource package after the HTML, more
connections for files not contained in the resource package), and
headers will be compressed. Also, the connection will live longer.
iGEL
Ted Mielczarek
> In addition, I don't think having pipelining on by default is a bad
> thing. Some sites don't work? So what? There is a big database of
> attack sites and web forgeries that Firefox checks anyhow. Why not
> augment that with 'poor-performance' sites? Publicly list them.
> Disable pipelining ("performance mode!") on them.
>
It turns out that when the site that doesn't work is someone's bank website,
people stop using your web browser. Idealism only gets you so far when you
have real users who have real things they need to get done. In addition, a
lot of the problems that were seen historically were with *proxies*,
including transparent proxies, so blacklisting specific sites or even
servers has no effect there.
I think it would be great to be able to ship with pipelining on, even in a
restricted capacity. Just flipping it on and seeing what breaks has not
proven to be a viable strategy, though. We could perhaps devise a
distributed testing strategy, where we provide a test plan for people to
use, which could run against a known-good server configuration, and report
if there's some proxy in the way breaking things, and additionally ask the
user to run against various websites they use on a daily basis and determine
if any of those servers break, and then report the results to us for
analysis. Even then, if we determine there are transparent proxies that are
broken, our only recourse is to ask the people running the proxies to fix
them.
-Ted
Steven Roussey
> On Wed, Nov 18, 2009 at 4:27 PM, Steven Roussey <srous...@gmail.com> wrote:
> > In addition, I don't think having pipelining on by default is a bad
> > thing. Some sites don't work? So what? There is a big database of
> > attack sites and web forgeries that Firefox checks anyhow. Why not
> > augment that with 'poor-performance' sites? Publicly list them.
> > Disable pipelining ("performance mode!") on them.
>
> It turns out that when the site that doesn't work is someone's bank website,
> people stop using your web browser. Idealism only gets you so far when you
> have real users who have real things they need to get done. In addition, a
> lot of the problems that were seen historically were with *proxies*,
> including transparent proxies, so blacklisting specific sites or even
> servers has no effect there.
So transparent proxies are the Achilles heal of http pipelining. So it
would basically take a replacement for http, or a new version
increment of http (HTTP/1.2 or whatever) to make it work. Assuming
that these proxies would die on protocol versions they didn't
understand, then the browser would notice and downgrade to HTTP/1.1.
If these assumptions held true, then someone with more karma than I
(which is to say, people with influence) should propose removing
pipelining from http/1.1 and moving it to 1.2. I'm sure politically,
people would try and get all their pet projects in there too, which
would be bad. But assuming not (big on assumptions here, and thus why
this proposal got started in the first place), most web servers (the
one's without the bugs -- more assumptions) would be able to update
their wares saying they accept 1.2 pretty easily.
> I think it would be great to be able to ship with pipelining on, even in a
> restricted capacity. Just flipping it on and seeing what breaks has not
> proven to be a viable strategy, though. We could perhaps devise a
> distributed testing strategy, where we provide a test plan for people to
> use, which could run against a known-good server configuration, and report
> if there's some proxy in the way breaking things, and additionally ask the
> user to run against various websites they use on a daily basis and determine
> if any of those servers break, and then report the results to us for
> analysis. Even then, if we determine there are transparent proxies that are
> broken, our only recourse is to ask the people running the proxies to fix
> them.
Mozilla has a distributed testing strategy, product, etc. in Mozilla
Test Pilot. Sounds exactly like something Test Pilot was build for.
The one thing I mentioned received no comment: using the manifest to
prioritize and prefetch items into the cache, even without a zip file.
-steve--
Steven Roussey
Apparently it doesn't matter with proxies mucking things up. But
having such an infrastructure will likely be necessary some day.
Everyone else does: MS/IE, Google/Chrome, and Apple/Safari (not quite
sure of their source material). I'm sure something is being worked on
though, there are enough tea leaves to look at.
-steve--
Michael Adams
like, may as well support 7z-style deflate for efficiency's sake. If
you want some real gains on the bandwidth side of things, we'll need
something even better: I've been tracking Matt Mahoney's research a
few years now, and him and his coders have GPL code for squishing
JPEGs.
http://www.mattmahoney.net/dc/
* Mainfests should be optional: if I make a package for my site and
have the browser dump those into the cache, then sites can package
content that stays put for a while and let the browser download the
rest as needed.
Gervase Markham
> File name encoding in zip files is a huge can of worms. Basically, only
> ASCII works interoperably.
In which case, we have a great opportunity to improve things, by
mandating the best existing method of supporting Unicode for this spec. :-)
Gerv
Ted Mielczarek
> But I am mainly writing to mention an issue with your choice of the
> ZIP format. At
> http://en.wikipedia.org/wiki/ZIP_%28file_format%29#The_structure_of_a_ZIP_file
> it says,
>
> "The location of a file is indicated in the so called central
> directory which is located at the end of the ZIP file. "
>
> If you look at the graphic below (
> http://en.wikipedia.org/wiki/File:ZIPformat.jpg
> ), the browser can't do much until the whole ZIP file is downloaded
> over the network,
> which looks like an unnecessary delay. Especially in the long run. Am
> I overlooking something? I guess you can say you start with ZIP, and
> add support for other formats later.
>
>
I think this is the most important response in this thread, and it raises a
critical problem with the proposal. I had forgotten this crucial detail
about the ZIP file format when I read the resource packages draft, but with
this as a given, you either need to use a format other than ZIP, or accept
that the browser will have to send at least three range requests to be able
to display files without having the entire archive downloaded (1st to
download the ZIP header, 2nd to download the central directory from the end
of the file, 3rd to download the rest of the data).
-Ted
Boris Zbarsky
> On Wed, Nov 18, 2009 at 1:06 PM, Stephan Wehner<stepha...@gmail.com>wrote:
>> "The location of a file is indicated in the so called central
>> directory which is located at the end of the ZIP file. "
> I think this is the most important response in this thread, and it raises a
> critical problem with the proposal.
Why?
The ZIP format has the following properties (based on that wikipedia
article, note):
1) The list of filename to position of file entry within archive
offsets is stored at the end of the file.
2) Some other metadata about files is stored at the end of the file.
3) The beginning of each file in the archive is clearly marked with a
specific 4-byte pattern.
4) Each file's entry includes its filename.
Due to property 1 and properties 3+4, an operation like "extract the
file named 'foo' from the archive" can be implemented in two ways:
* Look up "foo" in the directory and read data from the offset given by
that.
* Start scanning at the beginning of the archive looking for the file
entry start pattern. Each time it's found, compare the name of the
entry to "foo". Stop if found.
Obviously the first method is more efficient if nothing else is done.
However nothing prevents the browser from performing the scan of the
second method as the data streams in. The manifest tells you which
files you want to be getting out of the archive. You keep a list of the
files from that list that have been requested so far. As the archive
streams in, you look at the incoming data, and every time a new file
entry starts you compare its name available in the entry itself) to said
list; if someone asked for it, you hand it over. You also cache its
position in the archive for when someone else asks for it.
So in essence you rebuild the name+offset part of the directory as you
go, and can service requests in the order in which the files appear in
the archive (plus random order for the files that are in the part of the
archive you've already received, of course).
Alexander _did_ consider this part of things pretty carefully before
suggesting zip, as it happens...
-Boris
tg...@mozilla.com
>
> It seems like there might need to be some research to define some real
> world performance wins.
>
> There appear to be two parts to the proposal. One is a compression part
> and the other is a connection overhead part.
>
> Standards have been created and adopted by the browsers on the
> compression part, and some sites even have content systems that try to
> use that optimization
>
> see: .gz work from the past?
> http://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/
>
>
> the trade off has always been the overhead of building such management
> systems.
>
> It sounds to me like what the archive+compression proposal needs to
> demostrate is what the incremental performance win is over just the
> compression capabilities that already exist, and if it really makes
> anything easlier on the management side. if the management challenges
> stay the same, or get worse, I don't think we will see any greater
> adoption of this as a standard or something that gets used on sites.
I agree with Chris that we need to see some benchmarks. I also wonder if
it would be possible to prototype a transparent way to jar up a webpage
serverside.
If this zip thing isn't done right, we may end up with people doing
myth-based(as opposed to measurement) optimization with poorly-laid jar
files that hurt performance.
For example, browsers do (i'm not sure what firefox does) parallel http
connections to the the web server. Putting some important resource and
the end of a jar file will end up increasing delay before that item
renders significantly. These kind of errors may make the mobile surfing
experience worse, not better.
There may be some merit to doing an apache module that looks at referrer
headers to figure out which resources are requested with particular
pages and to generate these zip files automagically.
There are also implementation complexity issues to consider such as zip
file index being at the back of the file, whether to fetch files in
manifest or zip index order, etc.
Taras
tg...@mozilla.com
>
> It seems like there might need to be some research to define some real
> world performance wins.
>
> There appear to be two parts to the proposal. One is a compression part
> and the other is a connection overhead part.
>
> Standards have been created and adopted by the browsers on the
> compression part, and some sites even have content systems that try to
> use that optimization
>
> see: .gz work from the past?
> http://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/
>
>
> the trade off has always been the overhead of building such management
> systems.
>
> It sounds to me like what the archive+compression proposal needs to
> demostrate is what the incremental performance win is over just the
> compression capabilities that already exist, and if it really makes
> anything easlier on the management side. if the management challenges
> stay the same, or get worse, I don't think we will see any greater
> adoption of this as a standard or something that gets used on sites.
I agree with Chris that we need to see some benchmarks. I also wonder if
Joris
wrote:
I was not suggesting that the method in the article would be removed.
I was talking about a small addition that would help distribute
(downloadable) documentation in html form.
Ben Mathews
> Hi dev.platform,
>
> I have published the first public draft on Resource Packages, and
> would like to hear your feedback:
I think this is a great idea, but I'm curious why you chose .zip
instead of .tar.gz. I think that .tar.gz would generally produce
smaller files because of inter-file compression, and it still
maintains the other properties that you need, like progressive
unpacking.
To give an extreme example: here at Facebook, we have a large number
of small JavaScript files that we concatenate into larger packages.
For example, our base JS package is composed of 19 separate JS files.
If I split up those files and compress them, the .zip file is 14505
bytes, while the .tar.gz only takes 10366 bytes.
As I said, this is an extreme example: if we were to use these
resource packages, we would probably only have one JS file per
package. But .tar.gz would also allow the compressor to exploit
redundancies between CSS and JS, or between image headers. (Not to
mention that the file metadata is compressed in .tar.gz, also.)
-Ben
Boris Zbarsky
> I think this is a great idea, but I'm curious why you chose .zip
> instead of .tar.gz. I think that .tar.gz would generally produce
> smaller files because of inter-file compression, and it still
> maintains the other properties that you need, like progressive
> unpacking.
Does it? I thought the whole point of the inter-file compression
business was that you couldn't unpack the first file without also
unpacking some parts of some of the others.
-Boris
Ben Mathews
You're correct, but I think that's okay --- I don't think there's any
need to seek into the middle of the compressed stream.
Sure, you have to uncompress all previous files in order to access a
specific file, but I think that the browser should just uncompress the
whole resource package as it comes in. (Think of it as kind of like
a .tar file sent with "Content-Encoding: gzip" --- except that this is
better since there's no chance for intermediate proxies to disable
compression.)
Boris Zbarsky
> You're correct, but I think that's okay --- I don't think there's any
> need to seek into the middle of the compressed stream.
No, but there might be need to get files out of the beginning before
you've gotten some of the later data.
> Sure, you have to uncompress all previous files in order to access a
> specific file
Do you not also have to uncompress some future ones?
-Boris
Wawet76
be in UTF-8.
HTTP specs set default charset at ISO-8859-1. A same file will be
considered in this charset if fetched with a HTTP request, and in
UTF-8 if extracted from the resource package.
Charset handling is already hard with http headers, meta tags, charset
attributes... giving different fallback values to HTTP requested files
and resources packages files will add complexity, and could result in
nasty bugs hard to spot.
Ben Mathews
> > Sure, you have to uncompress all previous files in order to access a
> > specific file
>
> Do you not also have to uncompress some future ones?
No. Unlike bzip2 (which is block-based), gzip uses a stream-oriented
algorithm: all the data necessary to uncompress a symbol in the file
comes before that symbol.
You can see this with a command like 'head -c200 test.gz | zcat | wc -
c'. You can vary the number of input bytes and see that as you add
one byte at a time, the output length increases.
-Ben
Henri Sivonen
<507e9784-a94d-4666...@l35g2000vba.googlegroups.com>,
Wawet76 <thomas....@gmail.com> wrote:
> If no charset is specified, it could be a problem to assume files to
> be in UTF-8.
Whoa. I missed that. That's indeed a problem.
> HTTP specs set default charset at ISO-8859-1.
That's not how things work in reality. However, Resource Packages should
defer to the self-labeling mechanism of each file type (CSS @charset,
HTML <meta charset>, XML declaration).
This still poses the problem that files in the resource package vs.
files retrieved using plain HTTP may end up having out of sync effective
Content-Type values.
Gregg Tavares
I know I feel like I'm butting in here and I know his topic is 2 months
old but I wanted to point out this is false. It is not valid to start
scanning a zip file from the beginning looking for zip entries. Only the
directory says what parts of the file are valid data. It is a perfectly
valid file to have
offset : data
------------
0000 | offset to directory: 0400
0100 | zip entry "foo" (no longer valid)
0200 | directory (no longer valid)
"foo" : offset 0100
0300 | zip entry "foo"
0400 | directory
"foo": offset 0300
If you scan you'll see the first "foo" which is not a valid file. If you
scan for the directory you'll find the first one which also points to
invalid data. You have to seek to the valid directory and use the
offsets inside it. Anything else in the file is not part of the official
zip data though it's still a valid zip file
This was an intentional design of the original zip format which was
designed to run on machines with only floppy disks and so often there
was not enough space to add a file to an existing zip buy copying the
contents of the old zip to a new zip and then deleting the old one which
would require twice the space during the addition of files.
Instead they'd just append the new files (or updated files) to the end
of the existing zip, tack on a new directory and update the directory
offset.
Like others, I'd recommend going with gzipped tar files.
A few of reasons
1) Unlike zip, gzipped tar is streamable. (see above for why zip is not
streamable)
2) Gzipped tar compresses smaller for small files which is relatively
common for HTML/Web based stuff. This is because the headers are
compressed as well where as for zip they are not
3) Gzipped tar code is an HTTP standard. Why add more compression
formats when there is already one?
4) Because of #3 there is already gzipped tar code built into every
browser. (Then again, every browser has zip code too for png support)
To see an example of using gzipped tar as a streamable format see O3D
(o3d.googlecode.com) which uses gzipped tar files and can stream them so
when the first file in a gzipped tar has been downloaded it is
immediately available while the rest of the file keeps downloading. The
code is open source and goes through the NPAPI calls in Firefox so it's
probably already pretty close to the correct code in some small way.
We started with zip and specifically ditched it because of these issues.
Using gzipped tar files is no harder than using zip. On Mac and Linux it
is far more common. On Windows there is good free open source software
that will create them. 7zip will create them both from the command line
and from a gui.
There maybe be some reason to prefer zip but off the top of my head I
can't think of one.
Laurence Rowe
> On 11/20/2009 8:25 AM, Boris Zbarsky wrote:
>
>
>
>
>
> > On 11/20/09 6:46 AM, Ted Mielczarek wrote:
> >> On Wed, Nov 18, 2009 at 1:06 PM, Stephan
I know gzip is a standard part of http, but is tar? - I see no mention
of it in rfc1945 or rfc2616.
I agree that tar.gz is cleaner, but this is probably aiming for lowest
common denominator. There's always the option of using uncompressed
zip files transferred with Content-Encoding: gzip to benefit from
cross-file compression savings ;)
Laurence
Gregg Tavares
You're right. My bad. Still, gzipped tar is streamable, gzipped zip
is not. In fact any form of zip is not streamable. If the proposal
is to get data to the browser faster it seems like it should also
be about letting the browser use that data as fast as possible.
gzipped tar solves this, zip in any form does not.
>
> Laurence
Natanael L
Hmm, Google Groups don't work as I first thought...
Anyway:
Why no mentions of time stamps? The manifests *should* (not mandatory)
have time stamps for each file, and the HTML file could also have time
stamps on each file.
Imagine Google - occassionally they swap logo for a day. Why should
Fennec (Firefox Mobile) users have to reload the .tgz (.tar.gz) file?
Just load the cache and that single file with a fresher time stamp.
This works even better for huge sits with lots of static files that
occassionally are changed.
And of course - magnet links! (Optional)
http://en.wikipedia.org/wiki/Magnet_link
I just can't get the idea out of my head. Of course, it would be
optional. There would be a normal URL and a magnet URL for those who
wish.
If you have a magnet plugin in the browser, it would be used and
really large files (OGG/Theora videos?) could possibly be downloaded
from both the server as well as using torrents with DHT, increasing
the speed even more in under certain circumstances.
Alexander Limi
> You're right. My bad. Still, gzipped tar is streamable, gzipped zip
> is not. In fact any form of zip is not streamable. If the proposal
> is to get data to the browser faster it seems like it should also
> be about letting the browser use that data as fast as possible.
> gzipped tar solves this, zip in any form does not.
Sorry that I haven't answered to this thread earlier, tar.gz is now
added to the spec, in addition to zip:
http://limi.net/articles/resource-packages
Added you in the Acknowledgements section, let me know if I'm missing
anyone — or if you'd prefer not to be listed.
--
Alexander Limi · Firefox User Experience Team · http://limi.net
Gervase Markham
> Sorry that I haven't answered to this thread earlier, tar.gz is now
> added to the spec, in addition to zip:
>
> http://limi.net/articles/resource-packages
Is the solution really to make browsers support two formats? OK, so they
all might have the decoding code anyway, but it's more complexity and
more testing...
Gerv
Alexander Limi
It should be format-independent, I just mentioned the two that are most
likely to be implemented.
Major platforms don't have easy access to tar.gz (Windows), and an explicit
goal of the proposal is to make it integrator/webdev-friendly. Hence, we
support the two most commonly used compression formats that fit the bill
(bzip and rar do not, AFAIK).
Gervase Markham
> It should be format-independent, I just mentioned the two that are most
> likely to be implemented.
TBH, I think that's a terrible idea :-( If you don't specify what
compression formats are part of the standard, then all that will happen
is that a /de facto/ standard will emerge - which will be the easiest to
implement rather than the most technically appropriate.
> Major platforms don't have easy access to tar.gz (Windows),
Really? There's http://www.7-zip.org/ and loads of other free software.
> and an explicit
> goal of the proposal is to make it integrator/webdev-friendly. Hence, we
> support the two most commonly used compression formats that fit the bill
> (bzip and rar do not, AFAIK).
I think there's a significant advantage in picking formats that browsers
already have code for. No-one wants to extend their browser with a third
compression format, even if it makes things smaller (e.g. bzip). So I
agree the competition is between these two. The question is whether to
specify both, one, or neither as part of the standard.
Gerv