Ability for Chrome to read extension files from a .jar (or other archive) file without unzipping it

[Reivax]

Some extensions can have hundred of files (like Yoono for instance),
and Chrome needs to unzip them all during install.

It would be great if Chrome were able to handle a .jar file that would
contain all or most files like Firefox does.

The installation, update, removing times would drastically reduce,
giving the user a much better experience.

(Moreover, that would eliminate some FAT32 fs issues with file paths
too long to be properly handled by Windows XP).

What do you all think ?

[krtulmay]

Well, that "other archive" format already exists and it's called .crx
format, which is the packing format for extensions. And by the way,
this is just academic because .crx files use ZIP format, and .jar
files also use ZIP format; so the suggestion of using .jar files
offers no change in available functionality.

So you will not be able to solve any XP on FAT32 issues just by
changing the archiving extension's last 3 letters.

Your question is more like can the Chrome extensions system be
modified to not do entire extraction of the extension's content to
avoid issues like too long pathnames?

[Reivax]

Sorry, you didn't get it.

On chrome, the crx is totally unzipped when the extension is installed
(as I said).

On firefox, an xpi file can contain .jar files that need not be
unzipped at all for the add-on to work. Files are fetched from
the .jar files.

And yes it does solve path issues, since the individual file access is
no longer handled by the OS file system, but by the .jar handling.

That would make installation, un-installation and update much MUCH
faster.

Hope I clarified my first message.

[krtulmay]

Sorry, you are the one that didn't get it.

I am saying .jar files will not help for Chrome because .crx and .jar
files are both just zip files.

I know .jar files do not "need to" be unzipped. .crx do *not* need to
be unzipped either, since they are both ZIP file formats! *If* Chrome
wanted to just fetch the files from the .crx, they can! That's why
suggesting using .jar does not help the path issues and does *not*
provide any changes to installation, uninstallation, and updates.

*If* Chrome wants to change the way they install extensions now
without doing full extraction, they can already do that with .crx. It
is only a question of if they want to.

[Daniel Wagner-Hall]

How many files does you extension have so that this is actually an
issue? I can't imagine an extension needing more than 100 or so files,
and I can't imagine 100 files making that much difference for
loading/updating...

On the other hand, the load/update delay is a one-off (well, a few
times, but seldom), whereas loading files from a jar at runtime would
add delay every time the extension uses the files.

> --
> You received this message because you are subscribed to the Google Groups "Chromium-extensions" group.
> To post to this group, send email to chromium-...@chromium.org.
> To unsubscribe from this group, send email to chromium-extens...@chromium.org.
> For more options, visit this group at http://groups.google.com/a/chromium.org/group/chromium-extensions/?hl=en.
>
>

[Reivax]

You don't really get it do you ?
I don't care about the file format.
All I care is that this mode could be supported.
My initial message asks for nothing more.
We got your point, thanks.

[Reivax]

The crx file is 2.4Mb, and there are about 1800 files in it.
So, it kind of take some time :)

Upon installing, there are a few seconds during which the user does
not know what is happending.

As for reading the files, I guess the unitary jar file access time
from the disk would probably be better than reading each file
separately, but then I may depend on the caching applied...

On Jun 28, 7:03 pm, Daniel Wagner-Hall <dawag...@gmail.com> wrote:
> How many files does you extension have so that this is actually an
> issue? I can't imagine an extension needing more than 100 or so files,
> and I can't imagine 100 files making that much difference for
> loading/updating...
>
> On the other hand, the load/update delay is a one-off (well, a few
> times, but seldom), whereas loading files from a jar at runtime would
> add delay every time the extension uses the files.
>

> > To post to this group, send email to chromium-extensi...@chromium.org.
> > To unsubscribe from this group, send email to chromium-extensions+unsubscr...@chromium.org.

[Brian Kennish]

On Mon, Jun 28, 2010 at 10:20 AM, Reivax <xavier...@gmail.com> wrote:
> The crx file is 2.4Mb, and there are about 1800 files in it.

Wow, that's a lot of files! I can't say we've done much performance
optimizing for 1,800-file extensions. You can submit your suggestion
at http://crbug.com/new.

You might also want to host some of the files remotely and lazy-load
them. I imagine most are never accessed by the average user.

[aaafwd]

What about implementing a smart "incremental" unzipping of the *.crx file, so that only changed files get actually extracted? For example, IMO it would help immensely if only a dozen of files out of those 1800 were actually updated in the *.crx file. And this will not require any change in the API (unlike the JAR solution).

--
You received this message because you are subscribed to the Google Groups "Chromium-extensions" group.

To post to this group, send email to chromium-...@chromium.org.
To unsubscribe from this group, send email to chromium-extens...@chromium.org.

[Reivax]

The issue would remain at first install, which is when users get their
first impression.
For now, the impression is that user agreed to installation and then
nothing happens for a while, and he may wonder what is going on.

Supporting reading from an archive file might not require that many
changes in the API (okay, easier for me to tell than for you to
implement :), and these changes could be transparent to extension
developers.

Since all files are fetched through a chrome:// protocol, it could
read files either way, according to some declaration in the manifest,
for instance.

On Jun 30, 12:01 pm, aaafwd <aaa...@gmail.com> wrote:
> What about implementing a smart "incremental" unzipping of the *.crx file,
> so that only changed files get actually extracted? For example, IMO it would
> help immensely if only a dozen of files out of those 1800 were actually
> updated in the *.crx file. And this will not require any change in the API
> (unlike the JAR solution).
>

> On Tue, Jun 29, 2010 at 10:23 PM, Brian Kennish <bkenn...@chromium.org>wrote:

>
> > On Mon, Jun 28, 2010 at 10:20 AM, Reivax <xavier.yo...@gmail.com> wrote:
> > > The crx file is 2.4Mb, and there are about 1800 files in it.
>
> > Wow, that's a lot of files! I can't say we've done much performance
> > optimizing for 1,800-file extensions. You can submit your suggestion

> > athttp://crbug.com/new.

>
> > You might also want to host some of the files remotely and lazy-load
> > them. I imagine most are never accessed by the average user.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Chromium-extensions" group.

> > To post to this group, send email to chromium-extensi...@chromium.org.

> > To unsubscribe from this group, send email to

> > chromium-extensions+unsubscr...@chromium.org<chromium-extensions%2Bunsubscr...@chromium.org>
> > .

[Adam]

As XP shrinks in market share, it makes less and less sense for
developers to devote resources to elaborate solutions for a system
that will be irrelevant soon. I would suggest a hard look at
optimizing those files. Cut down the initial extension load by
combining, minimizing, etc. Or as another suggested, lazy load some
from remote servers after install.

Maybe load required files on install then background.html could fetch
the rest as needed. Plus you could trim filenames to eliminate the XP
filename limits.

Either way, I doubt its worth the resources for the Chromium team to
support jar files just so a very miniscule subset of developers can
use long file names for XP.

[Casey Banner]

As was mentioned previously, jar is equivalent to zip in this case,
they use the same compression. What I believe the OP wants is an
incremental unzip; however if your extension is so huge that this
becomes an issue I think you could probably take a look at optimizing
that before you ask for a feature.

-Casey

> --
> You received this message because you are subscribed to the Google Groups "Chromium-extensions" group.

> To post to this group, send email to chromium-...@chromium.org.
> To unsubscribe from this group, send email to chromium-extens...@chromium.org.

[Rainfly]

I think the general rule is, if you need more than 150 or so separate
files, then you're probably doing it wrong :/ Especially since the
filesize suggests that most of your 1800 files are text or very small
images, both of which are not hard to combine into fewer, bigger files
(though rewriting your extension to use them *will* be rough, you'll
probably be better off in the long run).

[Reivax]

@Adam:
I just mentioned the FAT32 issue as a sidenote, we solved it, no big
deal.
The true issue is the need for Chrome to unzip the extension files,
when other solutions exist.

@Casey:
No, incremental unzip won't do, what about first install? That is when
the issue is the most disturbing for users.

@Rainfly:

"general rule is, if you need more than 150 or so separate files, then
you're probably doing it wrong"

WOW ! what a bold statement. It is so irrelevant I won't even bother
to discuss it here.


Thanks for sharing your thoughts, guys, but you all sound like Steve
Jobs saying "Just don't hold it that way".

Why do you think VMs do use archive files ?
Why do you think Mozilla does it in Firefox ?
Our very same extension (I guess 95% code in common) is installed in
Firefox in a split second, while it takes more than 10 seconds on
Chrome.


Now, I'm not saying things should be done just because someone submits
the idea. But I would expect the discussion to be a little more
professional with points a little more elaborate than "optimize your
code" or "combine your files", or "you are doing it wrong", especially
when you don't know what the extension does and how many features it
has.

I guess the best valid point was that for now, not many extension may
really benefit from this feature, but extensions are developing fast,
and will reach further as the API is extended.

Thanks.

[Casey Banner]

I think you misunderstood what I suggested.

You cannot simply read zipped data and use it, it must be unzipped
first. If you unzip files upon access (which is w hat I think you are
asking for here), the you avoid the delay you seem to be experiencing
when all the files are unzipped at once. You cannot get around the
fact that the files are compressed. I think what you are asking for is
that files be decompressed when they are accessed (incremental unzip).

Also, you aren't going to get much help if you troll everyone who
tries to help you.

-Casey

[Reivax]

Thanks Casey for your reply.

I don't think I'm trolling people anymore than they troll me saying to
optimize stuff or just keep it under 150 files...
I don't think I've been trolling you at all, since your answer was not
of that kind either. Apologies if you feel differently.

I did in fact misunderstood your post, because incremental unzipping
had already been suggested, but with a totally different meaning than
yours:
See post by aaafwd :

"What about implementing a smart "incremental" unzipping of the *.crx
file, so that only changed files get actually extracted?"

Hence my answer.
Sorry for the misunderstanding!

As for unzipping, of course I realize that the code needs to be
unzipped at some point to be executed, but it needs not be written
into individual files. This is what is taking time, not unzipping
itself.

You may not be familiar with VMs (I'm no specialist either), but they
don't unzip on the disc individual files from their jar libraries when
they use them.
Agreed, it might impact the memory footprint.

With today's computers, unzipping data is really blazing fast compared
to accessing and then writing/reading files on a disk.
The gain in what I'm suggesting is not in unzipping, it's on writing
AND reading files, which would benefit each time it's executed, I
guess.

Sorry if I didn't make it clear at first (English is not my primary
language).

On Jul 1, 9:49 am, Casey Banner <kcban...@gmail.com> wrote:
> I think you misunderstood what I suggested.
>
> You cannot simply read zipped data and use it, it must be unzipped
> first. If you unzip files upon access (which is w hat I think you are
> asking for here), the you avoid the delay you seem to be experiencing
> when all the files are unzipped at once. You cannot get around the
> fact that the files are compressed. I think what you are asking for is
> that files be decompressed when they are accessed (incremental unzip).
>
> Also, you aren't going to get much help if you troll everyone who
> tries to help you.
>
> -Casey
>

> > To post to this group, send email to chromium-extensi...@chromium.org.
> > To unsubscribe from this group, send email to chromium-extensions+unsubscr...@chromium.org.

[Adam]

So what you're suggesting is that when an extension is installed, it
stays compressed and Chrome compresses/decompresses files each time
their used/modified?

Data compression is more important for things served over the
internet. (Minifying, gzip). But when files are stored locally,
uncompressed and un-minifed code isn't a big issue.

Have you benchmarked installing the unpacked extension vs. the packed
extension?

And while it may install faster on Firefox, at least Chrome doesn't
require a restart to begin using your extension. Also is your
extension in the chrome extension gallery yet? Post a link if possible
so others can verify the long install times.

[Reivax]

Hi Adam

I'm suggesting that files are not extracted on the disk at
installation, nor at run time.
They are read from the archive file (zip, jar, crx, whatever).

It's not a compression issue, but a file access issue, as I said.
Please do not focus on the compression, just on the "archive"
suggestion. It could even be an uncompressed tar file as far as I'm
concerned. But just ONE file instead of MANY.

As for benchmarking, since Chrome does not handle it, I can only
compare to Firefox, and the difference is huge.

Firefox having to be restarted is a Firefox issue that does not seem
relevant to counter my suggestion.

The extension is available in the gallery for a few days now :
https://chrome.google.com/extensions/detail/jkkenjlnjfemconejajakbijbheoffli

regards

[Ben]

Took about five seconds for me to download (so it might take a long time
on slow connections), but was available for use instantly (there was no
delay after the download). Of course, this might be different for
someone using Windows; especially if the data is on a fragmented FAT
partition, or any almost full partition.

Ubuntu Lucid Lynx (32-bit), for reference.

[Reivax]

Yes it's totally different on Windows.

On Jul 1, 5:47 pm, Ben <benjo...@gmail.com> wrote:
> Took about five seconds for me to download (so it might take a long time
> on slow connections), but was available for use instantly (there was no
> delay after the download). Of course, this might be different for
> someone using Windows; especially if the data is on a fragmented FAT
> partition, or any almost full partition.
>
> Ubuntu Lucid Lynx (32-bit), for reference.
>
> Reivax wrote:
> > Hi Adam
>
> > I'm suggesting that files are not extracted on the disk at
> > installation, nor at run time.
> > They are read from the archive file (zip, jar, crx, whatever).
>
> > It's not a compression issue, but a file access issue, as I said.
> > Please do not focus on the compression, just on the "archive"
> > suggestion. It could even be an uncompressed tar file as far as I'm
> > concerned. But just ONE file instead of MANY.
>
> > As for benchmarking, since Chrome does not handle it, I can only
> > compare to Firefox, and the difference is huge.
>
> > Firefox having to be restarted is a Firefox issue that does not seem
> > relevant to counter my suggestion.
>
> > The extension is available in the gallery for a few days now :

> >https://chrome.google.com/extensions/detail/jkkenjlnjfemconejajakbijb...

[Antony Sargent]

Having this many files in an extension is probably uncommon enough that it isn't a big priority for us to optimize for it, but it's definitely a poor user experience to not show some indication that the browser is working on downloading/unpacking if it's taking more than 1 sec or so. We may already have a bug on this, but if not please feel free to create one. 

--
You received this message because you are subscribed to the Google Groups "Chromium-extensions" group.

To post to this group, send email to chromium-...@chromium.org.
To unsubscribe from this group, send email to chromium-extens...@chromium.org.

[Reivax]

Thanks Antony for your answer.

It's uncommon for now :)
I really understand it can't be a high priority.
I'll check if there's a registered bug.

On Jul 2, 1:27 am, Antony Sargent <asarg...@chromium.org> wrote:
> Having this many files in an extension is probably uncommon enough that it
> isn't a big priority for us to optimize for it, but it's definitely a poor
> user experience to not show some indication that the browser is working on
> downloading/unpacking if it's taking more than 1 sec or so. We may already
> have a bug on this, but if not please feel free to create one.
>

> > To post to this group, send email to chromium-extensi...@chromium.org.

> > To unsubscribe from this group, send email to

> > chromium-extensions+unsubscr...@chromium.org<chromium-extensions%2Bunsubscr...@chromium.org>
> > .

[Adam]

My understanding was that extensions are downloaded as a .crx
(basically a zip archive). However, it might be a good idea to post on
your extensions page something about the 10 sec. install time. And I
agree with Antony that some UI adjustments to let the user know an
extension installation is in progress would be a better way to
approach the problem. Maybe even a status bar that indicates progress
instead of a static "installing" message.