How firefox keeps their flash size way down in Android
Gavin Peters (蓋文彼德斯)
Torne (Richard Coles)
Disclaimer: I've probably not thought of everything here and I'm happy
to be corrected :)
It's a nice trick, but it creates a lot of private dirty pages that
cannot be shared between processes. Chrome loads the same native code
into each of the browser and renderer processes, and would thus end up
using several times more RAM if this system was used.
Loading parts on demand into a chunk of memory shared across all the
processes would be difficult, since the protection mechanism it's
relying on to mark the bits of memory that aren't loaded yet is
per-process and so you'd need an additional layer of accounting to
keep track of which parts are loaded, and you may have to handle the
same fault once in *each* process before you stop getting faults for
that region.
There's also another problem: this increases memory pressure, because
the memory is dirty and thus cannot be paged out by the kernel unless
the device has a swapfile, which Android devices typically don't.
The "right" way to do this (for a very tenuous definition of right)
would be to introduce a way to mmap() files that handles the
decompression in the kernel: all programs could take advantage of
this, the memory could still be shared between processes and paged
out, and the only downside would be the performance cost of
decompressing the pages, which programs can judge for themselves if
it's worth the benefit.
> - Gavin
>
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
--
Torne (Richard Coles)
to...@google.com
Mike Frysinger
over 4 gb nand flashes ?
if there speedups are part of the crazy sharding, then perhaps it is
worth looking at. but if the only point is to save disk space at the
cost of startup speeds, then the answer is kind of obvious ...
-mike
2012/3/1 Gavin Peters (蓋文彼德斯) <gav...@google.com>:
Greg Simon
Mike Frysinger
"This is why Chrome for Android takes almost 50MB of internal flash
space after installation."
pretty sure the point of the work was to address space in flash, not
OTA transfers
-mike
2012/3/1 Greg Simon <greg...@chromium.org>:
Evan Martin
> Disclaimer: I've probably not thought of everything here and I'm happy
> to be corrected :)
>
> It's a nice trick, but it creates a lot of private dirty pages that
> cannot be shared between processes. Chrome loads the same native code
> into each of the browser and renderer processes, and would thus end up
> using several times more RAM if this system was used.
For plugins like Flash we always use a singleton process hosting the
plugin. (This is because plugins assume they're being used within a
single-process browser and don't know how to lock their disk data
structures against other instances of themselves.) So the dirty page
thing doesn't matter as much.
(Disclaimer: the previous paragraph may not be true on Android or with
PPAPI flash.)
The cool hacks in the original blog post are more important for
Firefox, which is constructed as a collection of shared objects, than
it is for us, where we put everything we can into the single binary.
Torne (Richard Coles)
> On Thu, Mar 1, 2012 at 8:45 AM, Torne (Richard Coles) <to...@google.com> wrote:
>> Disclaimer: I've probably not thought of everything here and I'm happy
>> to be corrected :)
>>
>> It's a nice trick, but it creates a lot of private dirty pages that
>> cannot be shared between processes. Chrome loads the same native code
>> into each of the browser and renderer processes, and would thus end up
>> using several times more RAM if this system was used.
>
> For plugins like Flash we always use a singleton process hosting the
> plugin. (This is because plugins assume they're being used within a
> single-process browser and don't know how to lock their disk data
> structures against other instances of themselves.) So the dirty page
> thing doesn't matter as much.
I think you misread the original post, Evan. This is talking about the
installed size in the phone's flash memory, not anything to do with
the Flash plugin. The native code I'm talking about is the actual
Chrome code itself, which on Android is a single large shared library
rather than an executable since Android apps' entry points are in
Java. This shared library is zipped into the application APK when it's
downloaded, but after installation is extracted onto the user's device
so that it can be loaded, which means that the installed size of the
Chrome for Android beta includes both a compressed and an uncompressed
copy of this library. It's this duplication that the trick Firefox for
Android is avoiding, by never decompressing the native library
containing their browser code onto disk.
> (Disclaimer: the previous paragraph may not be true on Android or with
> PPAPI flash.)
>
> The cool hacks in the original blog post are more important for
> Firefox, which is constructed as a collection of shared objects, than
> it is for us, where we put everything we can into the single binary.
--
Torne (Richard Coles)
to...@google.com
William Chan (陈智昌)
Torne (Richard Coles)
> (Disclaimer: the previous paragraph may not be true on Android or with
> PPAPI flash.)
>
> The cool hacks in the original blog post are more important for
> Firefox, which is constructed as a collection of shared objects, than
> it is for us, where we put everything we can into the single binary.
--
to...@google.com
Evan Martin
<will...@chromium.org> wrote:
> I think you misread Evan's post. His post discusses dirty pages and plugin
> processes. Flash is just an example.
I did misread the post, but it semi-coincidentally worked out. ;)
To clarify further, plugins are currently the *only* shared objects in Chrome.
Torne (Richard Coles)
No, on Chrome for Android the entire browser engine is a shared object
loaded via JNI, and the same is true for Firefox for Android (though
it may be multiple shared objects, I haven't looked) - this is how all
Android apps with native code work.
Torne (Richard Coles)
> On Thu, Mar 01, 2012 at 04:45:15PM +0000, Torne (Richard Coles) wrote:
>> It's a nice trick, but it creates a lot of private dirty pages that
>> cannot be shared between processes. Chrome loads the same native code
>> into each of the browser and renderer processes, and would thus end up
>> using several times more RAM if this system was used.
>
> base on the android system linker would share pages between processes.
> The new linker doesn't support that yet, but it's on the roadmap.
Does this rely on them being forked from a common ancestor, or does it
actually allocate shared memory? The Chrome processes on android are
*not* forked from a common ancestor as they are on desktop (well,
technically they are but it's the Android zygote, which doesn't have
any chrome code loaded).
>> Loading parts on demand into a chunk of memory shared across all the
>> processes would be difficult, since the protection mechanism it's
>> relying on to mark the bits of memory that aren't loaded yet is
>> per-process and so you'd need an additional layer of accounting to
>> keep track of which parts are loaded, and you may have to handle the
>> same fault once in *each* process before you stop getting faults for
>> that region.
>
> It's in my plans to try to handle this "properly", although this will
> obviously have a runtime cost. However, I expect the overhead to be well
> below the decompression overhead anyway. Facts may prove me wrong,
> though.
It might not be that expensive, no; I just meant that it's more
complicated than the current code.
> We discussed with Andi Kleen a more generic approach, which he agreed
> with. The idea is for the kernel to take hints at pages the program
> allows it to throw away when it sees fit and mark them such that they
> would segfault if accessed (unmapped would not work, since something
> else could then be mapped there, PROT_NONE would work ; could be a
> separate state, whatever). Kind of what ashmem allows with unpinning,
> except that you'd get a segfault instead of nulled out pages.
>
> Such facility could be useful for JIT code, memory caches, and many
> other use cases. That would also be a broader scope than compressed data
> in a given format. On the dynamic linker side of things, that would also
> allow on-demand relocation (which is also on the roadmap for our
> linker).
That's a nice idea, yeah; much more generally useful. I still do like
in principle the idea of having the decompression happen during paging
since this removes the need for cross-process bookkeeping in
userspace. It was just a thought, though :)
Iain Merrick
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
Iain Merrick
Software Engineer, Google
Tei
extension instead of ZIP. While most of Android is java, and java
classes are usually loaded from a ZIP archive (usually with the JAR
extension), Android applications using native code need to have native
libraries on the file system. These native libraries are found under
/data/data/$appid/lib, where $appid is the package name, as defined in
the AndroidManifest.xml file."
Uh.
Sounds like Android need a "virtual filesystem" layer. One so the OS
loader ask for files, and he don't know the files are receiving are
stored inside a jar file or other storing solution.
2012/3/2 Iain Merrick <hu...@google.com>:
> It *shouldn't* matter, but due to some unfortunate design decisions, many
> Android devices have a very small partition for the system image, and
> apparently repartitioning is too dangerous.
--
--
ℱin del ℳensaje.
Torne (Richard Coles)
> "Android applications are essentially zip archives, with the APK
> extension instead of ZIP. While most of Android is java, and java
> classes are usually loaded from a ZIP archive (usually with the JAR
> extension), Android applications using native code need to have native
> libraries on the file system. These native libraries are found under
> /data/data/$appid/lib, where $appid is the package name, as defined in
> the AndroidManifest.xml file."
>
> Uh.
>
> Sounds like Android need a "virtual filesystem" layer. One so the OS
> loader ask for files, and he don't know the files are receiving are
> stored inside a jar file or other storing solution.
Most files inside the APK *are* loaded from inside it without the app
having to do anything about it. Any resources that are stored inside
the archive uncompressed (e.g. image/sound/etc formats that are
already compressed and thus don't need to be deflated to be added to
the zip) are just mmap'ed out of the APK when accessed by the app, and
other, compressed resources are decompressed into ram when accessed.
The dynamic loader doesn't know how to do this, though, and even if it
did this would just be trading one problem for a different one, since
the .so would have to be stored in the APK uncompressed, making the
download size larger to save installed size, and most of the time
increasing download size significantly would not be a good tradeoff.
So, the Android system just extracts the .so on install, leaving all
the other files in the APK.
> 2012/3/2 Iain Merrick <hu...@google.com>:
>> It *shouldn't* matter, but due to some unfortunate design decisions, many
>> Android devices have a very small partition for the system image, and
>> apparently repartitioning is too dangerous.
>
>
>
> --
> --
> ℱin del ℳensaje.
>
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
--
Torne (Richard Coles)
to...@google.com
Markus Gutschke
Torne (Richard Coles)
> I think, Tei asked for something else. Android should have a fuse filesystem
> (or something similar) that allows you access the inside of an APK without
> your application having to explicitly unzip it. This would make the
> compressed content available to the kernel, which in turn would be able to
> allow user space to mmap() it.
Right, but because Android resources are already accessed while
zipped, this is only useful for shared objects (and most Android apps
do not have any native code). There is no general need for this, it's
already implemented at the Java level for most use cases. For the
shared object case, I would be concerned about the performance of
going back to userspace to retrieve pages on every fault... but
without that being implemented it's hard to test objectively. It is
certainly another option, but I don't think it's one that Android
system images are likely to be using any time soon.