Test MAR generation with Courgette?
Mike Beltzner
The sizes of our Windows MARs have been growing (when we started using
profile guided optimization for compiles, it really spiked) as have
they on all platforms. Everything Stephen Adams wrote in his blog post
(http://blog.chromium.org/2009/07/smaller-is-faster-and-safer-
too.html ) about why making software update file sizes smaller is
absolutely true, especially when we start considering updating mobile
products.
What would be required to figure out if Courgette can help us like
it's helped Chrome? Maybe we can get one or two interested people
looking into it? I hear that it's x86 only, but that will still be
helpful for 90% of our users!
cheers,
mike
(thanks to Asa and others for pointing it out to me)
Serge Gautherie
> ref:
> http://dev.chromium.org/developers/design-documents/software-updates-courgette
This ends with:
"We are writing a more detailed paper on Courgette and will post an
update when it is ready."
> What would be required to figure out if Courgette can help us like it's
> helped Chrome?
To know where binary(?) and manual(!) would be available...
> Maybe we can get one or two interested people looking into it?
Generating the update should be +/- simple;
then, the client side will need to be looked at...
> I hear that it's x86 only, but that will still be helpful for
> 90% of our users!
http://neugierig.org/software/chromium/notes/2009/05/courgette.html
"It only supports Windows x86 for now"
Mike Shaver
> To know where binary(?) and manual(!) would be available...
Don't think there's either, but the source is linked from your linked
page: http://src.chromium.org/viewvc/chrome/trunk/src/courgette/ and
it has tests to read.
Mike
Jean-Marc Desperrier
>> I hear that it's x86 only, but that will still be helpful for 90% of
>> our users!
>
> http://neugierig.org/software/chromium/notes/2009/05/courgette.html
> "It only supports Windows x86 for now"
From the source, it seems that the main reason for that is that they
wrote their own on-purpose PE parser, as well as their own on-purpose
disassembler. It doesn't seem to be a general usage disassembler, I
believe it just knows which opcodes will be followed by
relative/absolute address and their size.
It's quite well done, and seems very effective within a small code
volume. Very clean C++/STL code also.
The question is if it's worth it to reorganize the code to be able to
use 3rdParty components instead for those tasks, if it will make it
easier/faster to port to other architecture.
The answer is not so obvious, you might end up spending almost as much
time interpreting/transforming/integrating the output of those 3rd party
components as they did writing their own.
I think the choice depends quite strongly on the possibility to find
such a 3rd party tool that is cross-platform. Then it's really worth it.
Jean-Marc Desperrier
> I think the choice depends quite strongly on the possibility to find
> such a 3rd party tool that is cross-platform
But binutils' objdump (with the disassembly option) seems to fit the bill.
Jean-Marc Desperrier
Well after some more thinking, the problem is that the process is
symmetric, the client needs to be just as smart else you wouldn't get
such strong compression.
So it does make sense to re-implement everything to have a lighter load
on the client.
rhelmer
unpacked them and diffed the DLL/EXEs with Courgette.
All other files where left alone (same old bsdiff+bzip2), and I
bzip2'd each resulting binary (which seems to help on top of
Courgette):
1.9M patch
812K patch-courgette/
I made a script to create patch-courgette and also a script to apply
the patches to 3.5 and compare the result to 3.5.1, all checks out ok.
They aren't terribly interesting but I'd be happy to send them to
anyone that wants to see.
Here are the two largest patches (below is post-bzip2, courgette
versus mbsdiff, using the Mozilla-published 3.5.1 partial):
613K patch-courgette/xul.dll.patch
1.6M patch/xul.dll.patch
42K patch-courgette/js3250.dll.patch
110K patch/js3250.dll.patch
I built and ran this on Vista using VS Express, Chromium r20874,
release build.
Rerunning this on an update with fewer changes (like a security-only
update) would be an interesting test. I also don't know how much of a
wrench PGO throws in, that would be good to measure as well.
I'm unqualified to talk about code quality, what it would take to get
this into updater.exe etc. (which is probably the more involved part
as Serge already pointed out).
Mike Beltzner
to things.
cheers,
mike
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
Gary Kwong
> ref:
> http://dev.chromium.org/developers/design-documents/software-updates-courgette
>
>
> The sizes of our Windows MARs have been growing (when we started using
> profile guided optimization for compiles, it really spiked) as have they
> on all platforms. Everything Stephen Adams wrote in his blog post
> about why making software update file sizes smaller is absolutely true,
> especially when we start considering updating mobile products.
>
> What would be required to figure out if Courgette can help us like it's
> helped Chrome? Maybe we can get one or two interested people looking
> into it? I hear that it's x86 only, but that will still be helpful for
> 90% of our users!
>
> cheers,
> mike
>
> (thanks to Asa and others for pointing it out to me)
https://bugzilla.mozilla.org/show_bug.cgi?id=504624
A bug was already on file.
-Gary
nth10sd
Ted Mielczarek
effect of PGO. (Our nightly MARs pre-PGO were always smaller than 1MB, and
now they're almost always at or greater than 1MB.)
-Ted
On Fri, Jul 17, 2009 at 8:36 AM, Mike Beltzner <belt...@mozilla.com> wrote:
> Thanks for the analysis, Rob! We should definitely see what PGO does to
> things.
>
> cheers,
> mike
>
>
> On 17-Jul-09, at 12:02 AM, rhelmer wrote:
>
rhelmer
PGO turned off.
I haven't had a chance to build the MARs and do the courgette
comparison yet.
BTW is this mozconfig change enough to disable PGO?
ac_add_options --disable-profile-guided-optimization
I want to do a more controlled test, so I've set up the following
scenario (same revision in all cases):
1) no source changes, just built at different times
2) made one source change in (xpcom/io/nsLocalFileCommon.cpp flipped
"if (longName)" to "if (!longName)")
A lot of files are touched in #1, so I want to separate that from a
legitimate change like #2.
Then I'll redo 1/2 on the same revision with PGO turned on, which
should isolate it, I'm thinking.
On Jul 21, 12:11 pm, Ted Mielczarek <ted.mielcza...@gmail.com> wrote:
> FWIW, if he was testing 3.5 vs. 3.5.1, those numbers already include the
> effect of PGO. (Our nightly MARs pre-PGO were always smaller than 1MB, and
> now they're almost always at or greater than 1MB.)
>
> -Ted
>
> >> dev-platf...@lists.mozilla.org
> >>https://lists.mozilla.org/listinfo/dev-platform
>
> > _______________________________________________
> > dev-platform mailing list
> > dev-platf...@lists.mozilla.org
> >https://lists.mozilla.org/listinfo/dev-platform