Unified builds

[Ehsan Akhgari]

I've started to work on a project in my spare time to switch us to use
unified builds for C/C++ compilation. The way that unified builds work is
by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
files. With that, the build system creates files such as:

// Unified_cpp_path_0.cpp
#include "Source1.cpp"
#include "Source2.cpp"
// ...

And compiles them instead of the individual source files.

The advantage of this is that it speeds up the compilation (I've measured
between 6-15x speed improvement depending on the code in question
locally.) But there are also trade-offs with this approach. One trade-off
is that the source code might require changes before it can be compiled in
this way, due to things like name clashes, etc. The other one is that if
you change one .cpp file which is built in unified mode, we would spend
more time compiling it because we'll be compiling the unified .cpp file
which includes more than what you have changed. It's hard to come up with
numbers for this trade-off, but assuming that the linking step takes way
longer than the C++ compilation in the "touch one .cpp file" scenario, and
also that changes to headers will potentially trigger multiple .cpp
rebuilds in the same directory, I think doing unified builds in more parts
of the tree is a reasonable choice.

I'm going to continue working on this as time permits, if you're interested
in helping out, or if you have questions, please get in touch.

Cheers,
--
Ehsan
<http://ehsanakhgari.org/>

[Chris Peterson]

On 11/14/13, 2:49 PM, Ehsan Akhgari wrote:
> The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:
>
> // Unified_cpp_path_0.cpp
> #include "Source1.cpp"
> #include "Source2.cpp"
> // ...

Are the UNIFIED_SOURCES from one moz.build file ever unified with
another moz.build file's UNIFIED_SOURCES?

I see that UNIFIED_SOURCES only supports .c and .cpp (and .cc) sources.
Is there any reason why .mm files are not also supported? Do
Objective-C's #imports cause problems?


chris

[Ehsan Akhgari]

On 2013-11-14 6:29 PM, Chris Peterson wrote:
> On 11/14/13, 2:49 PM, Ehsan Akhgari wrote:
>> The way that unified builds work is
>> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
>> files. With that, the build system creates files such as:
>>
>> // Unified_cpp_path_0.cpp
>> #include "Source1.cpp"
>> #include "Source2.cpp"
>> // ...
>
> Are the UNIFIED_SOURCES from one moz.build file ever unified with
> another moz.build file's UNIFIED_SOURCES?

I don't think so, which is good. You don't want to do that since then
adding a file in one directory can break the build in another!

> I see that UNIFIED_SOURCES only supports .c and .cpp (and .cc) sources.
> Is there any reason why .mm files are not also supported? Do
> Objective-C's #imports cause problems?

Ironically I fixed that before seeing your email! See bug 938844.


One piece of interesting data point. I have a whole bunch of patches
locally which makes us build all of layout/ in unified mode. With those
patches applied, and after touching all of the .cpp files in layout/, I
can rebuild that directory in about 4 seconds on my machine. How long
this takes without those patches is left as an exercise for the readers'
imagination. :-)

Cheers,
Ehsan

[Mike Hommey]

On Thu, Nov 14, 2013 at 07:01:03PM -0500, Ehsan Akhgari wrote:
> On 2013-11-14 6:29 PM, Chris Peterson wrote:

> >On 11/14/13, 2:49 PM, Ehsan Akhgari wrote:
> >>The way that unified builds work is
> >>by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> >>files. With that, the build system creates files such as:
> >>
> >>// Unified_cpp_path_0.cpp
> >>#include "Source1.cpp"
> >>#include "Source2.cpp"
> >>// ...
> >
> >Are the UNIFIED_SOURCES from one moz.build file ever unified with
> >another moz.build file's UNIFIED_SOURCES?
>

> I don't think so, which is good. You don't want to do that since
> then adding a file in one directory can break the build in another!
>

> >I see that UNIFIED_SOURCES only supports .c and .cpp (and .cc) sources.
> >Is there any reason why .mm files are not also supported? Do
> >Objective-C's #imports cause problems?
>

> Ironically I fixed that before seeing your email! See bug 938844.
>
>
> One piece of interesting data point. I have a whole bunch of
> patches locally which makes us build all of layout/ in unified mode.
> With those patches applied, and after touching all of the .cpp files
> in layout/, I can rebuild that directory in about 4 seconds on my
> machine. How long this takes without those patches is left as an
> exercise for the readers' imagination. :-)

That sounds incredibly low, which got me curious, so I ran the test,
without unified sources, because if I am to believe my attempts last
year, layout doesn't build this way without modifications.

$ find objdir/layout -name '*.o' -delete
$ time make -s -C objdir/layout -j12 compile
real 1m36.949s
user 11m37.836s
sys 0m32.834s

With ccache and no cache miss.
$ find objdir/layout -name '*.o' -delete
$ time make -s -C objdir/layout -j12 compile
real 0m1.816s
user 0m9.273s
sys 0m2.616s

There is *no way* 1 minute and a half can get down to 4s. So I'll assume
your 4 seconds claim is with ccache with no cache miss. I doubt unified
vs. non unified makes much difference there. But really, ccache is not
the right metric. What's interesting is the actual compilation time, and
I'm sure it goes down. Just not down to 4s.

Mike

[Ehsan Akhgari]

On 2013-11-14 7:46 PM, Mike Hommey wrote:
> On Thu, Nov 14, 2013 at 07:01:03PM -0500, Ehsan Akhgari wrote:
>> On 2013-11-14 6:29 PM, Chris Peterson wrote:

>>> On 11/14/13, 2:49 PM, Ehsan Akhgari wrote:
>>>> The way that unified builds work is
>>>> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
>>>> files. With that, the build system creates files such as:
>>>>
>>>> // Unified_cpp_path_0.cpp
>>>> #include "Source1.cpp"
>>>> #include "Source2.cpp"
>>>> // ...
>>>
>>> Are the UNIFIED_SOURCES from one moz.build file ever unified with
>>> another moz.build file's UNIFIED_SOURCES?
>>

>> I don't think so, which is good. You don't want to do that since
>> then adding a file in one directory can break the build in another!
>>

>>> I see that UNIFIED_SOURCES only supports .c and .cpp (and .cc) sources.
>>> Is there any reason why .mm files are not also supported? Do
>>> Objective-C's #imports cause problems?
>>

You're right, I did in fact use ccache because I was using the wrong
mozconfig! (Rookie mistake.) With that fixed, we go down from 1m51s on
my machine to 31s, which is only about 4x faster!

Cheers,
Ehsan

[Neil]

Ehsan Akhgari wrote:

> we go down from 1m51s on my machine to 31s

And how does that compare to the case where you touch one .cpp file in
each subdirectory of layout?

--
Warning: May contain traces of nuts.

[Ehsan Akhgari]

On 2013-11-15 4:58 AM, Neil wrote:
> Ehsan Akhgari wrote:
>
>> we go down from 1m51s on my machine to 31s
>
> And how does that compare to the case where you touch one .cpp file in
> each subdirectory of layout?

It depends on which file, obviously, but here is an example with a
random file:

Before the patch:

$ touch layout/generic/FrameChildList.cpp
$ time make -s -j8 -C obj-ff-dbg/layout/generic/
FrameChildList.o
libgkgeneric_s.a.desc

real 0m0.793s
user 0m0.669s
sys 0m0.112s

After the patch:

$ touch layout/generic/FrameChildList.cpp
$ time make -s -j8 -C obj-ff-dbg/layout/generic/
Unified_cpp_layout_generic0.o
libgkgeneric_s.a.desc

real 0m0.855s
user 0m0.323s
sys 0m0.131s

As I said in my first post, touching one cpp file and building will be
slightly slower with unified builds.

Cheers,
Ehsan

[Ms2ger]

On 11/14/2013 11:49 PM, Ehsan Akhgari wrote:
> I've started to work on a project in my spare time to switch us to use
> unified builds for C/C++ compilation. The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:
>
> // Unified_cpp_path_0.cpp
> #include "Source1.cpp"
> #include "Source2.cpp"
> // ...
>
> And compiles them instead of the individual source files.

One issue I only thought of today: this means that Source2.cpp can use
all headers included into Source1.cpp–until enough files are added
between Source1 and Source2 that Source2 ends up in the next unified
file. This hasn't been much of an issue for autogenerated C++, but it
seems like it could be harder to get right with hand-written C++.

One way around it would be not to unify sources in automation. On one
hand, this could cause more bustage when changes that built locally turn
out not to have enough includes; on the other, it might be better than
having to fix up a dozen unrelated files whenever you add a new one.

Thoughts?
Ms2ger

[Gabriele Svelto]

On 14/11/2013 23:49, Ehsan Akhgari wrote:
> The advantage of this is that it speeds up the compilation (I've measured
> between 6-15x speed improvement depending on the code in question
> locally.)

Another advantage would be that for mostly self-contained modules we
could get most of the benefits of LTO without having to recur to LTO as
the compiler would have visibility over all the sources as a single
compilation-unit.

In turn this would also allow us to remove a lot of inline methods that
we only have in the headers for performance reasons. As it was pointed
out in the "Analyze C++/compiler usage and code stats easily" this could
bring significant compilation time improvements too.

Gabriele

[Ehsan Akhgari]

On Sat, Nov 16, 2013 at 4:34 AM, Ms2ger <ms2...@gmail.com> wrote:

> On 11/14/2013 11:49 PM, Ehsan Akhgari wrote:
>
>> I've started to work on a project in my spare time to switch us to use
>> unified builds for C/C++ compilation. The way that unified builds work is
>>
>> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
>> files. With that, the build system creates files such as:
>>
>> // Unified_cpp_path_0.cpp
>> #include "Source1.cpp"
>> #include "Source2.cpp"
>> // ...
>>
>> And compiles them instead of the individual source files.
>>
>
> One issue I only thought of today: this means that Source2.cpp can use all
> headers included into Source1.cpp–until enough files are added between
> Source1 and Source2 that Source2 ends up in the next unified file. This
> hasn't been much of an issue for autogenerated C++, but it seems like it
> could be harder to get right with hand-written C++.
>

Yes, that's correct.

> One way around it would be not to unify sources in automation. On one
> hand, this could cause more bustage when changes that built locally turn
> out not to have enough includes; on the other, it might be better than
> having to fix up a dozen unrelated files whenever you add a new one.
>

I don't think that we need to try to fix this problem any more than the
general problem of denoting our dependencies explicitly. It's common for
you to remove an #include from a header and find dozens of .cpp files in
the tree that implicitly depended on it. And that is much more likely to
happen than people adding/removing cpp files.

We have enough bustage caused by differences in the way we build on our
build slaves than the way that we build locally -- let's not add more of it.

[Ehsan Akhgari]

On Sat, Nov 16, 2013 at 5:26 AM, Gabriele Svelto <gsv...@mozilla.com>wrote:

> On 14/11/2013 23:49, Ehsan Akhgari wrote:
>
>> The advantage of this is that it speeds up the compilation (I've measured
>> between 6-15x speed improvement depending on the code in question
>> locally.)
>>
>
> Another advantage would be that for mostly self-contained modules we could
> get most of the benefits of LTO without having to recur to LTO as the
> compiler would have visibility over all the sources as a single
> compilation-unit.
>
> In turn this would also allow us to remove a lot of inline methods that we
> only have in the headers for performance reasons.

Well, remember that for now we don't #include *all* of your source files in
one .cpp file, so this is actually not entirely true. We currently group
the source files into batches of 16 and create one unified file per batch.

> As it was pointed out in the "Analyze C++/compiler usage and code stats
> easily" this could bring significant compilation time improvements too.

That is the whole point of this project. :-)

[Mike Hommey]

Another related problem is that of conflicting function or variable
names, macros, etc. We're obviously fixing those when they need to be
when unifying files in a given directory, but there is a chance a
conflict is introduced by a file moving from Unified1 to Unified2.

Mike

[Jonas Sicking]

On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> I've started to work on a project in my spare time to switch us to use
> unified builds for C/C++ compilation. The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:
>
> // Unified_cpp_path_0.cpp
> #include "Source1.cpp"
> #include "Source2.cpp"
> // ...

Doesn't this negate the advantage of static global variables. I.e.
when changing how you use a static global, rather than just auditing
the .cpp file where that static global lives, you now how to audit all
.cpp files that are "unified together".

/ Jonas

[L. David Baron]

Could we do a static analysis to look for things whose semantics are
changed by this unification, such as statics with the same name
between files that are/might be unified? (And could we make the
tree turn red if it failed?)

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Jonas Sicking]

On Sat, Nov 16, 2013 at 1:34 AM, Ms2ger <ms2...@gmail.com> wrote:
>
> One issue I only thought of today: this means that Source2.cpp can use all
> headers included into Source1.cpp–until enough files are added between
> Source1 and Source2 that Source2 ends up in the next unified file. This
> hasn't been much of an issue for autogenerated C++, but it seems like it
> could be harder to get right with hand-written C++.
>
> One way around it would be not to unify sources in automation. On one hand,
> this could cause more bustage when changes that built locally turn out not
> to have enough includes; on the other, it might be better than having to fix
> up a dozen unrelated files whenever you add a new one.

This sounds like a huge problem to me.

There's also the problem that some header files, in particular on
windows, add macros which can wreck havoc in our cpp files.

This happened recently with our autogenerated C++ files where adding a
.webidl file pushed a .cpp file from going from one unified file to
the next. This .cpp file pulled in one of these windows headers. The
result was that now a whole new set of files were exposed to the
windows macros and some of them broke.

This was a huge headache and I know that one of the solutions that
were seriously considered was to rename the new .webidl file from
foopy.webidl to zzfoopy.webidl.

/ Jonas

[Benoit Jacob]

Here is a wiki page to track our progress on this front and our means of
synchronizing on this work:

https://wiki.mozilla.org/Platform/Porting_to_unified_sources

Benoit


2013/11/14 Ehsan Akhgari <ehsan....@gmail.com>

> I've started to work on a project in my spare time to switch us to use
> unified builds for C/C++ compilation. The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:
>
> // Unified_cpp_path_0.cpp
> #include "Source1.cpp"
> #include "Source2.cpp"
> // ...
>

> And compiles them instead of the individual source files.
>

> The advantage of this is that it speeds up the compilation (I've measured
> between 6-15x speed improvement depending on the code in question

> locally.) But there are also trade-offs with this approach. One trade-off
> is that the source code might require changes before it can be compiled in
> this way, due to things like name clashes, etc. The other one is that if
> you change one .cpp file which is built in unified mode, we would spend
> more time compiling it because we'll be compiling the unified .cpp file
> which includes more than what you have changed. It's hard to come up with
> numbers for this trade-off, but assuming that the linking step takes way
> longer than the C++ compilation in the "touch one .cpp file" scenario, and
> also that changes to headers will potentially trigger multiple .cpp
> rebuilds in the same directory, I think doing unified builds in more parts
> of the tree is a reasonable choice.
>
> I'm going to continue working on this as time permits, if you're interested
> in helping out, or if you have questions, please get in touch.
>

> Cheers,
> --
> Ehsan
> <http://ehsanakhgari.org/>

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Neil]

Ehsan Akhgari wrote:

>I've started to work on a project in my spare time to switch us to use unified builds for C/C++ compilation. The way that unified builds work is by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build files. With that, the build system creates files such as:
>
>// Unified_cpp_path_0.cpp
>#include "Source1.cpp"
>#include "Source2.cpp"
>// ...
>
>And compiles them instead of the individual source files.
>
>

So basically whenever a moz.build contains UNIFIED_SOURCES and not
SOURCES then all of the old object files are now garbage?

[Boris Zbarsky]

On 11/17/13 5:26 PM, Ehsan Akhgari wrote:
> I don't think that we need to try to fix this problem any more than the
> general problem of denoting our dependencies explicitly. It's common for
> you to remove an #include from a header and find dozens of .cpp files in
> the tree that implicitly depended on it. And that is much more likely to
> happen than people adding/removing cpp files.

While true, in the new setup we have a different problem: adding or
removing a .cpp file makes other random .cpp files not compile.

This is especially a problem where windows.h is involved. For bindings
we simply disallowed including it in binding .cpp files, but for other
.cpp files that's not necessarily workable. Maybe we need a better
solution for windows.h bustage. :(

-Boris

[Benoit Jacob]

2013/11/18 Boris Zbarsky <bzba...@mit.edu>

While working on porting directories to UNIFIED_SOURCES, I too have found
that the main problem was system headers (not just windows.h but also Mac
and X11 headers) tend to define very polluting symbols in the root
namespace, which we collide with thanks to "using namespace" statements.

The solution I've employed so far has been to:
1) minimize the number of cpp files that need to #include such system
headers, by typically moving code out of header files and only #including
system headers in a few implementation files;
2) Keep these cpp files, that #include system headers, in plain old
SOURCES, not in UNIFIED_SOURCES.

In other words, I've been doing partial ports to UNIFIED_SOURCES only, not
full ports, but in this way we can still get 90% of the benefits and
sidestep problems caused by system headers. And 1) is generally considered
a good thing regardless.

Benoit



>
> -Boris

[Boris Zbarsky]

On 11/18/13 9:49 AM, Benoit Jacob wrote:
> 2) Keep these cpp files, that #include system headers, in plain old
> SOURCES, not in UNIFIED_SOURCES.

Maybe that's the right way to go. Enforce that no .cpp file that's in
UNIFIED_SOURCES ever includes the broken system headers, like we already
do for windows.h and the bindings files.

-Boris

[Neil]

Maybe a windows.h intercept that includes the real windows.h and then
undefines all the contentious symbols?

[Chris Peterson]

On 11/16/13, 1:34 AM, Ms2ger wrote:
> One way around it would be not to unify sources in automation. On one
> hand, this could cause more bustage when changes that built locally turn
> out not to have enough includes; on the other, it might be better than
> having to fix up a dozen unrelated files whenever you add a new one.

If we had a mozconfig flag to disable unified builds (i.e. treat
UNIFIED_SOURCES as SOURCES), interested parties can run non-unified
builds to find and fix accidental header dependencies at their leisure.

I agree with Ehsan, however, that we don't need to add extra bustage to
TBPL, so the builders should not use a non-unified flag. Plus we
especially want the speed and extra "pseudo-LTO" optimizations from
unified builds on the builders. :)


cpeterson

[Robert O'Callahan]

On Tue, Nov 19, 2013 at 3:31 AM, Boris Zbarsky <bzba...@mit.edu> wrote:

> While true, in the new setup we have a different problem: adding or
> removing a .cpp file makes other random .cpp files not compile.
>

I don't think we should worry much about this until we have more data on
how often it's a problem in practice. The bindings code is "special" in
that you have a single directory pulling in headers from all over Gecko and
files are added or removed from that directory frequently.

If unified builds turn out to cause too many problems, turning them off is
trivial. Most of the actual code changes going into the unified build work
are either good or neutral on their own merits.

Rob
--
Jtehsauts tshaei dS,o n" Wohfy Mdaon yhoaus eanuttehrotraiitny eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o Whhei csha iids teoa
stiheer :p atroa lsyazye,d 'mYaonu,r "sGients uapr,e tfaokreg iyvoeunr,
'm aotr atnod sgaoy ,h o'mGee.t" uTph eann dt hwea lmka'n? gBoutt uIp
waanndt wyeonut thoo mken.o w

[Ehsan Akhgari]

On 2013-11-17 5:39 PM, Mike Hommey wrote:
> On Sat, Nov 16, 2013 at 10:34:38AM +0100, Ms2ger wrote:

> Another related problem is that of conflicting function or variable
> names, macros, etc. We're obviously fixing those when they need to be
> when unifying files in a given directory, but there is a chance a
> conflict is introduced by a file moving from Unified1 to Unified2.

Yes, although I'm trying to be a bit smart here and look around for that
kind of problem when it's obvious in the patches that I write for this.

Cheers,
Ehsan

[Ehsan Akhgari]

Correct.

Ehsan

[Ehsan Akhgari]

On 2013-11-17 7:53 PM, Jonas Sicking wrote:
> On Sat, Nov 16, 2013 at 1:34 AM, Ms2ger <ms2...@gmail.com> wrote:
>>

>> One issue I only thought of today: this means that Source2.cpp can use all
>> headers included into Source1.cpp–until enough files are added between
>> Source1 and Source2 that Source2 ends up in the next unified file. This
>> hasn't been much of an issue for autogenerated C++, but it seems like it
>> could be harder to get right with hand-written C++.
>>
>> One way around it would be not to unify sources in automation. On one hand,
>> this could cause more bustage when changes that built locally turn out not
>> to have enough includes; on the other, it might be better than having to fix
>> up a dozen unrelated files whenever you add a new one.
>

> This sounds like a huge problem to me.
>
> There's also the problem that some header files, in particular on
> windows, add macros which can wreck havoc in our cpp files.
>
> This happened recently with our autogenerated C++ files where adding a
> .webidl file pushed a .cpp file from going from one unified file to
> the next. This .cpp file pulled in one of these windows headers. The
> result was that now a whole new set of files were exposed to the
> windows macros and some of them broke.
>
> This was a huge headache and I know that one of the solutions that
> were seriously considered was to rename the new .webidl file from
> foopy.webidl to zzfoopy.webidl.

A much easier solution for non-webidl unified code is to move foopy.cpp
to SOURCES (instead of UNIFIED_SOURCES) and then fix things up later on
if needed.

Cheers,
Ehsan

[Ehsan Akhgari]

On 2013-11-18 3:52 PM, Robert O'Callahan wrote:
> On Tue, Nov 19, 2013 at 3:31 AM, Boris Zbarsky <bzba...@mit.edu> wrote:
>

>> While true, in the new setup we have a different problem: adding or
>> removing a .cpp file makes other random .cpp files not compile.
>>
>

> I don't think we should worry much about this until we have more data on
> how often it's a problem in practice. The bindings code is "special" in
> that you have a single directory pulling in headers from all over Gecko and
> files are added or removed from that directory frequently.

Indeed.

> If unified builds turn out to cause too many problems, turning them off is
> trivial. Most of the actual code changes going into the unified build work
> are either good or neutral on their own merits.

Yeah. Specifically, none of them should make it impossible to build
each .cpp file individually.

Cheers,
Ehsan

[Ehsan Akhgari]

I wouldn't go all the way to that extreme. The list of broken system
headers is unfortunately quite large, and we already have lots of this
pattern in the tree:

#ifdef MyFunction
// screw you, windows.h/X.h/etc.
#undef MyFunction
#endif

Outlawing such inclusions altogether makes sense in the case of WebIDL
but I believe is too big of a shotgun for killing this fly in other
unified code. :-)

That being said, I have also excluded things that take a lot of effort
to fix from UNIFIED_SOURCES so far. For now, it's only the Mac system
headers which screw things up beyond repair.

Cheers,
Ehsan

[Ehsan Akhgari]

On 2013-11-17 7:45 PM, Jonas Sicking wrote:

> On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
>> I've started to work on a project in my spare time to switch us to use
>> unified builds for C/C++ compilation. The way that unified builds work is
>> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
>> files. With that, the build system creates files such as:
>>
>> // Unified_cpp_path_0.cpp
>> #include "Source1.cpp"
>> #include "Source2.cpp"
>> // ...
>

> Doesn't this negate the advantage of static global variables. I.e.
> when changing how you use a static global, rather than just auditing
> the .cpp file where that static global lives, you now how to audit all
> .cpp files that are "unified together".

It does, but code that relies on this is just buggy. I've been thinking
about doing periodic non-unified builds on try, and that will catch this
kind of problem.

Cheers,
Ehsan

[Ehsan Akhgari]

On 2013-11-17 7:50 PM, L. David Baron wrote:

> On Sunday 2013-11-17 16:45 -0800, Jonas Sicking wrote:
>> On Thu, Nov 14, 2013 at 2:49 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
>>> I've started to work on a project in my spare time to switch us to use
>>> unified builds for C/C++ compilation. The way that unified builds work is
>>> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
>>> files. With that, the build system creates files such as:
>>>
>>> // Unified_cpp_path_0.cpp
>>> #include "Source1.cpp"
>>> #include "Source2.cpp"
>>> // ...
>>
>> Doesn't this negate the advantage of static global variables. I.e.
>> when changing how you use a static global, rather than just auditing
>> the .cpp file where that static global lives, you now how to audit all
>> .cpp files that are "unified together".
>

> Could we do a static analysis to look for things whose semantics are
> changed by this unification, such as statics with the same name
> between files that are/might be unified? (And could we make the
> tree turn red if it failed?)

That analysis is quite hard to perform if we try to catch all kinds of
such leakage. I think a periodic non-unified build is a much better way
of discovering such problems.

Cheers,
Ehsan

[L. David Baron]

I'm inclined to think it'll need to be more than periodic, actually.

I expect that otherwise we'd get pretty frequent bustage with people
forgetting to add #includes, and that bustage would then show up
when we add or remove files, which would make it a huge pain to add
or remove files.

But I'm actually also worried (perhaps *more* worried) about cases
where there are semantic differences but things will still compile,
such as two static variables of the same name and type, in different
files (e.g., "static bool gInitialized"). We could end up with
breakage both because of code that expects such variables to be
distinct, or from new code that expects such variables to be merged.

[Mike Hommey]

The spilling of "using namespace" across source files could also cause
some headaches.

Mike

[Ehsan Akhgari]

This is something which we can easily measure over a period of time.
For example, this has not happened at least as of earlier today since
the first build unification patch that I have written. My gut feeling
is that this problem would not occur very often in practice, but I could
be wrong.

> But I'm actually also worried (perhaps *more* worried) about cases
> where there are semantic differences but things will still compile,
> such as two static variables of the same name and type, in different
> files (e.g., "static bool gInitialized"). We could end up with
> breakage both because of code that expects such variables to be
> distinct, or from new code that expects such variables to be merged.

We can definitely write static analyses for such specific cases
(especially if we have reason to believe that they may be common
problems.) If you have ideas for such analyses please file bugs.

Cheers,
Ehsan

[Ehsan Akhgari]

On 2013-11-18 10:16 PM, Mike Hommey wrote:

>> But I'm actually also worried (perhaps *more* worried) about cases
>> where there are semantic differences but things will still compile,
>> such as two static variables of the same name and type, in different
>> files (e.g., "static bool gInitialized"). We could end up with
>> breakage both because of code that expects such variables to be
>> distinct, or from new code that expects such variables to be merged.
>

> The spilling of "using namespace" across source files could also cause
> some headaches.

I haven't seen much of that yet. My anecdotal experience suggests that
files in the same directory tend to have similar namespace using statements.

Cheers,
Ehsan

[Robert O'Callahan]

On Tue, Nov 19, 2013 at 4:04 PM, L. David Baron <dba...@dbaron.org> wrote:

> I expect that otherwise we'd get pretty frequent bustage with people
> forgetting to add #includes, and that bustage would then show up
> when we add or remove files, which would make it a huge pain to add
> or remove files.
>
> But I'm actually also worried (perhaps *more* worried) about cases
> where there are semantic differences but things will still compile,
> such as two static variables of the same name and type, in different
> files (e.g., "static bool gInitialized").
>

Fortunately two static variables with the same name in the same translation
unit is an error in C++, at least with gcc.

[L. David Baron]

On Tuesday 2013-11-19 17:08 +1300, Robert O'Callahan wrote:
> Fortunately two static variables with the same name in the same translation
> unit is an error in C++, at least with gcc.

Ah, indeed. I'd tested in C, where it wasn't an error, but I also
see an error with gcc in C++.

[Gregory Szorc]

On 11/14/13, 2:49 PM, Ehsan Akhgari wrote:
> I've started to work on a project in my spare time to switch us to use
> unified builds for C/C++ compilation. The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:

This conversion will also decrease the benefit of ccache since any
source file change now logically invalidates the TU for N other sources
sharing the same unified source. Before, we could have 9/10 ccache hits
and recompile 1/10 files. Now, we have 0/10 ccache hits and
/effectively/ recompile 10/10 files.

However, I suspect the benefits of unified compilation outweigh the
benefits of ccache for many developer workflows, so this shouldn't be a
concern.

[Jonathan Watt]

Just a heads-up for other LLDB users...

The last few days I've been driven nuts by Xcode failing to stop at some
breakpoints (it just lists them as pending). I've now tracked this down to the
use of UNIFIED_SOURCES. The issue is explained here:

http://lldb.llvm.org/troubleshooting.html

Unfortunately, setting target.inline-breakpoint-strategy to "always" doesn't
actually seem to work with the LLDB that comes with Xcode 5.0.1, and my
experience of LLDB builds that I've created myself has been that they're pretty
unstable.

I'm not sure how best to handle this just yet, but given this...

On 19/11/2013 03:04, L. David Baron wrote:
> On Monday 2013-11-18 18:44 -0500, Ehsan Akhgari wrote:
>> On 2013-11-17 7:50 PM, L. David Baron wrote:
>>> Could we do a static analysis to look for things whose semantics are
>>> changed by this unification, such as statics with the same name
>>> between files that are/might be unified? (And could we make the
>>> tree turn red if it failed?)
>>
>> That analysis is quite hard to perform if we try to catch all kinds
>> of such leakage. I think a periodic non-unified build is a much
>> better way of discovering such problems.
>
> I'm inclined to think it'll need to be more than periodic, actually.
>
> I expect that otherwise we'd get pretty frequent bustage with people
> forgetting to add #includes, and that bustage would then show up
> when we add or remove files, which would make it a huge pain to add
> or remove files.
>
> But I'm actually also worried (perhaps *more* worried) about cases
> where there are semantic differences but things will still compile,
> such as two static variables of the same name and type, in different
> files (e.g., "static bool gInitialized"). We could end up with
> breakage both because of code that expects such variables to be
> distinct, or from new code that expects such variables to be merged.

I may end up being the guinea pig that is perpetually having his builds broken
because he has to have a patch applied to revert all UNIFIED_SOURCES lines back
to SOURCES lines in order to debug anything...

Jonathan

[Bobby Holley]

On Wed, Nov 20, 2013 at 10:01 AM, Jonathan Watt <jw...@jwatt.org> wrote:

> I may end up being the guinea pig that is perpetually having his builds
> broken because he has to have a patch applied to revert all UNIFIED_SOURCES
> lines back to SOURCES lines in order to debug anything...
>

Given that new versions of XCode don't even ship with gdb anymore, I think
we need to treat lldb as a supported tool, and figure something out here. I
don't think it makes sense to have every developer on a new mac platform
reverting all the UNIFIED_SOURCES stuff.

bholley

[Ehsan Akhgari]

On Wed, Nov 20, 2013 at 1:27 PM, Bobby Holley <bobby...@gmail.com> wrote:

> On Wed, Nov 20, 2013 at 10:01 AM, Jonathan Watt <jw...@jwatt.org> wrote:
>

> > I may end up being the guinea pig that is perpetually having his builds
> > broken because he has to have a patch applied to revert all
> UNIFIED_SOURCES
> > lines back to SOURCES lines in order to debug anything...
> >
>

> Given that new versions of XCode don't even ship with gdb anymore, I think
> we need to treat lldb as a supported tool, and figure something out here. I
> don't think it makes sense to have every developer on a new mac platform
> reverting all the UNIFIED_SOURCES stuff.
>

I think somebody needs to file a bug with the lldb project and describe the
problem to them and ask them for a fix. We are not the only project using
unified builds, so this problem is not specific to Mozilla. Given the fact
that they can handle setting breakpoints in header files, they should be
able to fix this in the general case.

[Jonathan Watt]

I'm still investigating this, and will contact them once I understand a bit
better what's going on. For now I still wanted to give other LLDB using mozilla
devs a heads-up though.

Jonathan

[Bobby Holley]

Ok. What should we do in the mean time? Does anyone know if it's possible
to get gdb going with XCode 5 on Mavericks?

On Wed, Nov 20, 2013 at 10:30 AM, Ehsan Akhgari <ehsan....@gmail.com>wrote:

> On Wed, Nov 20, 2013 at 1:27 PM, Bobby Holley <bobby...@gmail.com>wrote:
>
>> On Wed, Nov 20, 2013 at 10:01 AM, Jonathan Watt <jw...@jwatt.org> wrote:
>>

>> > I may end up being the guinea pig that is perpetually having his builds
>> > broken because he has to have a patch applied to revert all
>> UNIFIED_SOURCES
>> > lines back to SOURCES lines in order to debug anything...
>> >
>>

>> Given that new versions of XCode don't even ship with gdb anymore, I think
>> we need to treat lldb as a supported tool, and figure something out here.
>> I
>> don't think it makes sense to have every developer on a new mac platform
>> reverting all the UNIFIED_SOURCES stuff.
>>
>
> I think somebody needs to file a bug with the lldb project and describe
> the problem to them and ask them for a fix. We are not the only project
> using unified builds, so this problem is not specific to Mozilla. Given
> the fact that they can handle setting breakpoints in header files, they
> should be able to fix this in the general case.
>

[Ehsan Akhgari]

Thanks! Please keep us in the loop.

Cheers,
Ehsan

[Ehsan Akhgari]

On 2013-11-20 1:34 PM, Bobby Holley wrote:
> Ok. What should we do in the mean time? Does anyone know if it's
> possible to get gdb going with XCode 5 on Mavericks?

This should help in the mean time:

https://bugzilla.mozilla.org/show_bug.cgi?id=939583#c3

> On Wed, Nov 20, 2013 at 10:30 AM, Ehsan Akhgari <ehsan....@gmail.com

> <mailto:ehsan....@gmail.com>> wrote:
>
> On Wed, Nov 20, 2013 at 1:27 PM, Bobby Holley <bobby...@gmail.com
> <mailto:bobby...@gmail.com>> wrote:
>
> On Wed, Nov 20, 2013 at 10:01 AM, Jonathan Watt <jw...@jwatt.org

> <mailto:jw...@jwatt.org>> wrote:
>
> > I may end up being the guinea pig that is perpetually having
> his builds
> > broken because he has to have a patch applied to revert all
> UNIFIED_SOURCES
> > lines back to SOURCES lines in order to debug anything...
> >
>

> Given that new versions of XCode don't even ship with gdb
> anymore, I think
> we need to treat lldb as a supported tool, and figure something
> out here. I
> don't think it makes sense to have every developer on a new mac
> platform
> reverting all the UNIFIED_SOURCES stuff.
>
>
> I think somebody needs to file a bug with the lldb project and
> describe the problem to them and ask them for a fix. We are not the
> only project using unified builds, so this problem is not specific
> to Mozilla. Given the fact that they can handle setting breakpoints
> in header files, they should be able to fix this in the general case.
>

[Vladimir Vukicevic]

I just did a unified and non-unified build on my windows desktop -- non SSD. VS2012, using mozmake. Full clobber. (mozmake -s -j8)

Unified: 20 min
Non-Unified: 36 min

This is huge! I was curious about the cost for incremental builds...

touch gfx/2d/Factory.cpp (part of a unified file), rebuild using binaries target:

Unified: 53s
Non-Unified: 58s

touch gfx/thebes/gfxPlatform.cpp (note: this dir/file is not unified), rebuild using binaries target:

Unified: 56s
Non-Unified: 56s

(I need to rerun this on my computer with a SSD; I had a single-file binaries rebuild down to 10s there)

... and was very surprised to see no real difference, often non-unified taking slightly longer. So. Big win, thanks guys!

- Vlad

[Zack Weinberg]

On 2013-11-18 5:41 PM, Ehsan Akhgari wrote:
>
> I wouldn't go all the way to that extreme. The list of broken system
> headers is unfortunately quite large, and we already have lots of this
> pattern in the tree:
>
> #ifdef MyFunction
> // screw you, windows.h/X.h/etc.
> #undef MyFunction
> #endif

A small note about this pattern: the #ifdef wrapper is unnecessary, i.e.
it is safe to write only

// screw you, X.h
#undef MyFunction

because #undef is specified to be a silent NOP when applied to an
identifier with no macro definition.

I am uneasy with the entire UNIFIED_SOURCES concept; I worry that it
will lead to symbol-visibility problems down the road, when A.cpp
defines things that aren't meant to be accessible elsewhere, B.cpp
(which comes after it in the relevant unification block) starts using
them, and nobody notices until decoupling them has become a major
refactoring job.

But I'm not so uneasy that I'm going to get in the way of faster builds. :-)

zw

[Gregory Szorc]

Unified sources have probably saved sufficient CPU cycles across all of
automation to more than offset having a non-unified build-only job. I'll
defer to the Platform Team (Ehsan?) to request that from Release
Engineering.

[Ehsan Akhgari]

On 2013-11-20 5:17 PM, Gregory Szorc wrote:
> On 11/20/13, 2:02 PM, Zack Weinberg wrote:

> Unified sources have probably saved sufficient CPU cycles across all of
> automation to more than offset having a non-unified build-only job. I'll
> defer to the Platform Team (Ehsan?) to request that from Release
> Engineering.

I'll try to find somebody else to own this. I am the wrong person for
that task myself.

Cheers,
Ehsan

[Jonathan Watt]

Right now I'm seeing some weird behavior, but it seems
target.inline-breakpoint-strategy can basically be made to work.

When I first added the line to set target.inline-breakpoint-strategy in my
~/.lldbinit as per:

http://lldb.llvm.org/troubleshooting.html

it didn't work, even after restarting Xcode and starting a new debugging
session. Touching the file I had the breakpoint in, rebuilding and restarting
Xcode didn't help either. Some time later I pulled and rebuilt and suddenly
breakpoints were working again. Since then I've run a few experiments trying to
figure out what was needed to get the target.inline-breakpoint-strategy setting
to work, but without any luck. For now I'm giving up, and suggest that anyone
else that finds setting target.inline-breakpoint-strategy doesn't immediately
work shuts Xcode and does a clobber build before reopening it...or something.

I'll update the OS X Debugging wiki page. Since it seems setting
target.inline-breakpoint-strategy can basically be made to work I don't plan on
contacting the LLDB guys right now.

Jonathan

[Daniel Glastonbury]

On Thursday, November 21, 2013 9:37:05 AM UTC+10, Jonathan Watt wrote:
> When I first added the line to set target.inline-breakpoint-strategy in my
>
> ~/.lldbinit as per:
>
>
>
> http://lldb.llvm.org/troubleshooting.html
>
>
>
> it didn't work, even after restarting Xcode and starting a new debugging
>
> session.

I followed your advice. Quit Xcode, created .lldbinit, ./mach clobber && ./mach build and my missing breakpoints are back. Thank you.

cheers
Dan

[Ehsan Akhgari]

Can we add our own lldbinit to the tree the same way that we have a
gdbinit so that not everybody needs to go through this pain individually?

Cheers,
Ehsan

[Jonathan Watt]

On 21/11/2013 01:12, Daniel Glastonbury wrote:
> I followed your advice. Quit Xcode, created .lldbinit, ./mach clobber && ./mach build and my missing breakpoints are back. Thank you.

You're welcome.

[Jonathan Watt]

https://bugzilla.mozilla.org/show_bug.cgi?id=941539

[Michael Shal]

> From: "Gregory Szorc" <g...@mozilla.com>

> Unified sources have probably saved sufficient CPU cycles across all of
> automation to more than offset having a non-unified build-only job. I'll
> defer to the Platform Team (Ehsan?) to request that from Release
> Engineering.

How many CPU cycles would we have saved across all of automation if we didn't clobber all the time in the first place?

-Mike

[Ehsan Akhgari]

We don't, we clobber periodically. I'm not sure why that is better than
never clobbering...

Ehsan

[Ed Morley]

On 21 November 2013 15:38:28, Ehsan Akhgari wrote:
> We don't, we clobber periodically. I'm not sure why that is better
> than never clobbering...

We've been force-clobbering 1-2 times a day in automation for several
months now sadly, due to bug 928195 and similar prior. (Patches almost
ready to land at least :-))

[Gregory Szorc]

On 11/21/13, 7:06 AM, Michael Shal wrote:
>> From: "Gregory Szorc" <g...@mozilla.com>
>> Unified sources have probably saved sufficient CPU cycles across all of
>> automation to more than offset having a non-unified build-only job. I'll
>> defer to the Platform Team (Ehsan?) to request that from Release
>> Engineering.
>
> How many CPU cycles would we have saved across all of automation if we didn't clobber all the time in the first place?

Indeed. We'll get there. I was somewhat disappointed an in-tree Tup
backend didn't make the cut of Q4 build system goals.

But keep in mind that tests - not builds - represent the largest
financial cost to our automation. Obviously fast builds are important
because every other job depends on them completing and they are the
common component all developers need. But if shaving costs is your goal,
we should be heavily investing in testing improvements.

[1] contains the cost breakdown. Keep in mind that "build" jobs do a lot
more than just build. Here's a broken down Linux64 m-c build job from a
few hours ago:

* 80:00 total
* 3:20 chroot population (bug 851294 tracks improving)
* 2:58 repo updating (bug 851270 tracks improving)
* 35:22 building/compiling
* 5:57 symbols
* 1:02 package tests
* 1:20 packaging
* 4:11 uploading
* 2:00 misc other packaging
* 20:44 make check

I'd say ~45 minutes (56%) wall time is actual "build." The rest is tests
or automation overhead. This already makes build faster than browser
chrome mochitests. And considering the build system is one of a few
pieces of automation that attempts to use all CPU cores, it's one of the
most efficient pieces of automation. (AFAIK JS JIT tests and xpcshell
tests are the only other pieces of automation that scale out to N cores
reasonably effectively.) Need moar concurrency.

[1]
http://oduinn.com/blog/2013/11/20/the-financial-cost-of-a-checkin-part-1/

[Ehsan Akhgari]

Yes, but our periodic clobber has been in effect long before that bug
(and in fact for as long as I can remember.)

Cheers,
Ehsan

[ISHIKAWA,chiaki]

(2013/11/15 7:49), Ehsan Akhgari wrote:
> I've started to work on a project in my spare time to switch us to use
> unified builds for C/C++ compilation. The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:
>

> // Unified_cpp_path_0.cpp
> #include "Source1.cpp"
> #include "Source2.cpp"
> // ...
>

> And compiles them instead of the individual source files.
>

I just noticed my build locally has started to use unified build.
(I saw files like Unified_*.cpp files being compiled.)

Will this interfere interpreting the backtrace printed by
executable?

For example, this is a real example.
The backtrace of a crash in
https://bugzilla.mozilla.org/show_bug.cgi?id=931642#c87
https://tbpl.mozilla.org/php/getParsedLog.php?id=30894121&tree=Mozilla-Central

showed the following entries.

06:46:12 INFO - 0
XUL!mozilla::storage::Service::Observe(nsISupports*, char const*,
char16_t const*) [nsTArray.h : 871 + 0x0]
06:46:12 INFO - rbx = 0x00007fff74b3f630 r12 = 0x0000000103f088c0
[register dump omitted.]
06:46:12 INFO - Found by: given as instruction pointer in context
06:46:12 INFO - 1
XUL!_ZThn8_N7mozilla7storage7Service7ObserveEP11nsISupportsPKcPKDs
[Unified_cpp_storage_src1.cpp : 923 + 0x8]
06:46:12 INFO - rbx = 0x0000000000000005 r12 = 0x000000010692b4f8
[register dump omitted.]
06:46:12 INFO - Found by: call frame info

Note the file "Unified_cpp_storage_src1.cpp"
I looked for Unified_cpp_storage_src1.cpp using |mlocate| command under
linux and could not find it. So I thought this file was available only
under OS X (where the bug was reported.)

Only later, looking at this discussion on the mailing list, I realize
that this artificially looking file name is related to Unified Build.

How do people deal with the missing original file name in these dump or
during debugging?

TIA

PS: Is there a switch to disable Unified Build?

[Gregory Szorc]

Do we need periodic clobber? It's just wallpapering over legit
clobber-needed issues, which doesn't exactly help us fix them.

I'm fine with clobbering on "special" builds (such as Nightly and other
release builds).

[Ehsan Akhgari]

Hmm, I don't understand why you're seeing this. The debug information
should take the original file name, the same way that it does for other
#includes. How do you produce this build?

> PS: Is there a switch to disable Unified Build?

See bug 941097. But for now if you really need it you can manually edit
moz.build files and replace UNIFIED_SOURCES with SOURCES.

Cheers,
Ehsan

[ISHIKAWA,chiaki]

Well, I don't know exactly, but
https://tbpl.mozilla.org/php/getParsedLog.php?id=30894121&tree=Mozilla-Central

showed at near the top (the second line)

Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-5 on
2013-11-21 06:26:17 PST for push 7427eede548f

So this is a standard TBPL test run for MacOSX, I think, so it should
have used typical MacOSX tool chain. Someone in the list surely knows.

(I was asked to look into this bugzilla because a patch that I produced
*seemed* to increase the frequency of this bustage, but I think it is
totally unrelated. The particular bug looks like a MacOS X only thing IMHO.)

>> PS: Is there a switch to disable Unified Build?
>
> See bug 941097. But for now if you really need it you can manually edit
> moz.build files and replace UNIFIED_SOURCES with SOURCES.
>

If I need the original filename desperately, I would do this.
Mac OS X compiler or whatever tool chain should show the original file
name, indeed in the stack trace. (I don't know how the linux GCC
toolchain handles this. I will find out tomorrow.)

Thank you for the great work increasing the speed of compilation.

Chiaki Ishikawa

> Cheers,
> Ehsan
>
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
>

[Ehsan Akhgari]

Hmm, perhaps Ted knows why this happens? This is very surprising to me...

>>> PS: Is there a switch to disable Unified Build?
>>
>> See bug 941097. But for now if you really need it you can manually edit
>> moz.build files and replace UNIFIED_SOURCES with SOURCES.
>>
>
> If I need the original filename desperately, I would do this.
> Mac OS X compiler or whatever tool chain should show the original file
> name, indeed in the stack trace. (I don't know how the linux GCC
> toolchain handles this. I will find out tomorrow.)

FWIW if this proves to be common, it's a huge problem since it would
affect our crash stats etc... :(

> Thank you for the great work increasing the speed of compilation.

My pleasure!

Cheers,
Ehsan

[Ed Morley]

> On 11/21/13, 8:40 AM, Ehsan Akhgari wrote:
>> Yes, but our periodic clobber has been in effect long before that bug
>> (and in fact for as long as I can remember.)

Yes, but it's only once a week rather than a couple of times a day :-)

On 21/11/2013 16:45, Gregory Szorc wrote:
> Do we need periodic clobber? It's just wallpapering over legit
> clobber-needed issues, which doesn't exactly help us fix them.
>
> I'm fine with clobbering on "special" builds (such as Nightly and other
> release builds).

I guess they help clean up old repositories from our non-try build
slaves - however we could probably just clobber any directory unchanged
for longer than N days instead.

The default clobber time is set at:
http://mxr.mozilla.org/build-central/source/buildbot-configs/mozilla/config.py#47
and for mozharness jobs:
http://mxr.mozilla.org/build-central/source/mozharness/mozharness/mozilla/purge.py#31

The clobberer script can be found at:
http://mxr.mozilla.org/build-central/source/mozharness/external_tools/clobberer.py

[ISHIKAWA,chiaki]

(2013/11/22 2:17), Ehsan Akhgari wrote:
>
> FWIW if this proves to be common, it's a huge problem since it would
> affect our crash stats etc... :(

Well, if the problem is related to the symptom
that I observed with the use of -gsplit-dwarf option
with ccache, then we may be in for a big surprise.

But in that case, the debug information had a file name that
was produced by the initial CPP invocation (*.i), and despite the
#file, #line information, the invocation of gcc using this intermediate
file as input produced gdb debug information using "*.i" file name, and
thus later gdb could not locate source files at all.

Here we are NOT producing such a strange intermediate file, correct?

Hmm, but I think I see some trouble coming up on the horizon :-(
(A few tweaks to the front end of compilers affected may be necessary.)

TIA

[Karl Tomlinson]

Gregory Szorc writes:

> Do we need periodic clobber? It's just wallpapering over legit
> clobber-needed issues, which doesn't exactly help us fix them.

The problem is that unfixed bugs in dependences mean that code
bugs can sometimes not show up until the next clobber. There is
then a huge regression range if there is a week's worth of
check-ins.

A week interval is plenty long enough for any bugs to show up.

[Mike Hommey]

On Thu, Nov 14, 2013 at 05:49:33PM -0500, Ehsan Akhgari wrote:
> I've started to work on a project in my spare time to switch us to use
> unified builds for C/C++ compilation. The way that unified builds work is
> by using the UNIFIED_SOURCES instead of the SOURCES variable in moz.build
> files. With that, the build system creates files such as:
>
> // Unified_cpp_path_0.cpp
> #include "Source1.cpp"
> #include "Source2.cpp"
> // ...
>
> And compiles them instead of the individual source files.
>

> The advantage of this is that it speeds up the compilation (I've measured
> between 6-15x speed improvement depending on the code in question
> locally.) But there are also trade-offs with this approach. One trade-off
> is that the source code might require changes before it can be compiled in
> this way, due to things like name clashes, etc. The other one is that if
> you change one .cpp file which is built in unified mode, we would spend
> more time compiling it because we'll be compiling the unified .cpp file
> which includes more than what you have changed. It's hard to come up with
> numbers for this trade-off, but assuming that the linking step takes way
> longer than the C++ compilation in the "touch one .cpp file" scenario, and
> also that changes to headers will potentially trigger multiple .cpp
> rebuilds in the same directory, I think doing unified builds in more parts
> of the tree is a reasonable choice.
>
> I'm going to continue working on this as time permits, if you're interested
> in helping out, or if you have questions, please get in touch.

Well, it didn't take long. A build with unified sources disabled already
fails:

layout/style/nsStyleUtil.h:62:12: error: member access into incomplete
type 'nsAString_internal'
aResult.AppendFloat(aNumber);
^
../../dist/include/nsXPCOM.h:20:12: note: forward declaration
of 'nsAString_internal'
DECL_CLASS(nsAString);


Mike

[Ehsan Akhgari]

Hmm to the best of my knowledge, we don't generate the *.i files unless
if you explicitly request them. Is that what you did in that build?

Cheers,
Ehsan

[Ehsan Akhgari]

Filed bug 942193.

Ehsan

[Boris Zbarsky]

On 11/22/13 11:41 AM, Ehsan Akhgari wrote:
> Hmm to the best of my knowledge, we don't generate the *.i files unless
> if you explicitly request them. Is that what you did in that build?

ccache obvously always generates .i files, since those are what it uses
as its cache key.

And then it compiles the .o from the .i unless you explicitly tell it to
CCACHE_CPP2, right?

-Boris

[Ehsan Akhgari]

Yes, absolutely, but I thought that we don't do any of this for our
tinderbox builds. Are you saying that we're doing that?

--
Ehsan
<http://ehsanakhgari.org/>

[Gregory Szorc]

On Nov 22, 2013, at 13:16, Ehsan Akhgari <ehsan....@gmail.com> wrote:

> On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
>

> Yes, absolutely, but I thought that we don't do any of this for our
> tinderbox builds. Are you saying that we're doing that?

The build system adds CCACHE_CPP2 if building with Clang (not with GCC).

[Ehsan Akhgari]

On 2013-11-22 4:25 PM, Gregory Szorc wrote:
> On Nov 22, 2013, at 13:16, Ehsan Akhgari <ehsan....@gmail.com> wrote:
>
>> On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
>>

>> Yes, absolutely, but I thought that we don't do any of this for our
>> tinderbox builds. Are you saying that we're doing that?
>
> The build system adds CCACHE_CPP2 if building with Clang (not with GCC).

Why just for clang?

[Mike Hommey]

On Fri, Nov 22, 2013 at 07:24:14PM -0500, Ehsan Akhgari wrote:
> On 2013-11-22 4:25 PM, Gregory Szorc wrote:
> >On Nov 22, 2013, at 13:16, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> >
> >>On Fri, Nov 22, 2013 at 1:02 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
> >>

> >>Yes, absolutely, but I thought that we don't do any of this for our
> >>tinderbox builds. Are you saying that we're doing that?
> >
> >The build system adds CCACHE_CPP2 if building with Clang (not with GCC).
>
> Why just for clang?

https://bugzilla.mozilla.org/show_bug.cgi?id=754988

Mike

[ISHIKAWA,chiaki]

(2013/11/23 1:41), Ehsan Akhgari wrote:
> On 2013-11-21 1:12 PM, ISHIKAWA,chiaki wrote:
>> (2013/11/22 2:17), Ehsan Akhgari wrote:
>>>
>>> FWIW if this proves to be common, it's a huge problem since it would
>>> affect our crash stats etc... :(
>>
>> Well, if the problem is related to the symptom
>> that I observed with the use of -gsplit-dwarf option
>> with ccache, then we may be in for a big surprise.
>>
>> But in that case, the debug information had a file name that
>> was produced by the initial CPP invocation (*.i), and despite the
>> #file, #line information, the invocation of gcc using this intermediate
>> file as input produced gdb debug information using "*.i" file name, and
>> thus later gdb could not locate source files at all.
>>
>> Here we are NOT producing such a strange intermediate file, correct?
>>
>> Hmm, but I think I see some trouble coming up on the horizon :-(
>> (A few tweaks to the front end of compilers affected may be necessary.)
>

> Hmm to the best of my knowledge, we don't generate the *.i files unless
> if you explicitly request them. Is that what you did in that build?
>

No, the build　in question is from automated periodic test.
(So I have no control over how it is compiled, etc. It should be using
the standard OSX build toolchain with default setting, etc.)

TIA

> Cheers,
> Ehsan
>
>

[ISHIKAWA,chiaki]

I am trying to create a low-disk condition for testing I/O errors during
|make mozmill| test suite run of thunderbird.

I am not sure where the various files are stored during |make mozmill|.
I can think of TMP, TMPDIR, and a few other
settings.
Does anyone have an idea of overview of the file used?
(Like log files for captured during run?)

If I know where the files are stored, I can either create a mini file
system and store these in such file systems, OR
I can create a large file with dd if=/dev/zero ...
to use up enough space to cause TB to hit ENOSPC error during execution.

On my own, I found that mozmill seems to use the following directory for
a TB profile that is used during the test run, and that is where mail
folders are stored, for example.

Quote from |make mozmill| run.
Using profile dir:
/REF-OBJ-DIR/objdir-tb3/mozilla/_tests/mozmill/mozmillprofile

/REF-OBJ-DIR/objdir-tb3 is my MOZ_OBJDIR and so I monitored how much
storage I should set aside for this directory to start running |make
mozmill| and then yet get some I/O errors due to filled-up file system.

To my surprise, the du the-above-directory shows
4-14 MB usage during the |make mozmill| run, but at the end, it jumps up
to 30+ MiB and stays at this number after |make mozmill| finished. I was
surprised at this number

Here is what I found out about the used space of the mozmillprofile
directory after the |make mozmill| test suite finished.

Can anyone shed light on the BIG CACHE file that exist AFTER |make
mozmill| finished?

588 ./plugins
16 ./Mail/Local Folders
12 ./Mail/tinderbox
32 ./Mail
4 ./Cache/5
4 ./Cache/F
4 ./Cache/2
4 ./Cache/0
4 ./Cache/8
4 ./Cache/C
4 ./Cache/4
4 ./Cache/1
4 ./Cache/E
4 ./Cache/3
4 ./Cache/7
4 ./Cache/D
4 ./Cache/9
4 ./Cache/B
4 ./Cache/6
4 ./Cache/A
12380 ./Cache <==== THIS seems to the BIG JUMP at the end.
2440 ./startupCache
8 ./extensions/moz...@mozilla.com/skin
160 ./extensions/moz...@mozilla.com/resource/modules
256 ./extensions/moz...@mozilla.com/resource/stdlib
420 ./extensions/moz...@mozilla.com/resource
8 ./extensions/moz...@mozilla.com/defaults/preferences
g12 ./extensions/moz...@mozilla.com/defaults
12 ./extensions/moz...@mozilla.com/locale/en-US
16 ./extensions/moz...@mozilla.com/locale
76 ./extensions/moz...@mozilla.com/content/css/smoothness/images
108 ./extensions/moz...@mozilla.com/content/css/smoothness
124 ./extensions/moz...@mozilla.com/content/css
12 ./extensions/moz...@mozilla.com/content/img
16
./extensions/moz...@mozilla.com/content/editor/bespin/resources/screen_theme
20 ./extensions/moz...@mozilla.com/content/editor/bespin/resources
352 ./extensions/moz...@mozilla.com/content/editor/bespin
360 ./extensions/moz...@mozilla.com/content/editor
20 ./extensions/moz...@mozilla.com/content/test
308 ./extensions/moz...@mozilla.com/content/jquery
928 ./extensions/moz...@mozilla.com/content
1412 ./extensions/moz...@mozilla.com
12 ./extensions/jsbr...@mozilla.com/components
44 ./extensions/jsbr...@mozilla.com/resource/modules
48 ./extensions/jsbr...@mozilla.com/resource
12 ./extensions/jsbr...@mozilla.com/chrome/content
16 ./extensions/jsbr...@mozilla.com/chrome
88 ./extensions/jsbr...@mozilla.com
1504 ./extensions
4 ./minidumps
30180 .

I could probably create 15 MiB file system (using loopback device) and
copy the files (except for CACHE) and create a symlink to the new file
system and run |make mozmill|, but I wonder that impact of this large
CCACHE before |make mozmill| runs and at the end.

TOA

[Ehsan Akhgari]

It's hard to say exactly what's going on here without knowing more about
how this build is produced. It would be really great if you could file
a bug about this with more details on how to reproduce this broken build.

Thanks!
Ehsan

[Neil]

Ehsan Akhgari wrote:

> It's hard to say exactly what's going on here without knowing more
> about how this build is produced.

These are intermittent failures on TBPL, mostly starred by philor.

> It would be really great if you could file a bug about this with more
> details on how to reproduce this broken build.

It would be really great if you had followed his link to bug 931642 in
the first place.

--
Warning: May contain traces of nuts.