Estimating moz.build completion

[Gregory Szorc]

I've been asked to estimate how long it will take to complete the
migration of build config data from Makefile.in to moz.build files.

An estimation is very difficult to perform for a few reasons:

a) There are many parts of build config. A good estimation would look at
every one independently and that would take a lot of time and would
arguably not increase the accuracy of the estimation (given how bad
software estimates tend to be).

b) Estimation assumes we're working on just conversion. I don't believe
its wise or optimal for us to focus just on the conversion - I would
rather we simultaneously work on build speed enhancements as they become
available.

Anyway, I do have some numbers.

Using the BXR tool, I was able to filter out Makefile.in belonging to
external projects as well as "special" Makefile.in (such as the root
Makefile.in and those pertaining to packaging).

The net results on mozilla-central are:

1017 makefiles
204 makefiles with rules
813 makefiles with no rules
1154 variable names
551 rules
234 target names
578 targets
220 tier targets
358 non-tier targets
75 command names
724 commands

(This table is now available in the output of BXR)

In my post yesterday, I posted the list of most common variable
assignments. Variable assignments are one large source of conversion
work. The other source are make targets/rules. And, I argue

We have 551 rules executing 724 commands (a command is a line in a make
rule). It helps to break these commands down because each one
potentially warrants special consideration.

The following lists the number of Makefile.in popular commands occur in:

$(INSTALL) 95
$(NSINSTALL) 44
Preprocessor.py 29
$(MAKE) 20
$(SYSINSTALL) 19
cp 16
cd 16
rm 14
$(RM) 14
sed 11
echo 8
$^ 7
$(MKDIR) 7
... and a long tail of a few dozen more.

Many of these commands are things we already have better primitives for.
e.g. INSTALL_TARGETS and PP_TARGETS. If Makefile.in were following best
practices, about half of the existing rules would go away.

Anyway, let's estimate each rule in a Makefile.in takes 30 minutes to
port to moz.build. Simple install rules will take less. More complicated
actions (like say WebIDL or IPDL rules) will likely take a few hours.
551 rules x 30 minutes is ~280 hours, or 7 person-weeks. Multiply that
number by 4 (to account for variables porting and because this is
software), add 200 hours for reviews, and we're looking at ~1300 hours,
or 32 person-weeks. Figure that whoever is working on this won't be
doing it full time, and we're realistically looking at another year or
so until completion. But, this is software estimation, so there is a
giant asterisk on all these numbers.

I'd be interested in what other build peers think about these numbers.

[Benjamin Smedberg]

On 7/17/2013 9:36 PM, Gregory Szorc wrote:
> The following lists the number of Makefile.in popular commands occur in:
>
> $(INSTALL) 95
> $(NSINSTALL) 44
> Preprocessor.py 29

> $(SYSINSTALL) 19
> cp 16
It's likely that we need to fix all these in order to accomplish the
"remove the export tier" goal in Q3 anyway. If the answer is "port 95%
of those to use INSTALL_TARGETS and PP_TARGETS" this is hugely
parallelizable and we can just burn it down by publishing a list and
asking for volunteers.

Can you publish this list of rules?

The other data-point which might be interesting is determining which
phase (export/libs/tools) each of these targets runs at. Then we can
prioritize the ones which run during the export tier.

> 1154 variable names
Can we sort these into "variables used by config.mk/rules.mk" and
"variables only used in one particular makefile" and then by frequency?

--BDS

[ms...@mozilla.com]

On Wednesday, July 17, 2013 9:36:20 PM UTC-4, Gregory Szorc wrote:
> Many of these commands are things we already have better primitives for.
>
> e.g. INSTALL_TARGETS and PP_TARGETS. If Makefile.in were following best
>
> practices, about half of the existing rules would go away.
>
>
>
> Anyway, let's estimate each rule in a Makefile.in takes 30 minutes to
>
> port to moz.build. Simple install rules will take less. More complicated
>
> actions (like say WebIDL or IPDL rules) will likely take a few hours.
>
> 551 rules x 30 minutes is ~280 hours, or 7 person-weeks. Multiply that
>
> number by 4 (to account for variables porting and because this is
>
> software), add 200 hours for reviews, and we're looking at ~1300 hours,
>
> or 32 person-weeks. Figure that whoever is working on this won't be
>
> doing it full time, and we're realistically looking at another year or
>
> so until completion. But, this is software estimation, so there is a
>
> giant asterisk on all these numbers.

This time estimate is only for the rules listed in this post, not the variable conversions listed in 'moz.build conversion remaining variables', correct? Joey put together an estimate for the variable conversions based on our rate of progress and number of variables to convert (I can't seem to find the exact numbers at the moment, but I recall it was also fairly large).

A number of the high-frequency conversions are simple true/false variables, like LIBXUL_LIBRARY, FAIL_ON_WARNINGS, and others. These should be relatively straightforward. But we still do have some big ones that are more difficult to convert, like DEFINES, *FLAGS, *LIBS, *INCLUDES, etc.

Just want to make sure we aren't being overly ambitious here with our estimates.

-Mike

[plyf...@gmail.com]

"""""""""""""""""""""
http://sango.igetweb.com/
http://number1cosmetics.igetweb.com/
http://www.baldiny.com/