info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Python configure: status and discussing improvements
51 views
Skip to first unread message
Mike Hommey
Aug 16, 2016, 2:52:35 AM8/16/16
to dev-b...@lists.mozilla.org
Hi,
It's almost been 6 months since bug 1250294 landed, initiating the long
and slow move off autoconf.
Where are we, 6 months later? While there hasn't been constant progress
in the migration (we've all been busy with other things too), the
current status is that, pending in-flight bugs, the size of shell and m4
scripts is down below 70% of their original size.
https://plot.ly/~glandium/14/lines-vs-time/
The python configure script also does more checks than the autoconf
counterparts we moved. Most notably, we're now verifying that the
toolchain will actually build for the given --target.
Now, with 30% of the shell and m4 mess moved, and corresponding python
code written, it's time to sit back and look at what works and what
doesn't.
Here's my own list of things that I find suboptimal, with suggestions
when I have some.
- @depends('--disable-foo') and related are confusing. I still think
that at the sandbox level, it's a good thing that the options are
treated in consistent manner, ignoring their defaults. But referring
options by their name incurs some ambiguities along the negatives.
Maybe we should declare all options with their positive name
(--enable-foo, --with-foo), and set an explicit default of True when
the option is enabled by default. And make python configure reject
declarations with the negative name (--disable-foo, --without-foo).
- Relatedly, many of the help messages contain "Enable or disable",
and end up completely wrong when the default is opposite to what the
help says.
Maybe we can make all help messages start with e.g. "Enable"/
"Disable" or "Build with"/"Build without" automatically depending on
the default. The help argument would then *not* contain those.
Or we could have the --help handle replace "Enable" with "Disable" and
vice-versa, automatically...
- We have a lot of things like @depends_if(foo)(lambda x: True). Or
things largely equivalent to that, but in the full written form like
@depends(foo)
def bar(foo):
if foo:
return True
We probably need more helpers.
- It can be cumbersome to have to create @depends functions for
simple conditionals.
We have a bunch of templates that take a "when" argument. There also
is imply_option, set_config, set_define and
add_old_configure_assignment that all take @depends functions. In many
places, we do ad-hoc things like
depends(target)(lambda target: target.os == 'OSX')
or
delayed_getattr(milestone, 'is_nightly')
I know Nick didn't like the magic, but after having written to many of
those, I'm really considering magic would be nicer. What I was
thinking was to make calling a depends function possible, and it would
return a special class that create new pseudo depends function for
common operations.
Such that instead of
depends(target)(lambda target: target.os == 'OSX')
you could write:
target().os == 'OSX'
Instead of
delayed_getattr(milestone, 'is_nightly')
you could write:
milestone().is_nightly
Instead of
@depends(a, b)
def foo(a, b):
return a or b
you could write:
foo = a() or b()
etc.
- We need option defaults depending on the target. While I was looking
at some of the remaining options to move to python, I saw a lot of
them depend on the target. Option defaults need a dependency on
--help, which means the functions would run even when displaying
--help. Which makes a lot of sense, since --help is supposed to
display the defaults.
So far, we've avoided adding such a dependency for --target, because
it would mean running config.guess and config.sub during --help, and
it's not very desirable. I'm thinking about having --help imply a
--target derived in some minimalistic python code, avoiding both
config.guess and config.sub.
- I'm trying to enforce separation of concerns between generic things,
app-specific things and toolkit-specific things. This is something
that was essentially completely ignored in autoconf-base configure,
and leads to new problems. The main one is that the order in which
the python configure files are treated is, for most purposes, app first,
then toolkit, then generic. The problem is that e.g. toolchain tests are
in the latter, and sometimes we'd need for app or toolkit checks to do
toolchain-based checks (concrete example: --enable-valgrind needs to
check that valgrind.h can be found)
One thing I just thought about while writing the above is that we
could have a wrapper around the include() function that, when called
for foo.configure, would include all foo.configure files existing in
build/moz.configure/, toolkit/, and $build_project/. I haven't thought
too much about this one, it may or may not be a workable solution.
- There is no clear standard for app-overridable defaults. Sometimes we
have an imply_option in the app moz.configure, which means the option
can then never be overridden by a developer locally (however, that can
be the expected thing in some cases). Sometimes we have a default
function checking build_project. Sometimes we do a check with a
function that depends on the option (but I think those are usually
related to --target, rather than the app). I'm not sure there is
something nicer we could do here.
These are the items off the top of my head. Please add your own in a reply,
if you have others, and/or comments wrt the suggestions.
Cheers,
Mike
It's almost been 6 months since bug 1250294 landed, initiating the long
and slow move off autoconf.
Where are we, 6 months later? While there hasn't been constant progress
in the migration (we've all been busy with other things too), the
current status is that, pending in-flight bugs, the size of shell and m4
scripts is down below 70% of their original size.
https://plot.ly/~glandium/14/lines-vs-time/
The python configure script also does more checks than the autoconf
counterparts we moved. Most notably, we're now verifying that the
toolchain will actually build for the given --target.
Now, with 30% of the shell and m4 mess moved, and corresponding python
code written, it's time to sit back and look at what works and what
doesn't.
Here's my own list of things that I find suboptimal, with suggestions
when I have some.
- @depends('--disable-foo') and related are confusing. I still think
that at the sandbox level, it's a good thing that the options are
treated in consistent manner, ignoring their defaults. But referring
options by their name incurs some ambiguities along the negatives.
Maybe we should declare all options with their positive name
(--enable-foo, --with-foo), and set an explicit default of True when
the option is enabled by default. And make python configure reject
declarations with the negative name (--disable-foo, --without-foo).
- Relatedly, many of the help messages contain "Enable or disable",
and end up completely wrong when the default is opposite to what the
help says.
Maybe we can make all help messages start with e.g. "Enable"/
"Disable" or "Build with"/"Build without" automatically depending on
the default. The help argument would then *not* contain those.
Or we could have the --help handle replace "Enable" with "Disable" and
vice-versa, automatically...
- We have a lot of things like @depends_if(foo)(lambda x: True). Or
things largely equivalent to that, but in the full written form like
@depends(foo)
def bar(foo):
if foo:
return True
We probably need more helpers.
- It can be cumbersome to have to create @depends functions for
simple conditionals.
We have a bunch of templates that take a "when" argument. There also
is imply_option, set_config, set_define and
add_old_configure_assignment that all take @depends functions. In many
places, we do ad-hoc things like
depends(target)(lambda target: target.os == 'OSX')
or
delayed_getattr(milestone, 'is_nightly')
I know Nick didn't like the magic, but after having written to many of
those, I'm really considering magic would be nicer. What I was
thinking was to make calling a depends function possible, and it would
return a special class that create new pseudo depends function for
common operations.
Such that instead of
depends(target)(lambda target: target.os == 'OSX')
you could write:
target().os == 'OSX'
Instead of
delayed_getattr(milestone, 'is_nightly')
you could write:
milestone().is_nightly
Instead of
@depends(a, b)
def foo(a, b):
return a or b
you could write:
foo = a() or b()
etc.
- We need option defaults depending on the target. While I was looking
at some of the remaining options to move to python, I saw a lot of
them depend on the target. Option defaults need a dependency on
--help, which means the functions would run even when displaying
--help. Which makes a lot of sense, since --help is supposed to
display the defaults.
So far, we've avoided adding such a dependency for --target, because
it would mean running config.guess and config.sub during --help, and
it's not very desirable. I'm thinking about having --help imply a
--target derived in some minimalistic python code, avoiding both
config.guess and config.sub.
- I'm trying to enforce separation of concerns between generic things,
app-specific things and toolkit-specific things. This is something
that was essentially completely ignored in autoconf-base configure,
and leads to new problems. The main one is that the order in which
the python configure files are treated is, for most purposes, app first,
then toolkit, then generic. The problem is that e.g. toolchain tests are
in the latter, and sometimes we'd need for app or toolkit checks to do
toolchain-based checks (concrete example: --enable-valgrind needs to
check that valgrind.h can be found)
One thing I just thought about while writing the above is that we
could have a wrapper around the include() function that, when called
for foo.configure, would include all foo.configure files existing in
build/moz.configure/, toolkit/, and $build_project/. I haven't thought
too much about this one, it may or may not be a workable solution.
- There is no clear standard for app-overridable defaults. Sometimes we
have an imply_option in the app moz.configure, which means the option
can then never be overridden by a developer locally (however, that can
be the expected thing in some cases). Sometimes we have a default
function checking build_project. Sometimes we do a check with a
function that depends on the option (but I think those are usually
related to --target, rather than the app). I'm not sure there is
something nicer we could do here.
These are the items off the top of my head. Please add your own in a reply,
if you have others, and/or comments wrt the suggestions.
Cheers,
Mike
Mike Hommey
Aug 17, 2016, 5:30:17 AM8/17/16
to dev-b...@lists.mozilla.org
On Tue, Aug 16, 2016 at 03:51:51PM +0900, Mike Hommey wrote:
> Here's my own list of things that I find suboptimal, with suggestions
> when I have some.
> Here's my own list of things that I find suboptimal, with suggestions
> when I have some.
> - I'm trying to enforce separation of concerns between generic things,
> app-specific things and toolkit-specific things. This is something
> that was essentially completely ignored in autoconf-base configure,
> and leads to new problems. The main one is that the order in which
> the python configure files are treated is, for most purposes, app first,
> then toolkit, then generic. The problem is that e.g. toolchain tests are
> in the latter, and sometimes we'd need for app or toolkit checks to do
> toolchain-based checks (concrete example: --enable-valgrind needs to
> check that valgrind.h can be found)
>
> One thing I just thought about while writing the above is that we
> could have a wrapper around the include() function that, when called
> for foo.configure, would include all foo.configure files existing in
> build/moz.configure/, toolkit/, and $build_project/. I haven't thought
> too much about this one, it may or may not be a workable solution.
Partly related, there's another one:
> app-specific things and toolkit-specific things. This is something
> that was essentially completely ignored in autoconf-base configure,
> and leads to new problems. The main one is that the order in which
> the python configure files are treated is, for most purposes, app first,
> then toolkit, then generic. The problem is that e.g. toolchain tests are
> in the latter, and sometimes we'd need for app or toolkit checks to do
> toolchain-based checks (concrete example: --enable-valgrind needs to
> check that valgrind.h can be found)
>
> One thing I just thought about while writing the above is that we
> could have a wrapper around the include() function that, when called
> for foo.configure, would include all foo.configure files existing in
> build/moz.configure/, toolkit/, and $build_project/. I haven't thought
> too much about this one, it may or may not be a workable solution.
- there are optional parts of moz.configure that are completely unknown
when they are not included. This can complicate things in some cases.
See for example extra_toolchain_flags. IIRC, there are a few other
similar cases, and I can see other such cases appearing as we move
more things to python configure.
By extension, I've been revisiting the idea from bug 1259272 to have a
tool to do some introspection, and that not everything is visible can
make it awkward to use such a tool. The worst offender being
--enable-project/--enable-application.
I came up with a prototype patch that seems to work and would make
everything known. The idea is to add a context manager to the sandbox,
that works as follows:
with only_when(<some_depends_function>):
python_configure_stuff()
We can then rewrite things like depends_when or include_when using
that context manager. For example, include_when (which is the only
thing I tested so far) becomes:
@template
def include_when(filename, when):
with only_when(when):
include(filename)
I expect it would also allow to remove depends_win from
windows.configure, for example.
Thoughts?
Mike
Ted Mielczarek
Aug 17, 2016, 5:49:16 AM8/17/16
to dev-b...@lists.mozilla.org
moz.build is an inability to introspect the entire state of things
because of conditionals, it would be nice to fix that situation in
moz.configure.
-Ted
Christopher Manchester
Aug 18, 2016, 1:22:56 PM8/18/16
to dev-builds
On Mon, Aug 15, 2016 at 11:51 PM, Mike Hommey <m...@glandium.org> wrote:
Hi,
It's almost been 6 months since bug 1250294 landed, initiating the long
and slow move off autoconf.
Where are we, 6 months later? While there hasn't been constant progress
in the migration (we've all been busy with other things too), the
current status is that, pending in-flight bugs, the size of shell and m4
scripts is down below 70% of their original size.
https://plot.ly/~glandium/14/lines-vs-time/
The python configure script also does more checks than the autoconf
counterparts we moved. Most notably, we're now verifying that the
toolchain will actually build for the given --target.
Now, with 30% of the shell and m4 mess moved, and corresponding python
code written, it's time to sit back and look at what works and what
doesn't.
Here's my own list of things that I find suboptimal, with suggestions
when I have some.
- @depends('--disable-foo') and related are confusing. I still think
that at the sandbox level, it's a good thing that the options are
treated in consistent manner, ignoring their defaults. But referring
options by their name incurs some ambiguities along the negatives.
Maybe we should declare all options with their positive name
(--enable-foo, --with-foo), and set an explicit default of True when
the option is enabled by default. And make python configure reject
declarations with the negative name (--disable-foo, --without-foo).
This is probably one of the more confusing points when reading what we have so far. Related to this, I've always found it a bit disconcerting that @depends takes any depends function, or a string as an argument, but only if that string names an option. We could declare options without prefixes (requiring explicit defaults) and convert these to depends functions automatically so that option(name='foo', ...) would allow you to write @depends(foo) immediately thereafter, and never pass a string to @depends.
I support it. It would save a lot of boilerplate. We could manage some of this more templates instead, but writing lots of combinators for depends functions (making the last example "or(a, b)") doesn't seem that great either. Another idea that only differs syntactically, we could have a special function that converts a depends function into something that implements builtin operators in a way that produces other depends functions... so examples above would become something like "valueof(target).os" and "valueof(a) or valueof(b)". This might be a little less magical.
- We need option defaults depending on the target. While I was looking
at some of the remaining options to move to python, I saw a lot of
them depend on the target. Option defaults need a dependency on
--help, which means the functions would run even when displaying
--help. Which makes a lot of sense, since --help is supposed to
display the defaults.
So far, we've avoided adding such a dependency for --target, because
it would mean running config.guess and config.sub during --help, and
it's not very desirable. I'm thinking about having --help imply a
--target derived in some minimalistic python code, avoiding both
config.guess and config.sub.
- I'm trying to enforce separation of concerns between generic things,
app-specific things and toolkit-specific things. This is something
that was essentially completely ignored in autoconf-base configure,
and leads to new problems. The main one is that the order in which
the python configure files are treated is, for most purposes, app first,
then toolkit, then generic. The problem is that e.g. toolchain tests are
in the latter, and sometimes we'd need for app or toolkit checks to do
toolchain-based checks (concrete example: --enable-valgrind needs to
check that valgrind.h can be found)
One thing I just thought about while writing the above is that we
could have a wrapper around the include() function that, when called
for foo.configure, would include all foo.configure files existing in
build/moz.configure/, toolkit/, and $build_project/. I haven't thought
too much about this one, it may or may not be a workable solution.
- There is no clear standard for app-overridable defaults. Sometimes we
have an imply_option in the app moz.configure, which means the option
can then never be overridden by a developer locally (however, that can
be the expected thing in some cases). Sometimes we have a default
function checking build_project. Sometimes we do a check with a
function that depends on the option (but I think those are usually
related to --target, rather than the app). I'm not sure there is
something nicer we could do here.
These are the items off the top of my head. Please add your own in a reply,
if you have others, and/or comments wrt the suggestions.
Cheers,
Mike
_______________________________________________
dev-builds mailing list
dev-b...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds
Christopher Manchester
Aug 18, 2016, 1:23:29 PM8/18/16
to dev-builds
On Wed, Aug 17, 2016 at 2:29 AM, Mike Hommey <m...@glandium.org> wrote:
On Tue, Aug 16, 2016 at 03:51:51PM +0900, Mike Hommey wrote:
> Here's my own list of things that I find suboptimal, with suggestions
> when I have some.
> - I'm trying to enforce separation of concerns between generic things,
> app-specific things and toolkit-specific things. This is something
> that was essentially completely ignored in autoconf-base configure,
> and leads to new problems. The main one is that the order in which
> the python configure files are treated is, for most purposes, app first,
> then toolkit, then generic. The problem is that e.g. toolchain tests are
> in the latter, and sometimes we'd need for app or toolkit checks to do
> toolchain-based checks (concrete example: --enable-valgrind needs to
> check that valgrind.h can be found)
>
> One thing I just thought about while writing the above is that we
> could have a wrapper around the include() function that, when called
> for foo.configure, would include all foo.configure files existing in
> build/moz.configure/, toolkit/, and $build_project/. I haven't thought
> too much about this one, it may or may not be a workable solution.
Partly related, there's another one:
- there are optional parts of moz.configure that are completely unknown
when they are not included. This can complicate things in some cases.
See for example extra_toolchain_flags. IIRC, there are a few other
similar cases, and I can see other such cases appearing as we move
more things to python configure.
By extension, I've been revisiting the idea from bug 1259272 to have a
tool to do some introspection, and that not everything is visible can
make it awkward to use such a tool. The worst offender being
--enable-project/--enable-application.
I came up with a prototype patch that seems to work and would make
everything known. The idea is to add a context manager to the sandbox,
that works as follows:
with only_when(<some_depends_function>):
python_configure_stuff()
We can then rewrite things like depends_when or include_when using
that context manager. For example, include_when (which is the only
thing I tested so far) becomes:
@template
def include_when(filename, when):
with only_when(when):
include(filename)
I expect it would also allow to remove depends_win from
windows.configure, for example.
This looks highly desirable. I'm not sure exactly how this is going to work from looking at the example, but figuring out what's available based on what's possibly been included is one of the more confusing things about working with what we have.
Mike Hommey
Aug 18, 2016, 8:23:01 PM8/18/16
to Christopher Manchester, dev-builds
On Thu, Aug 18, 2016 at 10:22:24AM -0700, Christopher Manchester wrote:
> On Mon, Aug 15, 2016 at 11:51 PM, Mike Hommey <m...@glandium.org> wrote:
>
> > Hi,
> >
> > It's almost been 6 months since bug 1250294 landed, initiating the long
> > and slow move off autoconf.
> >
> > Where are we, 6 months later? While there hasn't been constant progress
> > in the migration (we've all been busy with other things too), the
> > current status is that, pending in-flight bugs, the size of shell and m4
> > scripts is down below 70% of their original size.
> > https://plot.ly/~glandium/14/lines-vs-time/
> >
> > The python configure script also does more checks than the autoconf
> > counterparts we moved. Most notably, we're now verifying that the
> > toolchain will actually build for the given --target.
> >
> > Now, with 30% of the shell and m4 mess moved, and corresponding python
> > code written, it's time to sit back and look at what works and what
> > doesn't.
> >
> On Mon, Aug 15, 2016 at 11:51 PM, Mike Hommey <m...@glandium.org> wrote:
>
> > Hi,
> >
> > It's almost been 6 months since bug 1250294 landed, initiating the long
> > and slow move off autoconf.
> >
> > Where are we, 6 months later? While there hasn't been constant progress
> > in the migration (we've all been busy with other things too), the
> > current status is that, pending in-flight bugs, the size of shell and m4
> > scripts is down below 70% of their original size.
> > https://plot.ly/~glandium/14/lines-vs-time/
> >
> > The python configure script also does more checks than the autoconf
> > counterparts we moved. Most notably, we're now verifying that the
> > toolchain will actually build for the given --target.
> >
> > Now, with 30% of the shell and m4 mess moved, and corresponding python
> > code written, it's time to sit back and look at what works and what
> > doesn't.
> >
> > Here's my own list of things that I find suboptimal, with suggestions
> > when I have some.
> >
> > when I have some.
> >
> > - @depends('--disable-foo') and related are confusing. I still think
> > that at the sandbox level, it's a good thing that the options are
> > treated in consistent manner, ignoring their defaults. But referring
> > options by their name incurs some ambiguities along the negatives.
> >
> > Maybe we should declare all options with their positive name
> > (--enable-foo, --with-foo), and set an explicit default of True when
> > the option is enabled by default. And make python configure reject
> > declarations with the negative name (--disable-foo, --without-foo).
> >
>
> This is probably one of the more confusing points when reading what we have
> so far. Related to this, I've always found it a bit disconcerting that
> @depends takes any depends function, or a string as an argument, but only
> if that string names an option. We could declare options without prefixes
> (requiring explicit defaults) and convert these to depends functions
> automatically so that option(name='foo', ...) would allow you to write
> @depends(foo) immediately thereafter, and never pass a string to @depends.
There are two awkwardnesses with that:
> > that at the sandbox level, it's a good thing that the options are
> > treated in consistent manner, ignoring their defaults. But referring
> > options by their name incurs some ambiguities along the negatives.
> >
> > Maybe we should declare all options with their positive name
> > (--enable-foo, --with-foo), and set an explicit default of True when
> > the option is enabled by default. And make python configure reject
> > declarations with the negative name (--disable-foo, --without-foo).
> >
>
> This is probably one of the more confusing points when reading what we have
> so far. Related to this, I've always found it a bit disconcerting that
> @depends takes any depends function, or a string as an argument, but only
> if that string names an option. We could declare options without prefixes
> (requiring explicit defaults) and convert these to depends functions
> automatically so that option(name='foo', ...) would allow you to write
> @depends(foo) immediately thereafter, and never pass a string to @depends.
- foo is not assigned to from the perspective of the python script. That
sure can be made to work in the sandbox, but it does seem like a
little too much magic (the kind of magic that doesn't happen when you
call a template, btw ; functions defined in a template don't magically
end up defined at the global level)
- options names have hyphens, and python identifiers can't contain them.
Also, declaring options without prefixes leaves you with the question:
what is the option supposed to be? --enable-foo, --with-foo, or --foo
(we have options in each of those kinds)
Now, I don't have data yet, and I will once I'm done with the
introspection work I'm doing right now, but the answer to this problem
might just be to stop defining options like they are defined today.
The original reason for the option() function to act as it does was that
there were complicated cases where you have a @depends function that
needs to depend on multiple options, and possibly other things. And
these cases exist. They've existed early, and there are a number of
them. But with progress being made, they may be appearing to be the
exception rather than the norm, and the norm is that we end up with more
or less pointless @depends functions that do generic things.
So depending on what the data says, it might be time to switch option to
a decorator. Whether we want to do that completely or not, we can already
start doing that with templates. There actually is precedent for it
with @deprecated_option.
Mike
Gregory Szorc
Aug 22, 2016, 2:34:17 PM8/22/16
to Ted Mielczarek, dev-builds
On Wed, Aug 17, 2016 at 2:48 AM, Ted Mielczarek <t...@mielczarek.org> wrote:
On Wed, Aug 17, 2016, at 05:29 AM, Mike Hommey wrote:
I really like this idea. I know one of the big issues we've had with
moz.build is an inability to introspect the entire state of things
because of conditionals, it would be nice to fix that situation in
moz.configure.
And that is a side-effect of us initially preferring variable-based/declarative patterns in moz.build as opposed to say something more function heavy (like Bazel's language) (if everything were a function, you could pass a "when" argument to the function to declare when it should actually be applied).
The good news is moz.configure leans heavily on functions, so we don't have this problem. I like the idea of a context manager receiving a condition function. We can provide a special mode to fake out the condition result to evaluate everything in a file. If someone wants to apply that to moz.build, I would not be opposed. However, glandium has been pushing to consolidate the moz.build reader with the emitter. If we start evaluating everything in a file regardless of conditions, the emitter may start barfing due to rules violations. One of the nice things about keeping the execution result and the emitter separate is you can postpone data validation for scenarios like these.
0 new messages