[Python-ideas] Rewriting the build system (part 2)

[Ryan Gonzalez]

So...I've been looking stuff up. Don't have too much free time right now, but it was mostly Googleing.

I have deemed three ideas to be best for Python:

- Write a custom makefile generator. I have some ideas, but they likely won't come to light any time soon.

- Use Boost.Build. It seems ugly and complicated, but it actually has a full configuring system that can detect libraries and try to compile source files. It's written in C89, not C++, and its dependency is the Boehm GC, which probably supports everywhere that Python runs on.

Only issue? It's undergoing a rewrite in Python. However, at the rate it's going, Python will likely be at version 7 by the time they finish.

- Stick with autotools. Trying to avoid this.

Thoughts? Again?

--

Ryan

[ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong.

http://kirbyfan64.github.io/

 

[Wes Turner]

Scons? http://www.scons.org/doc/production/HTML/scons-user.html

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Ryan Gonzalez]

That written in Python, so Python would require Python to build which would require Python to build which would require...

Besides, SCons sucks. If I were using a build system written in Python, I'd use fbuild or Waf.

[Wes Turner]

The subject line mentions "Part 2". Is there a link to "Part 1"? What is the rationale for changing?

Make, autoconf, and automake are already widely ported/portable.

What are the criteria for evaluating alternatives?

[Ryan Gonzalez]

https://mail.python.org/pipermail/python-ideas/2015-March/032381.html

[Ryan Gonzalez]

On Fri, Mar 13, 2015 at 2:47 PM, Chris Barker <chris....@noaa.gov> wrote:

On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rym...@gmail.com> wrote:

That written in Python, so Python would require Python to build which would require Python to build which would require...

Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer.

It it so wrong that python eats its own dog food in this manner?

It's not as much "Python eat its own dog food" as "cross-compiling natively would be even harder than it is right now."

C compilers are available on virtually every UNIX that's ever been created since the 70's, including AIX, Plan 9, Solaris, etc.

Python? Not as much.

 

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Wes Turner]

On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rym...@gmail.com> wrote:
>
> That written in Python, so Python would require Python to build which would require Python to build which would require...
>
> Besides, SCons sucks. If I were using a build system written in Python, I'd use fbuild or Waf.

Apparently, Waf started as a fork of SCons. [1][2]

[1] http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-in-large-projects
[2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf

I don't know anything about fbuild; others might.

A feature/alternative matrix could be helpful (in addition to a preliminary implementation) for making a cogent argument (in PEP form [3][4]). This is possible in RST with table or list-table syntax.

[3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates
[4] https://www.python.org/dev/peps/pep-0009/

[Ryan Gonzalez]

On Fri, Mar 13, 2015 at 3:01 PM, Wes Turner <wes.t...@gmail.com> wrote:

On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rym...@gmail.com> wrote:
>
> That written in Python, so Python would require Python to build which would require Python to build which would require...
>
> Besides, SCons sucks. If I were using a build system written in Python, I'd use fbuild or Waf.

Apparently, Waf started as a fork of SCons. [1][2]

[1] http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-in-large-projects
[2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf

But it's now better.

 

I don't know anything about fbuild; others might.

https://github.com/felix-lang/fbuild

 

A feature/alternative matrix could be helpful (in addition to a preliminary implementation) for making a cogent argument (in PEP form [3][4]). This is possible in RST with table or list-table syntax.

[3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates
[4] https://www.python.org/dev/peps/pep-0009/

[Chris Barker]

On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rym...@gmail.com> wrote:

That written in Python, so Python would require Python to build which would require Python to build which would require...

Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer.

It it so wrong that python eats its own dog food in this manner?

-CHB

[Robert Collins]

On 14 Mar 2015 09:44, "Chris Barker" <chris....@noaa.gov> wrote:
>
> On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>>
>> That written in Python, so Python would require Python to build which would require Python to build which would require...
>
>
> Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer.
>
> It it so wrong that python eats its own dog food in this manner?

Wrong, no.  harder yes. C compilers have mature cross compiler build chains including stage aware test suites and the like. When a tool is it's own build dependency bootstrapping it on a new platform gains a hard requirement on cross compiling rather than it being a thing for special cases like platforms that cannot compile the tool. This makes the life of distribution maintainers harder too, since cross arch platform bootstrapping becomes even more manual.

Rob

[rand...@fastmail.us]

On Fri, Mar 13, 2015, at 20:07, Robert Collins wrote:
> Wrong, no. harder yes. C compilers have mature cross compiler build
> chains
> including stage aware test suites and the like. When a tool is it's own
> build dependency bootstrapping it on a new platform gains a hard
> requirement on cross compiling rather than it being a thing for special
> cases like platforms that cannot compile the tool.

That's what bootstrapping is. If you're not doing this, you're not
bootstrapping.

[Antoine Pitrou]

On Sat, 14 Mar 2015 13:07:57 +1300
Robert Collins

<rob...@robertcollins.net> wrote:
> On 14 Mar 2015 09:44, "Chris Barker" <chris....@noaa.gov> wrote:
> >
> > On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
> >>
> >> That written in Python, so Python would require Python to build which
> would require Python to build which would require...
> >
> >
> > Well, you can't buid a C compiler without a C compiler either (or SOME
> compiler). And given that you should be able to do cross-compilation for
> platforms that don't currently have a python build, I'm not sure that's a
> killer.
> >
> > It it so wrong that python eats its own dog food in this manner?
>
> Wrong, no. harder yes. C compilers have mature cross compiler build chains
> including stage aware test suites and the like.

Not to mention that any decent platform manufacturer or system vendor
will do the job of writing or porting a C compiler for the platform.
Few of them will port Python for us, though.

Regards

Antoine.

[Chris Angelico]

On Sat, Mar 14, 2015 at 6:47 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>>
>> That written in Python, so Python would require Python to build which
>> would require Python to build which would require...
>
>
> Well, you can't buid a C compiler without a C compiler either (or SOME
> compiler). And given that you should be able to do cross-compilation for
> platforms that don't currently have a python build, I'm not sure that's a
> killer.
>
> It it so wrong that python eats its own dog food in this manner?
>
> -CHB

I've been just skimming this thread rather than reading in detail, so
please forgive me if this question is completely unrelated.

Will this affect the commands required to build Python from source, or
is all this just for actual development?

Consider two completely different workflows here: Firstly, a core
Python dev. S/he wants to be able to efficiently build, edit, etc, and
wants to be able to edit any part of the code. and secondly, someone
who has little knowledge of Python's internals, and just wants to
build the latest version (either from hg or from a source tarball).

In the first case, it's fine to require that some version of Python be
already installed. If editing file XYZ requires that file ABC be
rebuilt from it, and the rebuild process requires Python 3.3 or newer,
that's not going to hurt people. The modified ABC gets checked in
along with the change to XYZ, and all's well.

But in the second case, it'd be much safer to minimize the
requirements. If you can untar the source and just "./configure; make;
sudo make install" your way to having the latest Python, that's a LOT
easier than I've just been doing trying to get my Dad onto a newer
Mozilla Thunderbird. (I don't understand what it's doing all along
there, but it pulls a whole bunch of separate repositories when I say
update, and there's a separate bootstrap step, and ... I dunno.)

ChrisA

[anatoly techtonik]

On Fri, Mar 13, 2015 at 10:46 PM, Wes Turner <wes.t...@gmail.com> wrote:
> The subject line mentions "Part 2". Is there a link to "Part 1"? What is the
> rationale for changing?
>
> Make, autoconf, and automake are already widely ported/portable.

"Autotools has reached the Chandrasekhar mass limit of software and
become a black hole of suck that consumes endless hours of bright
peoples’ time without any good result. It is well past time for it to
die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

> What are the criteria for evaluating alternatives?

Python 3 support and beautiful output to console? =)

[Chris Barker]

>> That written in Python, so Python would require Python to build which
>> would require Python to build which would require...

A bit more here:

It seems there are two cases:

1) Development on and for a common, mature platform:

In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools.

2) Development for / porting to a new and/or obscure platform:

In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general):

  a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it...

  b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process:

  i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools

 ii) run the configure script to generate Makefiles, etc. configured for the platform at hand.

 iii) run make to do the actual build.

Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability.

I have no idea if any of the known tools can do this, but if so, it would be pretty usable.

And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work.

But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions.

Will this affect the commands required to build Python from source, or
is all this just for actual development?

well, it may or may not be both:

If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery.

 someone
who has little knowledge of Python's internals, and just wants to
build the latest version (either from hg or from a source tarball).

That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...

But in the second case, it'd be much safer to minimize the

requirements. If you can untar the source and just "./configure; make;
sudo make install" your way to having the latest Python, that's a LOT
easier than I've just been doing trying to get my Dad onto a newer
Mozilla Thunderbird. (I don't understand what it's doing all along
there, but it pulls a whole bunch of separate repositories when I say
update, and there's a separate bootstrap step, and ... I dunno.)

requiring _some_, not very recent, python is not exactly

 """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""

If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".

In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building.

-CHB

[Chris Angelico]

On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker <chris....@noaa.gov> wrote:
>> someone
>> who has little knowledge of Python's internals, and just wants to
>> build the latest version (either from hg or from a source tarball).
>
>
> That's the assumption I'm following -- the user wants the latest version --
> so probably already has easy access to an older version with which to run
> the build tools...

The tricky part here is defining the oldest supported version. "To
build Python X.Y, you must already have Python ??? or later installed"
- how do you pick the minimum? The older that minimum, the more warped
the build code has to be (especially if you want to support 2.x as
well as 3.x - imagine trying to make it possible to use the system
Python on RHEL, as well as any 3.[1-4]), but go too far the other way
and you end up with a multi-step build process ("first build Python
3.4, then use that to build Python 3.7") which is just going to be a
pain.

>> But in the second case, it'd be much safer to minimize the
>> requirements. If you can untar the source and just "./configure; make;
>> sudo make install" your way to having the latest Python, that's a LOT
>> easier than I've just been doing trying to get my Dad onto a newer
>> Mozilla Thunderbird. (I don't understand what it's doing all along
>> there, but it pulls a whole bunch of separate repositories when I say
>> update, and there's a separate bootstrap step, and ... I dunno.)
>
>
> requiring _some_, not very recent, python is not exactly
>
> """a whole bunch of separate repositories when I say update, and there's a
> separate bootstrap step"""
>
> If there is an already build python (like OS-X, any general purpose linux
> distro, Windows if you don't mind point and clicking the python.org
> installer, etc, then it's not really a "bootstrap".

No, that's not what I'm talking about. I'm talking about trying to
build Thunderbird and having to do a bunch of steps where I run this
program, then run that, then let it chug its way through stuff, and
then eventually it'll start building. Without detailed knowledge of
the build internals I have no way of knowing what steps I can skip, so
every time I want to test the build again, I have to go through them
all. It took three parts of forever to figure out what was going on.

I much prefer projects that ship with the standard "./configure; make"
pattern. I don't care if the configure script is built by some arcane
incantation, as long as it's checked into source control (or at very
least is in the downloadable source tarball for any particular
version). If I want to build your program, I don't want to have to
hunt down your notes file, read it through in detail, and jump through
a dozen hoops; I just want to point my compiler at your source, and
then go make myself a hot chocolate while it does its work.

[Ryan Gonzalez]

On Mon, Mar 16, 2015 at 11:50 AM, Chris Angelico <ros...@gmail.com> wrote:

On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker <chris....@noaa.gov> wrote:
>>  someone
>> who has little knowledge of Python's internals, and just wants to
>> build the latest version (either from hg or from a source tarball).
>
>
> That's the assumption I'm following -- the user wants the latest version --
> so probably already has easy access to an older version with which to run
> the build tools...

The tricky part here is defining the oldest supported version. "To
build Python X.Y, you must already have Python ??? or later installed"
- how do you pick the minimum? The older that minimum, the more warped
the build code has to be (especially if you want to support 2.x as
well as 3.x - imagine trying to make it possible to use the system
Python on RHEL, as well as any 3.[1-4]), but go too far the other way
and you end up with a multi-step build process ("first build Python
3.4, then use that to build Python 3.7") which is just going to be a
pain.

Sounds like Go is now. It's a pain.

 

>> But in the second case, it'd be much safer to minimize the
>> requirements. If you can untar the source and just "./configure; make;
>> sudo make install" your way to having the latest Python, that's a LOT
>> easier than I've just been doing trying to get my Dad onto a newer
>> Mozilla Thunderbird. (I don't understand what it's doing all along
>> there, but it pulls a whole bunch of separate repositories when I say
>> update, and there's a separate bootstrap step, and ... I dunno.)
>
>
> requiring _some_, not very recent, python is not exactly
>
>  """a whole bunch of separate repositories when I say update, and there's a
> separate bootstrap step"""
>
> If there is an already build python (like OS-X, any general purpose linux
> distro, Windows if you don't mind point and clicking the python.org
> installer, etc, then it's not really a "bootstrap".

No, that's not what I'm talking about. I'm talking about trying to
build Thunderbird and having to do a bunch of steps where I run this
program, then run that, then let it chug its way through stuff, and
then eventually it'll start building. Without detailed knowledge of
the build internals I have no way of knowing what steps I can skip, so
every time I want to test the build again, I have to go through them
all. It took three parts of forever to figure out what was going on.

All of Mozilla's build systems need help. They have a build system genius who wrote Tup working for them, and they somehow still have 20,000 stupid mozbuild files with some weird, intensely slow make-based system that everyone complains about.

 

I much prefer projects that ship with the standard "./configure; make"
pattern. I don't care if the configure script is built by some arcane
incantation, as long as it's checked into source control (or at very
least is in the downloadable source tarball for any particular
version). If I want to build your program, I don't want to have to
hunt down your notes file, read it through in detail, and jump through
a dozen hoops; I just want to point my compiler at your source, and
then go make myself a hot chocolate while it does its work.

Most of that is bad documentation. If there's no configure script but a file labeled INSTALLING, then that should contain simple instructions to build.

I hate most tools that ship with configure scripts because 70% of the time some stupid test does some stupid thing wrong and gives the worst errors, 29% of the time the configure system doesn't pick up my options correctly, and 1% of the time it actually *works*.

SCons also constantly gives me trouble.

Of course, you could always do it the plan9port way:

ryan@DevPC-LX:~/stuff/plan9port$ ./configure

read the README file.

ryan@DevPC-LX:~/stuff/plan9port$ 

 

ChrisA
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Chris Barker]

On Mon, Mar 16, 2015 at 9:50 AM, Chris Angelico <ros...@gmail.com> wrote:

The tricky part here is defining the oldest supported version. "To
build Python X.Y, you must already have Python ??? or later installed"
- how do you pick the minimum? The older that minimum, the more warped
the build code has to be

sure -- but it seems what's on the table now is: must be buildable by an old version of C -- not even C99 which is nominally 16 years old...

So yeah, the build system itself would need to support a "old" python -- though probably python2.7 would be OK at this point -- sorry, not py3 yet, but what can you do?

> If there is an already build python (like OS-X, any general purpose linux
> distro, Windows if you don't mind point and clicking the python.org
> installer, etc, then it's not really a "bootstrap".

No, that's not what I'm talking about. I'm talking about trying to
build Thunderbird and having to do a bunch of steps where I run this
program, then run that, then let it chug its way through stuff, and
then eventually it'll start building.

It sounds like that build setup suck, yes -- but not sure why that's relevant. Clearly we don't want something that ugly. We want something that is both easy and robust to configure for new platfroms, etc, and easy on the end user to "jsut build" the source. All I"m proposing that is requireing a not-very-recent python as a first step is only a big hurdle for folks poritn got new platforms -- your use case would be, in INSTALL.txt:

"""

1) You need at least python 2.7 to build this software. 

try:

python --version

if it starts up and reports a version greater than 2.7, then you are all set. If not, then It can probably be installed with your system package manager:

apt_get install python

or similar. On Windows, you can download an installer for Python2.7 from python.org

2) Once a python is working, you can build the this latest version from source with:

./buildme

"""

Of course, there may be options, etc, but that's the basic deal.

And heck, if people REALLY want the ./configure && make && make install dance, then we could ship a little configure script that does nothing but call the python-based one...

Of course, all this is predicated on there BEING some miraculous as-powerful-and-flexible-and-robust-as-autotools-but-much-easier-to-use python-based build system.....

One more note: so adding a python dependency is a pain. But the current system requires separate stems for Windows and *nix-y systems, and who knows what for hypothetical systems that don't support autotools. So maybe a python-based build system that worked on all the common platforms, with an additional makefile that you have to hand-mangle to bootstrap a new platform isn't any more to maintain.

-CHB

[Ryan Gonzalez]

The best build systems I know in Python are fbuild, Waf, and Meson. Meson generates Ninja scripts, and Ninja's in C++, so that's out of the question.

I have used fbuild a LOT and can say that, after you get through the small amount of command-line boilerplate, it works very well. It's also very easy to extend with host/target compiler differences. Not sure about Waf, but I do know it does about anything you'd guess.

I guess this narrows the result down to Boost.Build or a Python-based build system.

 

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Andrew Barnert]

Something nobody's mentioned yet in this thread:

The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.).

Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems?

Sent from my iPhone

_______________________________________________

[Stefan Krah]

Ryan Gonzalez <rymg19@...> writes:
> I hate most tools that ship with configure scripts because 70% of the time
some stupid test does some stupid thing wrong and gives the worst errors,
29% of the time the configure system doesn't pick up my options correctly,
and 1% of the time it actually *works*.

That's just not true. Also, the small build issue that triggered
this thread is already tracked here:

http://bugs.python.org/issue22625


It seems more productive to me to fix that rather than rewrite the
build system. Users don't have an unlimited tolerance for pointless
churn.



Stefan Krah

[Ryan Gonzalez]

Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on.

[Ryan Gonzalez]

On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah <ste...@bytereef.org> wrote:

Ryan Gonzalez <rymg19@...> writes:
> I hate most tools that ship with configure scripts because 70% of the time
some stupid test does some stupid thing wrong and gives the worst errors,
29% of the time the configure system doesn't pick up my options correctly,
and 1% of the time it actually *works*.

That's just not true.  Also, the small build issue that triggered
this thread is already tracked here:

   http://bugs.python.org/issue22625

You've evidently had better luck than I usually have.

 

It seems more productive to me to fix that rather than rewrite the
build system.  Users don't have an unlimited tolerance for pointless
churn.

Lots and lots of projects have moved from autotools to about everything else.

It's slow, annoying, and generates huge files (the 5k-line configure.ac generates a 17k-line configure script). And largely pointless, since nowadays few like it anymore.

 

Stefan Krah

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Mark Lawrence]

On 16/03/2015 19:24, Ryan Gonzalez wrote:
> On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah
> <ste...@bytereef.org

> <mailto:ste...@bytereef.org>> wrote:
>
> Ryan Gonzalez <rymg19@...> writes:
> > I hate most tools that ship with configure scripts because 70% of the time
> some stupid test does some stupid thing wrong and gives the worst
> errors,
> 29% of the time the configure system doesn't pick up my options
> correctly,
> and 1% of the time it actually *works*.
>
> That's just not true. Also, the small build issue that triggered
> this thread is already tracked here:
>
> http://bugs.python.org/issue22625
>
>
> You've evidently had better luck than I usually have.

Luck doesn't enter in to it. More likely a bad workan always blames his
tools?

>
>
> It seems more productive to me to fix that rather than rewrite the
> build system. Users don't have an unlimited tolerance for pointless
> churn.
>
>
> Lots and lots of projects have moved from autotools to about everything
> else.
>
> It's slow, annoying, and generates huge files (the 5k-line configure.ac

> <http://configure.ac> generates a 17k-line configure script). And

> largely pointless, since nowadays few like it anymore.
>

Perhaps everybody should move to MSVC, that way at least everything
would be consistent :)

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[Chris Barker]

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <tech...@gmail.com> wrote:

"Autotools has reached the Chandrasekhar mass limit of software and
become a black hole of suck that consumes endless hours of bright
peoples’ time without any good result. It is well past time for it to
die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

Is it even used for building Python on Windows (for the "normal" python.org builds)?

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

-CHB

[1] http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-government-except-for-all

[Ryan Gonzalez]

On Mon, Mar 16, 2015 at 3:04 PM, Mark Lawrence <bream...@yahoo.co.uk> wrote:

On 16/03/2015 19:24, Ryan Gonzalez wrote:

On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah
<ste...@bytereef.org
<mailto:ste...@bytereef.org>> wrote:

    Ryan Gonzalez <rymg19@...> writes:
    > I hate most tools that ship with configure scripts because 70% of the time
    some stupid test does some stupid thing wrong and gives the worst
    errors,
    29% of the time the configure system doesn't pick up my options
    correctly,
    and 1% of the time it actually *works*.

    That's just not true.  Also, the small build issue that triggered
    this thread is already tracked here:

    http://bugs.python.org/issue22625


You've evidently had better luck than I usually have.

Luck doesn't enter in to it.  More likely a bad workan always blames his tools?

:P

I never have that much trouble with other build tools, or even hand-written makefiles/configure scripts. Just autotools.

 

    It seems more productive to me to fix that rather than rewrite the
    build system.  Users don't have an unlimited tolerance for pointless
    churn.


Lots and lots of projects have moved from autotools to about everything
else.

It's slow, annoying, and generates huge files (the 5k-line configure.ac
<http://configure.ac> generates a 17k-line configure script). And
largely pointless, since nowadays few like it anymore.

Perhaps everybody should move to MSVC, that way at least everything would be consistent :)

And non-standards-compliant. :)

 

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Ryan Gonzalez]

On Mon, Mar 16, 2015 at 11:31 AM, Chris Barker <chris....@noaa.gov> wrote:

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <tech...@gmail.com> wrote:

"Autotools has reached the Chandrasekhar mass limit of software and
become a black hole of suck that consumes endless hours of bright
peoples’ time without any good result. It is well past time for it to
die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

Is it even used for building Python on Windows (for the "normal" python.org builds)?

No. Some random batch file is used. That's all I know.

However, when building without MSVC/VS2008, you need MinGW, which means you'll end up using autotools.

 

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

 

-CHB

[1] http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-government-except-for-all

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Ned Deily]

In article <loom.2015031...@post.gmane.org>,

Stefan Krah <ste...@bytereef.org> wrote:
> That's just not true. Also, the small build issue that triggered
> this thread is already tracked here:
>
> http://bugs.python.org/issue22625
>
> It seems more productive to me to fix that rather than rewrite the
> build system. Users don't have an unlimited tolerance for pointless
> churn.

+10

How many different platforms and configurations on each platform do we
explicitly or implicitly support today for current CPython 2 and CPython
3 releases? I don't know (and I should since I help release them) but
it's clearly at least in the dozens. We do not currently have formal
tests or test platforms (e.g. buildbots) for many of them like we should
and it would be a monumental undertaking to try to migrate the current
build system to something substantially different. It's fine to let off
steam about frustrations with build systems but talking about it here is
not gonna cause it to change. And it won't change unless someone (or,
more likely, some big company) is willing to invest an enormous effort
in people time and machine resources to do so. Stefan's suggestion is
much more practical. Along with it, if someone is motivated, better
documenting the current processes for cross-compilation and which pairs
of build/target systems are supported would be a welcome and extremely
useful improvement. Much of what is there today for cross-compilation
has slipped in over the years with little discussion or documentation.
It's easy to break because it's not always clear how it is supposed to
work and because it isn't regularly tested.

--
Ned Deily,
n...@acm.org

[Chris Barker]

On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily <n...@acm.org> wrote:

 it would be a monumental undertaking to try to migrate the current
build system to something substantially different.  It's fine to let off
steam about frustrations with build systems but talking about it here is
not gonna cause it to change.  And it won't change unless someone (or,
more likely, some big company) is willing to invest an enormous effort
in people time and machine resources to do so.  Stefan's suggestion is
much more practical. 

This doesn't seem to me to be a winner takes all proposition -- there is no reason someone couldn't set up a new build system for Python, get it all working nicely on their platform(s) of choice, persuade others how wonderful it is, get contributions, etc.

If it turns out to really be better/easier etc. than autoconf, then , and only then, would it make any sense to talk about replacing the existing official build system. And even then, the two could live in parallel for quite some time...

-Chris

[Ned Deily]

In article
<CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC...@mail.gmail.com>,

Chris Barker <chris....@noaa.gov> wrote:
> This doesn't seem to me to be a winner takes all proposition -- there is no
> reason someone couldn't set up a new build system for Python, get it all
> working nicely on their platform(s) of choice, persuade others how
> wonderful it is, get contributions, etc.
>
> If it turns out to really be better/easier etc. than autoconf, then , and
> only then, would it make any sense to talk about replacing the existing
> official build system. And even then, the two could live in parallel for
> quite some time...

Sorry, that's just not realistic. We already have two different build
systems, one for Windows and one for all other platforms, and it's
difficult enough to keep both of them running. Adding yet another
system in parallel is not in the cards without a major infusion of
resources. There are frequent minor changes to the current build
systems as part of new Python features or bug fixes and those run the
risk of breaking things today and too often do. Major changes to build
systems impact most parts of the Python ecosphere, for example,
development, automated testing, manual testing, release management,
downstream distributors of CPython (many of them managing their own sets
of local modifications to the current processes), etc etc. To be even
able to evaluate the merits of a major build system migration, we would
first need to understand what we support today, what are the actual
shortcomings of what we have today, and what we want to support in the
future. Then, given that, we'd need to decide what are the options and
what are their costs. One prerequisite to that is having a much more
comprehensive test build environment so we could make even substantial
incremental changes with a reasonable degree of confidence that we're
not breaking support of a defined set of platforms and configurations.
We don't have that today. Working towards that would be a necessary and
important step.

Also, note that, in all this talk about autoconf, be aware the CPython
build processes don't even make full use of current automake/autoconf
features. Much of the current Python build process predates the
addition of features to those tools. Just bringing the current
configure scripts and Makefiles up-to-date to use all the current
features would be a big task that would imply a big testing effort on
resources we don't currently have.

Also keep in mind that builds and installations are areas that few core
CPython developers are all that interested in and with good reason.
It's usually tedious and generally unrewarding work especially when
operating in a vacuum. Let's first work towards properly identifying
and testing current important environments (for example, from typical
installed Python configurations, not build directories and, as noted
before, the almost totally undocumented and tested world of
cross-building) and then we can start incrementally improving things
with some degree of confidence that we're not going to break things for
us and our users.

[Stephen J. Turnbull]

Ryan Gonzalez writes:

> I have used fbuild a LOT

Actually, I suspect that in the relevant sense, you haven't used it a
lot. Have you used it for a single program on

1. more than 3 CPUs, AND
2. on each CPU, more than one (and preferably more than two) OSes, AND
3. for each CPU-OS combination, more than two different configurations
of the program being built, AND
4. for each CPU-OS-configuration combination, at least 3 versions of
the program source (preferably spanning a major version bump)?

The number of programs/versions you have built on a single (or small
selection of) CPU-OS combinations is irrelevant to the discussion.
The problem autotools helps to solve better than any other system I
know of is bootstrapping on a new system. It's also quite good at
handling foibles (== regressions) across version bumps of the source
program.

I agree it's suboptimal (at best :-) to have a PyPI module that
depends on the autotools. I've found it to be a real PITA for Windows
users who want to develop certain Emacs Lisp applications, for
example, which is a similar kind of development environment to Python.

But for bootstrapping Python itself, what else is there, really?
Every time I've seen a project move to something like SCons, I've also
seen their port rate to new platforms decrease dramatically. (Hard to
interpret that trend, though -- if a project has the free cycles to
change build systems, maybe they have ported to almost all of the
plausible platforms already.) Autotools alternatives just have way
fewer modules for dealing with new and/or old ;-) platforms.

Bright! Shiny! New! system use crufty old system :-(
+ => +
missing library :-( library

Caveat: Autotools is the only build system (well, Imake -- now you
know how old I am!) I have substantial experience with. But I've
heard stories from colleagues about the limitations of Scons and
cmake in this respect, and I find it hard to believe that there are
systems out there with the accumulated knowledge of platforms that has
been assembled in Autotools over the decades.

[Mark Lawrence]

On 16/03/2015 23:45, Chris Barker wrote:
> On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily
> <n...@acm.org

> <mailto:n...@acm.org>> wrote:
>
> it would be a monumental undertaking to try to migrate the current
> build system to something substantially different. It's fine to let off
> steam about frustrations with build systems but talking about it here is
> not gonna cause it to change. And it won't change unless someone (or,
> more likely, some big company) is willing to invest an enormous effort
> in people time and machine resources to do so. Stefan's suggestion is
> much more practical.
>
>
> This doesn't seem to me to be a winner takes all proposition -- there is
> no reason someone couldn't set up a new build system for Python, get it
> all working nicely on their platform(s) of choice, persuade others how
> wonderful it is, get contributions, etc.
>

This sounds really simple to me so when do you start work on it?

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[Andrew Barnert]

On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez <rym...@gmail.com> wrote:

Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on.

How does trapping an error and moving on give you a user-configurable option to select which OpenSSL to use, whether he wants a fat i386/x86_64 build or just x86_64, whether he wants a framework bundle or a traditional layout...? (These are the first three flags that Homebrew passes to Python's configure aside from the install prefix, which I'd hope everything can handle.)

[Chris Barker]

On Mon, Mar 16, 2015 at 7:28 PM, Mark Lawrence <bream...@yahoo.co.uk> wrote:

On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily

This doesn't seem to me to be a winner takes all proposition -- there is

no reason someone couldn't set up a new build system for Python, get it
all working nicely on their platform(s) of choice, persuade others how
wonderful it is, get contributions, etc.

This sounds really simple to me so when do you start work on it?

I think I was misunderstood -- my point was that there is no reason to start all this with trying to persuade the python dev team to accept a new build system up front.

The only possible way it could happen were if someone(s) did the work and got a new system in place that worked well and proved to be better than autoconf in various ways, while being just as capable. And that would be LOT of work to get that far.

But it could be done without having to persuade anyone up front that it 's a good idea.

Frankly, I doubt anyone's going to put that work in -- I know I won't -- remember that I'm one of the folks that's skeptical that this mythical build system exists at all -- never mind the work to transition to it.

-Chris

[Ryan Gonzalez]

On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <ste...@xemacs.org> wrote:

Ryan Gonzalez writes:

 > I have used fbuild a LOT

Actually, I suspect that in the relevant sense, you haven't used it a
lot.  Have you used it for a single program on

1.  more than 3 CPUs, AND
2.  on each CPU, more than one (and preferably more than two) OSes, AND
3.  for each CPU-OS combination, more than two different configurations
    of the program being built, AND
4.  for each CPU-OS-configuration combination, at least 3 versions of
    the program source (preferably spanning a major version bump)?

No...but I know that Felix uses it for iPhone, OSX, Linux, ... with separation of host and target and multiple configurations.

 

The number of programs/versions you have built on a single (or small
selection of) CPU-OS combinations is irrelevant to the discussion.
The problem autotools helps to solve better than any other system I
know of is bootstrapping on a new system.  It's also quite good at
handling foibles (== regressions) across version bumps of the source
program.

I agree it's suboptimal (at best :-) to have a PyPI module that
depends on the autotools.  I've found it to be a real PITA for Windows
users who want to develop certain Emacs Lisp applications, for
example, which is a similar kind of development environment to Python.

But for bootstrapping Python itself, what else is there, really?
Every time I've seen a project move to something like SCons, I've also
seen their port rate to new platforms decrease dramatically.  (Hard to
interpret that trend, though -- if a project has the free cycles to
change build systems, maybe they have ported to almost all of the
plausible platforms already.)  Autotools alternatives just have way
fewer modules for dealing with new and/or old ;-) platforms.

This is also SCons. Everybody loved it until they had to maintain it. :)

 

Bright! Shiny! New! system         use crufty old system :-(
           +                  =>              +
   missing library :-(                     library

Caveat: Autotools is the only build system (well, Imake -- now you
know how old I am!) I have substantial experience with.  But I've
heard stories from colleagues about the limitations of Scons and
cmake in this respect, and I find it hard to believe that there are
systems out there with the accumulated knowledge of platforms that has
been assembled in Autotools over the decades.

SCons is very annoying. CMake is weird and only really useful for those huge, 20,000 file C++ projects.

[Ryan Gonzalez]

On Mon, Mar 16, 2015 at 10:38 PM, Andrew Barnert <abar...@yahoo.com> wrote:

On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez <rym...@gmail.com> wrote:

Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on.

How does trapping an error and moving on give you a user-configurable option to select which OpenSSL to use, whether he wants a fat i386/x86_64 build or just x86_64, whether he wants a framework bundle or a traditional layout...? (These are the first three flags that Homebrew passes to Python's configure aside from the install prefix, which I'd hope everything can handle.)

Just mix a command-line argument with a try-catch block. It's actually much easier/less-hackish than it sounds.

The nice thing is that fbuild allows multiple builders, so separating target/host stuff and command-line args is super-easy.

[anatoly techtonik]

On Mon, Mar 16, 2015 at 8:18 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
> They have a build system genius who wrote Tup working for them

I like the conceptual brevity of the Tup page http://gittup.org/tup/
It is the core of that I am try to bring from SCons internals to the
surface. It is very often that you need to process some DAG
and touch nodes based on their changed dependencies.

The problem is so common that it is even worth to spend time
on visualization for it. So far no build system has this stuff, and
I run of time to complete the SCons plumbing.
--
anatoly t.

[anatoly techtonik]

On Mon, Mar 16, 2015 at 8:39 PM, Chris Barker <chris....@noaa.gov> wrote:
> One more note: so adding a python dependency is a pain. But the current
> system requires separate stems for Windows and *nix-y systems, and who knows
> what for hypothetical systems that don't support autotools. So maybe a
> python-based build system that worked on all the common platforms, with an
> additional makefile that you have to hand-mangle to bootstrap a new platform
> isn't any more to maintain.

Manual instruction with lists of files that can be parsed out of it is
better than
Makefile IMO. Pumped up Markdown or something..
--
anatoly t.

[anatoly techtonik]

On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
> On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <ste...@xemacs.org>
> wrote:
>>
>> Ryan Gonzalez writes:
>>
>> > I have used fbuild a LOT
>>
>> Actually, I suspect that in the relevant sense, you haven't used it a
>> lot. Have you used it for a single program on
>>
>> 1. more than 3 CPUs, AND
>> 2. on each CPU, more than one (and preferably more than two) OSes, AND
>> 3. for each CPU-OS combination, more than two different configurations
>> of the program being built, AND
>> 4. for each CPU-OS-configuration combination, at least 3 versions of
>> the program source (preferably spanning a major version bump)?
>>
>
> No...but I know that Felix uses it for iPhone, OSX, Linux, ... with
> separation of host and target and multiple configurations.

What or who is Felix?

> This is also SCons. Everybody loved it until they had to maintain it. :)

I know a blind girl, who made a voice synthesizer, which works on
Windows, Linux and Android, and she used SCons to build it. That
gave me another reason to stick with this build system and try to
enhance it.

> SCons is very annoying. CMake is weird and only really useful for those
> huge, 20,000 file C++ projects.

For me the advantage of SCons is that I already know its codebase.
There a lot of room for improvement if you can tag yourself as a Python
hacker it addition to inherent potential user property.

In any, I'd appreciate the feedback on most annoying SCons feature,
because it will give a criteria for comparing it with others and for future
development.
--
anatoly t.

[anatoly techtonik]

On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <tech...@gmail.com>
> wrote:
>>
>> "Autotools has reached the Chandrasekhar mass limit of software and
>> become a black hole of suck that consumes endless hours of bright
>> peoples’ time without any good result. It is well past time for it to
>> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
>>
>> I couldn't sum up this better. It looks even worse from Windows machine.
>
> Is it even used for building Python on Windows (for the "normal" python.org
> builds)?

There was some success with MinGW toolchain, but I am not tracking if it was
integrated into official source. The FUD was that compiled extensions are not
compatible between MSVC builds and MinGW builds, but I could not confirm
that. If I understand correctly, the incompatibility is only for C++ programs,
because of strange SEH strategy and proprietary name mangling.

> But to miss-paraphrase Winston Churchill [1]:
>
> "Autotools is the worst build system except for all the others."
>
> Is there really something better out there?

A lot of attempts are made, but reviewing them requires a lot of
time and effort.

[anatoly techtonik]

On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>>
>> But to miss-paraphrase Winston Churchill [1]:
>>
>> "Autotools is the worst build system except for all the others."
>>
>> Is there really something better out there?
>>
>
> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

Is there anything Javascriptey? Python needs to move to web. I heard
something about Grunt, which was replaced by Gulp, but no idea if they
really mature enough to include "system features detection" and "user
configured builds" (or work with C/C++ compilers at all).

[anatoly techtonik]

On Tue, Mar 17, 2015 at 2:45 AM, Chris Barker <chris....@noaa.gov> wrote:
>
> This doesn't seem to me to be a winner takes all proposition -- there is no
> reason someone couldn't set up a new build system for Python, get it all
> working nicely on their platform(s) of choice, persuade others how wonderful
> it is, get contributions, etc.
>
> If it turns out to really be better/easier etc. than autoconf, then , and
> only then, would it make any sense to talk about replacing the existing
> official build system. And even then, the two could live in parallel for
> quite some time...

Both Blender and Wesnoth have SCons and CMake running in parallel
for a long time. Both camps are just maintaining their own chains and it
works well.

[anatoly techtonik]

On Tue, Mar 17, 2015 at 5:27 AM, Stephen J. Turnbull <ste...@xemacs.org> wrote:
> systems out there with the accumulated knowledge of platforms that has
> been assembled in Autotools over the decades.

Given that Autotools still doesn't have native Windows support, the value of
that knowledge is overstated.

[Florian Bruhin]

* anatoly techtonik <tech...@gmail.com> [2015-03-18 10:21:46 +0300]:

> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
> >>
> >> But to miss-paraphrase Winston Churchill [1]:
> >>
> >> "Autotools is the worst build system except for all the others."
> >>
> >> Is there really something better out there?
> >>
> >
> > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>
> Is there anything Javascriptey? Python needs to move to web. I heard
> something about Grunt, which was replaced by Gulp, but no idea if they
> really mature enough to include "system features detection" and "user
> configured builds" (or work with C/C++ compilers at all).

What does "Python needs to move to web" to do with it's build system
being "Javascriptey"?

Florian

--
http://www.the-compiler.org | m...@the-compiler.org (Mail/XMPP)
GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
I love long mails! | http://email.is-not-s.ms/

[Ryan Gonzalez]

Then you need Node.js installed, which is kind of big.

On Wed, Mar 18, 2015 at 2:21 AM, anatoly techtonik <tech...@gmail.com> wrote:

On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>>
>> But to miss-paraphrase Winston Churchill [1]:
>>
>> "Autotools is the worst build system except for all the others."
>>
>> Is there really something better out there?
>>
>
> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

Is there anything Javascriptey? Python needs to move to web. I heard
something about Grunt, which was replaced by Gulp, but no idea if they
really mature enough to include "system features detection" and "user
configured builds" (or work with C/C++ compilers at all).

[Ryan Gonzalez]

On Wed, Mar 18, 2015 at 2:44 AM, anatoly techtonik <tech...@gmail.com> wrote:

On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
> On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <ste...@xemacs.org>
> wrote:
>>
>> Ryan Gonzalez writes:
>>
>>  > I have used fbuild a LOT
>>
>> Actually, I suspect that in the relevant sense, you haven't used it a
>> lot.  Have you used it for a single program on
>>
>> 1.  more than 3 CPUs, AND
>> 2.  on each CPU, more than one (and preferably more than two) OSes, AND
>> 3.  for each CPU-OS combination, more than two different configurations
>>     of the program being built, AND
>> 4.  for each CPU-OS-configuration combination, at least 3 versions of
>>     the program source (preferably spanning a major version bump)?
>>
>
> No...but I know that Felix uses it for iPhone, OSX, Linux, ... with
> separation of host and target and multiple configurations.

What or who is Felix?

A programming language. http://felix-lang.org/. The code base is pretty huge; since it comes with most of its dependencies (re2, Judy, etc.), it consists of about 1600 source files. Fbuild works pretty well, other than the fact that the language's creator is 100% positive that dynamic languages are cruel, evil, and unmaintainable.

 

> This is also SCons. Everybody loved it until they had to maintain it. :)

I know a blind girl, who made a voice synthesizer, which works on
Windows, Linux and Android, and she used SCons to build it. That
gave me another reason to stick with this build system and try to
enhance it.

> SCons is very annoying. CMake is weird and only really useful for those
> huge, 20,000 file C++ projects.

For me the advantage of SCons is that I already know its codebase.
There a lot of room for improvement if you can tag yourself as a Python
hacker it addition to inherent potential user property.

In any, I'd appreciate the feedback on most annoying SCons feature,
because it will give a criteria for comparing it with others and for future
development.

Ah, right...you work on SCons...please don't take anything I said personally. :)

It's just that 99% of the time I try to build a project with SCons, some horribly stupid thing goes wrong because of some weird API call. Also, variant directories are WAY to weird for me.

It's also kind of slow. I'm one of those people that gets impatient if I have more than 1.01s for my build to complete, which is why I don't really like make too much.

 

--
anatoly t.

[Nathaniel Smith]

On Mar 18, 2015 8:14 AM, "anatoly techtonik" <tech...@gmail.com> wrote:
>
> On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris....@noaa.gov> wrote:
> > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <tech...@gmail.com>
> > wrote:
> >>
> >> "Autotools has reached the Chandrasekhar mass limit of software and
> >> become a black hole of suck that consumes endless hours of bright
> >> peoples’ time without any good result. It is well past time for it to
> >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
> >>
> >> I couldn't sum up this better. It looks even worse from Windows machine.
> >
> > Is it even used for building Python on Windows (for the "normal" python.org
> > builds)?
>
> There was some success with MinGW toolchain, but I am not tracking if it was
> integrated into official source. The FUD was that compiled extensions are not
> compatible between MSVC builds and MinGW builds, but I could not confirm
> that. If I understand correctly, the incompatibility is only for C++ programs,
> because of strange SEH strategy and proprietary name mangling.

This isn't FUD, alas. mingw in almost all of its variants uses effectively a different "libc" than MSVC, so code compiled with two compilers can't even agree on the meaning of file numbers. It is mostly possible to set up a mingw-w64 build that produces MSVC-compatible extensions, but it requires using a configuration that's undocumented and untested upstream (I.e. you will run into mingw bugs). NumPy *may* switch to this at some point -- we've been working on it for like a year now -- but it's extremely non-trivial.

-n

[Steve Dower]

And unfortunately, a lot of that work may need to be redone for 3.5 and VC14... Sorry about that, but once the other compilers have caught up with the new CRT interface then it will (should) all be better in the long run.

Cheers,
Steve

Top-posted from my Windows Phone

---

From: Nathaniel Smith
Sent: ‎3/‎18/‎2015 10:31
To: anatoly techtonik
Cc: python...@python.org
Subject: Re: [Python-ideas] Rewriting the build system (part 2)

[anatoly techtonik]

On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <m...@the-compiler.org> wrote:
> * anatoly techtonik <tech...@gmail.com> [2015-03-18 10:21:46 +0300]:
>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>> >>
>> >> But to miss-paraphrase Winston Churchill [1]:
>> >>
>> >> "Autotools is the worst build system except for all the others."
>> >>
>> >> Is there really something better out there?
>> >>
>> >
>> > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>>
>> Is there anything Javascriptey? Python needs to move to web. I heard
>> something about Grunt, which was replaced by Gulp, but no idea if they
>> really mature enough to include "system features detection" and "user
>> configured builds" (or work with C/C++ compilers at all).
>
> What does "Python needs to move to web" to do with it's build system
> being "Javascriptey"?

Because things like Twisted are de-facto standard in JavaScript world
(Node.js), and Python will benefit more if people look for how things are
implemented in non-C worlds.
--
anatoly t.

[Andrew Barnert]

On Mar 19, 2015, at 4:12 AM, anatoly techtonik <tech...@gmail.com> wrote:
>
>> On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <m...@the-compiler.org> wrote:
>> * anatoly techtonik <tech...@gmail.com> [2015-03-18 10:21:46 +0300]:
>>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>>>>>
>>>>> But to miss-paraphrase Winston Churchill [1]:
>>>>>
>>>>> "Autotools is the worst build system except for all the others."
>>>>>
>>>>> Is there really something better out there?
>>>>
>>>> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>>>
>>> Is there anything Javascriptey? Python needs to move to web. I heard
>>> something about Grunt, which was replaced by Gulp, but no idea if they
>>> really mature enough to include "system features detection" and "user
>>> configured builds" (or work with C/C++ compilers at all).
>>
>> What does "Python needs to move to web" to do with it's build system
>> being "Javascriptey"?
>
> Because things like Twisted are de-facto standard in JavaScript world
> (Node.js), and Python will benefit more if people look for how things are
> implemented in non-C worlds.

Is this a sarcastic comment?

Or are you seriously suggesting that because people use things like the stdlib's asyncio, gevent, Tornado, etc. in Python, which have different paradigms than Twisted, the one Node copied, we need to force some other part of Python to be more like Node in some other way, because "move to the web" means "be more like the latest (but nowhere near most popular) framework for building web services"?

Also, correct me if I'm wrong, but isn't Node implemented in C, with a build system in C and Python? So if we really want to be more like Node for some reason, we should make the build system Pythony instead of Javascripty?

[Ryan Gonzalez]

On Thu, Mar 19, 2015 at 1:48 PM, Andrew Barnert <abar...@yahoo.com.dmarc.invalid> wrote:

On Mar 19, 2015, at 4:12 AM, anatoly techtonik <tech...@gmail.com> wrote:
>
>> On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <m...@the-compiler.org> wrote:
>> * anatoly techtonik <tech...@gmail.com> [2015-03-18 10:21:46 +0300]:
>>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rym...@gmail.com> wrote:
>>>>>
>>>>> But to miss-paraphrase Winston Churchill [1]:
>>>>>
>>>>> "Autotools is the worst build system except for all the others."
>>>>>
>>>>> Is there really something better out there?
>>>>
>>>> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>>>
>>> Is there anything Javascriptey? Python needs to move to web. I heard
>>> something about Grunt, which was replaced by Gulp, but no idea if they
>>> really mature enough to include "system features detection" and "user
>>> configured builds" (or work with C/C++ compilers at all).
>>
>> What does "Python needs to move to web" to do with it's build system
>> being "Javascriptey"?
>
> Because things like Twisted are de-facto standard in JavaScript world
> (Node.js), and Python will benefit more if people look for how things are
> implemented in non-C worlds.

Is this a sarcastic comment?

Or are you seriously suggesting that because people use things like the stdlib's asyncio, gevent, Tornado, etc. in Python, which have different paradigms than Twisted, the one Node copied, we need to force some other part of Python to be more like Node in some other way, because "move to the web" means "be more like the latest (but nowhere near most popular) framework for building web services"?

Also, correct me if I'm wrong, but isn't Node implemented in C, with a build system in C and Python? So if we really want to be more like Node for some reason, we should make the build system Pythony instead of Javascripty?

You're not wrong. Node uses Gyp, although, if Chromium switches to GN (written in C++), Node might switch, too.

 

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/