[Python-Dev] Bug in build system for cross-platform builds

[Russell Keith-Magee]

Hi all,

I’ve been working on developing Python builds for mobile platforms, and I’m looking for some help resolving a bug in Python’s build system.

The problem affects cross-platform builds - builds where you are compiling python for a CPU architecture other than the one on the machine that is doing the compilation. This requirement stems from supporting mobile platforms (iOS, Android etc) where you compile on your laptop, then ship the compiled binary to the device.

In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing parallel builds. However, as a side effect, this patch broke (as far as I can tell) *all* cross platform builds. This was reported in issue 22625 [2]. 

Since that time, the problem has gotten slightly worse; the addition of changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been able to hack together a fix that enables me to get a set of binaries, but the patch is essentially reverting 22359, and making some (very dubious) assumptions about the order in which things are built.

Autoconf et al aren’t my strong suit; I was hoping someone might be able to help me resolve this issue.

Yours,

Russ Magee %-)

[1] http://bugs.python.org/issue22359

[2] http://bugs.python.org/issue22625

[3] https://hg.python.org/cpython/rev/565b96093ec8

[4] https://hg.python.org/cpython/rev/02e3bf65b2f8

[Martin Panter]

On 15 February 2016 at 08:24, Russell Keith-Magee

<rus...@keith-magee.com> wrote:
> Hi all,
>
> I’ve been working on developing Python builds for mobile platforms, and I’m
> looking for some help resolving a bug in Python’s build system.
>
> The problem affects cross-platform builds - builds where you are compiling
> python for a CPU architecture other than the one on the machine that is
> doing the compilation. This requirement stems from supporting mobile
> platforms (iOS, Android etc) where you compile on your laptop, then ship the
> compiled binary to the device.
>
> In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing parallel
> builds. However, as a side effect, this patch broke (as far as I can tell)
> *all* cross platform builds. This was reported in issue 22625 [2].
>
> Since that time, the problem has gotten slightly worse; the addition of
> changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been able
> to hack together a fix that enables me to get a set of binaries, but the
> patch is essentially reverting 22359, and making some (very dubious)
> assumptions about the order in which things are built.
>
> Autoconf et al aren’t my strong suit; I was hoping someone might be able to
> help me resolve this issue.

Would you mind answering my question in
<https://bugs.python.org/issue22625#msg247652>? In particular, how did
cross-compiling previously work before these changes. AFAIK Python
builds a preliminary Python executable which is executed on the host
to complete the final build. So how do you differentiate between host
and target compilers etc?
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Russell Keith-Magee]

In order to build for a host platform, you have to compile for a local platform first - for example, to compile an iOS ARM64 binary, you have to compile for OS X x86_64 first. This gives you a local platform version of Python you can use when building the iOS version.

Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points at the CPU-local version of Python that can be invoked, which is used for module builds, and for compiling the standard library source code. This is set by —host and —build flags to configure, plus the use of CC and LDFLAGS environment variables to point at the compiler and libraries for the platform you’re compiling for, and a PATH variable that provides the local platform’s version of Python.

There are two places where special handling is required: the compilation and execution of the parser generator, and _freeze_importlib. In both cases, the tool needs to be compiled for the local platform, and then executed. Historically (i.e., Py3.4 and earlier), this has been done by spawning a child MAKE to compile the tool; this runs the compilation phase with the local CPU environment, before returning to the master makefile and executing the tool. By spawning the child MAKE, you get a “clean” environment, so the tool is built natively. However, as I understand it, it causes problems with parallel builds due to race conditions on build rules. The change in Python3.5 simplified the rule so that child MAKE calls weren’t used, but that means that pgen and _freeze_importlib are compiled for ARM64, so they won’t run on the local platform.

As best as I can work out, the solution is to:

(1) Include the parser generator and _freeze_importlib as part of the artefacts of local platform. That way, you could use the version of pgen and _freeze_importlib that was compiled as part of the local platform build. At present, pgen and _freeze_importlib are used during the build process, but aren’t preserved at the end of the build; or

(2) Include some concept of the “local compiler” in the build process, which can be used to compile pgen and _freeze_importlib; or

There might be other approaches that will work; as I said, build systems aren’t my strength.

Yours,

Russ Magee %-)

[Martin Panter]

Hi Russell. Sorry for the minor ~1 month delay in replying :)

I have been doing some experimenting to see what is involved in
cross-compiling Python (Native host = Linux, target = Windows via
mingw and some patches). So I have a slightly better understanding of
the problem than before.

On 16 February 2016 at 01:41, Russell Keith-Magee

<rus...@keith-magee.com> wrote:
> In order to build for a host platform, you have to compile for a local
> platform first - for example, to compile an iOS ARM64 binary, you have to
> compile for OS X x86_64 first. This gives you a local platform version of
> Python you can use when building the iOS version.
>
> Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points at
> the CPU-local version of Python that can be invoked, which is used for
> module builds, and for compiling the standard library source code. This is
> set by —host and —build flags to configure, plus the use of CC and LDFLAGS
> environment variables to point at the compiler and libraries for the
> platform you’re compiling for, and a PATH variable that provides the local
> platform’s version of Python.

So far I haven’t succeeded with my Min GW cross build and am
temporarily giving up due to incompatibilities. But my attempts looked
a bit like this:

make clean # Work around confusion with existing in-source build
mkdir native
(cd native/ && ../configure)
make -C native/ Parser/pgen
mkdir mingw
(cd mingw/ && ../configure --host=i486-mingw32 --build=x86)
make -C mingw/ PGEN=../native/Parser/pgen

Actually it was not as smooth as the above commands, because pgen
tends to get overwritten with a cross-compiled version. Perhaps we
could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used
at <https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html>.

> There are two places where special handling is required: the compilation and
> execution of the parser generator, and _freeze_importlib. In both cases, the
> tool needs to be compiled for the local platform, and then executed.
> Historically (i.e., Py3.4 and earlier), this has been done by spawning a
> child MAKE to compile the tool; this runs the compilation phase with the
> local CPU environment, before returning to the master makefile and executing
> the tool. By spawning the child MAKE, you get a “clean” environment, so the
> tool is built natively. However, as I understand it, it causes problems with
> parallel builds due to race conditions on build rules. The change in
> Python3.5 simplified the rule so that child MAKE calls weren’t used, but
> that means that pgen and _freeze_importlib are compiled for ARM64, so they
> won’t run on the local platform.

You suggest that the child Make command happened to compile pgen etc
natively, rather than with the cross compiler. But my understanding is
that when you invoke $(MAKE), all the environment variables, configure
settings, etc, including the cross compiler, would be inherited by the
child.

Would it be more correct to say instead that in 3.4 you did a separate
native build step, precompiling pgen and _freeze_importlib for the
native build host? Then you hoped that the child Make was _not_
invoked in the cross-compilation stage and your precompiled
executables would not be rebuilt?

> As best as I can work out, the solution is to:
>
> (1) Include the parser generator and _freeze_importlib as part of the
> artefacts of local platform. That way, you could use the version of pgen and
> _freeze_importlib that was compiled as part of the local platform build. At
> present, pgen and _freeze_importlib are used during the build process, but
> aren’t preserved at the end of the build; or

I don’t understand. After I run Make, it looks like I get working
executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
you mean to install these programs with “make install” or something?

> (2) Include some concept of the “local compiler” in the build process, which
> can be used to compile pgen and _freeze_importlib; or

On the surface solution (2) sounds like the ideal fix. But I guess the
local native compiler might also require a separate set of CPPFLAGS,
pyconfig.h settings etc. In other words it is sounding like a whole
separate “configure” run. I am thinking it might be simplest to just
require this native “configure” run to be done manually.

[Russell Keith-Magee]

That might fix the pgen problem,  but _freeze_importlib still remains. I suppose the same thing might be possible for _freeze_importlib as well…

> There are two places where special handling is required: the compilation and
> execution of the parser generator, and _freeze_importlib. In both cases, the
> tool needs to be compiled for the local platform, and then executed.
> Historically (i.e., Py3.4 and earlier), this has been done by spawning a
> child MAKE to compile the tool; this runs the compilation phase with the
> local CPU environment, before returning to the master makefile and executing
> the tool. By spawning the child MAKE, you get a “clean” environment, so the
> tool is built natively. However, as I understand it, it causes problems with
> parallel builds due to race conditions on build rules. The change in
> Python3.5 simplified the rule so that child MAKE calls weren’t used, but
> that means that pgen and _freeze_importlib are compiled for ARM64, so they
> won’t run on the local platform.

You suggest that the child Make command happened to compile pgen etc
natively, rather than with the cross compiler. But my understanding is
that when you invoke $(MAKE), all the environment variables, configure
settings, etc, including the cross compiler, would be inherited by the
child.

Would it be more correct to say instead that in 3.4 you did a separate
native build step, precompiling pgen and _freeze_importlib for the
native build host? Then you hoped that the child Make was _not_
invoked in the cross-compilation stage and your precompiled
executables would not be rebuilt?

Yes - as far as I can make out (with my admittedly hazy understanding), that appears to be what is going on. Although it’s not that I “hoped” the build wouldn’t happen on the second pass - it was the behavior that was previously relied, and on was altered.

 

> As best as I can work out, the solution is to:
>
> (1) Include the parser generator and _freeze_importlib as part of the
> artefacts of local platform. That way, you could use the version of pgen and
> _freeze_importlib that was compiled as part of the local platform build. At
> present, pgen and _freeze_importlib are used during the build process, but
> aren’t preserved at the end of the build; or

I don’t understand. After I run Make, it looks like I get working
executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
you mean to install these programs with “make install” or something?

Making them part of the installable artefacts would be one option, but they don’t have to be installed, just preserved.

For example, as a nasty hack, I’ve been able to use this approach to get the build working for 3.5. After the native build, I copy _freeze_importlib to a “safe” location. I then copy it back into place prior to the target build. It works, but it’s in no way suitable for a final build.

 

> (2) Include some concept of the “local compiler” in the build process, which
> can be used to compile pgen and _freeze_importlib; or

On the surface solution (2) sounds like the ideal fix. But I guess the
local native compiler might also require a separate set of CPPFLAGS,
pyconfig.h settings etc. In other words it is sounding like a whole
separate “configure” run. I am thinking it might be simplest to just
require this native “configure” run to be done manually.

That run is going to happen anyway, since you have to compile and build for the native platform. 

Yours,

Russ Magee %-)

[Martin Panter]

On 11 March 2016 at 23:16, Russell Keith-Magee <rus...@keith-magee.com> wrote:
>
> On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmi...@gmail.com> wrote:
>> make clean # Work around confusion with existing in-source build
>> mkdir native
>> (cd native/ && ../configure)
>> make -C native/ Parser/pgen
>> mkdir mingw
>> (cd mingw/ && ../configure --host=i486-mingw32 --build=x86)
>> make -C mingw/ PGEN=../native/Parser/pgen
>>
>> Actually it was not as smooth as the above commands, because pgen
>> tends to get overwritten with a cross-compiled version. Perhaps we
>> could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used
>> at
>> <https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html>.
>>
> That might fix the pgen problem, but _freeze_importlib still remains. I
> suppose the same thing might be possible for _freeze_importlib as well…

Yes. I never got up to it failing in my experiments, but I think I
would propose a FREEZE_IMPORTLIB override variable for that too.

>> Would it be more correct to say instead that in 3.4 you did a separate
>> native build step, precompiling pgen and _freeze_importlib for the
>> native build host? Then you hoped that the child Make was _not_
>> invoked in the cross-compilation stage and your precompiled
>> executables would not be rebuilt?
>
>
> Yes - as far as I can make out (with my admittedly hazy understanding), that
> appears to be what is going on. Although it’s not that I “hoped” the build
> wouldn’t happen on the second pass - it was the behavior that was previously
> relied, and on was altered.

Do you have a copy/patch/link/etc to the actual commands that you
relied on? It’s hard to guess exactly what you were doing that broke
without this information.

>> > As best as I can work out, the solution is to:
>> >
>> > (1) Include the parser generator and _freeze_importlib as part of the
>> > artefacts of local platform. That way, you could use the version of pgen
>> > and
>> > _freeze_importlib that was compiled as part of the local platform build.
>> > At
>> > present, pgen and _freeze_importlib are used during the build process,
>> > but
>> > aren’t preserved at the end of the build; or
>>
>> I don’t understand. After I run Make, it looks like I get working
>> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> you mean to install these programs with “make install” or something?
>
>
> Making them part of the installable artefacts would be one option, but they
> don’t have to be installed, just preserved.

What commands are you running that cause them to not be preserved at
the end of the build?

> For example, as a nasty hack, I’ve been able to use this approach to get the
> build working for 3.5. After the native build, I copy _freeze_importlib to a
> “safe” location. I then copy it back into place prior to the target build.
> It works, but it’s in no way suitable for a final build.
>
>>
>> > (2) Include some concept of the “local compiler” in the build process,
>> > which
>> > can be used to compile pgen and _freeze_importlib; or
>>
>> On the surface solution (2) sounds like the ideal fix. But I guess the
>> local native compiler might also require a separate set of CPPFLAGS,
>> pyconfig.h settings etc. In other words it is sounding like a whole
>> separate “configure” run. I am thinking it might be simplest to just
>> require this native “configure” run to be done manually.
>
>
> That run is going to happen anyway, since you have to compile and build for
> the native platform.

[Martin Panter]

On 13 March 2016 at 01:13, Russell Keith-Magee <rus...@keith-magee.com> wrote:
> The patches that I've uploaded to Issue23670 [1] show a full cross-platform
> [1] http://bugs.python.org/issue23670
> build process. After you apply that patch, the iOS directory contains a
> meta-Makefile that manages the build process.

Thanks very much for pointing that out. This has helped me understand
a lot more things. Only now do I realize that the four files generated
by pgen and _freeze_importlib are actually already committed into the
Mercurial repository:

Include/graminit.h
Python/graminit.c
Python/importlib.h
Python/importlib_external.h

A question for other Python developers: Why are these generated files
stored in the repository? The graminit ones seem to have been there
since forever (1990). It seems the importlib ones were there due to a
bootstrapping problem, but now that is solved. Antoine
<https://bugs.python.org/issue14928#msg163048> said he kept them in
the repository on purpose, but I want to know why.

If we ignore the cross compiling use case, would there be any other
consequences of removing these generated files from the repository?
E.g. would it affect the Windows build process?

I have two possible solutions in mind: either remove the generated
files from the repository and always build them, or keep them but do
not automatically regenerate them every build. Since they are
generated files, not source files, I would prefer to remove them, but
I want to know the consequences first.

> On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter <vadmi...@gmail.com> wrote:
>> On 11 March 2016 at 23:16, Russell Keith-Magee <rus...@keith-magee.com>
>> wrote:
>> >
>> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmi...@gmail.com>
>> > wrote:
>> >> I don't understand. After I run Make, it looks like I get working
>> >> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> >> you mean to install these programs with "make install" or something?
>> >
>> >
>> > Making them part of the installable artefacts would be one option, but
>> > they
>> > don't have to be installed, just preserved.
>>
>> What commands are you running that cause them to not be preserved at
>> the end of the build?
>
>

> I don't know - this is where I hit the end of my understanding of the build
> process. All I know for certain is that 3.4.2 doesn't have this problem;
> 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> subsequent fix [4] introduced an additional problem with _freeze_importlib.

>
> [3] https://hg.python.org/cpython/rev/565b96093ec8
> [4] https://hg.python.org/cpython/rev/02e3bf65b2f8

After my realization about the generated files, I think I can solve
this one. Before the changes you identified, the build process
probably thought the generated files were up to date, so it didn't
need to cross-compile pgen or _freeze_importlib. If the generated file
timestamps were out of date (e.g. depending on the order they are
checked out or extracted), the first native build stage would have
fixed them up.

[Gregory P. Smith]

They should not be regenerated every build, if they are, that seems like a bug in the makefile to me (or else the timestamp checks that make does vs how your code checkout happened).  Having them checked in is convenient for cross builds as it is one less thing that needs a build-host-arch build.

my 2 cents,

-gps

 

Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org

[Steve Dower]

Simply removing them would break the Windows build, and it may not be easily fixable as the dependency system we use doesn't allow building a project twice.

Currently we fail the build if a generated file changes and then the user has to trigger a second build with the new file, but typically they're fine and the first build succeeds.

Cheers,
Steve

Top-posted from my Windows Phone

---

From: Martin Panter
Sent: ‎3/‎13/‎2016 19:43
To: Russell Keith-Magee; python-dev
Cc: ant...@python.org
Subject: Re: [Python-Dev] Bug in build system for cross-platform builds

Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

[Nick Coghlan]

On 14 March 2016 at 13:04, Gregory P. Smith <gr...@krypto.org> wrote:
> They should not be regenerated every build, if they are, that seems like a
> bug in the makefile to me (or else the timestamp checks that make does vs
> how your code checkout happened). Having them checked in is convenient for
> cross builds as it is one less thing that needs a build-host-arch build.

It's also two less things to go wrong for folks just wanting to work
on the 99.9% of CPython that is neither the Grammar file nor
importlib._bootstrap.

I'm trying to remember the problem I was solving in making
freezeimportlib.o explicitly depend on the Makefile (while cursing
past me for not writing it down in the commit message), and I think
the issue was that it wasn't correctly picking up changes to the
builtin module list. If that's right, then fixing the dependency to be
on "$(srcdir)/Makefile.pre.in" instead of the generated Makefile
should keep it from getting confused in the cross-compilation
scenario, and also restore the behaviour of using the checked in copy
rather than rebuilding it just because you ran "./configure".

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[R. David Murray]

The repo-timestamp problem is addressed by the 'make touch' target.

And yes, checking in these platform-independent artifacts is very
intentional: less to build, fewer external dependencies in the build
process...you don't need to *have* python to *build* python, which you
would have to if they were not checked in.

--David

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Xavier de Gaye]

On 03/14/2016 02:26 PM, R. David Murray wrote:
>
> The repo-timestamp problem is addressed by the 'make touch' target.
>
> And yes, checking in these platform-independent artifacts is very
> intentional: less to build, fewer external dependencies in the build
> process...you don't need to *have* python to *build* python, which you
> would have to if they were not checked in.

Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect
uses of recursive make. The changeset added executable binaries as
prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)).
This broke cross-compilation:
* the executables do not exist and must be cross-compiled
* then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since
the prerequisites are newer
* but the executables cannot run on the build system

Actually the files need not be re-generated as their timestamps have been
setup for that purpose with 'make touch'. So a solution to the problem
introduced by this changeset when cross-compiling could be to remove the
binaries as prerequisites of these rules and include the recipe of their
corresponding rules, the one used to build the executable, into the recipes of
the original rule. Also IMHO the Programs/_freeze_importlib.c can be used
instead of Programs/_freeze_importlib.o as a prerequisite in the
Python/importlib.h rule.

[1] https://hg.python.org/cpython/rev/c2a53aa27cad/
[2] http://bugs.python.org/issue22359

Xavier

[Xavier de Gaye]

On 03/14/2016 05:34 PM, Xavier de Gaye wrote:
> Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect
> uses of recursive make. The changeset added executable binaries as
> prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)).
> This broke cross-compilation:
> * the executables do not exist and must be cross-compiled
> * then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since
> the prerequisites are newer
> * but the executables cannot run on the build system
>
> Actually the files need not be re-generated as their timestamps have been
> setup for that purpose with 'make touch'. So a solution to the problem
> introduced by this changeset when cross-compiling could be to remove the
> binaries as prerequisites of these rules and include the recipe of their
> corresponding rules, the one used to build the executable, into the recipes of
> the original rule. Also IMHO the Programs/_freeze_importlib.c can be used
> instead of Programs/_freeze_importlib.o as a prerequisite in the
> Python/importlib.h rule.
>
> [1] https://hg.python.org/cpython/rev/c2a53aa27cad/
> [2] http://bugs.python.org/issue22359

The pgen dependencies are lost when following my previous suggestion, which is
wrong. I have uploaded a patch at issue 22359 that uses a conditional to change the
rules, based on whether a cross-compilation is being run.

[Martin Panter]

On 14 March 2016 at 13:26, R. David Murray <rdmu...@bitdance.com> wrote:
> On Mon, 14 Mar 2016 03:04:08 -0000, "Gregory P. Smith" <gr...@krypto.org> wrote:
>> On Sun, Mar 13, 2016 at 7:41 PM Martin Panter <vadmi...@gmail.com> wrote:
>> > Include/graminit.h
>> > Python/graminit.c
>> > Python/importlib.h
>> > Python/importlib_external.h
>> >
>> > A question for other Python developers: Why are these generated files

>> > stored in the repository? [. . .]

>>
>> They should not be regenerated every build, if they are, that seems like a
>> bug in the makefile to me (or else the timestamp checks that make does vs
>> how your code checkout happened).

The reason the current Python 3 build regenerates some files, is
because of the makefile prerequisites. For example, Include/graminit.h
currently depends on Parser/pgen, which needs to be compiled for the
native build host.

>> Having them checked in is convenient for
>> cross builds as it is one less thing that needs a build-host-arch build.
>

> [. . .]

> And yes, checking in these platform-independent artifacts is very
> intentional: less to build, fewer external dependencies in the build
> process...you don't need to *have* python to *build* python, which you
> would have to if they were not checked in.

Okay so it sounds like the generated files (more listed in .hgtouch)
have to stay. Reasons given:

* Some need Python to generate them (bootstrap problem)
* Relied on by Windows build system
* General convenience (less build steps, less prerequisites, less
things to go wrong)

One more idea I am considering is to decouple the makefile rules from
the main build. So to update the generated files you would have to run
a separate command like “make graminit” or “make frozen”. The normal
build would never regenerate them; although perhaps it could still
result in an error or warning if they appear out of date.

[Nick Coghlan]

On 15 March 2016 at 10:49, Martin Panter <vadmi...@gmail.com> wrote:
> One more idea I am considering is to decouple the makefile rules from
> the main build. So to update the generated files you would have to run
> a separate command like “make graminit” or “make frozen”. The normal
> build would never regenerate them; although perhaps it could still
> result in an error or warning if they appear out of date.

Some of them used to work that way and it's an incredible PITA when
you actually *are* working on one of the affected bits of the
interpreter - working on those parts is rare, so if there are special
cases to remember, you can pretty much guarantee we'll have forgotten
them by the time we work on that piece again.

However, it would be worth reviewing the explicit dependencies on
"Makefile" and see whether they could be replaced by dependencies on
Makefile.pre.in instead. I'm confident that will work just fine for
the importlib bootstrapping, and I suspect it will work for the other
pregenerated-and-checked-in files as well.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Martin Panter]

On 15 March 2016 at 04:32, Nick Coghlan <ncog...@gmail.com> wrote:
> On 15 March 2016 at 10:49, Martin Panter <vadmi...@gmail.com> wrote:
>> One more idea I am considering is to decouple the makefile rules from
>> the main build. So to update the generated files you would have to run
>> a separate command like “make graminit” or “make frozen”. The normal
>> build would never regenerate them; although perhaps it could still
>> result in an error or warning if they appear out of date.
>
> Some of them used to work that way and it's an incredible PITA when
> you actually *are* working on one of the affected bits of the
> interpreter - working on those parts is rare, so if there are special
> cases to remember, you can pretty much guarantee we'll have forgotten
> them by the time we work on that piece again.

Perhaps if we wrapped them all up in a common “make regenerate”
target, so it is only one special case to remember? Maybe you could
include other stuff like “make clinic” in that as well. Or you could
include the special commands in the warning messages.

> However, it would be worth reviewing the explicit dependencies on
> "Makefile" and see whether they could be replaced by dependencies on
> Makefile.pre.in instead. I'm confident that will work just fine for
> the importlib bootstrapping, and I suspect it will work for the other
> pregenerated-and-checked-in files as well.

The problem is not the reference to Makefile. The graminit files do
not depend on Makefile. The bigger problem is that the checked-in
files depend on compiled programs. This is a summary of the current
rules for importlib:

_freeze_importlib.o: _freeze_importlib.c Makefile

_freeze_importlib: _freeze_importlib.o [. . .]
$(LINKCC) [. . .]

importlib_external.h: _bootstrap_external.py _freeze_importlib
_freeze_importlib _bootstrap_external.py importlib_external.h

importlib.h: _bootstrap.py _freeze_importlib
_freeze_importlib _bootstrap.py importlib.h

So importlib.h depends on the _freeze_importlib compiled program (and
only indirectly on Makefile). The makefile says we have to compile
_freeze_importlib before checking if importlib.h is up to date.

Gnu Make has order-only prerequisites
<https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html>,
which it seems we could abuse to do most of what we want. But (1) I’m
not sure if we can restrict ourselves to Gnu Make, and (2) it is a
horrible hack and would always compile _freeze_importlib even if it is
never run.

[Nick Coghlan]

On 15 March 2016 at 15:15, Martin Panter <vadmi...@gmail.com> wrote:
> The problem is not the reference to Makefile. The graminit files do
> not depend on Makefile. The bigger problem is that the checked-in
> files depend on compiled programs. This is a summary of the current
> rules for importlib:
>
> _freeze_importlib.o: _freeze_importlib.c Makefile
>
> _freeze_importlib: _freeze_importlib.o [. . .]
> $(LINKCC) [. . .]
>
> importlib_external.h: _bootstrap_external.py _freeze_importlib
> _freeze_importlib _bootstrap_external.py importlib_external.h
>
> importlib.h: _bootstrap.py _freeze_importlib
> _freeze_importlib _bootstrap.py importlib.h
>
> So importlib.h depends on the _freeze_importlib compiled program (and
> only indirectly on Makefile). The makefile says we have to compile
> _freeze_importlib before checking if importlib.h is up to date.

Ah, I understand now - the fundamental problem is with a checked in
file depending on a non-checked-in file, so if you clean out all the
native build artifacts when cross-compiling, the makefile will attempt
to create target versions of all the helper utilities (pgen,
_freeze_importlib, argument clinic, etc).

Would it help to have a "make bootstrap" target that touched all the
checked in generated files with build dependencies on non-checked-in
files, but only after first touching the expected locations of the
built binaries they depend on?

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Martin Panter]

On 15 March 2016 at 08:04, Nick Coghlan <ncog...@gmail.com> wrote:
> On 15 March 2016 at 15:15, Martin Panter <vadmi...@gmail.com> wrote:
>> _freeze_importlib.o: _freeze_importlib.c Makefile
>>
>> _freeze_importlib: _freeze_importlib.o [. . .]
>> $(LINKCC) [. . .]
>>
>> importlib_external.h: _bootstrap_external.py _freeze_importlib
>> _freeze_importlib _bootstrap_external.py importlib_external.h
>>
>> importlib.h: _bootstrap.py _freeze_importlib
>> _freeze_importlib _bootstrap.py importlib.h
>

> Ah, I understand now - the fundamental problem is with a checked in
> file depending on a non-checked-in file, so if you clean out all the
> native build artifacts when cross-compiling, the makefile will attempt
> to create target versions of all the helper utilities (pgen,
> _freeze_importlib, argument clinic, etc).
>
> Would it help to have a "make bootstrap" target that touched all the
> checked in generated files with build dependencies on non-checked-in
> files, but only after first touching the expected locations of the
> built binaries they depend on?

That sounds similar to “make touch”, with a couple differences. One
trouble I forsee is the conflict with shared prerequisites. E.g. “make
bootstrap” would have to create some dummy object files as
prerequisites of the pgen program, but we would first have build
others e.g. Parser/acceler.o properly for the main Python library. It
all feels way too complicated to me. The Python build system is
complicated enough as it is.

Maybe it is simplest to just add something in the spirit of Xavier’s
suggested patch. This would mean that we keep the generated files
checked in (to help with Windows and cross compiled builds), we keep
the current rules that force normal makefile builds to blindly
regenerate the files, but we add some flag or configure.ac check to
disable this regeneration if desired.