Teaching gyp about architectures and crosscompiling?

[Torne (Richard Coles)]

Hi gyp folk,

gyp currently knows the OS it's building for ("flavor") which is both used by generators to make decisions about what to generate, and also exported as a variable for gyp files to use. Would people be averse to it also gaining an awareness of what architecture it's building for? It looks like some of the Windows machinery already does this in a generator-specific way (msvs and winja).

Also, there are some patches that were proposed a while ago to teach gyp about crosscompilation a bit more robustly by also having it distinguish host/target OS/flavor, which I imagine could also be extended to the architecture, if this is something we want to have.

The specific problem I'm trying to solve is that for the Android.mk-based build system backend, the build system expects to be able to build for more than one target architecture in the same build tree, and currently this breaks for any host binary which is target-arch specific such as v8's mksnapshot; I need to do something like mangling the target name with the architecture included to prevent this from clashing, but at the moment chromium just passes the target architecture as a normal gyp variable in GYP_DEFINES and so it's not immediately available to the generator in a reliable way.

Any objections to making the host/target OS/architecture a more first-class concept in gyp? I realise that simple projects may not need this complexity, but the defaults can reasonably just be detected..

--
Torne (Richard Coles)
to...@google.com

[Ryan Sleevi]

When using other generators, such as Ninja, the ability exists to play
the build files (build.ninja and friends) in a separate output
directory independent of the source files (eg: out_ArchA/Debug,
out_ArchB/Debug, etc)

I'm guessing that the Android.mk-based build follows a pattern similar
to the Makefile generator, in which, by default, it places build
artifacts in the source hierarchy. The Makefile generator
support(s/ed?) the ability to relocate the Makefiles, similar to
Ninja, via the -Goutput_dir generator flag.

This, combined with the toolset (target vs host), should be sufficient
to handle cross-compilation, shouldn't it?

[bungeman]

Just wanted to point out that I believe --generator-output is where the make generator puts the makefiles, -Goutput_dir is where the toplevel of the actual build should be (if relative, then this is relative to --generator-output).

[Torne (Richard Coles)]

No, this isn't the problem. The issue is that all android build output goes to a single central location and there is just one directory for host binaries. mksnapshot needs to be built with specific knowledge of the target arch, so if you build for arm you get a mksnapshot that will only work for arm, and then when you build for x86 it will use the already compiled arm-specific binary and fail. There's no problem with generating the makefiles; I need to be able to mangle the target names for things built for host.

There are other issues we are running into that also would be easier to solve if gyp understood cross compilation and architectures; the chromium gyp files need to know all four of the host and target os and architecture but obtains them in very haphazard ways, passed from build/android/envsetup.sh or guessed in a variety of ways in different places..

[Ryan Sleevi]

On Mon, Feb 25, 2013 at 11:01 AM, Torne (Richard Coles)
<to...@google.com> wrote:
> No, this isn't the problem. The issue is that all android build output goes
> to a single central location and there is just one directory for host
> binaries. mksnapshot needs to be built with specific knowledge of the target
> arch, so if you build for arm you get a mksnapshot that will only work for
> arm, and then when you build for x86 it will use the already compiled
> arm-specific binary and fail. There's no problem with generating the
> makefiles; I need to be able to mangle the target names for things built for
> host.

Given that GYP lacks an intrinsic concept of host/target arch (beyond
the fact that such toolsets exist), and the most common way (eg: in
Chromium and Node) is to use a variable named target_arch, can you not
just incorporate target_arch into the output name of such
target-arch-dependent host targets?
{
'target_name': 'mksnapshot',
'product_name': 'mksnapshot-<(target_arch)'
'toolsets': ['host'],
...
}

It may be an unfounded concern, but I'm just trying to think of a
clean, cross-compiler friendly syntax that also keeps in line with
GYP's goal of being expressed in the lowest common denominator between
generators. (eg: why you cannot do configuration-specific sources,
even though all generator BUT xcode support them)

>
> There are other issues we are running into that also would be easier to
> solve if gyp understood cross compilation and architectures; the chromium
> gyp files need to know all four of the host and target os and architecture
> but obtains them in very haphazard ways, passed from
> build/android/envsetup.sh or guessed in a variety of ways in different
> places..

I don't know if, in the context of other generators (particularly MSVS
and XCode), it becomes necessarily easier. For example, the MSVS
generator can only build one target name/configuration at a time. For
Chromium's use of NACL on Windows - which requires both a 32-bit and
64-bit helper executable - this has necessitated creating multiple
targets, distinguished by name, in order to fully express the build
dependency that a '32-bit Chromium' still needs a 64-bit helper
binary, for if/when it's executed on a 64-bit Windows.

I understand the desire to solve the problem for "just one generator",
but those approaches seem to make GYP even more unwieldy for the
general case (eg: Chromium's previous use of XCode's arch conditionals
- GCC_PREPROCESSOR_DEFINITIONS[arch=x86_64] ), so if a syntax is to
be proposed, I think it will have to be able to work well with all
generators.

[Torne (Richard Coles)]

On 25 February 2013 19:11, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Feb 25, 2013 at 11:01 AM, Torne (Richard Coles)
<to...@google.com> wrote:
> No, this isn't the problem. The issue is that all android build output goes
> to a single central location and there is just one directory for host
> binaries. mksnapshot needs to be built with specific knowledge of the target
> arch, so if you build for arm you get a mksnapshot that will only work for
> arm, and then when you build for x86 it will use the already compiled
> arm-specific binary and fail. There's no problem with generating the
> makefiles; I need to be able to mangle the target names for things built for
> host.

Given that GYP lacks an intrinsic concept of host/target arch (beyond
the fact that such toolsets exist), and the most common way (eg: in
Chromium and Node) is to use a variable named target_arch, can you not
just incorporate target_arch into the output name of such
target-arch-dependent host targets?
{
  'target_name': 'mksnapshot',
  'product_name': 'mksnapshot-<(target_arch)'
  'toolsets': ['host'],
  ...
}

Hm, I can try that, it should work. The problem there is you need to know that the binary is target-arch specific, as if you forget then things will break in unpredictable ways..

 

It may be an unfounded concern, but I'm just trying to think of a
clean, cross-compiler friendly syntax that also keeps in line with
GYP's goal of being expressed in the lowest common denominator between
generators. (eg: why you cannot do configuration-specific sources,
even though all generator BUT xcode support them)

I think I maybe haven't explained what I'm thinking of well enough. This shouldn't add any new requirements on generators.

 

>
> There are other issues we are running into that also would be easier to
> solve if gyp understood cross compilation and architectures; the chromium
> gyp files need to know all four of the host and target os and architecture
> but obtains them in very haphazard ways, passed from
> build/android/envsetup.sh or guessed in a variety of ways in different
> places..

I don't know if, in the context of other generators (particularly MSVS
and XCode), it becomes necessarily easier. For example, the MSVS
generator can only build one target name/configuration at a time. For
Chromium's use of NACL on Windows - which requires both a 32-bit and
64-bit helper executable - this has necessitated creating multiple
targets, distinguished by name, in order to fully express the build
dependency that a '32-bit Chromium' still needs a 64-bit helper
binary, for if/when it's executed on a 64-bit Windows.

I wasn't intending to support building more than one of these things "at once", (i.e. with one set of gyp-generated files), and yes, it won't address the weird cases where you need to build stuff explicitly for a foreign arch.

 

I understand the desire to solve the problem for "just one generator",
but those approaches seem to make GYP even more unwieldy for the
general case (eg: Chromium's previous use of XCode's arch conditionals
-  GCC_PREPROCESSOR_DEFINITIONS[arch=x86_64] ), so if a syntax is to
be proposed, I think it will have to be able to work well with all
generators.

It should work with all generators.

Currently gyp detects the OS/flavor and passes this info to the generators, which may or may not make use of it, and also exposes it to the gyp files as the variable "OS". You can override this at gyp time. What I was suggesting is to do the same for the architecture, and to also have a separate TARGET_OS and TARGET_ARCH or similar variables which always default to the same as the host ones, but can be overridden separately. gyp files that currently do their own os/arch detection could then drop this in favour of using these standard vars (and the same for wrappers/environment scripts like build/android/envsetup.sh), and generators which want to do something OS/arch specific can just directly test these configuration parameters instead of having to guess themselves. There shouldn't be any changes to generators *required*.

If you think this is too complicated or too much work for too little benefit, then that's fair enough, but I don't think there is a problem with cross-generator compatibility here; my goal is just to formalise something that chromium's gyp files already do awkwardly for themselves and in the process make that information available to the generator easily if it has a use for it.

[Ryan Sleevi]

In that case, a tentative +1. I know there has been discussion about trying to formalize some of the GYP best practice (eg: a bare bones equivalent to Chromium's build/common.gypi), and this seems in line with that.

I am not entirely sure it needs to be a built-in like OS, vs. part of a default set of variables, but I have no strong feelings either way.

[Torne (Richard Coles)]

Well, the point is just that some of the generators need to know this information anyway, and that many projects may need to detect them for the host system in the same way, so it saves duplicating logic to do it in gyp itself.

I recalled another example of where this information would be useful: currently you can't crosscompile stuff for Android (with the normal make/ninja backends) on Mac using gyp, because the generators assume that when OS==android it's okay to use linux-specific flags when linking host binaries. To fix this the generator needs to know the host OS separately from the target. :)

I'll see if I can put a concrete CL together for this soon and see what people think.

 

>
> --
> Torne (Richard Coles)
> to...@google.com

[Torne (Richard Coles)]

On 25 February 2013 19:11, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Feb 25, 2013 at 11:01 AM, Torne (Richard Coles)
<to...@google.com> wrote:
> No, this isn't the problem. The issue is that all android build output goes
> to a single central location and there is just one directory for host
> binaries. mksnapshot needs to be built with specific knowledge of the target
> arch, so if you build for arm you get a mksnapshot that will only work for
> arm, and then when you build for x86 it will use the already compiled
> arm-specific binary and fail. There's no problem with generating the
> makefiles; I need to be able to mangle the target names for things built for
> host.

Given that GYP lacks an intrinsic concept of host/target arch (beyond
the fact that such toolsets exist), and the most common way (eg: in
Chromium and Node) is to use a variable named target_arch, can you not
just incorporate target_arch into the output name of such
target-arch-dependent host targets?
{
  'target_name': 'mksnapshot',
  'product_name': 'mksnapshot-<(target_arch)'
  'toolsets': ['host'],
  ...
}

Having tried this, the problem is worse; it's not just mksnapshot itself but all its dependencies in v8 which need mangling, and the target_name has to be mangled, not just the product_name, since it's not just the name of the final binary that mustn't conflict but also all the intermediate build files/directories, and that's not affected by product_name. This basically boils down to renaming every target in v8, which I'm not sure the v8 developers will be very happy about, as it means you can't run "make v8" any more and will have to do "make v8_arm" or whatever. :/

Can't think of a better gypfile-side workaround, though.

[Sam Clegg]

On Mon, Feb 25, 2013 at 11:01 AM, Torne (Richard Coles)
<to...@google.com> wrote:
> No, this isn't the problem. The issue is that all android build output goes
> to a single central location and there is just one directory for host
> binaries. mksnapshot needs to be built with specific knowledge of the target
> arch, so if you build for arm you get a mksnapshot that will only work for
> arm, and then when you build for x86 it will use the already compiled
> arm-specific binary and fail. There's no problem with generating the
> makefiles; I need to be able to mangle the target names for things built for
> host.

IIUC you are saying that you build for arm, then build for x86 and the
mksnapshot binary (which is target specific) doesn't get rebuilt when
you switch? If it is target specific, then surely when you switch
target the compile flags (or something) change which should cause the
host binary to get rebuilt and overwrite the old one. Is this not
happening? Am I missing something here?

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "gyp-developer" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gyp-develope...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Torne (Richard Coles)]

On 27 February 2013 17:33, Sam Clegg <s...@google.com> wrote:

On Mon, Feb 25, 2013 at 11:01 AM, Torne (Richard Coles)
<to...@google.com> wrote:
> No, this isn't the problem. The issue is that all android build output goes
> to a single central location and there is just one directory for host
> binaries. mksnapshot needs to be built with specific knowledge of the target
> arch, so if you build for arm you get a mksnapshot that will only work for
> arm, and then when you build for x86 it will use the already compiled
> arm-specific binary and fail. There's no problem with generating the
> makefiles; I need to be able to mangle the target names for things built for
> host.

IIUC you are saying that you build for arm, then build for x86 and the
mksnapshot binary (which is target specific) doesn't get rebuilt when
you switch?   If it is target specific, then surely when you switch
target the compile flags (or something) change which should cause the
host binary to get rebuilt and overwrite the old one.  Is this not
happening?  Am I missing something here?

The Android build system doesn't know how to rebuild files when the commands used to build them change; it doesn't do the indirection that the normal make backend for gyp generates calling do-cmd (and we can't generate this ourselves because the rules that actually compile files are not under our control, they are from the core android build makefiles).

Also, even if this worked this would not really be an acceptable implementation; the Android build system expects that you can build any number of target configurations in parallel in one tree, and switch between them at will, and doing so shouldn't cause a binary to get recompiled every time. The build system assumes that host binaries only need to be built once no matter how many targets are built, and the only way I can see to work around this is to change the names of the host targets depending on the target architecture if the binaries in question depend on that.

[Torne (Richard Coles)]

On 27 February 2013 17:52, Sam Clegg <s...@google.com> wrote:

This sounds like a bug in the android build system to me.  Surely if
the CFLAGS change it would rebuild the host binary?  Or is that you
need the different mksnapshot binaries to exist on disk at the same
time, rather than one at a time?

Both of these things. It is a bug, or rather a missing feature, in the Android build system (traditional make-based build systems don't have this feature, and Android has not chosen to go to the effort required to add it). But, *also*, the different mksnapshot binaries need to exist at the same time, rather than one at a time. You can build android for dozens of devices and all the output files exist at once, in different subdirectories of the same source tree.

 

> Also, even if this worked this would not really be an acceptable
> implementation; the Android build system expects that you can build any
> number of target configurations in parallel in one tree, and switch between
> them at will, and doing so shouldn't cause a binary to get recompiled every
> time. The build system assumes that host binaries only need to be built once
> no matter how many targets are built, and the only way I can see to work
> around this is to change the names of the host targets depending on the
> target architecture if the binaries in question depend on that.

So the android build system fundamentally can't cope with a host
binary that varies with target arch?

That is exactly the problem, yes. It assumes that host binaries only depend on the host's OS and architecture.

 

Is there a possibility to come at this from the other end of the
problem and create mksnapshot that is multi-arch?

I doubt it, it has to build the entire v8 engine for the architecture in question; I doubt v8 is able to have multiple copies of itself in the same binary.

On 27 February 2013 17:52, Sam Clegg <s...@google.com> wrote:

This sounds like a bug in the android build system to me.  Surely if
the CFLAGS change it would rebuild the host binary?  Or is that you
need the different mksnapshot binaries to exist on disk at the same
time, rather than one at a time?

> Also, even if this worked this would not really be an acceptable
> implementation; the Android build system expects that you can build any
> number of target configurations in parallel in one tree, and switch between
> them at will, and doing so shouldn't cause a binary to get recompiled every
> time. The build system assumes that host binaries only need to be built once
> no matter how many targets are built, and the only way I can see to work
> around this is to change the names of the host targets depending on the
> target architecture if the binaries in question depend on that.

So the android build system fundamentally can't cope with a host
binary that varies with target arch?

Is there a possibility to come at this from the other end of the
problem and create mksnapshot that is multi-arch?

[Torne (Richard Coles)]

OK, I have filed https://code.google.com/p/gyp/issues/detail?id=325 to track this: feel free to comment there if my description isn't sufficient or you disagree with this direction. I'm going to take a look at doing this pretty soon; I'll try and make the CLs small and incremental so we can flush out any problems with the approach before I spend a lot of time on it.