Adding Ninja features needed for Fortran
1,106 views
Skip to first unread message
Brad King
Mar 31, 2015, 11:07:14 AM3/31/15
to ninja...@googlegroups.com, Bill Hoffman, Ben Boeckel
Hi Folks,
In CMake land we are interested in extending Ninja with functionality
needed to support Fortran module dependencies. We'd like to discuss
the design here first in order to make sure we stay on an upstream-
acceptable track.
I've attached a markdown document that introduces Fortran module
dependencies, proposes an approach to handling them automatically
with Ninja, and describes the features we propose to add to Ninja
to support this. (I chose markdown so we can paste this document
into a Github issue or PR if needed.) Detecting Fortran module
dependencies in real projects is a bit more complicated than has
been discussed here previously (in threads linked below). This
design is based on our experience with Fortran dependencies in the
CMake Makefile generators, but is tailored to Ninja's paradigm.
Readers interested only in the proposed Ninja features can skip to
the "Ninja Features Needed" section at the end of the document.
The rest of the document explains the problem, justifies the need
for the features, and explains how they can be used to implement
Fortran support. The quick summary is:
1. Add a notion of *implicit outputs* on build plan nodes that
are not part of `$out` but that Ninja knows are generated.
2. Teach Ninja to recognize a `depgen` option on a rule and
load the named file immediately after an instance of the
rule is brought up to date. The file will specify updates
to the build plan.
We'd appreciate feedback on the overall approach and proposed Ninja
features, as well as guidance or suggestions on the format design
mentioned at the end of the document.
Thanks,
-Brad
P.S. For reference, the Fortran use case has been discussed before:
Rules with multiple outputs, again
https://groups.google.com/forum/#!topic/ninja-build/j-2RfBIOd_8
Fortran 95 with ninja
https://groups.google.com/forum/#!topic/ninja-build/b1-AF3pRJuE
The latter thread links to another older thread:
Re-reading dep files for auto-generated files while building
https://groups.google.com/forum/#!topic/ninja-build/Zqygj8hQbdI
from which a related "deps reload" patch is linked. The
functionality added by that patch will likely be subsumed by
the functionality we propose to support Fortran, but is not
enough on its own.
In CMake land we are interested in extending Ninja with functionality
needed to support Fortran module dependencies. We'd like to discuss
the design here first in order to make sure we stay on an upstream-
acceptable track.
I've attached a markdown document that introduces Fortran module
dependencies, proposes an approach to handling them automatically
with Ninja, and describes the features we propose to add to Ninja
to support this. (I chose markdown so we can paste this document
into a Github issue or PR if needed.) Detecting Fortran module
dependencies in real projects is a bit more complicated than has
been discussed here previously (in threads linked below). This
design is based on our experience with Fortran dependencies in the
CMake Makefile generators, but is tailored to Ninja's paradigm.
Readers interested only in the proposed Ninja features can skip to
the "Ninja Features Needed" section at the end of the document.
The rest of the document explains the problem, justifies the need
for the features, and explains how they can be used to implement
Fortran support. The quick summary is:
1. Add a notion of *implicit outputs* on build plan nodes that
are not part of `$out` but that Ninja knows are generated.
2. Teach Ninja to recognize a `depgen` option on a rule and
load the named file immediately after an instance of the
rule is brought up to date. The file will specify updates
to the build plan.
We'd appreciate feedback on the overall approach and proposed Ninja
features, as well as guidance or suggestions on the format design
mentioned at the end of the document.
Thanks,
-Brad
P.S. For reference, the Fortran use case has been discussed before:
Rules with multiple outputs, again
https://groups.google.com/forum/#!topic/ninja-build/j-2RfBIOd_8
Fortran 95 with ninja
https://groups.google.com/forum/#!topic/ninja-build/b1-AF3pRJuE
The latter thread links to another older thread:
Re-reading dep files for auto-generated files while building
https://groups.google.com/forum/#!topic/ninja-build/Zqygj8hQbdI
from which a related "deps reload" patch is linked. The
functionality added by that patch will likely be subsumed by
the functionality we propose to support Fortran, but is not
enough on its own.
Brad King
Mar 31, 2015, 2:48:25 PM3/31/15
to Peter Collingbourne, ninja...@googlegroups.com, Bill Hoffman, Ben Boeckel
On 03/31/2015 02:42 PM, Peter Collingbourne wrote:
> The proposal is: add a flag to build rules that indicates that if the process
> exits with a specific exit code, Ninja should re-run the generator. CMake
> will scan all Fortran files in the project at generation time and generate
> rules based on them.
That is not scalable because many different Fortran translation units
need to be preprocessed and scanned individually. With generated files
and other ordering dependencies there is no single "scan now" step.
Furthermore, the dependency scanner CMake uses for the Makefile generator
just does approximate preprocessing. With Ninja we aim to be much more
precise by using the real preprocessor.
The features we propose for Ninja are quite generic and will have
other uses besides Fortran.
-Brad
> The proposal is: add a flag to build rules that indicates that if the process
> exits with a specific exit code, Ninja should re-run the generator. CMake
> will scan all Fortran files in the project at generation time and generate
> rules based on them.
That is not scalable because many different Fortran translation units
need to be preprocessed and scanned individually. With generated files
and other ordering dependencies there is no single "scan now" step.
Furthermore, the dependency scanner CMake uses for the Makefile generator
just does approximate preprocessing. With Ninja we aim to be much more
precise by using the real preprocessor.
The features we propose for Ninja are quite generic and will have
other uses besides Fortran.
-Brad
Matthew Woehlke
Mar 31, 2015, 3:26:09 PM3/31/15
to ninja...@googlegroups.com
On 2015-03-31 11:07, Brad King wrote:
> 2. Teach Ninja to recognize a `depgen` option on a rule and
> load the named file immediately after an instance of the
> rule is brought up to date. The file will specify updates
> to the build plan.
> [...]
> 2. Teach Ninja to recognize a `depgen` option on a rule and
> load the named file immediately after an instance of the
> rule is brought up to date. The file will specify updates
> to the build plan.
> * Add implicit outputs to an existing build statement.
Does this mean we might *finally* be able to teach ninja at least about
things like sphinx/doxygen that produce a large number of output files
whose names are not necessarily easily known to the CMakeLists.txt
author? (Bonus points if it means we can *also* somehow automagically
generate the list of input file dependencies beyond the documentation
generation configuration file.)
For that matter, the latter would be tremendously useful for e.g. qrc,
which presently forces a reconfigure if the .qrc changes in order to
update the dependencies.
--
Matthew
Matthew Woehlke
Mar 31, 2015, 4:03:50 PM3/31/15
to ninja...@googlegroups.com
would be off topic to this list. The point here is that it seems to me
that the changes Brad is proposing would *finally* give us a path where
we might eventually be able to implement them. That is, I think the
proposed features sound like they could be very, very useful, and that
their value is by no means limited to supporting Fortran.
p.s. I also really hope ninja can finally get Fortran sorted. As a user
of both ninja and Eigen, I think that objective is useful in its own
right, *aside* from that Brad seems to have a well thought out plan that
will achieve that goal in a generic and flexible manner that has the
potential to allow other great improvements in the future.
--
Matthew
Evan Martin
Mar 31, 2015, 6:09:24 PM3/31/15
to Brad King, ninja-build, Bill Hoffman, Ben Boeckel
I uploaded the document to github here for their nice formatting:
https://gist.github.com/anonymous/439f8f367ba088c10a6a
> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
https://gist.github.com/anonymous/439f8f367ba088c10a6a
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Peter Collingbourne
Apr 1, 2015, 3:06:59 PM4/1/15
to Brad King, ninja...@googlegroups.com, Bill Hoffman, Ben Boeckel
Hi Brad,
May I suggest a simpler alternative proposal? It seems to satisfy the
requirements for Fortran, if I understand them correctly.
The proposal is: add a flag to build rules that indicates that if the process
exits with a specific exit code, Ninja should re-run the generator. CMake
will scan all Fortran files in the project at generation time and generate
rules based on them. The dependency scanner will check if the dependencies
have changed since CMake scanned the files, and if so, exit with this exit
code, causing the rules to be re-generated.
You could probably optimize this process so that CMake does not need to do
all its regular work if nothing but the Fortran dependencies have changed. For
example, CMake could maintain the Fortran dependencies in a separate include
file, and update only that file if only the Fortran dependencies have changed.
Assuming that the dependencies don't change very often, this seems reasonable
to me.
Thanks,
Peter
> =================
>
> Ninja has excellent support for implicit dependencies of languages that
> use the `C` preprocessor. Fortran adds another level of complexity for
> build systems because its "module" system adds implicit *outputs* to
> compilation nodes that are referenced as implicit dependencies of
> other compilation nodes. These implicit outputs and dependencies may
> be discovered only during the build process when all (possibly generated)
> files are available for a given source file to be preprocessed.
>
> Below we introduce Fortran modules and their build system requirements.
> Then we propose new Ninja features to support them.
>
> Fortran Modules
> ---------------
>
> The Fortran module system is defined in section 11.2 of the
> [Fortran 2008 Standard][ISO/IEC 1539-1:2010], also available as a
> [Draft PDF][].
>
> [ISO/IEC 1539-1:2010]: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=50459
> [Draft PDF]: http://www.j3-fortran.org/doc/year/10/10-007.pdf
>
> Each Fortran source file may provide zero or more modules:
>
> $ cat provider.F90
> MODULE mymod
> ...
> END
>
> and also consume zero or more modules:
>
> $ cat consumer.F90
> USE MODULE mymod
> ...
>
> Compilation of a source file consuming a module requires the `.mod`
> file to be present:
>
> $ f95 -c consumer.F90
> ...
> Fatal Error: Can't open module file 'mymod.mod' for reading...
>
> Compilation of a source file providing a module generates a `.mod`
> file for it:
>
> $ f95 -c provider.F90
> $ ls *.o *.mod
> provider.o
> mymod.mod
>
> Once the module has been provided then it can be used:
>
> $ f95 -c consumer.F90
> $ ls *.o *.mod
> consumer.o
> provider.o
> mymod.mod
>
> If the source providing a module is modified and recompiled, the
> module file *may or may not* be updated, depending on whether the
> interface exposed in the module changes.
>
> In general modules used by a source file may be found anywhere
> in a search path (e.g. `-I` paths). They may be provided by
> another source in the same build or may be found in external
> directories existing on the system. The only restriction on
> module dependencies is that they are acyclic.
>
> Manual Module Dependencies
> --------------------------
>
> The above example source files may be built correctly by Ninja
> using manual specification of the module dependencies. We'll
> ignore preprocessor implicit dependencies for now:
>
> $ cat build.ninja
> rule FC
> command = f95 -o $OBJ -c $in
> restat = $RESTAT
>
> build provider.o mymod.mod: FC provider.F90
> OBJ = provider.o
> RESTAT = 1
>
> build consumer.o: FC consumer.F90 | mymod.mod
> OBJ = consumer.o
>
> We use the `OBJ` variable in `FC` because `$out` would
> contain the module as well as the object file. We use `restat`
> for the provider because `mymod.mod` may not be updated by the
> compiler. We can see that the above works in a test session:
>
> $ ninja -v consumer.o
> [1/2] f95 -o provider.o -c provider.F90
> [2/2] f95 -o consumer.o -c consumer.F90
>
> $ ninja -v consumer.o
> ninja: no work to do.
>
> $ touch provider.F90
> $ ninja -v consumer.o
> [1/2] f95 -o provider.o -c provider.F90
>
> $ ninja -v consumer.o
> ninja: no work to do.
>
> This example serves as a conceptual model for the basic dependency
> structure Fortran needs. In practice Fortran source files provide
> and consume many modules so maintaining the dependencies by hand
> is impractical. The build system must be able to discover these
> implicit outputs and implicit dependencies automatically.
>
> Automatic Module Dependencies
> -----------------------------
>
> In order to scan for modules provided or consumed by a source file,
> it must first be preprocessed. Therefore we split compilation into
> three steps:
>
> 1. A `FP` rule to preprocess a source file:
>
> rule FP
> command = f95 -o $out -E $in -MMD -MT $out -MF $out.d
> deps = gcc
> depfile = $out.d
>
> This rule uses normal `depfile` for its implicit preprocessing
> dependencies. Preprocessing does not require the `.mod` files
> to exist.
>
> 2. A `FD` rule to scan for module dependency information and tell
> Ninja to load it using a `depgen` feature proposed below.
>
> rule FD
> command = scan $in $OBJ $out
> depgen = $out
>
> The user-provided `scan` tool will read the preprocessed source
> from `$in`, detect the implicit outputs and dependencies of the
> object file named by `$OBJ`, and write the build plan updates
> to the `$out` file also specified as the `depgen` file. The
> `depgen` file will tell Ninja, through some syntax, how to
> update its build plan. It will inject implicit outputs and
> dependencies into the compilation node for `$OBJ`.
>
> 3. A `FC` rule will compile the already-preprocessed source to
> produce an object file and possibly some `.mod` files.
>
> rule FC
> command = f95 -o $out -c $in
>
> Build statements using this rule can have an implicit or
> order-only dependency on the corresponding `FD` rule output
> to make sure implicit outputs and dependencies are known.
>
> With these rules we may use the following build statements for
> our above example:
>
> build provider.pp.f90: FP provider.F90
> build provider.pp.f90.g: FD provider.pp.f90
> OBJ = provider.o
> build provider.o: FC provider.pp.f90 || provider.pp.f90.g
>
> build consumer.pp.f90: FP consumer.F90
> build consumer.pp.f90.g: FD consumer.pp.f90
> OBJ = consumer.o
> build consumer.o: FC consumer.pp.f90 || consumer.pp.f90.g
>
> The `provider.pp.f90.g` file will:
>
> * Add `mymod.mod` as an *implicit output* of the
> `build provider.o` statement.
>
> * Mark the `build provider.o` node with the `restat` option
> so that consumers do not rebuild after `provider.o` if the
> `mymod.mod` implicit output does not change.
>
> * Add or update an *implicit build statement* of the form:
>
> build mymod.mod.proxy: phony mymod.mod
>
> The `consumer.pp.f90.g` file will:
>
> * Add or update an *implicit build statement* of the form:
>
> build mymod.mod.proxy: phony fallback
>
> * Add `mymod.mod.proxy` as an *implicit dependency* of the
> `build consumer.o` statement.
>
> The phony `mymod.mod.proxy` implicit build statement is needed
> because the `scan` tool reading sources that consume `mymod`
> does not know where the `mymod.mod` file will be located on
> disk or even if it is provided by another source in the build:
>
> * Compilations consuming `mymod.mod` get an implicit dependency
> on `mymod.mod.proxy` for which an implicit build statement
> is always provided.
>
> * The compilation providing `mymod.mod`, if any, knows where
> the file will be located on disk. The path is added as
> a dependency of `mymod.mod.proxy` so that the phony target
> will be out of date when `mymod.mod` is modified and the
> consumers will recompile.
>
> * The `fallback` file is created once by the `scan` tool and
> never updated. It satisfies proxy dependencies for external
> (system) modules not provided by another source file in the
> build. In such cases no modification to sources within the
> build would cause the proxy to become out of date.
>
> Ninja Features Needed
> ---------------------
>
> The above approach requires Ninja to be taught some new features.
> In particular:
>
> The `depgen` file, through some syntax, will specify updates
> to existing build nodes and/or add new build nodes:
>
> * Add a new build statement (at least phony rules).
> * Add implicit dependencies to an existing build statement.
>
> We have not yet designed a specific syntax for the `depgen` file
> and are open to suggestions. It should be a format that is easy
> to generate, fast to parse, and extensible beyond the above
> operations in the future.
>
--
Peter
May I suggest a simpler alternative proposal? It seems to satisfy the
requirements for Fortran, if I understand them correctly.
The proposal is: add a flag to build rules that indicates that if the process
exits with a specific exit code, Ninja should re-run the generator. CMake
will scan all Fortran files in the project at generation time and generate
have changed since CMake scanned the files, and if so, exit with this exit
code, causing the rules to be re-generated.
You could probably optimize this process so that CMake does not need to do
all its regular work if nothing but the Fortran dependencies have changed. For
example, CMake could maintain the Fortran dependencies in a separate include
file, and update only that file if only the Fortran dependencies have changed.
Assuming that the dependencies don't change very often, this seems reasonable
to me.
Thanks,
Peter
> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> Ninja and Fortran
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> =================
>
> Ninja has excellent support for implicit dependencies of languages that
> use the `C` preprocessor. Fortran adds another level of complexity for
> build systems because its "module" system adds implicit *outputs* to
> compilation nodes that are referenced as implicit dependencies of
> other compilation nodes. These implicit outputs and dependencies may
> be discovered only during the build process when all (possibly generated)
> files are available for a given source file to be preprocessed.
>
> Below we introduce Fortran modules and their build system requirements.
> Then we propose new Ninja features to support them.
>
> Fortran Modules
> ---------------
>
> The Fortran module system is defined in section 11.2 of the
> [Fortran 2008 Standard][ISO/IEC 1539-1:2010], also available as a
> [Draft PDF][].
>
> [ISO/IEC 1539-1:2010]: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=50459
> [Draft PDF]: http://www.j3-fortran.org/doc/year/10/10-007.pdf
>
> Each Fortran source file may provide zero or more modules:
>
> $ cat provider.F90
> MODULE mymod
> ...
> END
>
> and also consume zero or more modules:
>
> $ cat consumer.F90
> USE MODULE mymod
> ...
>
> Compilation of a source file consuming a module requires the `.mod`
> file to be present:
>
> $ f95 -c consumer.F90
> ...
> Fatal Error: Can't open module file 'mymod.mod' for reading...
>
> Compilation of a source file providing a module generates a `.mod`
> file for it:
>
> $ f95 -c provider.F90
> $ ls *.o *.mod
> provider.o
> mymod.mod
>
> Once the module has been provided then it can be used:
>
> $ f95 -c consumer.F90
> $ ls *.o *.mod
> consumer.o
> provider.o
> mymod.mod
>
> If the source providing a module is modified and recompiled, the
> module file *may or may not* be updated, depending on whether the
> interface exposed in the module changes.
>
> In general modules used by a source file may be found anywhere
> in a search path (e.g. `-I` paths). They may be provided by
> another source in the same build or may be found in external
> directories existing on the system. The only restriction on
> module dependencies is that they are acyclic.
>
> Manual Module Dependencies
> --------------------------
>
> The above example source files may be built correctly by Ninja
> using manual specification of the module dependencies. We'll
> ignore preprocessor implicit dependencies for now:
>
> $ cat build.ninja
> rule FC
> command = f95 -o $OBJ -c $in
> restat = $RESTAT
>
> build provider.o mymod.mod: FC provider.F90
> OBJ = provider.o
> RESTAT = 1
>
> build consumer.o: FC consumer.F90 | mymod.mod
> OBJ = consumer.o
>
> We use the `OBJ` variable in `FC` because `$out` would
> contain the module as well as the object file. We use `restat`
> for the provider because `mymod.mod` may not be updated by the
> compiler. We can see that the above works in a test session:
>
> $ ninja -v consumer.o
> [1/2] f95 -o provider.o -c provider.F90
> [2/2] f95 -o consumer.o -c consumer.F90
>
> $ ninja -v consumer.o
> ninja: no work to do.
>
> $ touch provider.F90
> $ ninja -v consumer.o
> [1/2] f95 -o provider.o -c provider.F90
>
> $ ninja -v consumer.o
> ninja: no work to do.
>
> This example serves as a conceptual model for the basic dependency
> structure Fortran needs. In practice Fortran source files provide
> and consume many modules so maintaining the dependencies by hand
> is impractical. The build system must be able to discover these
> implicit outputs and implicit dependencies automatically.
>
> Automatic Module Dependencies
> -----------------------------
>
> In order to scan for modules provided or consumed by a source file,
> it must first be preprocessed. Therefore we split compilation into
> three steps:
>
> 1. A `FP` rule to preprocess a source file:
>
> rule FP
> command = f95 -o $out -E $in -MMD -MT $out -MF $out.d
> deps = gcc
> depfile = $out.d
>
> This rule uses normal `depfile` for its implicit preprocessing
> dependencies. Preprocessing does not require the `.mod` files
> to exist.
>
> 2. A `FD` rule to scan for module dependency information and tell
> Ninja to load it using a `depgen` feature proposed below.
>
> rule FD
> command = scan $in $OBJ $out
> depgen = $out
>
> The user-provided `scan` tool will read the preprocessed source
> from `$in`, detect the implicit outputs and dependencies of the
> object file named by `$OBJ`, and write the build plan updates
> to the `$out` file also specified as the `depgen` file. The
> `depgen` file will tell Ninja, through some syntax, how to
> update its build plan. It will inject implicit outputs and
> dependencies into the compilation node for `$OBJ`.
>
> 3. A `FC` rule will compile the already-preprocessed source to
> produce an object file and possibly some `.mod` files.
>
> rule FC
> command = f95 -o $out -c $in
>
> Build statements using this rule can have an implicit or
> order-only dependency on the corresponding `FD` rule output
> to make sure implicit outputs and dependencies are known.
>
> With these rules we may use the following build statements for
> our above example:
>
> build provider.pp.f90: FP provider.F90
> build provider.pp.f90.g: FD provider.pp.f90
> OBJ = provider.o
> build provider.o: FC provider.pp.f90 || provider.pp.f90.g
>
> build consumer.pp.f90: FP consumer.F90
> build consumer.pp.f90.g: FD consumer.pp.f90
> OBJ = consumer.o
> build consumer.o: FC consumer.pp.f90 || consumer.pp.f90.g
>
> The `provider.pp.f90.g` file will:
>
> * Add `mymod.mod` as an *implicit output* of the
> `build provider.o` statement.
>
> * Mark the `build provider.o` node with the `restat` option
> so that consumers do not rebuild after `provider.o` if the
> `mymod.mod` implicit output does not change.
>
> * Add or update an *implicit build statement* of the form:
>
> build mymod.mod.proxy: phony mymod.mod
>
> The `consumer.pp.f90.g` file will:
>
> * Add or update an *implicit build statement* of the form:
>
> build mymod.mod.proxy: phony fallback
>
> * Add `mymod.mod.proxy` as an *implicit dependency* of the
> `build consumer.o` statement.
>
> The phony `mymod.mod.proxy` implicit build statement is needed
> because the `scan` tool reading sources that consume `mymod`
> does not know where the `mymod.mod` file will be located on
> disk or even if it is provided by another source in the build:
>
> * Compilations consuming `mymod.mod` get an implicit dependency
> on `mymod.mod.proxy` for which an implicit build statement
> is always provided.
>
> * The compilation providing `mymod.mod`, if any, knows where
> the file will be located on disk. The path is added as
> a dependency of `mymod.mod.proxy` so that the phony target
> will be out of date when `mymod.mod` is modified and the
> consumers will recompile.
>
> * The `fallback` file is created once by the `scan` tool and
> never updated. It satisfies proxy dependencies for external
> (system) modules not provided by another source file in the
> build. In such cases no modification to sources within the
> build would cause the proxy to become out of date.
>
> Ninja Features Needed
> ---------------------
>
> The above approach requires Ninja to be taught some new features.
> In particular:
>
> 1. Add a notion of *implicit outputs* on build plan nodes that
> are not part of `$out` but that Ninja knows are generated.
>
> 2. Teach Ninja to recognize the `depgen` option on a rule and
> 1. Add a notion of *implicit outputs* on build plan nodes that
> are not part of `$out` but that Ninja knows are generated.
>
> load the named file immediately after an instance of the rule
> is brought up to date (whether or not the instance runs).
>
> The `depgen` file, through some syntax, will specify updates
> to existing build nodes and/or add new build nodes:
>
> * Add a new build statement (at least phony rules).
> * Add implicit dependencies to an existing build statement.
> * Add implicit outputs to an existing build statement.
> * Enable `restat` on an existing build statement.
>
> We have not yet designed a specific syntax for the `depgen` file
> and are open to suggestions. It should be a format that is easy
> to generate, fast to parse, and extensible beyond the above
> operations in the future.
>
--
Peter
Nils Gladitz
Apr 13, 2015, 9:59:45 AM4/13/15
to Evan Martin, Brad King, ninja-build, Bill Hoffman, Ben Boeckel
On 04/01/2015 12:09 AM, Evan Martin wrote:
> I uploaded the document to github here for their nice formatting:
> https://gist.github.com/anonymous/439f8f367ba088c10a6a
I don't savvy fortran myself but I am eagerly looking forward to being
> I uploaded the document to github here for their nice formatting:
> https://gist.github.com/anonymous/439f8f367ba088c10a6a
able to build a larger C++ project with fortran components with ninja.
What is the outlook here? The thread seems to have gone suspiciously quiet.
Is this being discussed somewhere else?
Has the proposal itself been accepted?
Has someone already started implementing this?
Nils
Nico Weber
Apr 13, 2015, 10:01:40 AM4/13/15
to Nils Gladitz, Evan Martin, Brad King, ninja-build, Bill Hoffman, Ben Boeckel
Hi Nis,
I replied off-thread that I think that being able to build Fortran files with ninja is a good goal. Sadly, I haven't had time to look at this yet. (Looking at my check-ins, I'm currently doing work I thought I'd be doing 3 weeks ago, so I'll hopefully get to this soon.)
Nico
Nils Gladitz
Apr 14, 2015, 3:05:41 AM4/14/15
to Nico Weber, Evan Martin, Brad King, ninja-build, Bill Hoffman, Ben Boeckel
On 04/13/2015 04:01 PM, Nico Weber wrote:
> I replied off-thread that I think that being able to build Fortran files
> with ninja is a good goal. Sadly, I haven't had time to look at this
> yet. (Looking at my check-ins, I'm currently doing work I thought I'd be
> doing 3 weeks ago, so I'll hopefully get to this soon.)
Thanks, I'll get my hopes up then! :)
> I replied off-thread that I think that being able to build Fortran files
> with ninja is a good goal. Sadly, I haven't had time to look at this
> yet. (Looking at my check-ins, I'm currently doing work I thought I'd be
> doing 3 weeks ago, so I'll hopefully get to this soon.)
Nils
Brad King
Dec 11, 2015, 11:50:17 AM12/11/15
to ninja-build, Nils Gladitz, Nico Weber, Bill Hoffman, Ben Boeckel
Hi Folks,
On 04/14/2015 03:05 AM, Nils Gladitz wrote:
> On 04/13/2015 04:01 PM, Nico Weber wrote:
>> I think that being able to build Fortran files with ninja
>> is a good goal.
>
https://github.com/bradking/ninja
The changes include implementation, documentation, and tests
for the proposed approach. The details differ a bit from the
design document I posted previously, but the overall approach
is still the same: we dynamically update the build graph using
information discovered during the build, as needed to support
Fortran module dependencies.
I've also filed some PRs with preliminary changes:
* De-duplicate disk abstraction layers:
https://github.com/ninja-build/ninja/pull/1060
* Avoid duplicate EdgeScheduled call:
https://github.com/ninja-build/ninja/pull/1059
* Implicit outputs:
https://github.com/ninja-build/ninja/pull/989
As PRs are merged I will rebase and submit more commits from the
above-mentioned branch as new PRs to have the overall change
reviewed incrementally in manageable chunks.
For those interested in CMake+Ninja+Fortran, I've also published
a work-in-progress 'ninja-fortran' branch of CMake here:
https://github.com/bradking/CMake
Note that I may force-push to any of these branches as revisions
are made.
-Brad
On 04/14/2015 03:05 AM, Nils Gladitz wrote:
> On 04/13/2015 04:01 PM, Nico Weber wrote:
>> I think that being able to build Fortran files with ninja
>> is a good goal.
>
> Thanks, I'll get my hopes up then! :)
I've published a 'features-for-fortran' branch on my fork:
https://github.com/bradking/ninja
The changes include implementation, documentation, and tests
for the proposed approach. The details differ a bit from the
design document I posted previously, but the overall approach
is still the same: we dynamically update the build graph using
information discovered during the build, as needed to support
Fortran module dependencies.
I've also filed some PRs with preliminary changes:
* De-duplicate disk abstraction layers:
https://github.com/ninja-build/ninja/pull/1060
* Avoid duplicate EdgeScheduled call:
https://github.com/ninja-build/ninja/pull/1059
* Implicit outputs:
https://github.com/ninja-build/ninja/pull/989
As PRs are merged I will rebase and submit more commits from the
above-mentioned branch as new PRs to have the overall change
reviewed incrementally in manageable chunks.
For those interested in CMake+Ninja+Fortran, I've also published
a work-in-progress 'ninja-fortran' branch of CMake here:
https://github.com/bradking/CMake
Note that I may force-push to any of these branches as revisions
are made.
-Brad
Nico Weber
Dec 11, 2015, 1:07:54 PM12/11/15
to Brad King, ninja-build, Nils Gladitz, Bill Hoffman, Ben Boeckel
Brad and I chatted off-list a bit today. We hope to get this merged over the next few weeks. I'll set aside some dedicated time for reviewing each Monday. Comments from everyone else are welcome too, of course :-)
For people who are curious, Brad's branch has a diff to the manual explaining the new features: https://github.com/ninja-build/ninja/compare/master...bradking:features-for-fortran?name=features-for-fortran#diff-29236bdc5ce1bf0b4c795c2f263fcd1f (It might be possible to simplify some of how the "these deps need to be applied on this build, not the next" bit is expressed; we'll discuss this on pull requests.)
This does seem like a non-trivial addition to ninja, but so far nobody could think of a simpler addition that still makes it possible to build Fortran files. And the approach Brad has chosen does have the nice property that no work is done on empty builds. The "dynamic" depfiles can only add edges between existing nodes from the manifest -- no work is added to the build graph, only more constraints between existing nodes get added.
(This feature might even be useful for regular C++ builds, when considering generated header files: At the moment, dependencies on rules that generate a header need to be threaded through explicitly with a stamp file. With an "add edges from a depfile directly" and gcc's -MG flag, a generator could write ninja files that first get the depfiles for all cc files with -MG (without actually compiling) and then reads those as "dynamic" depfiles. That would add an edge between the .h file generator and the actual compilation would know to only start after the header file generation has completed.)
Fredrik Medley
Dec 16, 2015, 2:59:58 AM12/16/15
to ninja-build, brad...@kitware.com, nilsg...@gmail.com, bill.h...@kitware.com, ben.b...@kitware.com
Impressive feature!
Den fredag 11 december 2015 kl. 19:07:54 UTC+1 skrev Nico Weber:
Brad and I chatted off-list a bit today. We hope to get this merged over the next few weeks. I'll set aside some dedicated time for reviewing each Monday. Comments from everyone else are welcome too, of course :-)For people who are curious, Brad's branch has a diff to the manual explaining the new features: https://github.com/ninja-build/ninja/compare/master...bradking:features-for-fortran?name=features-for-fortran#diff-29236bdc5ce1bf0b4c795c2f263fcd1f (It might be possible to simplify some of how the "these deps need to be applied on this build, not the next" bit is expressed; we'll discuss this on pull requests.)
Why is there a restriction in depending on only one dyndep file per build rule? (Just noting that there is no restriction on the producer side.)
Can the dyndep loading be moved to the rule producing the file instead of all the rules consuming it?
This does seem like a non-trivial addition to ninja, but so far nobody could think of a simpler addition that still makes it possible to build Fortran files. And the approach Brad has chosen does have the nice property that no work is done on empty builds. The "dynamic" depfiles can only add edges between existing nodes from the manifest -- no work is added to the build graph, only more constraints between existing nodes get added.
Is there a potential future possibility to dynamically load .ninja files? This will lead to all targets not existing from the start, but it could add rules to e.g. running checks on each file extracted from a .tar.
(This feature might even be useful for regular C++ builds, when considering generated header files: At the moment, dependencies on rules that generate a header need to be threaded through explicitly with a stamp file. With an "add edges from a depfile directly" and gcc's -MG flag, a generator could write ninja files that first get the depfiles for all cc files with -MG (without actually compiling) and then reads those as "dynamic" depfiles. That would add an edge between the .h file generator and the actual compilation would know to only start after the header file generation has completed.)
For generated headers C/C++, 'gcc -MG' will only work if the generated headers do not include other generated headers or should 'gcc -MG' be run on the generated headers individually as well?
/Fredrik
Brad King
Dec 17, 2015, 9:38:36 AM12/17/15
to Fredrik Medley, ninja-build
On 12/16/2015 02:59 AM, Fredrik Medley wrote:
> Impressive feature!
Thanks!
> Why is there a restriction in depending on only one dyndep file per build rule?
It may be possible to allow multiple inputs to be listed in the dyndep binding
but for simplicity we kept it to just one for now. One should be sufficient
in practice.
> (Just noting that there is no restriction on the producer side.)
The producing side is just a normal build statement.
> Can the dyndep loading be moved to the rule producing the file instead of all
> the rules consuming it?
Actually the implementation does load the dyndep file immediately after the
build statement producing it finishes. However, the consuming build statements
need to know ahead of time that their dependency information may not yet be
complete. This mark must be associated with the build statement (Edge) so
the best place to put it is as a binding on the build statement. It is also
useful for a human reader to understand that not all dependencies and outputs
may be listed in the build manifest when a dyndep binding is present.
> Is there a potential future possibility to dynamically load .ninja files?
> This will lead to all targets not existing from the start, but it could add
> rules to e.g. running checks on each file extracted from a .tar.
A simplifying property of the dyndep feature design is that it does not add
new build statements, only new inputs/outputs to existing build statements.
Loading entire .ninja files would be much more involved and is beyond the
scope of the dyndep work so I haven't thought much about it.
-Brad
> Impressive feature!
Thanks!
> Why is there a restriction in depending on only one dyndep file per build rule?
but for simplicity we kept it to just one for now. One should be sufficient
in practice.
> (Just noting that there is no restriction on the producer side.)
> Can the dyndep loading be moved to the rule producing the file instead of all
> the rules consuming it?
build statement producing it finishes. However, the consuming build statements
need to know ahead of time that their dependency information may not yet be
complete. This mark must be associated with the build statement (Edge) so
the best place to put it is as a binding on the build statement. It is also
useful for a human reader to understand that not all dependencies and outputs
may be listed in the build manifest when a dyndep binding is present.
> Is there a potential future possibility to dynamically load .ninja files?
> This will lead to all targets not existing from the start, but it could add
> rules to e.g. running checks on each file extracted from a .tar.
new build statements, only new inputs/outputs to existing build statements.
Loading entire .ninja files would be much more involved and is beyond the
scope of the dyndep work so I haven't thought much about it.
-Brad
Brad King
Nov 15, 2016, 2:49:53 PM11/15/16
to ninja-build
On 12/11/2015 11:50 AM, Brad King wrote:
> I've published a 'features-for-fortran' branch on my fork:
I've moved this branch to a new location:
> I've published a 'features-for-fortran' branch on my fork:
* https://github.com/Kitware/ninja/tree/features-for-fortran#readme
and rebased it on `master` after the 1.7.2 release. The branch adds
the needed features and updates the version number to indicate that
it is a non-upstream version with specific additional features.
> For those interested in CMake+Ninja+Fortran, I've also published
> a work-in-progress 'ninja-fortran' branch of CMake here:
* https://cmake.org/cmake/help/v3.7/generator/Ninja.html#fortran-support
Support for CMake+Ninja+Fortran is activated automatically by the
CMake Ninja generator when it detects that the available "ninja"
tool is of a version like that in the branch above.
-Brad
Reply all
Reply to author
Forward
0 new messages