Fortran failing example

[Peter Koval]

https://www.dropbox.com/s/q78qjwseldjg48h/ex3-meson.tar.bz2

Hello Jussi,

The example under the link is a simplification of my failing case.
While first generating a meson project it works (builds, compiles and runs correctly). However, if you add an (unused) argument to the subroutine report_precision in the module m1 and try to recompile in a usual way
ninja -v
, then only the module m1 gets recompiled, but not module m2. This leads to a segmentation fault while running the program. I could not simplify the example further and still get segfaulting.

An example of modification of the subroutine report_precision is given by the subr report_precision_3 which also locates in the module m1.

The version of gfortran I have is 4.5...

Best regards, Peter

Send from phone. Please excuse my brevity.

[Jussi Pakkanen]

On Fri, Aug 15, 2014 at 10:16 PM, Peter Koval <koval...@gmail.com> wrote:

 

The example under the link is a simplification of my failing case.
While first generating a meson project it works (builds, compiles and runs correctly). However, if you add an (unused) argument to the subroutine report_precision in the module m1 and try to recompile in a usual way
ninja -v
, then only the module m1 gets recompiled, but not module m2. This leads to a segmentation fault while running the program. I could not simplify the example further and still get segfaulting.

I looked into this and apparently we have triggered some issues in Meson's dependencies.

First of all there's a deficiency in Ninja that is reported here:

https://groups.google.com/forum/#!topic/ninja-build/j-2RfBIOd_8

And then there's a GCC bug which generates invalid dependency files. That one is reported here:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=62162

I don't want to add workarounds for Meson unless necessary, so I'd like to see what the upstream responses are to these bugs before deciding what to do about them.

[Peter Koval]

Hopefully it is ninja that has to be corrected.

Petr

Send from phone. Please excuse my brevity.

[Jussi Pakkanen]

On Sun, Aug 17, 2014 at 3:28 AM, Peter Koval <koval...@gmail.com> wrote:

Hopefully it is ninja that has to be corrected.

I changed the way dependency tracking works in Meson. Due to the specified bugs and the fact that Debian's Ninja package is quite old I have to do all this manually. The failing example you had now recompiles correctly if you change m1.

However there is an issue that can not be easily fixed given the way Ninja works. It assumes that the dependency grid is somewhat static whereas in Fortran that is not the case (editing one file can add arbitrary nodes to the build graph). I have not been able to think of a way to do this both reliably and efficiently. However there is a simple workaround. If you set your build command to this:

touch meson-private/coredata.dat && ninja

You are guaranteed to always have a reliable build, but the compilation will take a bit longer.

If your edits do not create new modules or new uses of them, you can just run "ninja".

[Peter Koval]

Dear Jussi,

I confirm that the problem is resolved. Namely, if I change the
arguments of a subroutine from a given module, then the all modules
which depend on this module will get rebuild. From the other hand, if
I just add a comment in the given module (which does not change
anything for object file), then only this module will get rebuild, but
not the other modules and project will get correctly linked. Thank
you! This is basically the effect I was looking for by using meson
instead of make utility.

I have a few comments though:
1. In the current version, there are a lot of 'echo' targets built.
Maybe they can be reduced unless they are not remnants of some
debugging process?

2. In present version the dependencies are caugth better than by using
make utility (as I mentioned already). However, I think the number of
rebuilds can be cut down further. Imagine a situation when there are
modules m1, m2, m3 and a subroutine s1 locates in the module m1 and it
is called in the module m2, but not called in the module m3, although
the module m1 us USEd in both modules m2 and m3. In this situation, if
the interface of the subroutine s1 got changes, then only module m2 is
necessary to rebuild, but module m3 may be left in peace. However (as
far as I understood) in the present version all three modules will get
rebuild. I will have to confirm that i) any rebuild of the module m3
is indeed not required and ii) in present version all three modules
will get rebuild.
What do you think?

3. In previous version of meson by default a debugging version of the
project has been generated, while in the present version of meson any
optimization flags got removed by default and the compiler's defaults
are -O2 option which is much slower than -O0 option which was
explicitly present in the previous version of meson (if I recall
correctly).

4. May I suggest for you to consider some features for build.meson
script by using which one would be able to specify enviroment
variables from inside of the build.meson script. This feature is
present in the Makefiles and in many other build systems. A way of
specifying enviroment variables from inside the build script would
save an additional scripting level for me.

5. If item 4 is not impossible then the include feature would make my
work with meson completely similar to what I do with Makefiles.
Namely, I could then to separate machine/compiler dependent
information and project-related information into different files.

Best regards,

Peter

--
Dr. Peter Koval
inet: http://sites.google.com/site/kovalpeter/
phone: +34 943 01 8770

[Jussi Pakkanen]

On Thu, Aug 21, 2014 at 2:38 AM, Peter Koval <koval...@gmail.com> wrote:

 

I confirm that the problem is resolved. Namely, if I change the
arguments of a subroutine from a given module, then the all modules
which depend on this module will get rebuild. From the other hand, if
I just add a comment in the given module (which does not change
anything for object file), then only this module will get rebuild, but
not the other modules and project will get correctly linked. Thank
you! This is basically the effect I was looking for by using meson
instead of make utility.

Good to hear. This behaviour was also my goal so good to know it works.

 

I have a few comments though:
1. In the current version, there are a lot of 'echo' targets built.
Maybe they can be reduced unless they are not remnants of some
debugging process?

These are to work around Ninja's deficiency to have multiple outputs with deps. What happens is that Meson creates a dummy rule that "creates" a .mod file from the .o file. In reality this is a no-op (the echo command you see) but it fools Ninja into doing the right thing.

These will get removed once this issue is fixed in Ninja. For the moment they have to be there.

 

2. In present version the dependencies are caugth better than by using
make utility (as I mentioned already). However, I think the number of
rebuilds can be cut down further. Imagine a situation when there are
modules m1, m2, m3 and a subroutine s1 locates in the module m1 and it
is called in the module m2, but not called in the module m3, although
the module m1 us USEd in both modules m2 and m3. In this situation, if
the interface of the subroutine s1 got changes, then only module m2 is
necessary to rebuild, but module m3 may be left in peace. However (as
far as I understood) in the present version all three modules will get
rebuild. I will have to confirm that i) any rebuild of the module m3
is indeed not required and ii) in present version all three modules
will get rebuild.
  What do you think?

This is possible but not simple. This behaviour for Fortran is the same as other languages such as C, where changing a header file recompiles all sources that include it regardless of whether they use the changed bits. You could do finer grained tracking but that would be much more complicated and the speed gains you get from skipping a few compiles every now and then would probably be lost by the slowdowns caused by stricter dependency parsing, maintaining the dependency database and so on.

 

3. In previous version of meson by default a debugging version of the
project has been generated, while in the present version of meson any
optimization flags got removed by default and the compiler's defaults
are -O2 option which is much slower than -O0 option which was
explicitly present in the previous version of meson (if I recall
correctly).

I don't think Meson ever put -O0 in by default. I could add -O0 for compilers that default to -O2 but that presents a problem. If the user specifies, say, -O1 with external build flags, then the compilation line would have both -O0 and -O1 in an unpredictable order.

There's not really a good works-for-everyone solution here. However it is simple to make Meson work for you. When setting up a debug non-optimized build directory for a compiler that defaults to -O2, you can do this:

FFLAGS=-O0 meson <build dir>

And it will do what you want. You can also set this with meson_introspect afterwards. It's not fully automatic but at least it works.

 

4. May I suggest for you to consider some features for build.meson
script by using which one would be able to specify enviroment
variables from inside of the build.meson script. This feature is
present in the Makefiles and in many other build systems. A way of
specifying enviroment variables from inside the build script would
save an additional scripting level for me.

What are you trying to accomplish with these envvars? Note that you can already set envvars for unit tests. You should not need to set env vars during builds unless some crazy compiler requires options to be passed in via envvars.

 

5. If item 4 is not impossible then the include feature would make my
work with meson completely similar to what I do with Makefiles.
Namely, I could then to separate machine/compiler dependent
information and project-related information into different files.

Including files in some smarter way than currently is in the works but a clear design has not yet come to mind. You can work around this in the following way, though.

Put your configs in separate directories like this:

conf1/meson.build

conf2/meson.build

Then process them like this:

if something

  subdir('conf1')

else

  subdir('conf2')

endif

 

[Peter Koval]

Dear Jussi,

sorry for the delay. I could not concentrate before. 

I am trying to make with meson a build similar to Makefiles I am using now. Namely, I generate makefiles for several executables, dependencies are generated automatically as well. Only what I cannot generate automatically is so called architecture file in which the information about machine and libraries is concentrated. If there is a way to specify this information in a short convenient form explicitly and separately from the logics of the program itself (which modules depend on which other modules, the executables to build, etc) this would help me a lot. I also think this would be a prerequisite to use your build system on clusters (because normally it is more difficult to figure the working/optimal configuration of libraries/compilers on clusters)

I attach this file and other automatically generated files for your information.

Best regards,

Peter

 

[Jussi Pakkanen]

On Wed, Sep 3, 2014 at 12:17 PM, Peter Koval <koval...@gmail.com> wrote:

 

I am trying to make with meson a build similar to Makefiles I am using now. Namely, I generate makefiles for several executables, dependencies are generated automatically as well. Only what I cannot generate automatically is so called architecture file in which the information about machine and libraries is concentrated. If there is a way to specify this information in a short convenient form explicitly and separately from the logics of the program itself (which modules depend on which other modules, the executables to build, etc) this would help me a lot. I also think this would be a prerequisite to use your build system on clusters (because normally it is more difficult to figure the working/optimal configuration of libraries/compilers on clusters)

If I interpret your makefiles correctly, a way to achieve similar results is to create a script to initialise Meson with suitable arguments for each machine. An example script could look something like this:

-----

#!/bin/sh

export FFLAGS=<set your Fortran flags here>

export CFLAGS=<set your C flags here>

MESONOPTIONS=-Dyour_meson_options_here --buildtype=debugoptimized

BUILDDIR=build-cluster1

mkdir $BUILDDIR

meson ${MESONOPTIONS} ${BUILDDIR}

----

To use it put it in your source root and run ./yourscript.sh. This makes Meson grab all the options you specify and use them. Once you have done this, just cd into the build directory and run Ninja. Meson stores the flags and settings you give it on initialisation. They will be used for all subsequent builds. All you have to do is to run Ninja.

[Peter Koval]

Hello Jussi,

the way you propose would resolve the task, also I perceive this as unsatisfactory because of too much scripting levels. I order to build with meson you would need to be aware of bash script, meson script, python script which generates list of source files (in my case) and ninja script. This proliferation makes bad feeling, but I can try that. Next I will experiment with intel fortran as I promised.

Best regards,

Peter

 

[Jussi Pakkanen]

On Wed, Sep 3, 2014 at 5:59 PM, Peter Koval <koval...@gmail.com> wrote:

 

the way you propose would resolve the task, also I perceive this as unsatisfactory because of too much scripting levels. I order to build with meson you would need to be aware of bash script, meson script, python script which generates list of source files (in my case) and ninja script. This proliferation makes bad feeling, but I can try that. Next I will experiment with intel fortran as I promised.

Depending on your actual requirements, some (or all) of these can be done inside the Meson script, too. As an example if you always know you want to use certain compiler arguments for certain compilers, you can do something like this:

fc = meson.get_compiler('fortran')

if fc.get_id() == 'intel'

  add_global_arguments('-some-intel-flag', language : 'fortran')

elif fc.get_id() == 'gcc

  add_global_arguments('-some-gcc-flag', language : 'fortran')

elif ...

endif

Or if you want to set compiler arguments per target you'd do something like this:

if fc.get_id() == 'intel'

  extra_args = ['-some-intel']

elif fc.get_id() == 'gcc

  extra_args = ['-some-gcc']

else

  extra_args = []

endif

executable('prog', ..., fortran_args : extra_args)

Which one of these three methods you should use depends on how much and what kind of variation in compiler flags you need.

See the wiki for details:

https://github.com/jpakkane/meson/wiki/Adding%20arguments

https://github.com/jpakkane/meson/wiki/Compiler%20properties

There are also project options which might be exactly what you want:

https://github.com/jpakkane/meson/wiki/Build%20options

[Peter Koval]

Dear Jussi,

on my desktop I met following error with g95 compiler

g95: unrecognized option '-Jxyzedit.dir'

(after commanding ninja -v )

g95 --version
G95 (GCC 4.0.3 (g95 0.92!) Jun 24 2009)
Copyright (C) 2002-2008 Free Software Foundation, Inc.

G95 comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of G95
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING


ninja --version
1.5.1

Do you think it is difficult to correct?

Peter

[Jussi Pakkanen]

On Wed, Sep 3, 2014 at 7:13 PM, Peter Koval <koval...@gmail.com> wrote:

 

on my desktop I met following error with g95 compiler

g95: unrecognized option '-Jxyzedit.dir'

That flag comes from gfortran:

-Jdir

           This option specifies where to put .mod files for compiled modules.  It is also added to the list of directories to searched by an

           "USE" statement. 

To fix this I need to know what is the equivalent command line argument(s) for g95.

[Peter Koval]

Dear Jussi,

I think the option for g95 would be -fmod=directory

By default the .mod files appear at the same place where .o files. In turn, .o files appear by default in the current working directory. Maybe you can change meson in such a way that -J options will be avoided?

Best regards,

Peter

[Peter Koval]

Hello Jussi,

I almost got ifort working. There are two things that I could not do

1. Standard argument for all warnings would be -warn all for ifort, but I could not redefine it successfully in the derived class IntelFortranCompiler(FortranCompiler)

2. The option to put modules for ifort is -module path. However, if this option appears in quotes '-module path', then ifort could not recognize it

Basically two warnings plague my current solution
ifort: command line warning #10157: ignoring option '-W'; argument is of wrong type
ifort: command line warning #10159: invalid argument for option '-m'

I think it would be better to remove all the quotes from the commands

i.e. instead of the command

ifort '-O2' '-Wall' '-O2' '-g' '-Idp_import_chk.dir' '-module dp_import_chk.dir'  -o dp_import_chk.dir/m_siesta_wfsx.F90.o -c ../m_siesta_wfsx.F90

it is better to generate

ifort -O2 -Wall -g -Idp_import_chk.dir -module dp_import_chk.dir  -o dp_import_chk.dir/m_siesta_wfsx.F90.o -c ../m_siesta_wfsx.F90

Even better would be to change directory before build to dp_import_chk.dir and invoke

ifort -O2 -warn all -g -o m_siesta_wfsx.F90.o -c ../../m_siesta_wfsx.F90

Best regards,

Peter

[Jussi Pakkanen]

On Thu, Sep 4, 2014 at 11:51 AM, Peter Koval <koval...@gmail.com> wrote:

 

I almost got ifort working. There are two things that I could not do

FortranCompiler)

2. The option to put modules for ifort is -module path. However, if this option appears in quotes '-module path', then ifort could not recognize it

Meson deals with command arguments as arrays. One entry is one item. This may seem a bit strict for people used to Make lax command lines but unfortunately the only reliable way to make command invocation reliable on all platforms.

The fix is very simple. You just need to return an array with two entries. So instead of this:

return '-module ' + path

you should have

return ['-module', path]

then it will be properly handled. If it doesn't then that is a bug and I'll fix it.

Similarly you have this:

std_warn_args = ['']

What this means is "one argument that is the empty string". If you don't want to add anything, this should be the empty array [].

You also had a gcc_type for the constructor of Intel fortran compiler. You should not put that it. Gcc_type is only meaningful for gfortran. Just remove it.

[Peter Koval]

Hello Jussi,

I tried to return the list as you suggested, but this lead to an error

Traceback (most recent call last):
  File "/home/kovalp/service/meson/meson.py", line 184, in <module>
    app.generate()
  File "/home/kovalp/service/meson/meson.py", line 147, in generate
    g.generate()
  File "/home/kovalp/service/meson/ninjabackend.py", line 131, in generate
    [self.generate_target(t, outfile) for t in self.build.get_targets().values()]
  File "/home/kovalp/service/meson/ninjabackend.py", line 131, in <listcomp>
    [self.generate_target(t, outfile) for t in self.build.get_targets().values()]
  File "/home/kovalp/service/meson/ninjabackend.py", line 252, in generate_target
    obj_list.append(self.generate_single_compile(target, outfile, src, False, [], header_deps))
  File "/home/kovalp/service/meson/ninjabackend.py", line 1145, in generate_single_compile
    element.write(outfile)
  File "/home/kovalp/service/meson/ninjabackend.py", line 105, in write
    newelems.append(templ % ninja_quote(i))
  File "/home/kovalp/service/meson/ninjabackend.py", line 33, in ninja_quote
    return text.replace(' ', '$ ').replace(':', '$:')
AttributeError: 'list' object has no attribute 'replace'

Second issue is that in the sanity_check subroutine it is better to generate sanitycheckf.f90 file, not sanitycheckf.f95 (intel fortran does not recognize .f95 extension)

Best regards,

Peter

[Jussi Pakkanen]

On Fri, Sep 5, 2014 at 10:12 AM, Peter Koval <koval...@gmail.com> wrote:

 

Hello Jussi,

I tried to return the list as you suggested, but this lead to an error

Yeah, I figured one of these bugs would still linger around the codebase. Sorry about that, it's fixed now.

 

[Peter Koval]

Hello,

I succeed to compile and run my project with g95, sunf90, gfortran and ifort compilers on my desktop. I think this is a milestone which is worth to commit. Could you introduce changes to environment.py according to attached file?

Peter

[Jussi Pakkanen]

On Mon, Sep 8, 2014 at 11:27 AM, Peter Koval <koval...@gmail.com> wrote:

 

I succeed to compile and run my project with g95, sunf90, gfortran and ifort compilers on my desktop. I think this is a milestone which is worth to commit. Could you introduce changes to environment.py according to attached file?

Applied. Thanks a lot.

 

[Peter Koval]

Hello Jussi,

thank you for introducing my changes. I have to complain again about rebuild behaviour of my Fortran project.

If I use gfortran or g95 compilers, then a non substantial change of a module leads to recompilation of this particular module and linking of the program. This is the behaviour we expect from meson. However, in case of sunf90 and ifort, the same non substantial change causes ninja to recompile all dependencies as it would do make utility. I admit I still do not understand how your smart detection strategy actually works. Maybe, because I do not understand it, I suggested something wrong for ifort and sun compilers?

Are you willing to debug it? Sunf90 is a free compiler (not open source, but free). Intel compiler you could also try for free. Maybe I can provide my generated project and you will see immediately what is wrong ? Ok, I attach the directory generated by meson.

Best regards,

Peter
 

[Jussi Pakkanen]

On Tue, Sep 9, 2014 at 10:52 AM, Peter Koval <koval...@gmail.com> wrote:

 

If I use gfortran or g95 compilers, then a non substantial change of a module leads to recompilation of this particular module and linking of the program. This is the behaviour we expect from meson. However, in case of sunf90 and ifort, the same non substantial change causes ninja to recompile all dependencies as it would do make utility. I admit I still do not understand how your smart detection strategy actually works. Maybe, because I do not understand it, I suggested something wrong for ifort and sun compilers?

The reason this is happening is because Gnu Fortran is smart and the other two apparently aren't. When Gnu Fortran recompiles a file and generates a mod file it checks if the generated mod file would be identical to the already existing one. If it is, then it does not touch the file at all. This means that the mod file's timestamp is not updated and this tells Ninja that the interface has not changed so it can skip the recompilation steps. Apparently the other two don't do this but rather always write the mod file which triggers recompilation.

If ifort and sunf90 have a compiler flag that would tell them to do the smart mod file generation that Gnu Fortran does, then fixing this is simple. If they don't, then it gets a bit trickier.

[Peter Koval]

I see, timestamps. make utility also uses timestamps... Thus, it must be possible to generate Makefiles instead of ninja scripts... Right?

I will look into options. Maybe there is such flag available.

Peter

[Peter Koval]

Hello Jussi,

in another more involved case, with gfortran compiler I touched a module and it caused a chain recompilation as Makefiles would do. Maybe, in case of gfortran, it depends on size of the module whether .mod file would be regenerated or not.

However, g95 compiler withstands this test. Only with g95 compiler the recompilations are saved as we would expect.

Best regards,

Peter

[Peter Koval]

https://software.intel.com/en-us/forums/topic/270731

The issue seems to be known.

[Jussi Pakkanen]

On Tue, Sep 9, 2014 at 6:40 PM, Peter Koval <koval...@gmail.com> wrote:

https://software.intel.com/en-us/forums/topic/270731

The issue seems to be known.

This is a bit unfortunate, but unless unless a) ifort fixes its behaviour or b) they provide documentation for their mod files so I can write code to work around this issue, there is unfortunately no way to make this work reliably (short of writing a full Fortran parser which is a fair bit of work). Sorry.

If you have an example where gfortran produces different mod files for identical module definitions, then I recommend you file that in gfortran's bug tracker as this seems to be an issue in their code generator.

If sunf90 generates mod files that are identical but only have updated time stamps then that is something I can work around. If the files are different, then the situation is the same as with ifort.

The bright side in all this is that Meson will choose reliability over anything else so your built binaries will be up to date. The downside is that sometimes this will cause some source files to be rebuilt even though it is not strictly required.

I wish I had better news on this, but unfortunately this is the state of things as they currently stand.

[Peter Koval]

Hello Jussi,

I introduced 3 more Fortran compilers into environment.py (PGI, open64 and pathscale). I confirm that this works with my code on two machines. With this change I cover all compilers available to me now. There are abisoft and lahey compilers which are missing, but I do not have access to them currently.

Moreover, I attach a script I am using to get source code lists. Maybe it will of interest for the other fortran users.

I wish to see the corresponding changes in meson in the next commits.

Peter

[Jussi Pakkanen]

On Thu, Sep 18, 2014 at 3:20 PM, Peter Koval <koval...@gmail.com> wrote:

 

I introduced 3 more Fortran compilers into environment.py (PGI, open64 and pathscale). I confirm that this works with my code on two machines. With this change I cover all compilers available to me now. There are abisoft and lahey compilers which are missing, but I do not have access to them currently.

Added to trunk. Thanks.