Multiple logical units pointing to /dev/null.

[Lewis Lloyd]

Hi!

I am currently working on a piece of scientific software. Historically, the Intel fortran compiler has been used for development. Recently it was decided that we would go through the effort of attempting to use other compilers so that we could do performance benchmarks. gfortran was the first alternative compiler targeted. After much gnashing-of-teeth, the software is now able to compile and execute.

This particular piece of software has many optional output files. There are command line arguments that allow the user to name these output files. The user (and developer) community has a habit of using /dev/null as the file name for all of the files that they don't want written for a given simulation. These files can reach 10-100 gigs if the simulation is long enough.

Sadly, gfortran was kind enough to inform us that having multiple logical units connected to a single file (/dev/null in this case) is not allowed by the Fortran standard. The Intel compiler has had no problem allowing us to do this (even with -stand f08). The gfotran compiler appears to be quite adamant about not allowing this.

Luckily, calls to 'open' are wrapped in a subroutine, 'open_file', that does error our handling. I figured that I would just open /dev/null once and have a single logical unit attached to it. If the filename passed to the open_file routine matched '/dev/null', then I would just pass back the logical unit for the already opened /dev/null. However, that idea broke down when the gfortran compiler then informed us that we couldn't write both unformatted and formatted information to the same lu. I don't believe there is a standard conforming way to write both formatted and unformatted information to the same logical unit. So, I scratched that idea and instead the open_file routine will now return an appropriately formatted scratch file logical unit. So, at this point, the software will run.

However, previously, we were not writing to disk (at least that is my understanding of what happens when you attach a logical unit to /dev/null - no disk i/o occurs). Now, we are writing all of the additional files to disk, albeit temporary scratch files. I am concerned about the impact of this new I/O overhead on long simulations. The concern is driven by both performance and disk-space. I have thought about using a system call to make another null device for each file redirected to /dev/null.

Does anyone have any experience-nuggets on how to handle situations like this?
Am I missing an option?
Is writing to /dev/null not as efficient as I had thought and the problem really doesn't exist?

Thanks.

--
Lewis

[steve kargl]

Lewis Lloyd wrote:

> Sadly, gfortran was kind enough to inform us that having multiple
> logical units connected to a single file (/dev/null in this case) is not
> allowed by the Fortran standard. The Intel compiler has had no
> problem allowing us to do this (even with -stand f08). The gfotran
> compiler appears to be quite adamant about not allowing this.

From the Fortran 2008 standard (p.205):

A unit shall not be connected to more than one file at the same time,
and a file shall not be connected to more than one unit at the same
time.

It seems that you'll need to wrap the statements with an if-statement,
where a logical flag is set if /dev/null is seen.

logical :: file1_flag = .false., file2_flag = .false' ! etc.

if (filename1 /= '/dev/null') then
open(unit=###, file=filename1, etc)
file1_flag = .true.
end if
...

if (file1_flag) then
write(unit=###, ...)
end if

--
steve

[Louis Krupp]

Here's an off-the-wall idea:

Ask your system administrator to use mknod (with major type 1 and
minor type 3, I believe) to create some more null devices --
/dev/null1, /dev/null2, etc. Change open_file to open each of those
in turn whenever it's asked to open /dev/null.

I haven't tried this at home.

Louis

[herrman...@gmail.com]

On Monday, May 16, 2016 at 12:10:46 PM UTC-7, Lewis Lloyd wrote:

(snip)

> This particular piece of software has many optional output files.
> There are command line arguments that allow the user to name these output files.
> The user (and developer) community has a habit of using /dev/null as the file
> name for all of the files that they don't want written for a given simulation.
> These files can reach 10-100 gigs if the simulation is long enough.

> Sadly, gfortran was kind enough to inform us that having multiple logical units
> connected to a single file (/dev/null in this case) is not allowed by the Fortran
> standard. The Intel compiler has had no problem allowing us to do this
> (even with -stand f08). The gfotran compiler appears to be quite adamant
> about not allowing this.

Since /dev/null isn't a file, but a device, I would say that rule shouldn't apply.
Even more, it doesn't matter what order records get written to /dev/null, though
it might to other devices.

It is nice of the compiler (actually, run time library) to check, as the result of writing
to a one disk file with two units open at the same time is likely to be bad.

You could try symbolic links from different names to /dev/null, it might be that
it doesn't figure that out.

Though, as for the actual problem, it would be nice to detect writing to the
same file under different names, such as through symbolic links, hard links,
or names that look different but aren't.

dir/x.txt vs yyy/../dir/x.txt, for example.

[herrman...@gmail.com]

On Monday, May 16, 2016 at 1:53:18 PM UTC-7, Louis Krupp wrote:
> On Mon, 16 May 2016 12:10:43 -0700 (PDT), Lewis Lloyd

(snip on writing to /dev/null with two different units at the same time)

> Here's an off-the-wall idea:

> Ask your system administrator to use mknod (with major type 1 and
> minor type 3, I believe) to create some more null devices --
> /dev/null1, /dev/null2, etc. Change open_file to open each of those
> in turn whenever it's asked to open /dev/null.

Unix cp command detects attempts to copy a file to itself, and doesn't
do the copy in that case.

I just tested on my system, and it does detect and not copy with
a symbolic link to /dev/null, but works with a mknod /dev/null2 c 3 2

(Use ls -l to find the appropriate major and minor numbers.)

Even more, cp detects a copy of a hard link to /dev/null!

The library might use the same test as cp.

There should be no security problems, but it depends on how well you
know your system administrator.

Oh, be sure that after they monkey, they chmod 666 so you can write to it.

[Richard Maine]

<herrman...@gmail.com> wrote:

> On Monday, May 16, 2016 at 12:10:46 PM UTC-7, Lewis Lloyd wrote:

> > Sadly, gfortran was kind enough to inform us that having multiple
> > logical units connected to a single file (/dev/null in this case) is not
> > allowed by the Fortran standard. The Intel compiler has had no problem
> > allowing us to do this (even with -stand f08). The gfotran compiler
> > appears to be quite adamant about not allowing this.
>
> Since /dev/null isn't a file, but a device, I would say that rule
> shouldn't apply.

It is a file to Fortran. See the OPEN statement; that's what it does is
connect a unit to a "file". Whatever it might be considered from other
perspectives, it is a file by definition in Fortran. Since the rule in
question is a Fortran rule, that's particularly relevant. Compilers
might or might not enforce it, but I don't think there are any grounds
for claiming that the rule somehow doesn't apply and that compilers
can't enforce it.

> It is nice of the compiler (actually, run time library) to check, as the
> result of writing to a one disk file with two units open at the same time
> is likely to be bad.

Yep. And I've known other compilers to check. I don't recall whether it
was for /dev/null, but certainly for "real" files and possibly also for
devices.

> You could try symbolic links from different names to /dev/null, it might
> be that it doesn't figure that out.

I seem to recall trying tricks like that with symlinks once and finding
that the compiler I was using at th etime wasn't adequately fooled.
Might have been for a "real" file instead of /dev/null; I don't recall
that much detail. Perhaps it checked the inode or some such thing.

But for the particular case of /dev/null on a Unix system, Louis Krupp's
idea seems likely to work. I haven't tried it either, but it seems
likely. I will note that if he is worried about eficiency, there's a lot
to be said for catching things earlier in the code to avoid the write
statements instead of writing to a null file. Formatted writes in
particular can eat a lot of processing power even before hitting the
physical disk.

--
Richard Maine
email: last name at domain . net
dimnain: summer-triangle

[steve kargl]

herrman...@gmail.com wrote:

> On Monday, May 16, 2016 at 12:10:46 PM UTC-7, Lewis Lloyd wrote:
>
> (snip)
>
>> This particular piece of software has many optional output files.
>> There are command line arguments that allow the user to name these output files.
>> The user (and developer) community has a habit of using /dev/null as the file
>> name for all of the files that they don't want written for a given simulation.
>> These files can reach 10-100 gigs if the simulation is long enough.
>
>> Sadly, gfortran was kind enough to inform us that having multiple logical units
>> connected to a single file (/dev/null in this case) is not allowed by the Fortran
>> standard. The Intel compiler has had no problem allowing us to do this
>> (even with -stand f08). The gfotran compiler appears to be quite adamant
>> about not allowing this.
>
> Since /dev/null isn't a file, but a device, I would say that rule shouldn't apply.
> Even more, it doesn't matter what order records get written to /dev/null, though
> it might to other devices.

Well, if you are on a UNIX(tm) or UNIX-like system, then the underlying philosophy
is that "everything is a file".

--
steve

[herrman...@gmail.com]

On Monday, May 16, 2016 at 2:24:17 PM UTC-7, Richard Maine wrote:

(snip, I wrote)

> > It is nice of the compiler (actually, run time library) to check, as the
> > result of writing to a one disk file with two units open at the same time
> > is likely to be bad.

> Yep. And I've known other compilers to check. I don't recall whether it
> was for /dev/null, but certainly for "real" files and possibly also for
> devices.

And as I noted later, cp usually checks to avoid the problem of accidentally losing
a file, copying to itself.

> > You could try symbolic links from different names to /dev/null, it might
> > be that it doesn't figure that out.

> I seem to recall trying tricks like that with symlinks once and finding
> that the compiler I was using at th etime wasn't adequately fooled.
> Might have been for a "real" file instead of /dev/null; I don't recall
> that much detail. Perhaps it checked the inode or some such thing.

Yes, one way is to do a string compare, but that can fail, such as in the
case of hard or symbolic links. And the inode test isn't that hard at the
unix level.

It might still fail with some combinations of remote file systems, such
as two different NFS mount points on the same underlying file system.

> But for the particular case of /dev/null on a Unix system, Louis Krupp's
> idea seems likely to work. I haven't tried it either, but it seems
> likely. I will note that if he is worried about eficiency, there's a lot
> to be said for catching things earlier in the code to avoid the write
> statements instead of writing to a null file. Formatted writes in
> particular can eat a lot of processing power even before hitting the
> physical disk.

Since he already has the ability to change the unit number, it would seem
an easy test to detect open to "/dev/null" and return zero.

Then, on each WRITE, put an IF such that it doesn't do it if the
unit is zero. There might be one, or might be thousands of WRITE statements.

Sometimes it is nice to actually write to /dev/null, though. It should be faster
than a disk, but slower than not executing the WRITE at all.

[herrman...@gmail.com]

On Monday, May 16, 2016 at 2:37:07 PM UTC-7, steve kargl wrote:

(snip, I wrote)

> > Since /dev/null isn't a file, but a device, I would say that rule shouldn't apply.
> > Even more, it doesn't matter what order records get written to /dev/null, though
> > it might to other devices.

> Well, if you are on a UNIX(tm) or UNIX-like system, then the underlying philosophy
> is that "everything is a file".

Some say that, but more say "everything is a file descriptor" or "everything looks like a file".

I tried some more with cp:

cp /dev/null /dev/fd/1 > /dev/null

which seems to work in csh, but not in bash.

/dev/fd/1 is the device name for file descriptor 1. It seems that csh and bash do
the redirection in different ways.

In bash,

cp /dev/null /dev/fd/2 2>/dev/null

also fails in the same way.

Note that unix has nothing against more than one file descriptor open on
the same file, and in fact has a system call, dup(), to make a copy of a file
descriptor that is already open.

Unix has isatty() such that one can test if a file descriptor is open on
a terminal device, and do terminal-like operations on it that wouldn't
work on a file. Looks like a file, but isn't.

Seems that there are a few fixes. One is that Unix could declare this
a bug, and standardize the /dev/null0 /dev/null1 /dev/null2 ... devices.

Another is that the Fortran standard could allow for opening null
units, such as a NULL='TRUE' option on OPEN.

Or gfortran (or its library) could special case /dev/null and allow it.

The last one is likely to be faster to implement.

[Steve Lionel]

On 5/16/2016 3:56 PM, steve kargl wrote:
> Lewis Lloyd wrote:
>
>> > Sadly, gfortran was kind enough to inform us that having multiple
>> > logical units connected to a single file (/dev/null in this case) is not
>> > allowed by the Fortran standard. The Intel compiler has had no
>> > problem allowing us to do this (even with -stand f08). The gfotran
>> > compiler appears to be quite adamant about not allowing this.
> From the Fortran 2008 standard (p.205):
>
> A unit shall not be connected to more than one file at the same time,
> and a file shall not be connected to more than one unit at the same
> time.

Fortran 2015 will relax this restriction. The current draft says:

"If input/output operations are performed on more than one unit while
they are connected to the same external file, the results are processor
dependent."

The rule about a unit not being able to connect to multiple files remains.

I'll comment that for Intel Fortran, -stand looks only at what it can
see at compile-time. Run-time behavior such as this is not checked for
standards conformance.

--
Steve Lionel
Developer Products Division
Intel Corporation
Merrimack, NH

For email address, replace "invalid" with "com"

User communities for Intel Software Development Products
http://software.intel.com/en-us/forums/
Intel Software Development Products Support
http://software.intel.com/sites/support/
My Fortran blog
http://www.intel.com/software/drfortran

Refer to http://software.intel.com/en-us/articles/optimization-notice
for more information regarding performance and optimization choices in
Intel software products.

[Richard Maine]

<herrman...@gmail.com> wrote:

> On Monday, May 16, 2016 at 2:37:07 PM UTC-7, steve kargl wrote:
>
> (snip, I wrote)
>
> > > Since /dev/null isn't a file, but a device, I would say that rule
> > > shouldn't apply. Even more, it doesn't matter what order records get
> > > written to /dev/null, though it might to other devices.
>
> > Well, if you are on a UNIX(tm) or UNIX-like system, then the underlying
> > philosophy is that "everything is a file".
>
> Some say that, but more say "everything is a file descriptor" or
> "everything looks like a file".

But more importantly, this thread, and the whole newsgroup, is about
Fortran. And to Fortran, everything connected to a logical unit is a
file.

f2003 " 9.4.5 The OPEN statement

An OPEN statement initiates or modifies the connection between an
external file and a specified unit."

And simillarly all over the place in all versions of the Fortran
standard. Not a file descriptor, or something that looks like a file, or
anything else. Just plain a file. To be slightly more specific, an
external file as opposed to an internal one, but that doesn't pertain to
the question at hand. As I noted before, the restriction in question is
one in the Fortran standard and the definitions in the Fortran standard
apply. (Though Steve mentions a proposal to relax the restriction,
that's in the future). Now what constitutes the "same file" is a valid
question, which isn't addressed by Fortran. But arguing that something
doesn't really count as a file will get you nowhere in terms of Fortran.

[herrman...@gmail.com]

On Tuesday, May 17, 2016 at 3:55:09 PM UTC-7, Richard Maine wrote:

(snip regarding unix file names)

> > Some say that, but more say "everything is a file descriptor" or
> > "everything looks like a file".

> But more importantly, this thread, and the whole newsgroup, is about
> Fortran. And to Fortran, everything connected to a logical unit is a
> file.

> f2003 " 9.4.5 The OPEN statement

> An OPEN statement initiates or modifies the connection between an
> external file and a specified unit."

> And simillarly all over the place in all versions of the Fortran
> standard. Not a file descriptor, or something that looks like a file, or
> anything else. Just plain a file. To be slightly more specific, an
> external file as opposed to an internal one, but that doesn't pertain to
> the question at hand. As I noted before, the restriction in question is
> one in the Fortran standard and the definitions in the Fortran standard
> apply. (Though Steve mentions a proposal to relax the restriction,
> that's in the future). Now what constitutes the "same file" is a valid
> question, which isn't addressed by Fortran. But arguing that something
> doesn't really count as a file will get you nowhere in terms of Fortran.

So my last suggestion, the ability to open a UNIT to a null file
(or device), independent of the underlying system, would be a
good solution.

Not that there aren't already enough options for OPEN.

-- glen

[kargl]

You can already connect a unit to /dev/null (if such a file exists).
You cannot connect more than one unit to the same file. All of
the tricks and contrived "cp" and symlink examples won't help you.
I suspect most runtime libraries on a UNIX or UNIX-like system
will use realpath() to resolve filenames. This won't be fooled be
stupid-user tricks.

--
steve

[vladim...@gmail.com]

> Fortran 2015 will relax this restriction. The current draft says:
>
> "If input/output operations are performed on more than one unit while
> they are connected to the same external file, the results are processor
> dependent."
>
> The rule about a unit not being able to connect to multiple files remains.

So perhaps it is time to open a feature request in gfortran bugzilla? Of course, the volunteer maintainers likely have different priorities.

[JB]

The canonical UNIX way of checking whether two file descriptors or
paths refer to the same file is to compare the device and inode
numbers. Which, FWIW, is also what the GFortran runtime library does.

--
JB