Musings about redirection and (only) new information overwriting existing file

[Janis Papanagnou]

Occasionally it happens that I want to redirect output to a file, but
create that file or overwrite an existing file only if there's actually
new data available. (So I don't want to and cannot use '>' or '>>'.)

Since I haven't seen that feature in shell I usually use awk to achieve
that function by something like

... | awk -v fn="some-file" '{ print > fn }'

I used that pattern already a couple of times so that I now have it in a
local bin-directory as executable file 'cf'

# cf - conditionally create file
awk -v fn="${1:?}" '{ print > fn }'

using it in contexts like

news-process | cf latest-news

where the latest-news file gets not overwritten if there's no newer news.


Redirecting with '>' to a file in shell will always overwrite the file,
unless 'noclobber' shell option is set; in that case you need '>|' to
overwrite an existing file.

In ksh there's also the '>;' redirection available. It writes output to
a temporary file and creates/overwrites the file only if no error occurs.

The tee(1) command has also no option to create or overwrite files only
conditionally if data is present, as far as I can see.

Is there some tool or shell function that I missed that does conditional
overwrites as described?


To reproduce standard input to standard output as well I added option -t
so that I can see what's getting written to that file (or to process any
output further in a pipeline).

#!/bin/ksh
#
# cf - conditionally create file if data is present

t=0
while getopts ":t" opt
do
case ${opt} in
(t) t=1 ;;
(\?) printf "Usage: cf [-t] filename\n" ; exit 1 ;;
esac
done
shift OPTIND-1
fn=${1:?}
awk -v fn="${fn}" -v t="${t}" '{ print > fn } t'


I can use that function in my environment but I think it would better
fit in shell, maybe as a new shell redirection ('>@', like ksh's '>;'),
or have it as feature of existing standard tools, maybe in tee(1) with
a new option supporting this semantics, like the -a is used to support
'>>'.

Janis

[Oğuz]

On 8/14/22 2:40 PM, Janis Papanagnou wrote:
> Occasionally it happens that I want to redirect output to a file, but
> create that file or overwrite an existing file only if there's actually
> new data available. (So I don't want to and cannot use '>' or '>>'.)
>
> Since I haven't seen that feature in shell I usually use awk to achieve
> that function by something like
>
> ... | awk -v fn="some-file" '{ print > fn }'

Does this have any advantage over using a temporary file? Like:

... > temp-file
test -s temp-file && mv temp-file some-file

[Helmut Waitzmann]

Oğuz <oguzism...@gmail.com>:

>On 8/14/22 2:40 PM, Janis Papanagnou wrote:
>> Occasionally it happens that I want to redirect output to a file,
>> but create that file or overwrite an existing file only if
>> there's actually new data available. (So I don't want to and
>> cannot use '>' or '>>'.)
>>
>> Since I haven't seen that feature in shell I usually use awk to
>> achieve that function by something like
>>
>> ... | awk -v fn="some-file" '{ print > fn }'
>
>Does this have any advantage over using a temporary file?
>

It depends.  While

> ... > temp-file
> test -s temp-file && mv temp-file some-file

copies the access permissions from “temp-file” to “some-file” and
replaces “some-file” if it already exists,

>> ... | awk -v fn="some-file" '{ print > fn }'

as well as


... > temp-file &&
if test -s temp-file
then
cat -- temp-file >| some-file &&
rm -f -- temp-file
fi


will overwrite “some-file” rather than replace it.  Also, if
“some-file” doesn't already exist, it will be just created rather
than being created and having the access modes modified according to
the access modes of “temp-file”.

[Janis Papanagnou]

On 16.08.2022 17:38, Helmut Waitzmann wrote:
> Oğuz <oguzism...@gmail.com>:
>> On 8/14/22 2:40 PM, Janis Papanagnou wrote:
>>> Occasionally it happens that I want to redirect output to a file, but
>>> create that file or overwrite an existing file only if there's
>>> actually new data available. (So I don't want to and cannot use '>'
>>> or '>>'.)
>>>
>>> Since I haven't seen that feature in shell I usually use awk to
>>> achieve that function by something like
>>>
>>> ... | awk -v fn="some-file" '{ print > fn }'
>>
>> Does this have any advantage over using a temporary file?

I haven't pondered about advantages or disadvantages of the one or
the other option. Here are just a few obvious thoughts I have...

It doesn't create a temporary file. - An advantage for itself; since
it happens - as we see in your test/mv based code below demonstrated -
that you forget to clean up that temporary file. (So it's getting yet
more complex than the workaround already is if you want to maintain a
tidy runtime environment.)

It's also less complex; an additional test and a conditional command.
The awk pattern doesn't create the file in the first place, if it's
not needed.

Also if you're at the system disk memory limit your code will create
an additional file requiring (if only temporary) additional space
that the system might not be able to allocate.

If you dislike awk you can of course put your test/mv commands set
in a file 'cf' (or whatever you call it) and it behaves (mostly) the
same. The main point of my post was to have a compact pattern of a
command I find useful, and with the tee-extension I also suggested,
I can also use it in a pipe. The other point of my post was whether
that semantics is already present in shell in some way I missed, or
in some tools; I wouldn't want to use my own tool if there's already
some standard tool supporting it.

>
> It depends. [...]
> [ considerations about access modes ]

Janis

[Helmut Waitzmann]

Janis Papanagnou <janis_pa...@hotmail.com>:

>I haven't pondered about advantages or disadvantages of the one or
>the other option. Here are just a few obvious thoughts I have...

[…]

I totally agree with you (therefore I deleted your explanations).

>The other point of my post was whether that semantics is already
>present in shell in some way I missed, or in some tools;

I don't know of any.  Of course one can do that using the shell and
“cat”:


(
unset -v -- line &&
lf="$( printf '%s\n' '' '.' )" && lf="${lf%.}" &&
if
IFS= read -r -- line && line="${line}${lf}"
${line:+:} false
then
exec > some-file &&
printf '%s' "$line" &&
cat
fi
)

>I wouldn't want to use my own tool if there's already some standard
>tool supporting it.

I don't know of a better way to do that than yours.