Shell Languages Versus Programming Languages

[lawren...@gmail.com]

It seems to me important to distinguish between shell scripting languages (bash etc) and, shall we say, non-shell scripting languages (e.g. Perl, Python, Ruby). Heck, let’s just call the latter “programming languages” and be done with it.

In a shell language, everything you type is assumed to be a string literal by default, unless you use special syntax to indicate substitution of variables, expressions, nested commands etc.

In a programming language, everything you type is interpreted as a programming construct of some kind. You use special syntax to represent string literals.

Shell languages are particularly convenient for interactively executing a single command at a time--which is why we use their implementations as shells.

And now, I present to you Microsoft PowerShell. Why do I mention it? Because Microsoft is porting it to Linux <http://arstechnica.com/information-technology/2016/08/powershell-is-microsofts-latest-open-source-release-coming-to-linux-os-x/>, <http://phoronix.com/scan.php?page=news_item&px=Microsoft-PowerShell-On-Linux>. Trouble is, according to this “Python Developer’s Guide to PowerShell” <https://mohd-akram.github.io/2013/05/16/a-python-developers-guide-to-powershell>, I see a language that is trying to be both a shell language and a programming language, in one.

Is this a good idea?

And there is also this <http://www.theregister.co.uk/2016/08/23/your_wget_is_broken_and_should_die_powershellers_tell_microsoft/>.

[hymie!]

In our last episode, the evil Dr. Lacto had captured our hero,

lawren...@gmail.com <lawren...@gmail.com>, who said:
> It seems to me important to distinguish between shell scripting
> languages (bash etc) and, shall we say, non-shell scripting languages
> (e.g. Perl, Python, Ruby). Heck, let’s just call the latter
> “programming languages” and be done with it.

The concept of "programming" is "pre-determining a list of commands that
should be run in succession." It doesn't much matter whether your programs
are being run by the shell, or by the kernel, or by a robot. You have
programmed it.

When you log into the machine, the machine runs "a shell" for you
and gives you an interactive prompt where you can issue individual
commands. The same is true if you run the "bc" command and issue
individual commands, or you can program bc to run your commands from
a file.

The fact that you call bash "a shell" instead of "a program" does not
change the fact that you are programming it.

--hymie! http://lactose.homelinux.net/~hymie hy...@lactose.homelinux.net

[Barry Margolin]

In article <slrnnva3n1...@lactose.homelinux.net>,

hymie! <hy...@lactose.homelinux.net> wrote:

> The fact that you call bash "a shell" instead of "a program" does not
> change the fact that you are programming it.

In my mind, the difference between a shell script and other type of
programs is that the shell is a tool primarily (or perhaps initially)
intended for interactive use, and a shell script is a program that
contains saved interactive commands for automated execution.

Other languages, like PHP, Perl, and Python, might provide a way to
execute commands interactively, but this is more intended as a debugging
tool, not as a primary way of controlling the computer.

The specific syntax (e.g. strings are automatically quoted, while you
have to use extra notation to indicate special interpretation) is mostly
an outgrowth of the intended use -- you generally want to keep
interactive syntax reasonably terse.

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Kaz Kylheku]

On 2016-10-05, Barry Margolin <bar...@alum.mit.edu> wrote:
> In article <slrnnva3n1...@lactose.homelinux.net>,
> hymie! <hy...@lactose.homelinux.net> wrote:
>
>> The fact that you call bash "a shell" instead of "a program" does not
>> change the fact that you are programming it.
>
> In my mind, the difference between a shell script and other type of
> programs is that the shell is a tool primarily (or perhaps initially)
> intended for interactive use, and a shell script is a program that
> contains saved interactive commands for automated execution.

"Shell" has a notable use outside of operating systems: "expert system
shell": the interactive query listener for probing an expert system's
knowledge.

Microsoft defines "Shell" to include an interactive GUI for an operating
system. The Windows Explorer is also called the "Windows Shell" and its
functions are available in the "Shell API".

[Thomas 'PointedEars' Lahn]

Kaz Kylheku wrote:

> On 2016-10-05, Barry Margolin <bar...@alum.mit.edu> wrote:
>> hymie! <hy...@lactose.homelinux.net> wrote:
>>> The fact that you call bash "a shell" instead of "a program" does not
>>> change the fact that you are programming it.
>> In my mind, the difference between a shell script and other type of
>> programs is that the shell is a tool primarily (or perhaps initially)
>> intended for interactive use, and a shell script is a program that
>> contains saved interactive commands for automated execution.
>
> "Shell" has a notable use outside of operating systems: "expert system
> shell": the interactive query listener for probing an expert system's
> knowledge.

Yes. But you are the one saying “operating system” first, so this is a
straw man argument.

> Microsoft defines "Shell" to include an interactive GUI for an operating
> system.

Fortunately, Micro$~1 has no say in this (newsgroup).

> The Windows Explorer is also called the "Windows Shell" and its
> functions are available in the "Shell API".

Nonsense.

--
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.

[Janis Papanagnou]

On 05.10.2016 17:56, Barry Margolin wrote:
> In article <slrnnva3n1...@lactose.homelinux.net>,
> hymie! <hy...@lactose.homelinux.net> wrote:
>
>> The fact that you call bash "a shell" instead of "a program" does not
>> change the fact that you are programming it.
>
> In my mind, the difference between a shell script and other type of
> programs is that the shell is a tool primarily (or perhaps initially)
> intended for interactive use, and a shell script is a program that
> contains saved interactive commands for automated execution.

That may be true for csh, but it's unlikely for the yet older sh.
Concepts (programming constructs) for the shell language have been
even borrowed from Algol68; nothing that is easily typed on the
command line, specifically at times when there was no history.

Janis

>
> [...]

[lawren...@gmail.com]

On Thursday, October 6, 2016 at 3:32:04 AM UTC+13, hymie! wrote:
> The concept of "programming" is "pre-determining a list of commands that
> should be run in succession." It doesn't much matter whether your programs
> are being run by the shell, or by the kernel, or by a robot. You have
> programmed it.

Shell languages are frequently deficient in this regard. For example, bash has trouble looping over arbitrary file names.

[Chris F.A. Johnson]

I never have any problem looping over arbitrary file names in bash.

I either use globbing directly:

for file in *
do
: whatever with "$file"
done

Or put the filenames into an array:

files=( * )
for file in "${files[@]}"
do
: whatever with "$file"
done


--
Chris F.A. Johnson

[lawren...@gmail.com]

On Friday, October 7, 2016 at 2:08:06 PM UTC+13, Chris F.A. Johnson wrote:

> On 2016-10-06, Lawrence D’Oliveiro wrote:
>>
>> For example, bash has trouble looping over arbitrary file names.
>
> I never have any problem looping over arbitrary file names in bash.

What if the file names come from, e.g. the “find” command?

[Teemu Likonen]

#!/bin/bash

while read -r -d $'\0' file; do
# Do something with "$file"
true
done < <(find ... -print0)

--
/// Teemu Likonen - .-.. <https://github.com/tlikonen> //
// PGP: 4E10 55DC 84E9 DFF6 13D7 8557 719D 69D3 2453 9450 ///

[lawren...@gmail.com]

On Friday, October 7, 2016 at 5:29:54 PM UTC+13, Teemu Likonen wrote:
> while read -r -d $'\0' file; do
> # Do something with "$file"
> true
> done < <(find ... -print0)

Interesting bashism. I didn’t think that would work, but whaddaya know...

[Ian Zimmerman]

On 2016-10-07 07:29, Teemu Likonen wrote:

> while read -r -d $'\0' file; do
> # Do something with "$file"
> true
> done < <(find ... -print0)

Not to flog a dead ho^H^H FAQ, but this is considered Really Bad Style
by some in this newsgroup.

If do_something_with_files is a well behaved external program, the
cleanest solution is

find -exec do_something_with_files '{}' +

If do_something_with_files is something else (a shell function, a
command that only takes I arg at a time, etc), you can get creative.

--
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign: http://cr.yp.to/smtp/8bitmime.html

[lawren...@gmail.com]

On Saturday, October 8, 2016 at 11:21:18 AM UTC+13, Ian Zimmerman wrote:
> On 2016-10-07 07:29, Teemu Likonen wrote:
>
>> while read -r -d $'\0' file; do
>> # Do something with "$file"
>> true
>> done < <(find ... -print0)
>
> Not to flog a dead ho^H^H FAQ, but this is considered Really Bad Style
> by some in this newsgroup.

Not by me. I am impressed.

[Kaz Kylheku]

Then you will almost certainly also like that Gawk also handles null as
a record separator (RS):

# paths are processed as records

find /etc -print0 | gawk -v RS='\0' 1

[lawren...@gmail.com]

On Saturday, October 8, 2016 at 4:04:22 PM UTC+13, Kaz Kylheku wrote:
> Then you will almost certainly also like that Gawk also handles null as
> a record separator (RS):

Neat. However, I learned Perl before I came across awk/gawk, so I never saw the point in the latter.

[Ben Bacarisse]

... and let's not forget xargs -0 (though best used when -exec '{}' +
can't be used direcly).

--
Ben.

[Ivan Shmakov]

Awk is way easier to use in certain cases; that's no different
to the other common text processing tools, such as Grep or Sed.

Consider, e. g.:

$ grep -- foo.\*bar < file

$ sed -e '\,foo/bar,d; \,.*\sbaz=\(.*\),!d; s//\1/;' < file

$ gawk '/foo/ { next; } 1 { print $2 - $1, $3; }' < file

While the above are not all that difficult to implement in Perl,
I cannot readily think of Perl oneliners having the effect of
the commands above /and/ being shorter at the same time.

--
FSF associate member #7257 58F8 0F47 53F5 2EB2 F6A5 8916 3013 B6A0 230E 334A

[Ivan Shmakov]

>>>>> Ben Bacarisse <ben.u...@bsb.me.uk> writes:

[...]

>> find /etc -print0 | gawk -v RS='\0' 1

> ... and let's not forget xargs -0 (though best used when -exec '{}' +

> can't be used directly).

Technically, -exec cmd... {} + has the effect of xargs -r0, not
just -0. Makes a difference if the cmd is something that
resorts to reading stdin when no filenames are given.

For me, common uses to xargs(1) were along the lines of, say:

$ find storage/files/[pio]?*.deviantart.net/ \
-type f -size +64k -print0 \
| LC_ALL=C sort -rnzt/ -k6,7 \
| DISPLAY=:1.0 xargs -r0 -- edisplay &

Or, if I'm interested in the files newly added to my archive,
it'd rather be -printf %C@\\t%p\\0, -k1,1, and there will be an
additional sed -z command to remove the timestamp before passing
the list to xargs(1).

[Janis Papanagnou]

On 08.10.2016 20:12, Ivan Shmakov wrote:
>>>>>> lawrencedo99 <lawren...@gmail.com> writes:
>>>>>> On Saturday, October 8, 2016 at 4:04:22 PM UTC+13, Kaz Kylheku wrote:
>
> >> Then you will almost certainly also like that Gawk also handles null
> >> as a record separator (RS):
>
> > Neat. However, I learned Perl before I came across awk/gawk, so I
> > never saw the point in the latter.
>
> Awk is way easier to use in certain cases; that's no different
> to the other common text processing tools, such as Grep or Sed.
>
> Consider, e. g.:
>
> $ grep -- foo.\*bar < file
>
> $ sed -e '\,foo/bar,d; \,.*\sbaz=\(.*\),!d; s//\1/;' < file
>
> $ gawk '/foo/ { next; } 1 { print $2 - $1, $3; }' < file

I would be very astonished if it can't be shorter in perl. It can even be
shorter in awk...

awk '!/foo/ { print $2 - $1, $3 }' file


Janis

[Martin Vaeth]

Ivan Shmakov <iv...@siamics.net> wrote:
>
> > Neat. However, I learned Perl before I came across awk/gawk, so I
> > never saw the point in the latter.
>
> Awk is way easier to use in certain cases

No it is not. In fact, there is practically nothing which awk can
do which perl cannot do with almost the same syntax (see below).
In my experience there are only two valid reasons to use awk
instead of perl:
1. In some rare cases the availability on the system
2. perl regexp implementation is broken in the sense that
it can need exponential runtime even for expressions which
actually need only a deterministic FSM.

> that's no different to the other common text processing tools

> such as Grep or Sed.

These are sometimes easier to use if you really need only
their very specialized functionality. But I wouldn't call
these tools "programming language" (although with sed you
can _pretend_ it is one).

> $ gawk '/foo/ { next; } 1 { print $2 - $1, $3; }' < file

To mimick your syntax in perl:

perl -ane 'if(/foo/) { next } print $F[1] - $F[0]," $F[2]\n"' < file

A few characters longer but IMHO practically the same.
But already with the mentioned "find" you see the disadvantage
of awk: It will work only correctly if you have find's GNU
extensions - not only concerning the filenames but also
concerning races - but even then there are a lot of restrictions
of find, e.g. when you want to follow only _certain_ symlinks,
want to stop when the total filelength reaches a certain limit etc.
In perl, you can use simply File::Find which has neither of these
restrictions, and moreover, it is also simple to write code which
can be used unchanged on operating systems which use other
directory conventions like f:\BLA\Foo ...
Try to do that (with e.g. path related conditions) in GNU find...

[Kaz Kylheku]

awk '!/foo/&&$0=($2-$1)OFS$3'

[Ivan Shmakov]

>>>>> Ivan Shmakov <iv...@siamics.net> writes:

>>>>> Ben Bacarisse <ben.u...@bsb.me.uk> writes:

>>> find /etc -print0 | gawk -v RS='\0' 1

>> ... and let's not forget xargs -0 (though best used when -exec '{}'
>> + can't be used directly).

> Technically, -exec cmd... {} + has the effect of xargs -r0, not just
> -0. Makes a difference if the cmd is something that resorts to
> reading stdin when no filenames are given.

I stand corrected; thanks to xargs(1) redirecting the input of
the child from /dev/null, it does /not/ make a difference in
such cases. (Though it still avoids a useless call in the case
of empty input.)

There is a difference for the likes of display(1), however.

[...]

[Ivan Shmakov]

>>>>> Martin Vaeth <mar...@mvath.de> writes:
>>>>> Ivan Shmakov <iv...@siamics.net> wrote:

[...]

>> that's no different to the other common text processing tools such
>> as Grep or Sed.

> These are sometimes easier to use if you really need only their very
> specialized functionality. But I wouldn't call these tools
> "programming language" (although with sed you can _pretend_ it is
> one).

Sed is Turing-complete, which makes it a programming language in
my book. Even if one inconvenient for solving other than some
specific tasks.

>> $ gawk '/foo/ { next; } 1 { print $2 - $1, $3; }' < file

> To mimick your syntax in perl:

> perl -ane 'if(/foo/) { next } print $F[1] - $F[0]," $F[2]\n"' < file

> A few characters longer but IMHO practically the same.

(Sometimes I wonder if the specification for the Perl "core"
dwarfs that of Common Lisp.)

Indeed, I was unaware of the "autosplit" mode; thanks.

> But already with the mentioned "find" you see the disadvantage of
> awk: It will work only correctly if you have find's GNU extensions -
> not only concerning the filenames but also concerning races

Specifically? Although I should note that I use GNU almost
exclusively, and as such, have little interest in environments
lacking GNU Find. (Other than perhaps those based on BusyBox.)

> - but even then there are a lot of restrictions of find, e.g. when
> you want to follow only _certain_ symlinks,

Care to provide an example on when that may be needed?

> want to stop when the total filelength reaches a certain limit etc.
> In perl, you can use simply File::Find which has neither of these
> restrictions,

My preference is to leave the file selection criteria outside
the scope of the software I write, thus allowing the user supply
any such criteria he or she fancies -- which I gather is not
quite feasible with File::Find. Well, aside from the fact that
the user /can/ indeed use File::Find in his own code to form the
list of files for my code to process.

> and moreover, it is also simple to write code which can be used
> unchanged on operating systems which use other directory conventions
> like f:\BLA\Foo ... Try to do that (with e.g. path related
> conditions) in GNU find...

Some such conditions can be specified with -regex.

Moreover, the systems that rely on non-Unix file naming
conventions require some adaptational layer for much of the
tools designed for Unix-like systems to run anyway, which means
the filename above is likely to denote the same file as, say,
/f:/BLA/Foo.

It's not a solution I like; but then again, I use such systems
only occasionally.

[Martin Vaeth]

Ivan Shmakov <iv...@siamics.net> wrote:
> > awk: It will work only correctly if you have find's GNU extensions -
> > not only concerning the filenames but also concerning races
>
> Specifically?

man find
/execdir

> Care to provide an example on when that may be needed?

Here is a tool which I had originally implemented in shell
(as an init-script for openrc):
https://github.com/vaeth/squash_dir

The task for which find is needed here is to check a
whole file tree - with certain user-specified exceptions,
which partly need to invoke "diff" to find out -
whether it reaches a certain total size.

Have in particular a look at the ugly code to generate an
appropriate "find" command in NeedSquash():

https://github.com/vaeth/squash_dir/raw/master/init.d/squash_dir

This code is not only ugly but actually lacks a lot of tests
which would be convenient for the tool but could not be
implemented this way. Moreover, the time which "find" needs is
unnecessarily long, because for the application "find" could
actually return earlier when the threshold size is reached.

This was one of the motivations to use a "real" programming
language like perl for the reimplementation which does not
have these limitations:
https://github.com/vaeth/squashmount

The corresponding part here is "sub threshold_dir" in:
https://github.com/vaeth/squashmount/raw/master/bin/squashmount

This function is longer but mainly because of features which would
not be possible at all in the shell version (e.g. postponing the
lengthy "diff" calls and executing them only if they could make
the crucial difference in size; also a lot of more fine-tuning
about elimination of branches by the user and even user-specified
criteria are simpler possible).

> My preference is to leave the file selection criteria outside
> the scope of the software I write

Which is only possible if the software you write "naturally" needs
to access only a few single files.

> Some such conditions can be specified with -regex.

Not system independent.

> Moreover, the systems that rely on non-Unix file naming
> conventions require some adaptational layer for much of the
> tools designed for Unix-like systems to run anyway

Exactly this is avoided: In perl - if used in the proper way -
you can write tools without making implicit assumptions about
the underlying naming conventions: The _language_ (more precisely,
a core library of the language) provides the necessary abstraction
which you can (and should) use.
perldoc File::Spec

[lawren...@gmail.com]

On Sunday, October 9, 2016 at 9:43:57 PM UTC+13, Martin Vaeth wrote:
> Moreover, the time which "find" needs is
> unnecessarily long, because for the application "find" could
> actually return earlier when the threshold size is reached.

This immediately brings to mind Prolog, which also does a lot of tree searches. That has a “cut” operator to truncate the search without actually failing; I wonder if “find” could do with something similar?

> This function is longer but mainly because of features which would
> not be possible at all in the shell version (e.g. postponing the
> lengthy "diff" calls and executing them only if they could make
> the crucial difference in size; also a lot of more fine-tuning
> about elimination of branches by the user and even user-specified
> criteria are simpler possible).

Basically, you have written a query optimizer.

Maybe that’s something else “find” could do with...