Hard links on VMS ODS5 disks

[Chris Townley]

I am just about to install V9.2-1 and intrigued about hard links.

When I installed E9.2-1 it asked me if I wanted to enable hard links,
with a warning to read the docs, but I wasn't much wiser afterwards.

I get he impression that it is much cleaner with them enables, but what
are the downside risks?

I have used soft links on *nix for many years, but only for linking to a
directory, which you cannot do with hard links

Any thoughts?

--
Chris

[Johnny Billquist]

I always have a bit of a hard time understanding the topic. One person I
talked to was of the opinion that ODS "hard links" were not hard links
because they didn't include reference counting.

But essentially, if we talk about hard links under Unix, it's just
additional directory entries pointing to the same inode.
The equivalent of the inode number under Unix is in ODS the file id. And
you can have multiple directory entries giving the same file id, which
is exactly the same as the thing in Unix.

The one difference being that in Unix, a file is removed when the link
count goes to zero. And there is no way for a program to reference a
file through the inode number itself.

While in ODS, the file id, and file, is more unaware of what directory
entries might exist that refers to it, and a file can be deleted without
the directory entry being deleted. And also, programs can open files by
ID, completely bypassing the directory lookup. Which is also why ODS
fild ids have a generation number, so that stale file ids don't give you
some other file you didn't expect.

But I would still already call this hard links. I wonder if "enabling
hardlinks" in ODS5 is then adding the reference counting, and not
deleting files until there are no references to them. Or is there
something else/more hidden in here?

Johnny

[Chris Townley]

That is my understanding from recent doc reading.

--
Chris

[gah4]

On Wednesday, July 12, 2023 at 6:34:40 AM UTC-7, Chris Townley wrote:
> I am just about to install V9.2-1 and intrigued about hard links.

(snip)

> I have used soft links on *nix for many years, but only for linking to a
> directory, which you cannot do with hard links

Symbolic links are often used for files on modern Unix-like systems.
The cases that I know that are still hard links are programs that have
different names. For example, gzip, gunzip, zcat, and gzcat, are all the
same file, but do different things based on argv[0].

Reminds me, though, on early VMS, maybe 1.x, you could rename a
directory into itself.

ren x.dir [-.x]x.dir

(I think that is how I did it.)

In which case the directory is gone, but the space still counts for your quota.
That was in 1979, so maybe v 1.5 or 1.6.

[Steven Schweda]

> [...] For example, gzip, gunzip, zcat, and gzcat, are all the

> same file, but do different things based on argv[0].

Common in the past, but the GNU folks decided against that scheme
some time ago. (At least a dozen years?) For example:

proa$ pwd
/usr/local/gnu/gzip/gzip-1.12

proa$ ls -li gzip gunzip zcat
133419312 -r-xr-xr-x 1 sms wheel 2346 Jul 12 17:21 gunzip
133419311 -rwxr-xr-x 1 sms wheel 140108 Jul 12 17:21 gzip
133419314 -r-xr-xr-x 1 sms wheel 1984 Jul 12 17:21 zcat

"gunzip" and "zcat" are now shell scripts. Excerpt from gzip.c:

[...]
#ifndef GNU_STANDARD
# define GNU_STANDARD 1
#endif
#if !GNU_STANDARD
/* For compatibility with old compress, use program name as an option.
* Unless you compile with -DGNU_STANDARD=0, this program will behave as
* gzip even if it is invoked under the name gunzip or zcat.
*
* Systems which do not support links can still use -d or -dc.
* Ignore an .exe extension for MSDOS and OS/2.
*/
if (strncmp (program_name, "un", 2) == 0 /* ungzip, uncompress */
|| strncmp (program_name, "gun", 3) == 0) /* gunzip */
decompress = 1;
else if (strequ (program_name + 1, "cat") /* zcat, pcat, gcat */
|| strequ (program_name, "gzcat")) /* gzcat */
decompress = to_stdout = 1;
#endif
[...]

[Arne Vajhøj]

At the risk of sounding very old fashioned then I have a strategy
for soft and hard links on VMS:
* I don't use them
* if VMS itself use them then I assume VSI and previous maintainers
knows how to use them properly

I answered yes to the question.

Arne

[Steven Schweda]

> * I don't use them

I use them mostly for testing programs which should be able to handle
them (Info-ZIP Zip and UnZip, VMSTAR, and so on). (Symlinks need more
special handling than hard links. Hard links seem pretty harmless.)

As I recall, some version(s) of OpenSSL kits ("0.9.8%", et al.?) used
symlinks for a bunch of header files, making those kits hard to use if
they were unpacked using a program and/or onto a file system which did
not support symlinks. There may have been other cases, too.

[gah4]

On Wednesday, July 12, 2023 at 3:46:45 PM UTC-7, Steven Schweda wrote:
> > [...] For example, gzip, gunzip, zcat, and gzcat, are all the
> > same file, but do different things based on argv[0].
> Common in the past, but the GNU folks decided against that scheme
> some time ago. (At least a dozen years?)

Looking on computers near me.

A MacBook Air with OS 10.12.6 has two different files, not linked,
but with exactly the same contents.

ls -li compress uncompress
421074 -rwxr-xr-x 1 root wheel 27680 Jul 14 2017 compress
421907 -rwxr-xr-x 1 root wheel 27680 Jul 14 2017 uncompress
and cmp -l shows that they are the same.

FreeBSD does use the hard links, and also Linux.
The Linux system is 13 years old, so doesn't disagree with your dozen years.

[Johnny Billquist]

On 2023-07-12 22:32, gah4 wrote:
> On Wednesday, July 12, 2023 at 6:34:40 AM UTC-7, Chris Townley wrote:
>> I am just about to install V9.2-1 and intrigued about hard links.
>
> (snip)
>
>> I have used soft links on *nix for many years, but only for linking to a
>> directory, which you cannot do with hard links
>
> Symbolic links are often used for files on modern Unix-like systems.
> The cases that I know that are still hard links are programs that have
> different names. For example, gzip, gunzip, zcat, and gzcat, are all the
> same file, but do different things based on argv[0].

Works the same if you use soft links.

Soft links are slightly costlier, but can do some things hard links cannot.

In Unix, you are not allowed to create arbitrary hard links to directories.
And hard links cannot cross between file systems.

In ODS, hard links to directories are not the same kind of problem as in
Unix, and hard links works fine. Actually, might *maybe* be an issue in
ODS-2 and newer, but with ODS-1 there is nothing problematic with hard
links to directories.

However, the same limitation about crossing between file systems are
also true on ODS.

Johnny

[Single Stage to Orbit]

On Wed, 2023-07-12 at 15:46 -0700, Steven Schweda wrote:
>    Common in the past, but the GNU folks decided against that scheme
> some time ago.  (At least a dozen years?)  For example:
>
> proa$ pwd
> /usr/local/gnu/gzip/gzip-1.12
>
> proa$ ls -li gzip gunzip zcat
> 133419312 -r-xr-xr-x  1 sms  wheel    2346 Jul 12 17:21 gunzip
> 133419311 -rwxr-xr-x  1 sms  wheel  140108 Jul 12 17:21 gzip
> 133419314 -r-xr-xr-x  1 sms  wheel    1984 Jul 12 17:21 zcat

my system still uses:

ls -li `which gzip` `which gunzip` `which zcat`
11537290 lrwxrwxrwx 1 root root 16 Dec 27 2022 /bin/gunzip -> gunzip-
reference
11537275 lrwxrwxrwx 1 root root 14 Dec 27 2022 /bin/gzip -> gzip-
reference
11537294 lrwxrwxrwx 1 root root 14 Dec 27 2022 /bin/zcat -> zcat-
reference

--
Tactical Nuclear Kittens

[bill]

Which is yet another example of "throw more hardware at the problem".
A link would save the space, but who cares.

bill

[Steven Schweda]

> my system still uses:

Thanks for the detailed description of "my system", and for including
the "gzip -V" report.

[Single Stage to Orbit]

Never mind, it seems they are separate executables. Sorry.
--
Tactical Nuclear Kittens

[Stephen Hoffman]

Enable hardlinks, run ANALYZE /DISK /REPAIR, and continue with whatever
you were doing.

That prompt is... ill-considered. That change should have been packaged
and presented as a bug fix.

That change fixes a longstanding bug in the existing file system design.





Background, from most of a decade ago: "Maybe that gets better
documented? The official documentation in the HPE
V8.4-to-V8.2-inclusive System Manager's Manual /10.12 Understanding
Hard Links/ appears correct but also a little conflated, and that
material is old enough that it doesn't even mention softlinks."

That cited doc is now here:
https://docs.vmssoftware.com/docs/vsi-openvms-system-manager-s-manual-volume-1-essentials.pdf
page 402.

Quoth the docs: "If you have hard links enabled, a file is actually
deleted when there are no more links to file. If you do not have hard
links enabled, and you have not created an alias for a file,
essentially only one link to that file exists: the primary link. If you
create an alias for that file, and you delete the alias, the file still
exists because the primary link to that file has not been deleted. The
alias is just another name in a directory for this link. Deleting the
primary link deletes the file, leaving the alias entries."
That section of the doc could be worded more clearly, and it is also
stale as there are four sorts of links with current OpenVMS: primary
links, aliases, hardlinks, and softlinks.

Yes, apps that rummage the file system directly might need to know. I'd
expect a number of apps that do try to look at links probably get it
wrong, too.





Old grumping, from most of a decade ago: "Usual solution for
compatibility? Add knobs and APIs and run-time checks, and throw it all
over the wall and let somebody else figure it out when — when, not if —
it breaks. Design abdication."


--
Pure Personal Opinion | HoffmanLabs LLC

[Arne Vajhøj]

On 7/13/2023 9:25 AM, bill wrote:
> On 7/13/2023 1:50 AM, gah4 wrote:
>> On Wednesday, July 12, 2023 at 3:46:45 PM UTC-7, Steven Schweda wrote:
>>>> [...] For example, gzip, gunzip, zcat, and gzcat, are all the
>>>> same file, but do different things based on argv[0].
>>> Common in the past, but the GNU folks decided against that scheme
>>> some time ago. (At least a dozen years?)

>> A MacBook Air with OS 10.12.6 has two different files, not linked,
>> but with exactly the same contents.
>>
>> ls -li compress uncompress
>> 421074 -rwxr-xr-x  1 root  wheel  27680 Jul 14  2017 compress
>> 421907 -rwxr-xr-x  1 root  wheel  27680 Jul 14  2017 uncompress
>> and cmp -l shows that they are the same.
>>
>> FreeBSD does use the hard links, and also Linux.
>> The Linux system is 13 years old, so doesn't disagree with your dozen
>> years.
>
> Which is yet another example of "throw more hardware at the problem".
> A link would save the space, but who cares.

We can discuss the optimal choice between:
* paying more dollars to get bigger disk
* the complexity of programs with multiple behavior depending on name

But I don't think that is the case here. I think the case is that
the cheapest disk you can buy for new systems are so big that there
is enough space, so no money is saved by adding the complexity of
multiple behaviors.

Arne

[gah4]

On Friday, July 14, 2023 at 5:55:14 AM UTC-7, Arne Vajhøj wrote:
> On 7/13/2023 9:25 AM, bill wrote:

(snip)

> >> A MacBook Air with OS 10.12.6 has two different files, not linked,
> >> but with exactly the same contents.

> >> ls -li compress uncompress
> >> 421074 -rwxr-xr-x 1 root wheel 27680 Jul 14 2017 compress
> >> 421907 -rwxr-xr-x 1 root wheel 27680 Jul 14 2017 uncompress
> >> and cmp -l shows that they are the same.

(snip)

> We can discuss the optimal choice between:
> * paying more dollars to get bigger disk
> * the complexity of programs with multiple behavior depending on name

But it is still multiple behaviors depending on the name,
and still wasting (not very much) disk space.

And then there is busybox, which puts all of the usual Unix-like program
in one file:

[, [[, acpid, addgroup, adduser, adjtimex, ar, arp, arping, ash,
awk, basename, beep, blkid, brctl, bunzip2, bzcat, bzip2, cal, cat,
catv, chat, chattr, chgrp, chmod, chown, chpasswd, chpst, chroot,
chrt, chvt, cksum, clear, cmp, comm, cp, cpio, crond, crontab,
cryptpw, cut, date, dc, dd, deallocvt, delgroup, deluser, depmod,
devmem, df, dhcprelay, diff, dirname, dmesg, dnsd, dnsdomainname,
dos2unix, dpkg, du, dumpkmap, dumpleases, echo, ed, egrep, eject,
env, envdir, envuidgid, expand, expr, fakeidentd, false, fbset,
fbsplash, fdflush, fdformat, fdisk, fgrep, find, findfs, flash_lock,
flash_unlock, fold, free, freeramdisk, fsck, fsck.minix, fsync,
ftpd, ftpget, ftpput, fuser, getopt, getty, grep, gunzip, gzip, hd,
hdparm, head, hexdump, hostid, hostname, httpd, hush, hwclock, id,
ifconfig, ifdown, ifenslave, ifplugd, ifup, inetd, init, inotifyd,
insmod, install, ionice, ip, ipaddr, ipcalc, ipcrm, ipcs, iplink,
iproute, iprule, iptunnel, kbd_mode, kill, killall, killall5, klogd,
last, length, less, linux32, linux64, linuxrc, ln, loadfont,
loadkmap, logger, login, logname, logread, losetup, lpd, lpq, lpr,
ls, lsattr, lsmod, lzmacat, lzop, lzopcat, makemime, man, md5sum,
mdev, mesg, microcom, mkdir, mkdosfs, mkfifo, mkfs.minix, mkfs.vfat,
mknod, mkpasswd, mkswap, mktemp, modprobe, more, mount, mountpoint,
mt, mv, nameif, nc, netstat, nice, nmeter, nohup, nslookup, od,
openvt, passwd, patch, pgrep, pidof, ping, ping6, pipe_progress,
pivot_root, pkill, popmaildir, printenv, printf, ps, pscan, pwd,
raidautorun, rdate, rdev, readlink, readprofile, realpath,
reformime, renice, reset, resize, rm, rmdir, rmmod, route, rpm,
rpm2cpio, rtcwake, run-parts, runlevel, runsv, runsvdir, rx, script,
scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont,
setkeycodes, setlogcons, setsid, setuidgid, sh, sha1sum, sha256sum,
sha512sum, showkey, slattach, sleep, softlimit, sort, split,
start-stop-daemon, stat, strings, stty, su, sulogin, sum, sv,
svlogd, swapoff, swapon, switch_root, sync, sysctl, syslogd, tac,
tail, tar, taskset, tcpsvd, tee, telnet, telnetd, test, tftp, tftpd,
time, timeout, top, touch, tr, traceroute, true, tty, ttysize,
udhcpc, udhcpd, udpsvd, umount, uname, uncompress, unexpand, uniq,
unix2dos, unlzma, unlzop, unzip, uptime, usleep, uudecode, uuencode,
vconfig, vi, vlock, volname, watch, watchdog, wc, wget, which, who,
whoami, xargs, yes, zcat, zcip

[Chris Townley]

But does it run under OpenVMS?

--
Chris

[Dave Froble]

Oh, I hope not ...

--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: da...@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486

[Arne Vajhøj]

Some of them are available for VMS.

A lot of them are probably available on VMS using GNV.

(separate executables should run under DCL, but I believe
some of these are shell internal commands)

Arne

[gah4]

On Friday, July 14, 2023 at 12:44:48 PM UTC-7, Chris Townley wrote:

(snip regarding busybox)

> But does it run under OpenVMS?

It is supposed to be able to port to any system running gcc.

I don't see on the list that anyone has tried OpenVMS.

There are some applets (that is what they call them) that are OS specific,
and so won't work.

Reminds me of, years ago, porting gnu-utils to OS/2 and finding that
tail -f works. I don't know if tail -f is supposed to work on VMS, though.

The main target is embedded Linux, especially where the whole system
is in flash memory.

[gah4]

On Friday, July 14, 2023 at 2:07:18 PM UTC-7, Arne Vajhøj wrote:

(snip regarding busybox)

> Some of them are available for VMS.

> A lot of them are probably available on VMS using GNV.

> (separate executables should run under DCL, but I believe
> some of these are shell internal commands)

One of the applets is sh.

It seems to be a compile time option, for some of the internal
commands to be recognized directly. Or maybe it is that the
built-in sh will, optionally, recognize all of them as built-in.

[Craig A. Berry]

bash has some builtins but other things as separate executables. It
sounds like busybox just has everything as a builtin.

[Arne Vajhøj]

On 7/14/2023 6:14 PM, Craig A. Berry wrote:
> On 7/14/23 4:07 PM, Arne Vajhøj wrote:
>> On 7/14/2023 4:51 PM, Dave Froble wrote:
>>> On 7/14/2023 3:44 PM, Chris Townley wrote:
>>>> On 14/07/2023 17:58, gah4 wrote:
>>>>> [, [[, acpid, addgroup, adduser, adjtimex, ar, arp, arping, ash,
>>>>>          awk, basename, beep, blkid, brctl, bunzip2, bzcat, bzip2,

...

>>>> But does it run under OpenVMS?
>>>
>>> Oh, I hope not ...
>>
>> Some of them are available for VMS.
>>
>> A lot of them are probably available on VMS using GNV.
>>
>> (separate executables should run under DCL, but I believe
>> some of these are shell internal commands)
>
> bash has some builtins but other things as separate executables.  It
> sounds like busybox just has everything as a builtin.

Yes.

What I was trying to convey was that GNV bash has
some commands that are separate executables (like awk I assume)
and some that are internal commands (like printenv I assume). And
only the first category would potentially run under DCL which I why
I assume more from that list will run under GNV bash than under DCL.

Arne

[Chris Townley]

But type/tail does ISTR

--
Chris

[gah4]

On Friday, July 14, 2023 at 4:36:41 PM UTC-7, Chris Townley wrote:

(snip, I wrote)

> > Reminds me of, years ago, porting gnu-utils to OS/2 and finding that
> > tail -f works. I don't know if tail -f is supposed to work on VMS, though.

tail -f reads the end of a file, then waits and checks to see if it got bigger.
That is, if the EOF went away.

Not all file system, I suspect, can do that.

[Craig A. Berry]

On most of the interesting cases it fails with invalid file organization.

[Dave Froble]

Works very well here. Of course, I don't use invalid file organizations.

:-)

[gah4]

On Friday, July 14, 2023 at 3:14:23 PM UTC-7, Craig A. Berry wrote:

> bash has some builtins but other things as separate executables. It
> sounds like busybox just has everything as a builtin.

As well as I know it, in the usual case all those are symlinks.
The normal sh built-ins will still be built-in, and the others will use
the link, and run another busybox.

That works well for systems that can keep one copy in memory,
but run it more than once using that copy. And you can replace
the symlink with an actual program, or symlink to something else.

The standalone_shell configure option, not the default, has it check
for busybox applets that are not normally shell built-ins, before checking
the path. Especially convenient if there are no other files to run.

[Chris Townley]

Sorry, I should have said TYPE/TAIL/CONTINUOUS which works with some
caveats on the file

--
Chris

[Craig A. Berry]

On 7/14/23 9:58 PM, Dave Froble wrote:
> On 7/14/2023 9:48 PM, Craig A. Berry wrote:
>>
>> On 7/14/23 6:36 PM, Chris Townley wrote:
>>> On 14/07/2023 22:30, gah4 wrote:
>>
>>>> Reminds me of, years ago, porting gnu-utils to OS/2 and finding that
>>>> tail -f works.  I don't know if tail -f is supposed to work on VMS,
>>>> though.
>>>>
>>>> The main target is embedded Linux, especially where the whole system
>>>> is in flash memory.
>>>
>>> But type/tail does ISTR
>>
>> On most of the interesting cases it fails with invalid file organization.
>>
>
> Works very well here.  Of course, I don't use invalid file organizations.
>
> :-)
>

I don't either, but the job controller does, and log files from batch
jobs are about the only reason I would use this feature. If memory
serves this is a well-known problem and somebody (maybe Hein?) had a fix
in the works that never got prioritized.

$ type/tail somelogfile.log
%TYPE-W-OPENIN, error opening DSA0:[SOMEUSER]somelogfile.log;1 as input
-SYSTEM-E-UNSUPPORTED, unsupported operation or function
-RMS-F-ORG, invalid file organization value
$ dir/full somelogfile.log

Directory DSA0:[SOMEUSER]

somelogfile.log;1 File ID: (9158,13,0)
Size: 900/912 Owner: [SOMEGROUP,SOMEUSER]
Created: 15-JUL-2023 08:23:51.37
Modified: 15-JUL-2023 08:23:51.41 (1)
Expires: <None specified>
Backup: <No backup recorded>
Effective: <None specified>
Recording: <None specified>
Accessed: 15-JUL-2023 08:23:51.37
Attr Mod: 15-JUL-2023 08:23:51.41
Data Mod: 15-JUL-2023 08:23:51.37
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 912, Extend: 0, Global buffer count: 0,
No version limit
Record format: VFC, 2 byte header, maximum 0 bytes, longest 1024 bytes
Record attributes: Print file carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

Total of 1 file, 900/912 blocks.

[Jan-Erik Söderholm]

As I have understood this (veruy annoying) thing is that
the file org is not "invalid" as such, it is just not
supported by type/tail...

[Johnny Billquist]

I think you missed the point.

In this case, the program does deal with the different behaviors
depending on under which name it is invoked. It's identical binaries.

The issue is just that you could have had just one copy of the program,
but they installed two. For absolutely zero difference except they use
up more disk space.

Johnny

[Johnny Billquist]

Yes. That is the whole point of busybox. It's just one binary, that have
all commands as built-in. Meaning you are not depending on other
programs first of all being installed, and second, not be corrupted,
third, not being dependent on some dynamic library that you might have.

Busybox is usually involved when you have some embedded system, or for
the recovery tools in case your system has been corrupted or messed up,
so that you can get back to a good state.

Johnny

[Johnny Billquist]

I have a hard time imagining any file system that could not. Such a file
system would then also not be able to append to an existing file.

Johnny

[Johnny Billquist]

Right.

And the secondary problem being that it's a little tricky to find the
last n lines in such a file organization. You'll have to do some
heuristics, and hope you got it right. But that usually is possible to
do with a high chance of success. So it could definitely be done. (I
have such a tool for RSX...)

Johnny

[gah4]

On Saturday, July 15, 2023 at 7:44:36 AM UTC-7, Johnny Billquist wrote:
> On 2023-07-15 03:33, gah4 wrote:

(snip)

> > tail -f reads the end of a file, then waits and checks to see if it got bigger.
> > That is, if the EOF went away.

> > Not all file system, I suspect, can do that.

> I have a hard time imagining any file system that could not. Such a file
> system would then also not be able to append to an existing file.

I suspect all could do it if you close, and reopen the file.

But the Unix ones do it without close/open, and then I found that
also worked on OS/2.

[Johnny Billquist]

Also works on RSX, which basically also means any VMS system/filesystem
shouldn't be a problem.

In general, I'm still of the opinion that any system I can image would
be capable. Opening a file for append do mean that you need to be able
to find where the end of the file is, and seek to there, unless you have
a very specific "open for append" system call.
Otherwise any system would basically open a file for append by opening
it for writing, finding where the eof is, and seek there. Then you could
start writing.

And if you had such a special function, there is still the question of
how seeks then work. If you don't have a seek at all, then sure, you
might not be able to do a tail -f either. But otherwise, what happens if
you seek to beyond eof and try to read? And what if you seek to eof that
you found before, and just try reading again? If someone have added more
to the file, it either would just give eof error, in case the local
program cached where eof is, and just checked against that, or else it
would try to do a read, and then it would get the new data, right?

And if you cache the eof, there is no reason why you could not get it
updated.

But even if you don't have a seek function, the question is what happens
when you reach eof. Reading would return eof error, obviously. And I
would think it would also be true no matter how many times you try.
And then comes the question, what happens if someone did append to the
file in between?

With RSX (and I assume VMS), the eof point is sortof cached, but an
attempt to read beyond it is perfectly possible, and will succeed/fail
based on the actual file content.

Johnny

[Dave Froble]

OpenVMS V7.2 on node DFE90A 15-JUL-2023 18:34:41.36 Uptime 299 05:59:05

$ t/tail netserver.log

Connect request received at 25-OCT-2022 22:09:58.00
from remote process AS800::"0=DFE"
for object "SYS$COMMON:[SYSEXE]FAL.EXE"

--------------------------------------------------------


--------------------------------------------------------

Connect request received at 25-OCT-2022 22:11:51.99
from remote process AS800::"0=DFE"
for object "SYS$COMMON:[SYSEXE]FAL.EXE"

--------------------------------------------------------

DFE job terminated at 25-OCT-2022 22:16:53.76

Accounting information:
Buffered I/O count: 15046 Peak working set size: 490
Direct I/O count: 1867 Peak page file size: 4051
Page faults: 1499 Mounted volumes: 0
Charged CPU time: 0 00:00:17.85 Elapsed time: 0 00:07:49.56
$

NETSERVER.LOG;12 File ID: (4619,103,0)
Size: 4/4 Owner: [DFE]
Created: 25-OCT-2022 22:09:04.26
Revised: 25-OCT-2022 22:16:53.84 (1)

Expires: <None specified>
Backup: <No backup recorded>
Effective: <None specified>
Recording: <None specified>

File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough

File attributes: Allocation: 4, Extend: 0, Global buffer count: 0
No version limit
Record format: VFC, 2 byte header, maximum 0 bytes, longest 80 bytes

Record attributes: Print file carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

Ok, my test was on a VAX, but I doubt that has any bearing. Don't have a clue.
Unless you're trying to read the file while it's still open. That doesn't work.

[terry-...@glaver.org]

On Saturday, July 15, 2023 at 6:43:21 PM UTC-4, Dave Froble wrote:
> Unless you're trying to read the file while it's still open. That doesn't work.

That's kind of the whole point of 'tail -f' 8-}

I suspect all of the pieces are there, but nobody has put them together in
a single utility. For example, $ BACKUP/IGNORE=INTERLOCK doesn't care
if files are open or not. Years ago, I taught VMS C-Kermit about all of the
various permutations of RMS file structure, including a few well-known
cases of intentionally using undefined file types (True BASIC was one of
the offenders). It had to convert every possible file type to either canonical
ASCII, fixed-block binary, or a special format called "labeled"* (which was
invented by fdc and myself just to deal with VMS's myriad file types). The
only issue would seem to be that the tail program won't see a change in
EOF on slowly-growing files until the creating process either has a buffer
flush or closes the file, so the output might be bursty. But that's true on
Unix as well.

* Our test for this was to send a multi-key indexed file with RMS journaling
from VMS to TOPS-20 to a PC to IBM 370 VM/CMS and back to VMS and
having it arrive intact, complete with journal info.

[Dave Froble]

On 7/16/2023 5:25 AM, terry-...@glaver.org wrote:
> On Saturday, July 15, 2023 at 6:43:21 PM UTC-4, Dave Froble wrote:
>> Unless you're trying to read the file while it's still open. That doesn't work.
>
> That's kind of the whole point of 'tail -f' 8-}

RSTS had/has an open mode with "read regardless" as an option. This can also
be done on VMS. But if a utility on VMS doesn't use such a capability, then it
won't happen.

Perhaps the design of the utility figures that since reading only what is on
disk doesn't see anything not yet written, that it's worthless to try.

I've implemented logging in various programs to flush writes to disk
immediately, since I'd usually want to see up to date logging on a current
running program.

It's the design of a utility and user programs that defines whether a capability
is implemented.

[Arne Vajhøj]

On 7/15/2023 10:47 AM, Johnny Billquist wrote:
> And the secondary problem being that it's a little tricky to find the
> last n lines in such a file organization. You'll have to do some
> heuristics, and hope you got it right. But that usually is possible to
> do with a high chance of success. So it could definitely be done. (I
> have such a tool for RSX...)

It depends on the record format.

VAR,VFC => read from start or make a good guess
FIX, STM, STMLF, STMCR => no problem
UDF => "last n lines" is not defined

Arne

[Arne Vajhøj]

I got that.

> The issue is just that you could have had just one copy of the program,
> but they installed two. For absolutely zero difference except they use
> up more disk space.

The two cost are hardware and peoples time.

With todays TB disks it is very unlikely that having two copies of some
executables will require buying bigger and more expensive disk(s).

Having developer support the name test logic on multiple platforms
and having sys admins needing to worry about linked stuff
when they clean up is a very small effort but still an
effort greater than zero.

Arne

[Arne Vajhøj]

On 7/16/2023 2:28 PM, Arne Vajhøj wrote:
> The two cost are hardware and peoples time.
>
> With todays TB disks it is very unlikely that having two copies of some
> executables will require buying bigger and more expensive disk(s).
>
> Having developer support the name test logic on multiple platforms
> and having sys admins needing to worry about linked stuff
> when they clean up is a very small effort but still an
> effort greater than zero.

And just to make sure we are talking about the same.

VMS example:

$ type a.c
#include <stdio.h>

int main(int argc, char *argv[])
{
printf("I am A\n");
return 0;
}
$ cc a
$ link a
$ type b.c
#include <stdio.h>

int main(int argc, char *argv[])
{
printf("I am B\n");
return 0;
}
$ cc b
$ link b
$ run a
I am A
$ run b
I am B

$ del a.exe;*, b.exe;*
$ type ab.c
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[])
{
if(strstr(argv[0], "]a.exe;") != NULL)
{
printf("I am A\n");
}
if(strstr(argv[0], "]b.exe;") != NULL)
{
printf("I am B\n");
}
return 0;
}
$ cc ab
$ link ab
$ set file/enter=a.exe ab.exe
$ set file/enter=b.exe ab.exe
$ run a
I am A
$ run b
I am B
$ set file/remove a.exe;1
$ set file/remove b.exe;1

Arne

[gah4]

On Sunday, July 16, 2023 at 11:30:26 AM UTC-7, Arne Vajhøj wrote:

(snip)

> #include <stdio.h>
> #include <string.h>
>
> int main(int argc, char *argv[])
> {
> if(strstr(argv[0], "]a.exe;") != NULL)
> {
> printf("I am A\n");
> }
> if(strstr(argv[0], "]b.exe;") != NULL)
> {
> printf("I am B\n");
> }
> return 0;
> }

I suspect you need a slightly better way to recognize the program name.

I am not so sure what VMS passes for argv[0] in all cases.

For extra fun, you can use the Command Definition Utility.

I remember TeX does fun things with the command name and arguments.
You need CDU to make them work right.

[Johnny Billquist]

On 2023-07-16 20:30, Arne Vajhøj wrote:
> On 7/16/2023 2:28 PM, Arne Vajhøj wrote:
>> The two cost are hardware and peoples time.
>>
>> With todays TB disks it is very unlikely that having two copies of some
>> executables will require buying bigger and more expensive disk(s).
>>
>> Having developer support the name test logic on multiple platforms
>> and having sys admins needing to worry about linked stuff
>> when they clean up is a very small effort but still an
>> effort greater than zero.
>
> And just to make sure we are talking about the same.

[...]

Yes. And my point is that this detail is there no matter what. So you
didn't "save" anything by having two binaries. Since it's actually just
one binary, and how it behaves is based on which name it was invoked as.

Yes, you *could* have different binaries, with the different behaviors,
and not have to deal with all this.

But that was not what was going on here. The binary we are talking about
still have this logic you are hinting at. It's just that in addition to
that engineering work, which seems to be your concern, it also uses up
twice the space (or even more, depending on how many copies of the
program you happen to install).

I feel like you didn't get the point that the decision on what the
program should do based on under what name it was invoked was not absent
with these multiple copies... It is still there, and still made use of.

Sure disk space is cheap and all that. But in this case, it's really
questionable why not have a hard link. Even at a fraction of a cents
difference in cost, it's still not zero, and bad behavior should really
not be propagated. Sooner or later a small problem becomes a bigger problem.

Johnny

[Johnny Billquist]

On 2023-07-16 20:09, Arne Vajhøj wrote:
> On 7/15/2023 10:47 AM, Johnny Billquist wrote:
>> And the secondary problem being that it's a little tricky to find the
>> last n lines in such a file organization. You'll have to do some
>> heuristics, and hope you got it right. But that usually is possible to
>> do with a high chance of success. So it could definitely be done. (I
>> have such a tool for RSX...)
>
> It depends on the record format.

Right. But in this case, the record format was given.

> VAR,VFC => read from start or make a good guess
> FIX, STM, STMLF, STMCR => no problem
> UDF => "last n lines" is not defined

And in this case it was VFC. And as I said - you can usually make a good
guess.

Johnny

[Johnny Billquist]

On 2023-07-16 00:42, Dave Froble wrote:

> Ok, my test was on a VAX, but I doubt that has any bearing.  Don't have
> a clue. Unless you're trying to read the file while it's still open.
> That doesn't work.

Why wouldn't it work on an open file? As long as you are allowed shared
reading it should be just fine.

Johnny

[Jan-Erik Söderholm]

But, "read from start" is the same as TYPE without /TAIL. And that is
what I do when I get that error from TYPE/TAIL. Just TYPE the log file
and let it scroll to the end, the net result is the same as when using
/tail, the last 24 (or so) lines are on the screen. And besides, more
lines are available based on the size of my scroll-back buffer in the
emulator, usually 20000 or 200000 lines.

Jan-Erik.

[Arne Vajhøj]

Ah.

You are saying that they ditched the linking but kept the code
that made linking work.

As if in my example had done:

$ cc ab
$ link/exe=a ab
$ link/exe=b ab

That is silly in my opinion. They should have gotten rid of that code.

Simple is good - complex is bad.

Arne

[Dave Froble]

I'm not familiar with the TYPE utility.

I am a bit familiar with RMS, the DLM, and usage.

My guess is that the utility doesn't use any "read regardless" capability. So
"allowed shared reading" may not be in use. Just a guess.

[Johnny Billquist]

On 2023-07-17 15:30, Dave Froble wrote:
> On 7/17/2023 6:55 AM, Johnny Billquist wrote:
>> On 2023-07-16 00:42, Dave Froble wrote:
>>
>>> Ok, my test was on a VAX, but I doubt that has any bearing.  Don't
>>> have a
>>> clue. Unless you're trying to read the file while it's still open.  That
>>> doesn't work.
>>
>> Why wouldn't it work on an open file? As long as you are allowed
>> shared reading
>> it should be just fine.
>>
>>   Johnny
>>
>
> I'm not familiar with the TYPE utility.

Oh. You meand it more specifically in the context of that
program/command. Ok, fair enough. I wouldn't know about TYPE limitations
myself either. But as a general thing about files in ODS5, you can
certainly read them while they are open.

> I am a bit familiar with RMS, the DLM, and usage.
>
> My guess is that the utility doesn't use any "read regardless"
> capability.  So "allowed shared reading" may not be in use.  Just a guess.

Your guess is as good as mine on that specific question.

Under RSX, TYPE is under the hood using PIP. And PIP have a switch /SR
for shared reading. That switch don't seem to be exposed under DCL.
But with MCR, you can certainly type a file that is open.

Johnny

[Johnny Billquist]

*Yes*

That was the whole point. And it "works" for them even without the
hardlinks. It's just a question of the program inspecting what it was
invoked as.
Under Unix, every file is always a hardlink. It's just a question of if
you have just one, or multiple ones...

> As if in my example had done:
>
> $ cc ab
> $ link/exe=a ab
> $ link/exe=b ab
>
> That is silly in my opinion. They should have gotten rid of that code.

Agreed. And that is where they were being really silly, and I felt you
missed this.

And it could have been done even simpler:

$ cc ab
$ link/exe=a ab

$ copy a.exe b.exe

Which most probably is what was actually done... Instead of creating a
hard link for b.exe...

> Simple is good - complex is bad.

Always.

Johnny

[Johnny Billquist]

Notice the "or" keyword in there... ;-)

The "make a good guess" is basically about jumping close to the end of
the file, and trying to identify the VFC lenght fields, and from that
extract the actual lines, and show the last n of them.
Which usually can be done with a fair amount of confidence, since for
text files, lines are seldom extremely long, nor do they contain random
control characters. Thus, if you just scan for control characters
(probably NUL) on an odd byte, this is most likely the second byte of a
record length, and you have your line starting there. And then
everything else follows trivially from that.

No need to read through the whole file. But it requires that you
actually read the full disk blocks of content, and not the pre-processed
data coming out of RMS.

Johnny

[Arne Vajhøj]

On 7/17/2023 10:32 AM, Johnny Billquist wrote:
> On 2023-07-17 15:29, Arne Vajhøj wrote:
>> You are saying that they ditched the linking but kept the code
>> that made linking work.
>
> *Yes*

>> That is silly in my opinion. They should have gotten rid of that code.
>
> Agreed. And that is where they were being really silly, and I felt you
> missed this.

I did!

Arne

[Arne Vajhøj]

On 7/17/2023 12:37 AM, gah4 wrote:
> On Sunday, July 16, 2023 at 11:30:26 AM UTC-7, Arne Vajhøj wrote:
>> #include <stdio.h>
>> #include <string.h>
>>
>> int main(int argc, char *argv[])
>> {
>> if(strstr(argv[0], "]a.exe;") != NULL)
>> {
>> printf("I am A\n");
>> }
>> if(strstr(argv[0], "]b.exe;") != NULL)
>> {
>> printf("I am B\n");
>> }
>> return 0;
>> }
>
> I suspect you need a slightly better way to recognize the program name.
>
> I am not so sure what VMS passes for argv[0] in all cases.

Neither am I.

It just worked for my case.

I will certainly not work on other OS.

But that is the point. It is a complication.

Arne

[Dave Froble]

On 7/17/2023 9:29 AM, Arne Vajhøj wrote:

> Simple is good - complex is bad.

I'm a firm believer in KISS ...

[Robert A. Brooks]

$ help type /tail

TYPE

/TAIL

/TAIL[=n]

Displays the last several lines of a log file. The value, n,
defaults to p-2 where p is the current terminal page length. You
can use TYPE/TAIL only if all of the following criteria are true:

o File organization is sequential.

o The longest record is less than 512 bytes.

o The record format is either VAR, VFC, STM, STRCM or STMLF
(for more information, see the description of FAB$B_RFM in the
OpenVMS Record Management Services Reference Manual).

o The file being typed is on a device that supports random
access. The TYPE/TAIL command does not work on magnetic tape
drives.

Even with this criteria, some file conditions cannot be
anticipated and may not allow display of the last several lines
of a log file, resulting in the following error message:

%TYPE-W-READERR, error reading DEVICE:[DIRECTORY]FILE.EXT;1

-SYSTEM-E-UNSUPPORTED, unsupported operation or function

--
-- Rob

[Simon Clubley]

On 2023-07-15, Chris Townley <ne...@cct-net.co.uk> wrote:
> On 15/07/2023 02:33, gah4 wrote:

>> On Friday, July 14, 2023 at 4:36:41?PM UTC-7, Chris Townley wrote:
>>
>> (snip, I wrote)
>>

>>>> Reminds me of, years ago, porting gnu-utils to OS/2 and finding that
>>>> tail -f works. I don't know if tail -f is supposed to work on VMS, though.
>>

>> tail -f reads the end of a file, then waits and checks to see if it got bigger.
>> That is, if the EOF went away.
>>
>> Not all file system, I suspect, can do that.
>>
>

> Sorry, I should have said TYPE/TAIL/CONTINUOUS which works with some
> caveats on the file
>

Does that work these days when the file is still open for write by
another user ?

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.

[Simon Clubley]

On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>
>> Simple is good - complex is bad.
>
> I'm a firm believer in KISS ...
>

Then you should strongly prefer Unix over VMS... :-)

[Dave Froble]

On 7/18/2023 8:21 AM, Simon Clubley wrote:
> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>
>>> Simple is good - complex is bad.
>>
>> I'm a firm believer in KISS ...
>>
>
> Then you should strongly prefer Unix over VMS... :-)
>
> Simon.
>

You've totally lost your mind!

[Johnny Billquist]

On 2023-07-18 14:21, Simon Clubley wrote:
> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>
>>> Simple is good - complex is bad.
>>
>> I'm a firm believer in KISS ...
>>
>
> Then you should strongly prefer Unix over VMS... :-)

Depends on which corner you look at.

Asynchronous I/O in Unix is a mess.

Johnny

[Johnny Billquist]

On 2023-07-18 14:17, Simon Clubley wrote:
> On 2023-07-15, Chris Townley <ne...@cct-net.co.uk> wrote:
>> On 15/07/2023 02:33, gah4 wrote:
>>> On Friday, July 14, 2023 at 4:36:41?PM UTC-7, Chris Townley wrote:
>>>
>>> (snip, I wrote)
>>>
>>>>> Reminds me of, years ago, porting gnu-utils to OS/2 and finding that
>>>>> tail -f works. I don't know if tail -f is supposed to work on VMS, though.
>>>
>>> tail -f reads the end of a file, then waits and checks to see if it got bigger.
>>> That is, if the EOF went away.
>>>
>>> Not all file system, I suspect, can do that.
>>>
>>
>> Sorry, I should have said TYPE/TAIL/CONTINUOUS which works with some
>> caveats on the file
>>
>
> Does that work these days when the file is still open for write by
> another user ?

Not sure why you think that would be an unsurmountable problem...

Johnny

[Single Stage to Orbit]

On Tue, 2023-07-18 at 16:05 +0200, Johnny Billquist wrote:
> Asynchronous I/O in Unix is a mess.

THey've had three goes at getting it right. There was AIO years back,
something else that I forget the name of, and then there's io-uring,
the latest thing to land. From what I've gathered it can do a lot
without incurring context switches penalty.
--
Tactical Nuclear Kittens

[Arne Vajhøj]

On 7/18/2023 8:21 AM, Simon Clubley wrote:

> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>
>>> Simple is good - complex is bad.
>>
>> I'm a firm believer in KISS ...
>
> Then you should strongly prefer Unix over VMS... :-)

I am not so sure about that.

VMS is probably one of the smallest and leanest commercial
OS'es today.

It may be more a result of lack of investment for 20 years
than a deliberate strategy. And we we miss a lot of things.
But still small code base and small API.

I am pretty sure that if VMS kernel had most Macro-32 and
Bliss code rewritten to C, then VMS kernel would be
smaller than just systemd. :-)

Arne

[Simon Clubley]

On 2023-07-18, Dave Froble <da...@tsoft-inc.com> wrote:
> On 7/18/2023 8:21 AM, Simon Clubley wrote:
>> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>>
>>>> Simple is good - complex is bad.
>>>
>>> I'm a firm believer in KISS ...
>>>
>>
>> Then you should strongly prefer Unix over VMS... :-)
>>
>

> You've totally lost your mind!
>

Why ? It's a lot easier to write applications with modern requirements
under Unix, especially Linux, than it is under VMS.

[Simon Clubley]

At what point did I say that ?

I'm interested if you can use TYPE/TAIL/CONTINUOUS on files open for
write these days.

[Simon Clubley]

On 2023-07-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
> On 7/18/2023 8:21 AM, Simon Clubley wrote:
>> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>>
>>>> Simple is good - complex is bad.
>>>
>>> I'm a firm believer in KISS ...
>>
>> Then you should strongly prefer Unix over VMS... :-)
>
> I am not so sure about that.
>
> VMS is probably one of the smallest and leanest commercial
> OS'es today.
>
> It may be more a result of lack of investment for 20 years
> than a deliberate strategy. And we we miss a lot of things.
> But still small code base and small API.
>

How does that API compare to the APIs available on Unix (especially
Linux) when you want to write an application (especially one with
modern requirements) ?

VMS had a world-leading API 20-30 years ago, but things (and expectations)
have moved on since then.

[Arne Vajhøj]

On 7/18/2023 1:31 PM, Simon Clubley wrote:
> On 2023-07-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>> On 7/18/2023 8:21 AM, Simon Clubley wrote:
>>> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>>>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>>>
>>>>> Simple is good - complex is bad.
>>>>
>>>> I'm a firm believer in KISS ...
>>>
>>> Then you should strongly prefer Unix over VMS... :-)
>>
>> I am not so sure about that.
>>
>> VMS is probably one of the smallest and leanest commercial
>> OS'es today.
>>
>> It may be more a result of lack of investment for 20 years
>> than a deliberate strategy. And we we miss a lot of things.
>> But still small code base and small API.
>
> How does that API compare to the APIs available on Unix (especially
> Linux) when you want to write an application (especially one with
> modern requirements) ?
>
> VMS had a world-leading API 20-30 years ago, but things (and expectations)
> have moved on since then.

????

The third letter in KISS stands for simple not for feature-rich.

Linux as of today is definitely feature rich but it is not
simple.

Arne

[Simon Clubley]

On 2023-07-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
> On 7/18/2023 1:31 PM, Simon Clubley wrote:
>> On 2023-07-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>>> On 7/18/2023 8:21 AM, Simon Clubley wrote:
>>>> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>>>>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>>>>
>>>>>> Simple is good - complex is bad.
>>>>>
>>>>> I'm a firm believer in KISS ...
>>>>
>>>> Then you should strongly prefer Unix over VMS... :-)
>>>
>>> I am not so sure about that.
>>>
>>> VMS is probably one of the smallest and leanest commercial
>>> OS'es today.
>>>
>>> It may be more a result of lack of investment for 20 years
>>> than a deliberate strategy. And we we miss a lot of things.
>>> But still small code base and small API.
>>
>> How does that API compare to the APIs available on Unix (especially
>> Linux) when you want to write an application (especially one with
>> modern requirements) ?
>>
>> VMS had a world-leading API 20-30 years ago, but things (and expectations)
>> have moved on since then.
>
> ????
>
> The third letter in KISS stands for simple not for feature-rich.
>

I know that Arne, and that is _exactly_ my point.

> Linux as of today is definitely feature rich but it is not
> simple.
>

It's a hell of a lot simpler to implement an application on Linux
than it is to implement that application on VMS.

Linux gives you functionality that is expected these days, but is
missing from VMS. That's what I am primarily referring to above.

For one example, I refer you to Stephen's often posted diatribe about
the effort involved in writing a secure network application on VMS
versus doing the same thing on Linux.

[Johnny Billquist]

Well, context switching is yet another story. Shouldn't really have
anything to do with doing asynchronous I/O...

But yeah... AIO. Oh what fun. And at the moment, there is a lot of focus
on epoll, but that only exists in Linux. Of course, this also leads to
the comment that you need to have multiple solutions in the code to deal
with the different variants of Unix... epoll is sortof nice, since it
gathers a lot of file descriptors inside another single file descriptor,
and you can then do blocking operations on that one fd, and from there
work out what to do when anything happens. And it can be made rather
efficient, especially when you are doing lots of I/O to lots of file
descriptors. But it's also a bit complicated to actually work with if
you want to get the most out of it, and easy to get into a situation
where it gets stuck instead.

Another area where Unix/Linux is pretty complicated is shared libraries.
It works, but don't try to explain how it actually works. It's
definitely on the complicated side... (along with the dynamic loaded
libraries...)

In general, there is no one OS where everything is nice and simple
(except RSX of course... ;-) )

Johnny

[Arne Vajhøj]

On 7/18/2023 1:55 PM, Simon Clubley wrote:
> On 2023-07-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>> On 7/18/2023 1:31 PM, Simon Clubley wrote:
>>> On 2023-07-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>>>> On 7/18/2023 8:21 AM, Simon Clubley wrote:
>>>>> On 2023-07-17, Dave Froble <da...@tsoft-inc.com> wrote:
>>>>>> On 7/17/2023 9:29 AM, Arne Vajhøj wrote:
>>>>>>
>>>>>>> Simple is good - complex is bad.
>>>>>>
>>>>>> I'm a firm believer in KISS ...
>>>>>
>>>>> Then you should strongly prefer Unix over VMS... :-)
>>>>
>>>> I am not so sure about that.
>>>>
>>>> VMS is probably one of the smallest and leanest commercial
>>>> OS'es today.
>>>>
>>>> It may be more a result of lack of investment for 20 years
>>>> than a deliberate strategy. And we we miss a lot of things.
>>>> But still small code base and small API.
>>>
>>> How does that API compare to the APIs available on Unix (especially
>>> Linux) when you want to write an application (especially one with
>>> modern requirements) ?
>>>
>>> VMS had a world-leading API 20-30 years ago, but things (and expectations)
>>> have moved on since then.
>>
>> ????
>>
>> The third letter in KISS stands for simple not for feature-rich.
>
> I know that Arne, and that is _exactly_ my point.

Well - Linux is not simple.

>> Linux as of today is definitely feature rich but it is not
>> simple.
>
> It's a hell of a lot simpler to implement an application on Linux
> than it is to implement that application on VMS.
>
> Linux gives you functionality that is expected these days, but is
> missing from VMS. That's what I am primarily referring to above.

Linux is not simpler than VMS. Linux is more feature rich than VMS.

In theory that would mean than an application on Linux is
simpler than an application on VMS. Because given same functionality
then the OS doing more means the application has to do less.

But practice is more that an application on Linux has more features
than on VMS and it is more complex because of the code needed
to use all those Linux features.

At least I do not recall ever seeing all those simple Linux
applications.

Latest Debian distro is over 1.3 BLOC.

> For one example, I refer you to Stephen's often posted diatribe about
> the effort involved in writing a secure network application on VMS
> versus doing the same thing on Linux.

The basic application API's are the same. C/C++ applications use OpenSSL
(or LibreSSL or GnuTLS). Java application use the builtin JCE. PHP
applications get it via Apache and OpenSSL. Python has a builtin
(that is a wrapper around OpenSSL).

Linux got more tools on top of the basic.

But having more tools is not what I call simple.

Arne

[tridac]

Develop under FreeBSD here, but have spent a few days recently
evaluating current Linux distros, one of which is Debian. If anything,
the mass adoption of systemd has a been a disaster in terms of
system complexity, which only increases the possible attack surface
width. If you hold the nose, Debian Bookworm is pretty good at user
level but so is Devuan, the systemd free Debian, which seems every
bit as good. Also looked at Suse, like meeting an old friend, but again,
systemd encumbered and nothing is where one expects it to be. Still
have 11.4 on a couple of old laptops, which is how unix should be.
Fedora produced rubbish on screen, while Xunbuntu works out of the
box, just like Debian and Suse.

I need to build and test some code under Linux, as FreeBSD is not so
mainstream, though a thoroughly more straightforward and hassle free
experience than any current Linux. Always saw systemd as a power grab
by Redhat and their most recent antics over source licensing only
serve to reinforce that view. All about the money, as usual, sadly...

Chris

[Dave Froble]

I have to wonder why you think re-writing in C would be "smaller" (whatever that
is) than what's there today?

[Dave Froble]

First, define reasonably well "secure network application".

[Arne Vajhøj]

On 7/18/2023 8:34 PM, Dave Froble wrote:
> On 7/18/2023 1:13 PM, Arne Vajhøj wrote:
>> I am pretty sure that if VMS kernel had most Macro-32 and
>> Bliss code rewritten to C, then VMS kernel would be
>> smaller than just systemd. :-)
>

> I have to wonder why you think re-writing in C would be "smaller"
> (whatever that is) than what's there today?

Same functionality implemented in different languages
result in a different number of lines of code.

Usually measured in LoC/FP.

C is fewer lines than Macro-32.

I am not sure that Bliss to C will really reduce LoC that much - it
was just bundled in with Macro-32.

VMS Basic is fewer lines than C. :-)

Usually it is rather irrelevant, but when comparing complexity
of two code bases written in different languages it can be
relevant.

Arne

[Dave Froble]

On 7/18/2023 8:47 PM, Arne Vajhøj wrote:
> On 7/18/2023 8:34 PM, Dave Froble wrote:
>> On 7/18/2023 1:13 PM, Arne Vajhøj wrote:
>>> I am pretty sure that if VMS kernel had most Macro-32 and
>>> Bliss code rewritten to C, then VMS kernel would be
>>> smaller than just systemd. :-)
>>
>> I have to wonder why you think re-writing in C would be "smaller" (whatever
>> that is) than what's there today?
>
> Same functionality implemented in different languages
> result in a different number of lines of code.

So, you're talking about source files?

Do you also claim that the executable code would also be smaller?

> Usually measured in LoC/FP.
>
> C is fewer lines than Macro-32.
>
> I am not sure that Bliss to C will really reduce LoC that much - it
> was just bundled in with Macro-32.
>
> VMS Basic is fewer lines than C. :-)
>
> Usually it is rather irrelevant, but when comparing complexity
> of two code bases written in different languages it can be
> relevant.
>
> Arne
>
>
>

[Arne Vajhøj]

On 7/18/2023 9:25 PM, Dave Froble wrote:
> On 7/18/2023 8:47 PM, Arne Vajhøj wrote:
>> On 7/18/2023 8:34 PM, Dave Froble wrote:
>>> On 7/18/2023 1:13 PM, Arne Vajhøj wrote:
>>>> I am pretty sure that if VMS kernel had most Macro-32 and
>>>> Bliss code rewritten to C, then VMS kernel would be
>>>> smaller than just systemd. :-)
>>>
>>> I have to wonder why you think re-writing in C would be "smaller"
>>> (whatever
>>> that is) than what's there today?
>>
>> Same functionality implemented in different languages
>> result in a different number of lines of code.
>
> So, you're talking about source files?

Yes.

> Do you also claim that the executable code would also be smaller?

No.

Arne

[Dave Froble]

Then, what would be the benefit?

For example:

Stat% = SYS$QIOW( , ! Event flag &
Ch% By Value, ! VMS channel &
IO$_ACPCONTROL By Value,! Operation &
IOSB::Stat%, ! I/O status block &
, ! AST routine &
, ! AST parameter &
Temp% By Desc, ! P1 sub-func code &
URL$, ! P2 URL string &
RetLen%, ! P3-return len &
IP$, ! P4-output string &
, ! P5 &
) ! P6

The above is rather easy to understand, from some perspectives. Would it be
"smaller" if strung out on one line? Sure. Would it be a bit harder to
understand? Most definitely. Prone to mistakes also.

Compilers, assemblers, and such ignore the "white space", so it really doesn't
matter to the executable. However, the white space and comments can make code
easier to understand, and avoid mistakes.

So, what's the issue with larger source files?

[Simon Clubley]

On 2023-07-19, Dave Froble <da...@tsoft-inc.com> wrote:
> On 7/18/2023 10:01 PM, Arne Vajhøj wrote:
>> On 7/18/2023 9:25 PM, Dave Froble wrote:
>>> On 7/18/2023 8:47 PM, Arne Vajhøj wrote:
>>>> On 7/18/2023 8:34 PM, Dave Froble wrote:
>>>>> On 7/18/2023 1:13 PM, Arne Vajhøj wrote:
>>>>>> I am pretty sure that if VMS kernel had most Macro-32 and
>>>>>> Bliss code rewritten to C, then VMS kernel would be
>>>>>> smaller than just systemd. :-)

David, please read this again. Arne is talking about the VMS kernel
being smaller than it currently is. I have no way to compare it to
systemd however. :-)

>>>>>
>>>>> I have to wonder why you think re-writing in C would be "smaller" (whatever
>>>>> that is) than what's there today?
>>>>
>>>> Same functionality implemented in different languages
>>>> result in a different number of lines of code.
>>>
>>> So, you're talking about source files?
>>
>> Yes.
>>
>>> Do you also claim that the executable code would also be smaller?
>>
>> No.
>>
>

> Then, what would be the benefit?
>

For one thing, if you didn't have to worry about the Macro-32 and
Bliss crap, the VMS port would have been completed years ago.

It's also a hell of a lot easier, quicker, and more robust, to
write something in C than it is in Macro-32 or Bliss.

There are also many languages in turn that have the same advantage
over C, but that doesn't change the above.

> For example:
>
> Stat% = SYS$QIOW( , ! Event flag &
> Ch% By Value, ! VMS channel &
> IO$_ACPCONTROL By Value,! Operation &
> IOSB::Stat%, ! I/O status block &
> , ! AST routine &
> , ! AST parameter &
> Temp% By Desc, ! P1 sub-func code &
> URL$, ! P2 URL string &
> RetLen%, ! P3-return len &
> IP$, ! P4-output string &
> , ! P5 &
> ) ! P6
>
> The above is rather easy to understand, from some perspectives. Would it be
> "smaller" if strung out on one line? Sure. Would it be a bit harder to
> understand? Most definitely. Prone to mistakes also.
>

Hmmm David, at what point above did Arne talk about replacing Basic code
with C code ? He's talking about replacing lower-level language code
with higher-level C code.

> Compilers, assemblers, and such ignore the "white space", so it really doesn't
> matter to the executable. However, the white space and comments can make code
> easier to understand, and avoid mistakes.
>
> So, what's the issue with larger source files?
>

It's not the larger source files. It's the very painful architecture
specific coding and lower levels of abstraction that Macro-32 and Bliss
bring to the table.

[Dave Froble]

Simon, do you need some "reading lessons"?

From above:

>>>> So, you're talking about source files?
>>>
>>> Yes.
>>>
>>>> Do you also claim that the executable code would also be smaller?
>>>
>>> No.

So, Arne is talking about the size of the source code. And that is what I
responded to. My example was showing the benefits of readability vs lines of
source code. Regardless of language.

[Arne Vajhøj]

On 7/19/2023 12:03 PM, Dave Froble wrote:
> On 7/18/2023 10:01 PM, Arne Vajhøj wrote:
>> On 7/18/2023 9:25 PM, Dave Froble wrote:
>>> On 7/18/2023 8:47 PM, Arne Vajhøj wrote:
>>>> On 7/18/2023 8:34 PM, Dave Froble wrote:
>>>>> On 7/18/2023 1:13 PM, Arne Vajhøj wrote:
>>>>>> I am pretty sure that if VMS kernel had most Macro-32 and
>>>>>> Bliss code rewritten to C, then VMS kernel would be
>>>>>> smaller than just systemd. :-)
>>>>>
>>>>> I have to wonder why you think re-writing in C would be "smaller"
>>>>> (whatever
>>>>> that is) than what's there today?
>>>>
>>>> Same functionality implemented in different languages
>>>> result in a different number of lines of code.
>>>
>>> So, you're talking about source files?
>>
>> Yes.
>>
>>> Do you also claim that the executable code would also be smaller?
>>
>> No.
>

> Then, what would be the benefit?

I am not talking about benefits.

I am talking about how to compare complexity (functionality)
of software written in different languages.

X1 lines of C can be more complex than X2 lines of Macro-32
even though X1 < X2. Because if the Macro-32 was
rewritten to X3 lines of C then it could happen that X3 < X1.

> For example:
>
>         Stat% = SYS$QIOW(       ,                       !  Event flag &
>                                 Ch% By Value,           !  VMS channel &
>                                 IO$_ACPCONTROL By Value,!  Operation &
>                                 IOSB::Stat%,            !  I/O status
> block &
>                                 ,                       !  AST routine &
>                                 ,                       !  AST parameter &
>                                 Temp% By Desc,          !  P1 sub-func
> code &
>                                 URL$,                   !  P2 URL string &
>                                 RetLen%,                !  P3-return len &
>                                 IP$,                    !  P4-output
> string &
>                                 ,                       !  P5 &
>                                 )                       !  P6
>
> The above is rather easy to understand, from some perspectives.  Would
> it be "smaller" if strung out on one line?  Sure.  Would it be a bit
> harder to understand?  Most definitely.  Prone to mistakes also.
>
> Compilers, assemblers, and such ignore the "white space", so it really
> doesn't matter to the executable.  However, the white space and comments
> can make code easier to understand, and avoid mistakes.

The issue is well known.

Ideally the measurement should be SLOC which is counting "statements",
which is independent of formatting.

But it is easier to count plain LOC.

And the assumption is that when looking at millions of lines
of code by thousands of different developers, then SLOC is
proportional to LOC.

> So, what's the issue with larger source files?

The point was comparison of functionality.

Re the simple vs complex point.

Arne

[Arne Vajhøj]

On 7/19/2023 1:32 PM, Simon Clubley wrote:

> On 2023-07-19, Dave Froble <da...@tsoft-inc.com> wrote:
>> On 7/18/2023 10:01 PM, Arne Vajhøj wrote:
>>> On 7/18/2023 9:25 PM, Dave Froble wrote:
>>>> Do you also claim that the executable code would also be smaller?
>>>
>>> No.
>>
>> Then, what would be the benefit?
>
> For one thing, if you didn't have to worry about the Macro-32 and
> Bliss crap, the VMS port would have been completed years ago.

Are you sure?

It is not my impression that the various VMS ISA migrations has
caused rewrite of lots of Macro-32 and Bliss.

My impression is that:
- they create the Macro-32 and Bliss compilers for the new platform
- they compile the old Macro-32 and Bliss code dating back from the
70's and 80's
- the new code get written in C (plus a little bit of native
assembler where needed)

> It's also a hell of a lot easier, quicker, and more robust, to
> write something in C than it is in Macro-32 or Bliss.

I think they do write the new stuff in C.

Obviously if they have a need to read and understand some
code then reading C will be something like a factor 10
faster than reading Macro-32.

> It's not the larger source files. It's the very painful architecture
> specific coding and lower levels of abstraction that Macro-32 and Bliss
> bring to the table.

They are not architecture specific as as the same code
run on many platforms.

But Bliss is low level and Macro-32 is very low level. The same
Macro-32 code does not become high level just because it goes through
a compiler instead of an assembler.

Arne

[John Reagan]

You can keep saying that but won't make it true. Yes, we had to make 3 cross-compilers
to do the port. Even if the whole OS was written in DEC C with all the extensions in the code
base, we would have still had to make some sort of GEM-to-LLVM converter to get the DECC
compiler hooked to LLVM. (And Macro-32 doesn't even use that GEM-to-LLVM converter).
It would have save person years of work as we were working on XMACRO in parallel with
the G2L code for BLISS and C.

As for "easier, quick, robust", that's a matter of skill. The real benefits
of C over BLISS/MACRO include "floating point" and provided I/O package.

[Simon Clubley]

On 2023-07-19, Arne Vajhøj <ar...@vajhoej.dk> wrote:
> On 7/19/2023 1:32 PM, Simon Clubley wrote:
>> On 2023-07-19, Dave Froble <da...@tsoft-inc.com> wrote:
>>> On 7/18/2023 10:01 PM, Arne Vajhøj wrote:
>>>> On 7/18/2023 9:25 PM, Dave Froble wrote:
>>>>> Do you also claim that the executable code would also be smaller?
>>>>
>>>> No.
>>>
>>> Then, what would be the benefit?
>>
>> For one thing, if you didn't have to worry about the Macro-32 and
>> Bliss crap, the VMS port would have been completed years ago.
>
> Are you sure?
>
> It is not my impression that the various VMS ISA migrations has
> caused rewrite of lots of Macro-32 and Bliss.
>
> My impression is that:
> - they create the Macro-32 and Bliss compilers for the new platform
> - they compile the old Macro-32 and Bliss code dating back from the
> 70's and 80's
> - the new code get written in C (plus a little bit of native
> assembler where needed)
>

You have missed what I am saying above, so I may have been too subtle.
Let me reword it: If VMS didn't have any Macro-32 or Bliss code in it,
and didn't need to support them as application level programming languages,
VMS would look much more internally like any another OS written in C does,
and the port would have been completed years ago.

As of this month, the VMS port has been going on for 9 years and it is
still not finished.

It doesn't take 9 years to port Linux to a brand-new architecture, even
if you first have to implement the compilers for that brand-new architecture.

[Simon Clubley]

On 2023-07-19, John Reagan <xyzz...@gmail.com> wrote:

I keep looking at why the port is currently at 9 years and counting
compared to how long it takes to port Linux to a brand-new architecture
and keep wondering why it is taking so long.

There are some things you have had to implement, such as KESU emulation,
that Linux doesn't have to deal with, but I have a hard time seeing why
that would take more than a few months to design and implement.

I get the impression that you have had to spend a lot of effort to
duplicate low-level behaviour in a way that simply isn't a concern
with other operating systems due to the need, especially, to support
Macro-32 and ensure that code written at this low-level, and which
uses the various idioms as a result, continues to work.

> As for "easier, quick, robust", that's a matter of skill. The real benefits
> of C over BLISS/MACRO include "floating point" and provided I/O package.

No, the real benefits of C include the ability to use abstractions in
your code which make the code a lot quicker to write, where you can
focus much more on what you want, instead of having to lay out in
minute detail how to do it, and as a result your code also becomes
much more reusable and portable.

You can design complex data structures much more easily and get much
more help from the compiler in finding errors in your code.

For one really simple example, in C, you ask for a pointer and let the
compiler implement the pointer behind the scenes. Ie: you write:

char *ptr;

instead of:

uint32_t ptr;

In C, your normal application code works the same, regardless of whether
that pointer is 32-bits or 64-bits. If the lowest-level language in VMS
was C, this is what the RMS interface would conceptually look like:

struct {whatever}_rms_rab_t *rab;

instead of the current setup. Likewise for itemlists and other structures.

Things which are currently highly visible to the source code become
abstracted implementation details that normal source code simply doesn't
care about or have to deal with.

[Dave Froble]

It is obvious to the casual observer that some assembly language "tricks" would
make an assembly language compiler a bit (or more) tricky. Supporting Macro-32,
a basically 32 bit language (these days), is a significant task. Yeah, yeah, I
know, Macro-32 can do 64 bit stuff. However, it is not just for VMS, but also
for all those applications written in Macro-32 that are still in use. Now, I'm
rather sure Simon feels they should be re-implimented in another language. I
have to wonder whether the users of those apps feel the same way. I doubt it.
Nice of Simon to assign significant work to others.

Similar observations for Bliss.

> As of this month, the VMS port has been going on for 9 years and it is
> still not finished.

Better than abandoned ...

> It doesn't take 9 years to port Linux to a brand-new architecture, even
> if you first have to implement the compilers for that brand-new architecture.

Not the same thing at all.

For those with Linux apps, there are already platforms that can be used. The
same cannot be said for VMS. So VMS has additional needs, as in needed, and
just might be more work. Linux however, just isn't needed on additional platforms.

I can also imagine a "brand-new architecture" where a port of Linux just might
not be so easy.

[Dan Cross]

In article <u9bag8$2n8ej$1...@dont-email.me>,
Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>[snip]

>> As for "easier, quick, robust", that's a matter of skill. The real benefits
>> of C over BLISS/MACRO include "floating point" and provided I/O package.
>
>No, the real benefits of C include the ability to use abstractions in
>your code which make the code a lot quicker to write, where you can
>focus much more on what you want, instead of having to lay out in
>minute detail how to do it, and as a result your code also becomes
>much more reusable and portable.
>
>You can design complex data structures much more easily and get much
>more help from the compiler in finding errors in your code.
>
>For one really simple example, in C, you ask for a pointer and let the
>compiler implement the pointer behind the scenes. Ie: you write:
>
> char *ptr;
>
>instead of:
>
> uint32_t ptr;
>
>In C, your normal application code works the same, regardless of whether
>that pointer is 32-bits or 64-bits.

I don't know that I agree with this.

C is not a low-level language (seriously; it hasn't been for a
while: https://queue.acm.org/detail.cfm?id=3212479), but neither
is it a particularly high-level language.

And while, yes, there is a notion of pointer types as first
class objects built into the language, C programmers also pull a
lot of tricks like punning integers for pointers, and so forth
that make this fraught.

There are a few typedefs in e.g. <stdint.h> that can help out
here (`uintptr_t` and friends) but the language allows implicit,
lossy conversions and C code is often not as portable as one
would like to believe.

Moreover, in the past few decades, C _compilers_ have become
inhospitable to systems programmers, so much so that it is now
de facto impossible to write an OS kernel in well-defined C
code; that is, in C that doesn't invoke undefined behavior.
This, in turn, leads to needing lots of compiler-specific help
to force specific behaviors for these instances of UB
(https://arxiv.org/pdf/2201.07845.pdf).

>If the lowest-level language in VMS
>was C, this is what the RMS interface would conceptually look like:
>
> struct {whatever}_rms_rab_t *rab;
>
>instead of the current setup. Likewise for itemlists and other structures.
>
>Things which are currently highly visible to the source code become
>abstracted implementation details that normal source code simply doesn't
>care about or have to deal with.

It's certainly easier to think of C as a high(er) level macro
assembler, but whether or not that's a good mental model is
debateable. Personally, I wouldn't put a lot of new C code into
a kernel that wasn't already implemented in C post about 2018 or
so.

- Dan C.

[John Reagan]

Ah, so you are commenting on the choices made in 1977. C wasn't on the menu
of implementation languages.

Even if there was a C compiler, the PDP-11/RSX legacy of the designers would
probably still have itemlists, descriptors, etc. all with their 32-bit pointer bias. And
also for FABs/RABs/etc. Just having a C compiler doesn't make you "think"
differently.

For BLISS, SDL makes good macros to let you deal with structures in abstractions.
You don't have to know sizes or offsets to use the fields. For example, from inside
Pascal frontend. The definition of SYM_Symbol_Table_Entry comes from SDL. It
was SDL that computed the field sizes/offsets/positions into the .REQ file.

LOCAL Sym : SYM_Symbol_Table_Entry;

IF .Sym[SYM_CLASS] EQL SYM_K_Function
THEN
! something for function
ELSE
! something for not function

or

Sym[SYM_BLOCK_NUMBER] = .Sym[SYM_BLOCK_NUMBER] + 1;

etc.

BLISS can be awkward for coding up following pointers and accessing fields but there are
macros we use and BLISS syntax for doing that.

SDL also makes macros for Macro source, but you are correct that the sizes get encoded
into the VAX instruction (ie, MOVW vs MOVL vs MOVQ). Or if some field is encoded into
the middle of a FLAGS field. Macro programmers can get sloppy in their usage. So the
Macro code has been limited to extend/modify data structures. For ones that are still the
same since 1977 (think FAB/RAB/descriptors), the MOVL from 1977 is just fine today.

[John Reagan]

And looking at GEM sources (no Macro code), I see way more things like

LOCAL SYM : REF GEM_SYMBOL_NODE INITIAL(0);

and uses of

RETURN .GEM_TB_IS_DECIMAL_TYPE[.SYM[GEM_SYM_DATA_TYPE]]

I don't see many random "pointers to integer" or "pointer to char"" (which isn't
easy to write in BLISS, you end up with REF VECTOR[,LONG] or REF VECTOR[,BYTE]
but have to keep using [0] all over the place)

And the C code has things like

GEM_ENTRY *entry = (GEM_ENTRY*) elfst_entry->Node();

if (entry == NULL) return NULL;
if (entry->Kind() != GEM_NODE_K_SYMBOL) return NULL;
if (entry->Subkind() != GEM_SYM_K_ENTRY) return NULL;

return entry->EntrySignature();

Neither have any size/position dependencies (other than BLISS-32 only does
32-bit values and switching to BLISS-64 sometimes can be tricky)

I do see several

GEM_UINT32*

declarations in the C code. Writing in C does "encourage" things like pointer to
int or pointer to char more than BLISS does (at least in my opinion)

[Simon Clubley]

On 2023-07-20, Dave Froble <da...@tsoft-inc.com> wrote:
> On 7/20/2023 8:20 AM, Simon Clubley wrote:
>
>> It doesn't take 9 years to port Linux to a brand-new architecture, even
>> if you first have to implement the compilers for that brand-new architecture.
>
> Not the same thing at all.
>

Why ? I'm talking about how long it is taking.

> For those with Linux apps, there are already platforms that can be used. The
> same cannot be said for VMS. So VMS has additional needs, as in needed, and
> just might be more work. Linux however, just isn't needed on additional platforms.
>
> I can also imagine a "brand-new architecture" where a port of Linux just might
> not be so easy.
>

People didn't have much of a problem getting it onto RISC-V, which is
indeed a brand-new architecture that also needed major compiler work.

[Simon Clubley]

On 2023-07-20, John Reagan <xyzz...@gmail.com> wrote:

> On Thursday, July 20, 2023 at 8:20:43?AM UTC-4, Simon Clubley wrote:
>> You have missed what I am saying above, so I may have been too subtle.
>> Let me reword it: If VMS didn't have any Macro-32 or Bliss code in it,
>> and didn't need to support them as application level programming languages,
>> VMS would look much more internally like any another OS written in C does,
>> and the port would have been completed years ago.
> Ah, so you are commenting on the choices made in 1977. C wasn't on the menu
> of implementation languages.
>

Unix had been around for several years by then.

Other operating systems had been written in various higher-level languages
before that.

It really is a pity that VMS didn't come along several years later
when attitudes to writing full systems in assembly language had
well and truly changed.

[John Dallman]

In article <u9dslr$385j1$1...@dont-email.me>,

clubley@remove_me.eisner.decus.org-Earth.UFP (Simon Clubley) wrote:
> On 2023-07-20, Dave Froble <da...@tsoft-inc.com> wrote:
> > I can also imagine a "brand-new architecture" where a port of
> > Linux just might not be so easy.
> People didn't have much of a problem getting it onto RISC-V, which
> is indeed a brand-new architecture that also needed major compiler
> work.

There are network externalities here. The things that Linux assumes about
a machine are fairly simple. They probably amount to something like:

* Byte addressing. Endianness is not important.
* Flat address space big enough for any single program.
* Some kind of memory protection system. Paging is optional.
* Some kind of interrupt priority system.
* User and supervisor modes.

Designing a new architecture that cannot meet these requirements means
you /must/ design a new operating system for it, and you cannot readily
take advantage of the large amounts of useful open-source software that
are out there. So nobody will do that, unless they have a really
compelling idea about computing that can't be made compatible with Linux
requirements. There don't seem to be many of those coming along at
present.

Note that these requirements are a subset of VMS' requirements, and all
of VMS' hardware platforms are fully capable of running Linux. The only
one where that hasn't been done on any scale is VAX, simply because VAX
was already on the way out when Linux started to grow out of its original
x86 niche.

I also read comp.arch, where there is a community of people who design
new processor architectures, either professionally or for fun. There are
at least two people there who as hobby projects have designed a new
architecture, implemented it in an FPGA, and got Linux running on it,
either by themselves or with one or two people's help. I don't know how
long it's taken them, but I think it's less than nine years.

The strangest new architecture that gets discussed there regularly is The
Mill <https://millcomputing.com/>. This has no registers in the
conventional sense, and two program counters per thread, but the
operating system they're porting to it is Linux.

John

[Dan Cross]

In article <memo.20230721...@jgd.cix.co.uk>,

John Dallman <j...@cix.co.uk> wrote:
>In article <u9dslr$385j1$1...@dont-email.me>,
>clubley@remove_me.eisner.decus.org-Earth.UFP (Simon Clubley) wrote:
>> On 2023-07-20, Dave Froble <da...@tsoft-inc.com> wrote:
>> > I can also imagine a "brand-new architecture" where a port of
>> > Linux just might not be so easy.
>> People didn't have much of a problem getting it onto RISC-V, which
>> is indeed a brand-new architecture that also needed major compiler
>> work.
>
>There are network externalities here. The things that Linux assumes about
>a machine are fairly simple. They probably amount to something like:
>
>* Byte addressing. Endianness is not important.
>* Flat address space big enough for any single program.
>* Some kind of memory protection system. Paging is optional.
>* Some kind of interrupt priority system.
>* User and supervisor modes.
>
>Designing a new architecture that cannot meet these requirements means
>you /must/ design a new operating system for it, and you cannot readily
>take advantage of the large amounts of useful open-source software that
>are out there.

It strikes me that these two things (an OS for an "unusual"
architecture lacking one or more of the requirements listed
above, and the ability to make use of the large body of
open-source software) are mostly orthogonal. Indeed, many
useful OSS packages are available for a variety of wildly
different systems.

Sure, things that are Linux-specific (or Unix/POSIX-specific
more generally) won't be readily portable to other systems, but
the universe of useful open source userspace software is very
large, and much of it could run on, say, a machine that doesn't
support interrupts or user/supervisor modes.

>So nobody will do that, unless they have a really
>compelling idea about computing that can't be made compatible with Linux
>requirements. There don't seem to be many of those coming along at
>present.

I'm not sure about that; we see lots of accelerator processors
that don't implement a traditional ISA and they run lots of
stuff.

>Note that these requirements are a subset of VMS' requirements, and all
>of VMS' hardware platforms are fully capable of running Linux. The only
>one where that hasn't been done on any scale is VAX, simply because VAX
>was already on the way out when Linux started to grow out of its original
>x86 niche.
>
>I also read comp.arch, where there is a community of people who design
>new processor architectures, either professionally or for fun. There are
>at least two people there who as hobby projects have designed a new
>architecture, implemented it in an FPGA, and got Linux running on it,
>either by themselves or with one or two people's help. I don't know how
>long it's taken them, but I think it's less than nine years.
>
>The strangest new architecture that gets discussed there regularly is The
>Mill <https://millcomputing.com/>. This has no registers in the
>conventional sense, and two program counters per thread, but the
>operating system they're porting to it is Linux.

I'll believe in the Mill when I see it. :-D

- Dan C.

[Arne Vajhøj]

On 7/20/2023 8:20 AM, Simon Clubley wrote:
> On 2023-07-19, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>> On 7/19/2023 1:32 PM, Simon Clubley wrote:
>>> On 2023-07-19, Dave Froble <da...@tsoft-inc.com> wrote:
>>>> On 7/18/2023 10:01 PM, Arne Vajhøj wrote:
>>>>> On 7/18/2023 9:25 PM, Dave Froble wrote:
>>>>>> Do you also claim that the executable code would also be smaller?
>>>>>
>>>>> No.
>>>>
>>>> Then, what would be the benefit?
>>>
>>> For one thing, if you didn't have to worry about the Macro-32 and
>>> Bliss crap, the VMS port would have been completed years ago.
>>
>> Are you sure?
>>
>> It is not my impression that the various VMS ISA migrations has
>> caused rewrite of lots of Macro-32 and Bliss.
>>
>> My impression is that:
>> - they create the Macro-32 and Bliss compilers for the new platform
>> - they compile the old Macro-32 and Bliss code dating back from the
>> 70's and 80's
>> - the new code get written in C (plus a little bit of native
>> assembler where needed)
>>
>
> You have missed what I am saying above, so I may have been too subtle.
> Let me reword it: If VMS didn't have any Macro-32 or Bliss code in it,
> and didn't need to support them as application level programming languages,
> VMS would look much more internally like any another OS written in C does,

That is a point you have made several times.

> and the port would have been completed years ago.

Why?

It is not faster to recompile Macro-32 and Bliss with no changes
than to recompile C with no changes.

See above.

Arne

[Arne Vajhøj]

On 7/20/2023 8:49 AM, Simon Clubley wrote:
> I get the impression that you have had to spend a lot of effort to
> duplicate low-level behaviour in a way that simply isn't a concern
> with other operating systems due to the need, especially, to support
> Macro-32 and ensure that code written at this low-level, and which
> uses the various idioms as a result, continues to work.

I believe Macro-32 is a relative simple compiler. And when they got
that then they can compile the existing Macro-32 code.

Not a reason for delay.

> No, the real benefits of C include the ability to use abstractions in
> your code which make the code a lot quicker to write, where you can
> focus much more on what you want, instead of having to lay out in
> minute detail how to do it, and as a result your code also becomes
> much more reusable and portable.
>
> You can design complex data structures much more easily and get much
> more help from the compiler in finding errors in your code.

????

I believe that VSI is mostly reusing existing Macro-32 code
and writing new code in C.

So not relevant.

> For one really simple example, in C, you ask for a pointer and let the
> compiler implement the pointer behind the scenes. Ie: you write:
>
> char *ptr;
>
> instead of:
>
> uint32_t ptr;
>
> In C, your normal application code works the same, regardless of whether
> that pointer is 32-bits or 64-bits. If the lowest-level language in VMS
> was C, this is what the RMS interface would conceptually look like:
>
> struct {whatever}_rms_rab_t *rab;
>
> instead of the current setup. Likewise for itemlists and other structures.
>
> Things which are currently highly visible to the source code become
> abstracted implementation details that normal source code simply doesn't
> care about or have to deal with.

You have posted that numerous times.

It means that changing from 32 bit pointers to 64 bit pointers is
way easier in C than in Macro-32.

But based on what we know about the port, then VSI has not changed
the VMS kernel from 32 bit pointers to 64 bit pointers (aka moving
everything from S0 and S1 to S2).

So it is not relevant for the topic.

Arne

[Arne Vajhøj]

On 7/20/2023 8:49 AM, Simon Clubley wrote:

> I keep looking at why the port is currently at 9 years and counting
> compared to how long it takes to port Linux to a brand-new architecture
> and keep wondering why it is taking so long.
>
> There are some things you have had to implement, such as KESU emulation,
> that Linux doesn't have to deal with, but I have a hard time seeing why
> that would take more than a few months to design and implement.

The 9 years do raise some questions, but some answers are known.

Why has Itanium->x86-64 taken longer time than VAX->Alpha and
Alpha->Itanium?

It has been explained that is because VSI got fewer engineers than
DEC and HP.

Why does it take so long when any CS student can do a port of some
nix flavor in a semester?

VMS has 2 constraints:
- it has to be commercial grade (rock solid, complete etc.)
- it has to be 100% compatible

Everybody that has done software development knows that it
takes 10-100 times more time to do a commercial grade
production ready product than an experimental test product.

And 100% compatibility instead of just 98% compatibility
is also a burden that prevents the easy shortcuts.

Why does VMS port take longer than a Linux port?

First we need to make sure that the Linux port is
comparable per above. Some of the few that comes to my
mind is IBM's ports to Power and mainframe. I have
no idea how long time IBM spent on that.

But let us assume IBM did spend less time than VSI
on VMS x86-64.

VMS is probably less portable than Linux in design.

VMS was designed in parallel with VAX - the OS people
could ask the HW people for what they needed.

The 64 bit port to Alpha was also to something
controlled by DEC and they could create PALcode
for what they needed.

Linux was designed in a world where the HW specs
were externally given.

Now VMS is in that world as well - VSI cannot call
Intel/AMD and ask to get a few instructions added.

It is not just the only 2 modes.

The VSI people have also mentioned the lack of probe
as an issue.

There are probably a number of issues in this category -
let us say somewhere between 10 and 100.

Add that work on top of all the above and the 9 years
start to look less surprising.

Arne

[Arne Vajhøj]

On 7/21/2023 8:21 AM, Simon Clubley wrote:
> On 2023-07-20, John Reagan <xyzz...@gmail.com> wrote:
>> On Thursday, July 20, 2023 at 8:20:43?AM UTC-4, Simon Clubley wrote:
>>> You have missed what I am saying above, so I may have been too subtle.
>>> Let me reword it: If VMS didn't have any Macro-32 or Bliss code in it,
>>> and didn't need to support them as application level programming languages,
>>> VMS would look much more internally like any another OS written in C does,
>>> and the port would have been completed years ago.
>> Ah, so you are commenting on the choices made in 1977. C wasn't on the menu
>> of implementation languages.
>
> Unix had been around for several years by then.
>
> Other operating systems had been written in various higher-level languages
> before that.
>
> It really is a pity that VMS didn't come along several years later
> when attitudes to writing full systems in assembly language had
> well and truly changed.

When DEC made the decisions back in the late 70's using assembler
and a proprietary low level language was not unusual.

And C was not that big a language yet. I don't even think
VAX C was available for the first VMS versions.

If DEC had been able to see into the future then they
may have gone with C.

But ...

Arne

[Arne Vajhøj]

On 7/19/2023 10:32 PM, John Reagan wrote:
> On Wednesday, July 19, 2023 at 1:32:25 PM UTC-4, Simon Clubley wrote:
>> It's also a hell of a lot easier, quicker, and more robust, to
>> write something in C than it is in Macro-32 or Bliss.

> As for "easier, quick, robust", that's a matter of skill. The real benefits
> of C over BLISS/MACRO include "floating point" and provided I/O package.

I believe that it requires more skills to be good at
Macro-32 than to be good at C.

But OK the minimum skills to be allowed to write kernel
code in C (as opposed to application code in C) are
probably sufficient to also make them Macro-32 capable
as well.

It still takes more time to read Macro-32 than C.

And while your new hires (for OS development) already
knows C then I guess nobody below 50 knows Macro-32.

Arne

[Single Stage to Orbit]

On Sat, 2023-07-22 at 10:48 -0400, Arne Vajhøj wrote:
> If DEC had been able to see into the future then they
> may have gone with C.

IF they had, VMS would have had a shitload of exploits against it.
Thankfully they didn't and avoided a lot of issues that plagues most
operating systems written in C.
--
Tactical Nuclear Kittens

[Arne Vajhøj]

Maybe.

C is very unsafe compared to almost all other high level
languages.

But I don't think it is more unsafe than Macro-32.

:-)

Anyway: Windows and Linux has not failed commercially
due to the buffer overflows.

:-)

Arne