FREESPADRIFT

[Phillip Helbig (undress to reply)]

What causes the free-space count to become wrong? Since VMS obviously
knows about it (otherwise /REPAIR couldn't fix it), why isn't it always
correct?

Is there any reason not to run ANA/DISK/REPAIR, say, once per day?

[Johnny Billquist]

Guessing, based on RSX and ODS-1, the count is kept and updated as files
are created/deleted/extended/truncated. However, there is no guarantee
that nothing bad happens sometime, that is not reflected in an update in
that count. Such as if the system crash while a file system update have
not been flushed to disk yet. Or only partially updated.

ANALYZE, on the other hand, actually scans through all disk blocks used
by the file headers, and then update the block used bitmap, and block
used counts, based on this. So it is updated to become the true, correct
count of how things actually are on disk right now.

Reasons not to run it every day? Well, that would be because you need to
make sure the file system is not in use when you're running it, since
you do not want updates done while ANALYZE is running. Also, it can be a
rather CPU and I/O intensive operation to run ANALYZE. So it is
inconvenient and disruptive to run it.

Johnny

--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol

[Kerry Main]

$ Help set volume /rebuild

$ Set volume user1 / rebuild ! once per day will do it (batch or whatever).

Can be done anytime i.e. drives do not need to be dismounted.

On systems that get rebooted often (like lab systems), you can also
mount drives like this:

$ mount vol1 /norebuild/others
$ spawn set volume /rebuild

Or to force a disk block update immediately:
$ set volume /rebuild=force user1


Regards,

Kerry Main
Kerry dot main at starkgaming dot com

[Robert A. Brooks]

On 6/12/2016 3:04 PM, Phillip Helbig (undress to reply) wrote:

> Is there any reason not to run ANA/DISK/REPAIR, say, once per day?

When I was a customer, I did run it every night, right before I defragged.

--
-- Rob

[Phillip Helbig (undress to reply)]

In article <njkcgp$u8s$1...@Iltempo.Update.UU.SE>, Johnny Billquist

<b...@softjar.se> writes:

> > What causes the free-space count to become wrong? Since VMS obviously
> > knows about it (otherwise /REPAIR couldn't fix it), why isn't it always
> > correct?
> >
> > Is there any reason not to run ANA/DISK/REPAIR, say, once per day?
>
> Guessing, based on RSX and ODS-1, the count is kept and updated as files
> are created/deleted/extended/truncated. However, there is no guarantee
> that nothing bad happens sometime, that is not reflected in an update in
> that count. Such as if the system crash while a file system update have
> not been flushed to disk yet. Or only partially updated.

I can understand it being off if there is some hardware problem, or some
OS crash, or a program behaving badly. But I occasionally see it in
normal operation.

> ANALYZE, on the other hand, actually scans through all disk blocks used
> by the file headers, and then update the block used bitmap, and block
> used counts, based on this. So it is updated to become the true, correct
> count of how things actually are on disk right now.

OK. With /REPAIR, when it is finished, though, it issues the same
message as without /REPAIR, i.e. that they differ.

> Reasons not to run it every day? Well, that would be because you need to
> make sure the file system is not in use when you're running it, since
> you do not want updates done while ANALYZE is running. Also, it can be a
> rather CPU and I/O intensive operation to run ANALYZE. So it is
> inconvenient and disruptive to run it.

It takes a minute or so. Is it correct that while the disk is locked
while it is running, nothing actually suffers as a result?

Or, is there a reason to run it at all, as long as there is enough
usable free space?

[Phillip Helbig (undress to reply)]

In article
<mailman.301.1465761765.14...@info-vax.com>, Kerry

Main <kerry...@backtothefutureit.com> writes:

> > > What causes the free-space count to become wrong? Since VMS
> > obviously
> > > knows about it (otherwise /REPAIR couldn't fix it), why isn't it always
> > > correct?
> > >
> > > Is there any reason not to run ANA/DISK/REPAIR, say, once per day?

> $ Help set volume /rebuild

I do this once per day. I don't think they do the same things (though
there might be some overlap).

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Phillip Helbig undress to reply via Info-vax
> Sent: 12-Jun-16 4:23 PM
> To: info...@info-vax.com
> Cc: Phillip Helbig undress to reply <hel...@asclothestro.multivax.de>
> Subject: Re: [New Info-vax] FREESPADRIFT
>

They don't do the same thing, but if all you are worried about is free
space values, then this should do it. From help file-

"Recovers caching limits for a volume that was dismounted
improperly. If a disk volume was dismounted improperly (such
as during a system failure), and was then remounted with the
MOUNT/NOREBUILD command, you can use SET VOLUME/REBUILD to
recover the caching that was in effect at the time of the
dismount. The FORCE option forces the disk to be rebuilt
unconditionally, thus updating the free block count in the disk
volume's lock value block."

Analyze /disk obviously does a lot more.

[brendan welch]

On 6/12/2016 3:04 PM, Phillip Helbig (undress to reply) wrote:

We found that in a similar situation, running a program which
continuously defragged disk files, the result was an increase
in hardware errors.
E.g., if a disk had accumulated maybe 3 errors over a course of
months, running the defrag would give maybe 3 more in a few days.

[Johnny Billquist]

On 2016-06-12 22:22, Phillip Helbig (undress to reply) wrote:
> In article <njkcgp$u8s$1...@Iltempo.Update.UU.SE>, Johnny Billquist
> <b...@softjar.se> writes:
>
>>> What causes the free-space count to become wrong? Since VMS obviously
>>> knows about it (otherwise /REPAIR couldn't fix it), why isn't it always
>>> correct?
>>>
>>> Is there any reason not to run ANA/DISK/REPAIR, say, once per day?
>>
>> Guessing, based on RSX and ODS-1, the count is kept and updated as files
>> are created/deleted/extended/truncated. However, there is no guarantee
>> that nothing bad happens sometime, that is not reflected in an update in
>> that count. Such as if the system crash while a file system update have
>> not been flushed to disk yet. Or only partially updated.
>
> I can understand it being off if there is some hardware problem, or some
> OS crash, or a program behaving badly. But I occasionally see it in
> normal operation.

That would sound more like a bug in that case.

>> ANALYZE, on the other hand, actually scans through all disk blocks used
>> by the file headers, and then update the block used bitmap, and block
>> used counts, based on this. So it is updated to become the true, correct
>> count of how things actually are on disk right now.
>
> OK. With /REPAIR, when it is finished, though, it issues the same
> message as without /REPAIR, i.e. that they differ.

That's the same in VFY under RSX then. In RSX it is because the report
gives the information as it was before the repair was done. If you run
it one more time, the numbers should stabilize, I would think.

>> Reasons not to run it every day? Well, that would be because you need to
>> make sure the file system is not in use when you're running it, since
>> you do not want updates done while ANALYZE is running. Also, it can be a
>> rather CPU and I/O intensive operation to run ANALYZE. So it is
>> inconvenient and disruptive to run it.
>
> It takes a minute or so. Is it correct that while the disk is locked
> while it is running, nothing actually suffers as a result?

A minute? Really? Wow. That sounds like a much more light weight operation.

> Or, is there a reason to run it at all, as long as there is enough
> usable free space?

As long as it only misses in that it reports less free than is the
actual case, then it's not that important. But at least in RSX, the
errors can also go in the other direction, so that you have blocks that
are in use, which the system also consider to be free. If those are not
taken care of, you can end up with the same disk block being allocated
to two files, causing data loss and strange corruptions. Not a good thing.

In essence, there are two different structures that keep track of
allocated blocks, and when they don't agree, something is wrong.

One is the BITMAP.SYS file, which keeps track of all free and allocated
blocks on the volume. The other is the INDEXF.SYS file, which keeps
record of all blocks allocated to files. If these two sources of
information disagree, then you have a potential problem coming up.
And figuring out what/where the problem is requires scanning through all
the blocks allocated to files in INDEXF.SYS, and check that against the
bitmap in BITMAP.SYS, to find discrepancies, as well as doubly allocated
blocks to several files. Which is why a run of just a minute or two
sounds impressive. It can be a lot of data. Now, PDP-11s aren't exactly
the speed of Alphas, but analyzing a 1G disk under RSX can take hours...

Johnny

[Bob Koehler]

In article <njkbn6$tuj$1...@news.kjsl.com>, hel...@asclothestro.multivax.de (Phillip Helbig (undress to reply)) writes:
> What causes the free-space count to become wrong? Since VMS obviously
> knows about it (otherwise /REPAIR couldn't fix it), why isn't it always
> correct?

It is normally cached, with more blocks marked on the disk as in use than
are actually in use, so allocation doesn't need to update the on-disk
value. During proper dismount, the value on the disk is corrected.
Improper dismount will leave it stale.

> Is there any reason not to run ANA/DISK/REPAIR, say, once per day?

Overkill. See also "set volum/rebuild".

[Phillip Helbig (undress to reply)]

In article
<mailman.133.1465763820.14...@info-vax.com>, Kerry

Main <kerry...@backtothefutureit.com> writes:

> > > > > Is there any reason not to run ANA/DISK/REPAIR, say, once per
> > day?
> >
> > > $ Help set volume /rebuild
> >
> > I do this once per day. I don't think they do the same things (though
> > there might be some overlap).
> >
>
> They don't do the same thing, but if all you are worried about is free
> space values, then this should do it. From help file-
>
> "Recovers caching limits for a volume that was dismounted
> improperly. If a disk volume was dismounted improperly (such
> as during a system failure), and was then remounted with the
> MOUNT/NOREBUILD command, you can use SET VOLUME/REBUILD to
> recover the caching that was in effect at the time of the
> dismount. The FORCE option forces the disk to be rebuilt
> unconditionally, thus updating the free block count in the disk
> volume's lock value block."
>
> Analyze /disk obviously does a lot more.

Right. In my case, the free-space drift doesn't come from having been
improperly dismounted, so probably SET VOLUME/REBUILD won't fix it
(again, I do this nightly anyway, just in case the has been some crash).

[lawren...@gmail.com]

On Tuesday, June 14, 2016 at 1:21:50 AM UTC+12, Bob Koehler wrote:

> It is normally cached, with more blocks marked on the disk as in use than
> are actually in use, so allocation doesn't need to update the on-disk
> value. During proper dismount, the value on the disk is corrected.
> Improper dismount will leave it stale.

No journalling, then?

This from the company that invented Spiralog...

[Michael Moroney]

Don't forget that Spiralog was newer than ODS-5/ODS-2. And Spiralog was
to be a different file system for VMS (not a mod to ODS-5) but was
cancelled.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Michael Moroney via Info-vax
> Sent: 16-Jun-16 2:24 PM
> To: info...@info-vax.com
> Cc: Michael Moroney <mor...@world.std.spaamtrap.com>
> Subject: Re: [New Info-vax] FREESPADRIFT
>

Walk down memory lane - Spiralog explained:
http://www.hpl.hp.com/hpjournal/dtj/vol8num2/vol8num2art1.pdf

[lawren...@gmail.com]

On Friday, June 17, 2016 at 6:24:46 AM UTC+12, Michael Moroney wrote:

> Don't forget that Spiralog was newer than ODS-5/ODS-2. And Spiralog was
> to be a different file system for VMS (not a mod to ODS-5) but was
> cancelled.

Linux was able to add journalling to an existing filesystem (turning ext2 into ext3). VMS never did the same?

[Stephen Hoffman]

Spiralog: http://www.hpl.hp.com/hpjournal/dtj/vol8num2/vol8num2art1.pdf
RMS Journaling: http://h71000.www7.hp.com/doc/72final/4454/4454pro.html


--
Pure Personal Opinion | HoffmanLabs LLC

[lawren...@gmail.com]

On Friday, June 17, 2016 at 11:57:02 AM UTC+12, Stephen Hoffman wrote:

> RMS Journaling: http://h71000.www7.hp.com/doc/72final/4454/4454pro.html

But not filesystem journalling.

[Craig A. Berry]

There is a new filesystem on the roadmap for this year or next. I
believe Clair Grant said something like "not Spiralog but has some
shared concepts." That's a paraphrase from memory, not a direct quote.
Of course the competition have been busy with their own implementations
of some of those same concepts plus a lot more, most recently APFS:

<https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/Introduction/Introduction.html>

<https://en.wikipedia.org/wiki/Apple_File_System>

[Michael Moroney]

VMS Engineering was working on a new file system when HP canned VMS in the
US, including that project. I don't know the details. VSI is likely to
come up with something to supersede ODS-5.

[lawren...@gmail.com]

On Friday, June 17, 2016 at 2:16:29 PM UTC+12, Craig A. Berry wrote:

> On 6/16/16 7:23 PM, Lawrence D’Oliveiro wrote:
>> On Friday, June 17, 2016 at 11:57:02 AM UTC+12, Stephen Hoffman wrote:
>>
>>> RMS Journaling: http://h71000.www7.hp.com/doc/72final/4454/4454pro.html
>>
>> But not filesystem journalling.
>
> There is a new filesystem on the roadmap for this year or next. I
> believe Clair Grant said something like "not Spiralog but has some
> shared concepts."

Sure, next-generation stuff is all very well (see btrfs and ZFS), but what about *current-generation* stuff, namely (relatively) simple journalling?

[Paul Sture]

A mentioned by another poster, work was started on a new filesystem.

We're back to where you spend the available resources - retrofitting
a new feature to an existing system or implementing a replacement?

The existing VMS filesystem doesn't cache writes as much as other file
systems, and on-disk pointers are built in such a way that a power loss
doesn't result in the damage you see on other file systems.

I'd put it that the current VMS filesystem didn't need journalling as
urgently as other filesystems.

And yes, the relative certainty that when you write something to disk
it the current VMS world, it does actually hit the disk, has been a
limiting factor in performance.

--
There are two hard things in computer science, and they are cache invalidation,
naming, and off-by-one errors.

[David Froble]

But it sure helped when that $10M in accounts receivable wasn't lost ....

[Jan-Erik Soderholm]

Den 2016-06-17 kl. 07:24, skrev Paul Sture:
> On 2016-06-16, lawren...@gmail.com <lawren...@gmail.com> wrote:
>> On Friday, June 17, 2016 at 6:24:46 AM UTC+12, Michael Moroney wrote:
>>
>>> Don't forget that Spiralog was newer than ODS-5/ODS-2. And Spiralog was
>>> to be a different file system for VMS (not a mod to ODS-5) but was
>>> cancelled.
>>
>> Linux was able to add journalling to an existing filesystem (turning
>> ext2 into ext3). VMS never did the same?
>
> A mentioned by another poster, work was started on a new filesystem.
>
> We're back to where you spend the available resources - retrofitting
> a new feature to an existing system or implementing a replacement?
>
> The existing VMS filesystem doesn't cache writes as much as other file
> systems, and on-disk pointers are built in such a way that a power loss
> doesn't result in the damage you see on other file systems.

Yes, that was also my line of thought here.

If you do not have native file versioning so that ever new file
directly overwrites any older version, you might see a higher
need for journaling on the file system level.

RMS already have journaling as an option.

And Rdb of course always had full database journaling by default.

Jan-Erik.

[Stephen Hoffman]

The OpenVMS ODS-2 and ODS-5 file systems uses careful write ordering,
which means it typically doesn't lose data in the event of a partial or
complete power failure or system crash, and doesn't corrupt itself.
Application data that's in flight might not fair well in these cases —
this being at the core of why I'm skeptical around storage-level
clustering and storage-level replication, in the absence of insight
into the host software activity and state — but the file system should
survive intact.

The combination of careful write ordering and the fondness for writing
everything to disk is also related to why the OpenVMS file systems are
comparatively slow.

Applications that depend on multi-part writes — few don't depend on
that, of course — tend to either use a database or they use RMS
journaling, This if they need to ensure all writes related to an
operation are committed, or that no writes are committed.

Spiralog was a log structured file system, which effectively means the
whole file system was the journal.

Spiralog did not react particularly gracefully to a full-disk
condition, but that's fodder for another day.

This is part of why I've been requesting an OpenVMS-integrated database
— more than a few sites and OpenVMS itself aren't in a position to fund
an Oracle purchase for this purpose — and the changes in storage
technologies presently underway are also why I've been wondering
whether the VSI new file system is flash / 3D Xpoint / SSD aware.

q.v.
http://arstechnica.com/apple/2016/06/digging-into-the-dev-documentation-for-apfs-apples-new-file-system/

[Paul Sture]

To be honest I cannot remember a single incident where either a crash
or power failure caused wholesale loss of data on a VMS disk.

Yes, that backup/image restore or indexed file reorganisation which
didn't complete due to a crash or power failure would be toast, but it
would be pretty obvious what had happened.

Real life example from the early 2000s when a less-uninterruptible
power supply took out the power to a whole building:

The VMS systems came back minus a few disks which didn't spin up.
Volume shadowing and the replacement of dead disks brought the
systems back to normal working order. The applications on these
systems used a mix of Oracle Rdb and Oracle Classic, set up by
competent DBAs.

In contrast some of the Unix systems (there were no Linux
deployments at that customer at that time) required full filesystem
restores from tape before they could be brought back into
production. This took several days for the larger systems.

Footnote: some concrete figures for money lost by extended downtime
came out of that incident and management was not afraid to invest
in beefing up their disaster recovery capabilities.

[David Froble]

Too bad that it appears to be a case of closing the barn doors after the horses
are out ....

:-)

But, better late than never ....

[Paul Sture]

On 2016-06-17, David Froble <da...@tsoft-inc.com> wrote:
> Paul Sture wrote:

<snip>

>>
>> Footnote: some concrete figures for money lost by extended downtime
>> came out of that incident and management was not afraid to invest
>> in beefing up their disaster recovery capabilities.
>>
>
> Too bad that it appears to be a case of closing the barn doors after
> the horses are out ....

Q: Why close the door after the horse has bolted?
A: To stop the chickens coming home to roost!

>
>:-)
>
> But, better late than never ....

Actually this was more a case of critical systems being disaster
tolerant (DT versus DR). I don't think anyone had thought of the case
where a whole building was out.

Initial solution: move selected systems to another building; just across
the road would do for starters, but leave you vulnerable to a street
level failure. Fortunately this customer was large enough to have
existing data centres in different locations.

This is where the then recent arrival of Fibrechannel opened up
possibilities - you could put 'local' disks in another building.

Longer term solution: build another data centre in another location
and add a third data centre later.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Paul Sture via Info-vax
> Sent: 17-Jun-16 2:53 PM
> To: info...@info-vax.com
> Cc: Paul Sture <nos...@sture.ch>
> Subject: Re: [New Info-vax] FREESPADRIFT
>

The dual active-active DC's within 100km of each other is the DR/DT
strategy that most Customers are looking at these days. For those that
need to accommodate a major incident taking out both of these sites,
then a third site is recommended much further from the first two i.e.
often called active-active-passive.

Why 100km? This is the rule of thumb distance which many folks in
the industry state is a balance between site separation and yet still
allow sync writes to occur with acceptable performance. YMMV as
it all depends on things like the read-write ratio of the critical apps.

A-A-P is certainly more expensive, but DR/DT is like insurance - you
may never use it, but if you do, you will be glad you made the
investment.

What is usually not considered these days are pandemic issues like
SARS and [insert major pandemic]. In these days of global travel, it
Is a major concern for large Customers. A pandemic issue could
easily take out both "local" sites - all it takes is an OPS person with
something like SARS to visit both sites (perhaps picking up tapes?)
and inadvertently contaminate both sites.

That's it - both sites are closed immediately once the guys in white
suits get involved. Everyone goes home.

And in case anyone thinks this is pie-in-the-sky stuff, check out what
happened to HP in Toronto during the SARS scare back In 2003 - luckily
only 1 site was impacted: (the closed office included one major DC site)

http://www.cnet.com/news/sars-sends-hp-workers-home-in-canada/

http://www.itbusiness.ca/news/sars-scare-leaves-hp-canada-in-limbo/5669

[lawren...@gmail.com]

On Saturday, June 18, 2016 at 2:26:45 AM UTC+12, Stephen Hoffman wrote:

> The OpenVMS ODS-2 and ODS-5 file systems uses careful write ordering,
> which means it typically doesn't lose data in the event of a partial or
> complete power failure or system crash, and doesn't corrupt itself.

Then where is the inconsistency in free-space usage coming from? You know, the issue that started this thread?

[Jan-Erik Soderholm]

But that doesn't lose data, just some free space.

[Michael Moroney]

I can't tell for certain without more info, but in a cluster wide mount,
some stuff like file headers and disk space bitmaps are allocated to each
cluster member (and marked as allocated on the disk) so the cluster member
can create/extend files without all the inter-cluster traffic required to
update cluster members every time this happens. When a disk is properly
dismounted, unused allocated space is returned. If a cluster member
crashes, it is possible for other members to not be able to mark space
allocated to the departed member free, it remains as a ghost allocation.
MOUNT/REBUILD normally clears this stuff up.

[Stephen Hoffman]

Caching. The displayed value is stored in a lock value block and —
bugs aside, and there have been bugs here — the value that's cached in
that lock value block can be lost when a host crashes. That causes a
skew. Accessing that free space data from disk is slow, the accuracy
of that value was not considered a requirement, and hence it's cached
in memory; in a lock value block.

The volume rebuild operations releases storage that is reserved on the
disk (the extent caching, as well as files marked for delete), though
that storage is reported (or released) with the ANALYZE /DISK (/REPAIR)
command.

It's this second area that you're drilling into with your questions on
journaling and log-structured file systems and such. Which is
arguably different from the free space skew — whether that skew was
secondary to a crash, corruption and/or bug — that started the thread.
Disk data is correct and consistent. OpenVMS itself can be configured
to perform a rebuild on boot for the system disk and the rebuild can be
controlled on volume mount, or that step can be skipped to allow faster
access to the storage for any of various reasons, such as to allow the
rebuild to be performed on a host with a faster I/O connection to the
disk. (Rebuilding a disk via NI — which is pretty much Phillip's
entire cluster — is much slower, and can end up with the whole cluster
boot wedged behind some rinky-dink box with a soda-straw served pipe to
the disk. Rebuilding the disk with a more direct I/O path and not via
an NI-based MSCP-served path is usually preferable.)

Here is a discussion of this particular topic from some sixteen years
ago: http://h71000.www7.hp.com/wizard/wiz_4080.html

[David Froble]

Hmmm .... perhaps this explains why I never see such problems on my
non-clustered systems?

[Paul Sture]

On 2016-06-17, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:

>
> q.v.
> <http://arstechnica.com/apple/2016/06/digging-into-the-dev-documentation-for-apfs-apples-new-file-system/>

The following sentence leaped out:

"A feature called "fast directory sizing" is purported to give macOS a
fast way to query the size of a directory and all its child objects,
rather than having to wait while a bunch of stat calls complete."

Rather useful when determining how much space is required for a
Time Machine Backup, I imagine.

cf VMS only storing the block sizes of files rather than the number of
bytes; this can be problematic for utilities ported from a system which
has an easy/cheap way to supply the exact size in bytes.

FWIW, from time to time a process running 'du' consumes high amounts of
CPU on OS X Server. Observations indicate that it's part of Time
Machine backing up the OS X Server data itself (not that I have much of
that in the grand scheme of things). I can see why Apple want to
implement the above feature.

[lawren...@gmail.com]

On Saturday, June 18, 2016 at 8:56:38 PM UTC+12, Paul Sture wrote:

> Rather useful when determining how much space is required for a
> Time Machine Backup, I imagine.

Time Machine plays fast and loose with your filesystem integrity (multiple hard links to directories), and you pay the price in reliability <http://arstechnica.com/civis/viewtopic.php?t=1252811>.

> cf VMS only storing the block sizes of files rather than the number of
> bytes; this can be problematic for utilities ported from a system which
> has an easy/cheap way to supply the exact size in bytes.

This was the example Linus Torvalds gave of why he hated VMS...

[Paul Sture]

On 2016-06-18, lawren...@gmail.com <lawren...@gmail.com> wrote:
> On Saturday, June 18, 2016 at 8:56:38 PM UTC+12, Paul Sture wrote:
>
>> Rather useful when determining how much space is required for a
>> Time Machine Backup, I imagine.
>
> Time Machine plays fast and loose with your filesystem integrity
> (multiple hard links to directories), and you pay the price in
> reliability
>
> <http://arstechnica.com/civis/viewtopic.php?t=1252811>.

Yes, I also went through problems with TM, but they appear to be fixed.

TM can be terribly slow in certain circumstances. I take regular
clone copies using other software.

>> cf VMS only storing the block sizes of files rather than the number of
>> bytes; this can be problematic for utilities ported from a system which
>> has an easy/cheap way to supply the exact size in bytes.
>
> This was the example Linus Torvalds gave of why he hated VMS...

Eh? There are many reasons to dislike an OS but 'hating VMS' for that
*perceived* problem? Really?

From another perspective, I ran straight into the spat between Torvalds
and Ted Ts'o when ext4 became the default on Ubuntu - this one did
produce data loss for me (until I worked out what was going on):

"Linus Torvalds Upset over Ext3 and Ext4"

<http://www.linux-magazine.com/Online/News/Linus-Torvalds-Upset-over-Ext3-and-Ext4>

[Stephen Hoffman]

On 2016-06-18 08:54:54 +0000, Paul Sture said:

> cf VMS only storing the block sizes of files rather than the number of
> bytes; this can be problematic for utilities ported from a system which
> has an easy/cheap way to supply the exact size in bytes.

See the XAB$W_FFB first free byte value, for those file formats that
aren't block-storage-oriented.

With OpenVMS, there's also the fairly user-visible distinctions among
the file user content size in bytes, the file storage being used for
the user data in blocks/sectors, and the total storage allocated to the
file in blocks/sectors due to pre-extending and the rounding up
inherent in the cluster factor. Then there's the metadata embedded in
various of the common formats, even within the various sequential
formats. Unix doesn't default to anything like VFC sequential files,
after all. Do you count that metadata in the file size, or not?
Arguably no, but it's inherently part of the file on OpenVMS and not at
all easy to exclude — whether VFC or the indexed file structures or
otherwise.

Not that OpenVMS APIs and the OpenVMS docs even particularly lend
themselves to presenting this or related information, either. Having
some decently-designed OO methods for retrieving this and other
metadata would changes this into something rather easier to deal with;
less glue code than is involved dredging through XABs.

[John E. Malmberg]

On 6/18/2016 3:54 AM, Paul Sture wrote:
> cf VMS only storing the block sizes of files rather than the number of
> bytes; this can be problematic for utilities ported from a system which
> has an easy/cheap way to supply the exact size in bytes.

Part of that is because VMS does not always store all the bytes in the
file that will show up when the file is read by the *nix utility.

The file attributes tell if and what line terminators should be supplied
when the file is read.

The only way to get the byte count for certain file types is to read the
entire file.

Regards,
-John
wb8...@qsl.net_work

[Paul Sture]

On 2016-06-18, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:
> On 2016-06-18 08:54:54 +0000, Paul Sture said:
>
>> cf VMS only storing the block sizes of files rather than the number of
>> bytes; this can be problematic for utilities ported from a system which
>> has an easy/cheap way to supply the exact size in bytes.
>
> See the XAB$W_FFB first free byte value, for those file formats that
> aren't block-storage-oriented.

I've always understood calculating sizes from the furst free byte value
to be an approximation. Here's a simple example:

$ create x.x
A
BB
CCC
^Z
$ dump/dec/byte/width=80
Virtual block number 1 (00000001), 512 (0200) bytes

66 66 0 2 0 65 0 1 ..A...BB 0
-1 -1 0 67 67 67 0 3 ..CCC... 8
0 0 0 0 0 0 0 0 ........ 16

Note that the record size fields get word aligned, as does the
EOF marker. Thus adding up the record sizes, plus 2 bytes for
the record size doesn't accurately reflect the actual bytes
consumed.

And IIRC, further padding can occur at a block boundary, even where
records can span blocks; I'm thinking of VFC records here but I'm pretty
sure I encountered longword padding, possibly to keep the record size
field and contents together.

> With OpenVMS, there's also the fairly user-visible distinctions among
> the file user content size in bytes, the file storage being used for
> the user data in blocks/sectors, and the total storage allocated to the
> file in blocks/sectors due to pre-extending and the rounding up
> inherent in the cluster factor. Then there's the metadata embedded in
> various of the common formats, even within the various sequential
> formats. Unix doesn't default to anything like VFC sequential files,
> after all.

According to:

<http://h71000.www7.hp.com/doc/731final/4523/4523pro_015.html#index_x_697>

both XAB$L_EBK and XAB$W_FFB Field are "meaningful for sequential files
only" (caveat, the above URL points to the V7.3-1 documentation)

> Do you count that metadata in the file size, or not? Arguably no, but
> it's inherently part of the file on OpenVMS and not at all easy to
> exclude — whether VFC or the indexed file structures or otherwise.

Tricky. If you are looking at the output of TYPE, then the record
attributes (e.g. Record attributes Carriage return carriage control)
make a difference to the output.

> Not that OpenVMS APIs and the OpenVMS docs even particularly lend
> themselves to presenting this or related information, either. Having
> some decently-designed OO methods for retrieving this and other
> metadata would changes this into something rather easier to deal with;
> less glue code than is involved dredging through XABs.

Having done this stuff from COBOL, I emphatically agree.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Paul Sture via Info-vax
> Sent: 18-Jun-16 7:52 AM
> To: info...@info-vax.com
> Cc: Paul Sture <nos...@sture.ch>

With OpenVMS and its default write-through strategies, data integrity
has always been the primary driver.

Yes, this strategy does put it at a disadvantage when doing benchmarks
that *only* consider performance.

As illustrated in this article, with other platforms, there was a
conscious
decision to implement write back (or async via periodic time frames)
strategies as a way to increase perf even if the designers acknowledge
that things like crashes, power fails could potentially cause data
integrity
issues.

One only has to power off a system in the middle of a benchmark on that
system to see what the effects could be in the real world.

With new disks like SSD and really big memory technologies emerging, one

needs to ask whether the difference in perf is really worth the
potential for
data integrity issues? Especially when one considers the considerable
pain
one must go through to recover from some significant event like a system

crash or hardware error or power failure on a write back designed
system.

Perhaps we will see more strategies like mirroring a RAM disk with a SSD

disk so that reads come from the RAM disk, but writes also go to SSD?

[Simon Clubley]

On 2016-06-18, Paul Sture <nos...@sture.ch> wrote:
> On 2016-06-18, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:
>
>> Do you count that metadata in the file size, or not? Arguably no, but
>> it's inherently part of the file on OpenVMS and not at all easy to
>> exclude — whether VFC or the indexed file structures or otherwise.
>
> Tricky. If you are looking at the output of TYPE, then the record
> attributes (e.g. Record attributes Carriage return carriage control)
> make a difference to the output.
>

IMHO, it depends on the context. If you are doing block level reads,
then you count the size of the on-disk metadata in the file size.

If you are doing record level reads, then you do not include the
metadata IMHO but you _do_ add in any additional terminator bytes to
the length (which might not actually be stored on disk).

You need the former if you are doing an image copy. You need the latter
if you want to tell a webserver at which point it should resume the
download.

Also, IMHO I think the default sequential record type for today's
world should be stream (and hence the terminator is also included in
the file data.) I do not think it should be variable length records.

The problem is that today's protocols are simply not designed to
handle the case of metadata buried within the file contents
themselves; the people who design this stuff have probably never
even encountered that case.

I do wonder how the IBM mainframe people handle this problem.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of Paul
> Sture via Info-vax
> Sent: 18-Jun-16 7:52 AM
> To: info...@rbnsn.com
> Cc: Paul Sture <nos...@sture.ch>
> Subject: Re: [Info-vax] Re; Spiralog, RMS Journaling (was Re:
> FREESPADRIFT)
>

[snip]

>
> >> cf VMS only storing the block sizes of files rather than the number
of
> >> bytes; this can be problematic for utilities ported from a system
which
> >> has an easy/cheap way to supply the exact size in bytes.
> >
> > This was the example Linus Torvalds gave of why he hated VMS...
>
> Eh? There are many reasons to dislike an OS but 'hating VMS' for that
> *perceived* problem? Really?
>

Btw, it's not that well known, but while the default display of file
size and
device free space is blocks, with OpenVMS, you can also get these values

in bytes as well.

>From V8.4-2 system:

SYS003> dir sys$system:sysdump.dmp/siz=all

Directory SYS$SYSROOT:[SYSEXE]

SYSDUMP.DMP;1 1604596/1604608

Total of 1 file, 1604596/1604608 blocks.
SYS003> set proc/unit=bytes
SYS003> dir sys$system:sysdump.dmp/siz=all

Directory SYS$SYSROOT:[SYSEXE]

SYSDUMP.DMP;1 783.49MB/783.50MB

Total of 1 file, 783.49MB/783.50MB
SYS003> sho dev dsa20

Device Device Error Volume Free
Trans Mnt
Name Status Count Label Space
Count Cnt
DSA20: Mounted 0 USER2 1.66TB
51 1
$1$DKC1: (SYS003) ShadowSetMember 0 (member of DSA20:)
SYS003>

[Stephen Hoffman]

On 2016-06-18 18:07:05 +0000, Paul Sture said:

> both XAB$L_EBK and XAB$W_FFB Field are "meaningful for sequential files only"

I'd prefer that any (errant) requests violating that would kick back a
run-time error when used on inappropriate file formats, but that
concept and that capability simply does not exist within the (current)
OpenVMS RMS APIs.

Ugly case would be an explicit NaN/NaT returned in these fields
secondary to an errant access; a Not a Number or Not a Thing response.
OpenVMS and various databases certainly have have examples akin to
that, but OpenVMS has no particularly consistent implementation of
that. In more than a few fields in the various data structures, RMS
doesn't, though.

Best case would be a compile-time error flagging the error, but that's
likely not happening here.

But yes, within the usual way OpenVMS works (now), It'd certainly be
useful to have an exact count of user data in the object as well as an
exact size of the container, though. But I'd still prefer to not (have
to) deal with that differentiation; around how OpenVMS works here and
how OpenVMS stores the file. I'd prefer to just deal with the data and
preferably with my data structures, and — best case — not have to
(write the code to) marshal or unmarshal the data.

As for the discussion that started this off, it'd be handy of OpenVMS
cleaned up that cache skew by itself. Autonomously. But that's not
likely happening anytime soon, either. (But this being OpenVMS, I'm
sure there are folks that cannot or will not want these sorts of
autonomous repairs, or OO interfaces, or code that marshals and
unmarshals data. For any of various reasons.)

[Jan-Erik Soderholm]

Den 2016-06-18 kl. 20:48, skrev Kerry Main:
>> -----Original Message-----
>> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of Paul
>> Sture via Info-vax
>> Sent: 18-Jun-16 7:52 AM
>> To: info...@rbnsn.com
>> Cc: Paul Sture <nos...@sture.ch>
>> Subject: Re: [Info-vax] Re; Spiralog, RMS Journaling (was Re:
>> FREESPADRIFT)
>>
> [snip]
>
>>
>>>> cf VMS only storing the block sizes of files rather than the number
> of
>>>> bytes; this can be problematic for utilities ported from a system
> which
>>>> has an easy/cheap way to supply the exact size in bytes.
>>>
>>> This was the example Linus Torvalds gave of why he hated VMS...
>>
>> Eh? There are many reasons to dislike an OS but 'hating VMS' for that
>> *perceived* problem? Really?
>>
>
> Btw, it's not that well known, but while the default display of file
> size and
> device free space is blocks, with OpenVMS, you can also get these values
>
> in bytes as well.

Yes, but that is a simple calculation with 1 block = 512 bytes.
It does not correctly display partially filled blocks. I guess
that what most here are talking about is the size up to the
marker. Convering the used/allocated number of blocks is simple.

Jan-Erik.

[Stephen Hoffman]

On 2016-06-18 18:31:49 +0000, Kerry Main said:

> One only has to power off a system in the middle of a benchmark on that
> system to see what the effects could be in the real world.

I've worked more than a few OpenVMS systems that have to be manually
rebooted, and with non-trivial recovery processing. There's no APIs
for coordinating with backups, no integrated means of checkpointing
application data, and a whole pile of other limits.

> With new disks like SSD and really big memory technologies emerging,

> one needs to ask whether the difference in perf is really worth the=

> potential for data integrity issues? Especially when one considers the
> considerable pain one must go through to recover from some significant
> event like a system crash or hardware error or power failure on a write
> back designed system.

Yeah, and I've had that same issue with long-running jobs on OpenVMS,
where the primitive tools force that into the database and particularly
onto the end developer, There's nothing even remotely close to Time
Machine in OpenVMS, either. Then there's that RMS is incapable of
getting consistent live backups.

> Perhaps we will see more strategies like mirroring a RAM disk with a
> SSD disk so that reads come from the RAM disk, but writes also go to
> SSD?

Mirror the whole server. Then there's the support for configuring and
deploying servers, for replicating servers and checkpointing and
process fail-overs in a cluster, for containers and app-stacking, and
any number of other features that might leverage the strengths of
OpenVMS. Which all tend toward the "here's a box of parts kiddo, go
have some fun" answer.

VSI needs to acquire and maintain stable and preferably growing profits
and — with apps and vendors still leaving the platform — the folks ay
VSI also need to grow the installed base, too.

That means taking a very careful look at when write-through is
appropriate and when it's not, when server replication — server
shadowing, in OpenVMS-speak — might be the way to go, hauling the
cluster configuration and management forward from the 1980s,
implementing a file system that's targeted for flash memory and — for
those folks using older servers and older hardware — SSD storage, and a
whole pile of other projects far too large to list. Which will be no
small effort to design and fund and schedule and deploy and..., of
course.

[John Reagan]

On Saturday, June 18, 2016 at 3:02:29 PM UTC-4, Stephen Hoffman wrote:
> On 2016-06-18 18:07:05 +0000, Paul Sture said:
>
> > both XAB$L_EBK and XAB$W_FFB Field are "meaningful for sequential files only"
>
> I'd prefer that any (errant) requests violating that would kick back a
> run-time error when used on inappropriate file formats, but that
> concept and that capability simply does not exist within the (current)
> OpenVMS RMS APIs.
>
> Ugly case would be an explicit NaN/NaT returned in these fields
> secondary to an errant access; a Not a Number or Not a Thing response.
> OpenVMS and various databases certainly have have examples akin to
> that, but OpenVMS has no particularly consistent implementation of
> that. In more than a few fields in the various data structures, RMS
> doesn't, though.

NaNs are floating point. NaTs don't exist in memory, only in registers (and are Itanium specific).

>
> Best case would be a compile-time error flagging the error, but that's
> likely not happening here.

The compiler has no way of knowing if the file you'll eventually open is sequential or not.

>
> But yes, within the usual way OpenVMS works (now), It'd certainly be
> useful to have an exact count of user data in the object as well as an
> exact size of the container, though. But I'd still prefer to not (have
> to) deal with that differentiation; around how OpenVMS works here and
> how OpenVMS stores the file. I'd prefer to just deal with the data and
> preferably with my data structures, and — best case — not have to
> (write the code to) marshal or unmarshal the data.
>

Consider a ZIP file or some other file system that compresses the on-disk file for space reasons. Asking "how many octets of data are in that file" is only somewhat related to how much physical disk space the file system requires to store and retrieve your data. Lets think back to early Macintosh days with separate data forks and resource forks.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: 18-Jun-16 3:24 PM
> To: info...@info-vax.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [New Info-vax] Re; Spiralog, RMS Journaling (was Re:
> FREESPADRIFT)
>

> On 2016-06-18 18:31:49 +0000, Kerry Main said:
>
> > One only has to power off a system in the middle of a benchmark on
> that
> > system to see what the effects could be in the real world.
>
> I've worked more than a few OpenVMS systems that have to be
> manually
> rebooted, and with non-trivial recovery processing. There's no APIs
> for coordinating with backups, no integrated means of checkpointing
> application data, and a whole pile of other limits.
>
> > With new disks like SSD and really big memory technologies emerging,
> > one needs to ask whether the difference in perf is really worth the=
> > potential for data integrity issues? Especially when one considers the
> > considerable pain one must go through to recover from some
> significant
> > event like a system crash or hardware error or power failure on a write
> > back designed system.
>
> Yeah, and I've had that same issue with long-running jobs on OpenVMS,
> where the primitive tools force that into the database and particularly
> onto the end developer, There's nothing even remotely close to Time
> Machine in OpenVMS, either. Then there's that RMS is incapable of
> getting consistent live backups.
>

You are missing the point.

Similar to asynch replication, and regardless of the OS platform, if one
adopts write back caching strategies, you are saying that you are ok with
potentially losing some data because the benefits of faster performance
is worth the risk of losing some tbd amount of data and whatever prob's
that might create.

No platform, including OpenVMS, can ever state they will never lose
data, but when the impact of lost or inconsistent data is high, you use
sync replication and write through strategies to significantly reduce the
risk.

These are not rocket science concepts.

[snip]

[Stephen Hoffman]

On 2016-06-18 20:59:14 +0000, John Reagan said:

> On Saturday, June 18, 2016 at 3:02:29 PM UTC-4, Stephen Hoffman wrote:
>> On 2016-06-18 18:07:05 +0000, Paul Sture said:
>>
>>> both XAB$L_EBK and XAB$W_FFB Field are "meaningful for sequential files only"
>>
>> I'd prefer that any (errant) requests violating that would kick back a
>> run-time error when used on inappropriate file formats, but that
>> concept and that capability simply does not exist within the (current)
>> OpenVMS RMS APIs.
>>
>> Ugly case would be an explicit NaN/NaT returned in these fields
>> secondary to an errant access; a Not a Number or Not a Thing response.
>> OpenVMS and various databases certainly have have examples akin to
>> that, but OpenVMS has no particularly consistent implementation of
>> that. In more than a few fields in the various data structures, RMS
>> doesn't, though.
>
> NaNs are floating point. NaTs don't exist in memory, only in registers
> (and are Itanium specific).

Okay. Let me try this a different way. The RMS interface — and many
of the other OpenVMS interfaces — has no mechanism for differentiating
a data field that is invalid or uninitialized from a field that
contains a valid value, and RMS further has no ready means to flag that
mere accesses to these fields are invalid at run-time. This is the
software equivalent of NaN or NaT that can be implemented in various
contexts, or of a database field that's marked as null. This
differentiation is analogous to what Itanium and other systems are
using NaN, NaT or such to represent. Something which really isn't.
This as differentiated from a data field that's empty, zero or some
other such explicitly-set value.

As a different and newer and software alternative, passing a message —
an OO or method invocation — the (entirely hypothetical) OORMS
framework can detect this access and this condition and can toss back a
run-time error, rather than leaving the user to read the fine manuals,
or to get silently bad data. Which is where the RTFM approach leaves
developers. The OO approach also means needing rather less glue code,
and also avoids revisiting data structure changes such as that from the
NAM to the NAML. Beyond reducing the piles of code chunder, the method
call approach also avoids the need for an analog to NaN or NaT in the
data structures. But then the chances of seeing OORMS or OO system
service calls and frameworks is zilch, and any wholesale replacement
for RMS unlikely at best.

I don't expect to see either OpenVMS or RMS dragged this far forward
anytime soon (if at all), however. But this is what other operating
systems are using now, so it's what many developers are already
familiar with.

[Stephen Hoffman]

On 2016-06-18 22:57:50 +0000, Kerry Main said:

> You are missing the point.

Um, no. I'm not. I know exactly what approach you are following, I
am very familiar with DT as implemented on OpenVMS, and I've seen it
fail utterly for products and for the competitiveness of more than a
few organizations. General requirements and expectations — not losing
transactions — can be provided in different ways. Sometimes in very
different ways. Speeds, feeds, features, tools and designs change.
Sometimes for worse. Often for better. Existing software and
existing designs and existing assumptions either adapt, or they get
replaced. Failure to consider and examine and adopt newer approaches
is how various products and various organizations fail. This is where
I certainly hope that new file system VSI is looking at is way past
SSD-ready and pondering newer non-volatile mechanisms and not
compromising for HDD support, for instance. Where clustering adopts
faster host-to-host and server-level replication designs — whether
that's akin to MC or Galaxy or RDMA or otherwise; host-based server
shadowing, etc.

[lawren...@gmail.com]

On Saturday, June 18, 2016 at 11:51:46 PM UTC+12, Paul Sture wrote:

> On 2016-06-18, Lawrence D’Oliveiro wrote:
> > On Saturday, June 18, 2016 at 8:56:38 PM UTC+12, Paul Sture wrote:
> >
> >> Rather useful when determining how much space is required for a
> >> Time Machine Backup, I imagine.
> >
> > Time Machine plays fast and loose with your filesystem integrity
> > (multiple hard links to directories), and you pay the price in
> > reliability
> >
> > <http://arstechnica.com/civis/viewtopic.php?t=1252811>.
>
> Yes, I also went through problems with TM, but they appear to be fixed.
>
> TM can be terribly slow in certain circumstances. I take regular
> clone copies using other software.

The funny thing is, rsync can perform mirror backups with deduping of unchanged files, without messing around with filesystem integrity; why couldn’t Apple have done the same?

> >> cf VMS only storing the block sizes of files rather than the number of
> >> bytes; this can be problematic for utilities ported from a system which
> >> has an easy/cheap way to supply the exact size in bytes.
> >
> > This was the example Linus Torvalds gave of why he hated VMS...
>
> Eh? There are many reasons to dislike an OS but 'hating VMS' for that
> *perceived* problem? Really?

A real problem. And yes, I saw his point.

> From another perspective, I ran straight into the spat between Torvalds
> and Ted Ts'o when ext4 became the default on Ubuntu - this one did
> produce data loss for me (until I worked out what was going on):
>
> "Linus Torvalds Upset over Ext3 and Ext4"
>
> <http://www.linux-magazine.com/Online/News/Linus-Torvalds-Upset-over-Ext3-and-Ext4>

That seemed to be a matter of default settings. You could change the options at mount time to prioritize what you wanted.

[lawren...@gmail.com]

On Sunday, June 19, 2016 at 6:35:05 AM UTC+12, Kerry Main wrote:

> With OpenVMS and its default write-through strategies, data integrity
> has always been the primary driver.

I fully accept that. And I, too, deplore vendor strategies that prioritize good-looking benchmarks over data integrity. Particularly insidious are hard drives with on-board cache, where a request to “make sure you really, really write this data to disk” can be met with “OK, I really, really have done it” right down to the SATA interface level, only to have the data still sitting in some controller cache buffer, not really on the disk at all, so it could still be lost on the next power failure.

Filesystem journalling is a way of improving your confidence in the integrity of the volume structure across crashes. So instead of doing a fsck (MOUNT/VERIFY?) on every reboot, you can defer that to, say, once every six months (I think that’s what the Linux systems usually do).

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: 18-Jun-16 8:03 PM
> To: info...@info-vax.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [New Info-vax] Re; Spiralog, RMS Journaling (was Re:
> FREESPADRIFT)
>

> On 2016-06-18 22:57:50 +0000, Kerry Main said:
>
> > You are missing the point.
>
> Um, no. I'm not. I know exactly what approach you are following, I
> am very familiar with DT as implemented on OpenVMS, and I've seen it
> fail utterly for products and for the competitiveness of more than a
> few organizations. General requirements and expectations — not losing
> transactions — can be provided in different ways. Sometimes in very
> different ways. Speeds, feeds, features, tools and designs change.

I am not talking about just OpenVMS . This is platform agnostic and not
specific to any OS. It is also a timeless core principle that is not likely
going to change any time soon.

As I stated in my previous reply, by its very nature, write back & asynch
update strategies increase the risk of a significant event (site lost or
system halted due to HW issue?) causing one to lose data.

When you tell an application that the update was completed, and yet
no form of data persistence has taken place that transcends at least
2 sites, then there is a risk that data will be lost in the event of a site
and or server being lost.

Remember that a site being lost could be complete and permanent,
so system and SAN battery type solutions are only short term risk
mitigations.

This is not to say that asynch/write back are not valid strategies, but
they are cost/benefit/risk tradeoffs that depends on what the potential
impact is should the significant event really occur.

This is the basics of RPO=0 discussions when doing DR planning. If the
business states it can live with something other than RPO=0, then
certainly it makes sense to look at alternative solutions.

[lawren...@gmail.com]

On Sunday, June 19, 2016 at 6:41:48 AM UTC+12, Simon Clubley wrote:

> Also, IMHO I think the default sequential record type for today's
> world should be stream (and hence the terminator is also included in
> the file data.) I do not think it should be variable length records.

Hands up all those who think RMS should die already...

[David Froble]

Stephen Hoffman wrote:
> On 2016-06-18 22:57:50 +0000, Kerry Main said:
>
>> You are missing the point.
>
> Um, no. I'm not. I know exactly what approach you are following, I am
> very familiar with DT as implemented on OpenVMS, and I've seen it fail
> utterly for products and for the competitiveness of more than a few
> organizations. General requirements and expectations — not losing
> transactions — can be provided in different ways. Sometimes in very
> different ways. Speeds, feeds, features, tools and designs change.
> Sometimes for worse. Often for better. Existing software and existing
> designs and existing assumptions either adapt, or they get replaced.
> Failure to consider and examine and adopt newer approaches is how
> various products and various organizations fail. This is where I
> certainly hope that new file system VSI is looking at is way past
> SSD-ready and pondering newer non-volatile mechanisms and not
> compromising for HDD support, for instance.

I think this is a good point, and I have the same hopes. I'd ask, why re-invent
the wheel? VMS already has ODS-2 (and that other thingy I don't use) for
rotating rust. Something new should be FOR something new.

[David Froble]

I would suggest "what does it matter?"

It's not like some other file is going to use the unused portion of some disk
block, or, as suggested, unused part of 4096 bytes ....

[David Froble]

Simon Clubley wrote:
> On 2016-06-18, Paul Sture <nos...@sture.ch> wrote:
>> On 2016-06-18, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:
>>
>>> Do you count that metadata in the file size, or not? Arguably no, but
>>> it's inherently part of the file on OpenVMS and not at all easy to
>>> exclude — whether VFC or the indexed file structures or otherwise.
>> Tricky. If you are looking at the output of TYPE, then the record
>> attributes (e.g. Record attributes Carriage return carriage control)
>> make a difference to the output.
>>
>
> IMHO, it depends on the context. If you are doing block level reads,
> then you count the size of the on-disk metadata in the file size.
>
> If you are doing record level reads, then you do not include the
> metadata IMHO but you _do_ add in any additional terminator bytes to
> the length (which might not actually be stored on disk).
>
> You need the former if you are doing an image copy. You need the latter
> if you want to tell a webserver at which point it should resume the
> download.

Hmmm ... Never considered that. Then again, for most HW, you're probably going
to read and write whole blocks anyway. If I was solving the problem, which I'm
not, I do think that I'd re-start on block boundaries.

> Also, IMHO I think the default sequential record type for today's
> world should be stream (and hence the terminator is also included in
> the file data.) I do not think it should be variable length records.

I'm still thinking about that one. Perhaps I'm blinded by past expectations.

> The problem is that today's protocols are simply not designed to
> handle the case of metadata buried within the file contents
> themselves; the people who design this stuff have probably never
> even encountered that case.

Perhaps we need different designers ???

:-)

> I do wonder how the IBM mainframe people handle this problem.

However they want, and I'm sure they don't care about other environments ..

[David Froble]

Perhaps you are ready to re-write billions of dollars of existing software?

Why cannot the old and new co-exist?

[lawren...@gmail.com]

On Sunday, June 19, 2016 at 2:48:34 PM UTC+12, David Froble wrote:

>
> Lawrence D’Oliveiro wrote:
>
>> Hands up all those who think RMS should die already...
>
> Perhaps you are ready to re-write billions of dollars of existing software?

This is the point I brought up before, about discounted cost versus total cost. As every new headache with the legacy technology appears, it is cheaper to spend that small fortune on patching it up, rather than a large fortune on getting rid of it. After a while, all those small fortunes add up to much more than the large fortune. And you are still stuck with the legacy technology.

> Why cannot the old and new co-exist?

The new technology does not have the problem adapting to the old, it is the old technology that has trouble adapting to the new.

[David Froble]

Ok, dumb Dave is having a problem with this. And yes, I had problems with the
concept of Null in Visual Basic.

A data field is a place where data is stored. Even if it is zeros, or whatever,
it's still data. An application can determine if it doesn't like the value of
the data. I'm not sure anything else can do so.

[David Froble]

lawren...@gmail.com wrote:
> On Sunday, June 19, 2016 at 2:48:34 PM UTC+12, David Froble wrote:
>> Lawrence D’Oliveiro wrote:
>>
>>> Hands up all those who think RMS should die already...
>> Perhaps you are ready to re-write billions of dollars of existing software?
>>
>
> This is the point I brought up before, about discounted cost versus total
> cost. As every new headache with the legacy technology appears, it is cheaper
> to spend that small fortune on patching it up, rather than a large fortune on
> getting rid of it. After a while, all those small fortunes add up to much
> more than the large fortune. And you are still stuck with the legacy
> technology.

Only if there are problems. No blood, no foul, right?

>> Why cannot the old and new co-exist?
>
> The new technology does not have the problem adapting to the old, it is the
> old technology that has trouble adapting to the new.

That assumes the old has any need to adapt to the new.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
> lawrencedo99--- via Info-vax
> Sent: 18-Jun-16 10:57 PM
> To: info...@rbnsn.com
> Cc: lawren...@gmail.com
> Subject: Re: [Info-vax] RMS record metadata, was: Re: Re; Spiralog, RMS
> Journaling (was Re: FREESPADRIFT)
>

"New" technology does not necessarily mean "better".

What is so often over looked with "new technology" is that there is
always a period of bugs, tweaks, unforeseen challenges & workarounds
that the early adopter has to address.

In addition, techie marketing hype touting the "benefits" of new tech
often overlooks the challenges of highly customized & rock solid business
logic integrated with the "old" code.

That rock solid business logic is where the real added value of IT exists &
Is often supported by more junior resources who do simple tweaks, but
in no way would they pretend that they could re-write the business logic
using some form of "cool" new technology.

The marketing hype will often tout the expense of hiring resources to
maintain the code so will spend $10M+ in a major redo of the application
that takes 2+ years (often freezing current state enhancements) in order
to save the cost of simply paying their current resources a bit more or
hiring resources at more than what they would get at other companies.

Smarter companies have woken up to the hype and are now looking
at "upgrade-and-integrate" strategies vs. "rip-and-replace".

[Simon Clubley]

On 2016-06-19, David Froble <da...@tsoft-inc.com> wrote:

> Simon Clubley wrote:
>>
>> IMHO, it depends on the context. If you are doing block level reads,
>> then you count the size of the on-disk metadata in the file size.
>>
>> If you are doing record level reads, then you do not include the
>> metadata IMHO but you _do_ add in any additional terminator bytes to
>> the length (which might not actually be stored on disk).
>>
>> You need the former if you are doing an image copy. You need the latter
>> if you want to tell a webserver at which point it should resume the
>> download.
>
> Hmmm ... Never considered that. Then again, for most HW, you're probably going
> to read and write whole blocks anyway. If I was solving the problem, which I'm
> not, I do think that I'd re-start on block boundaries.
>

You've missed the point David. HTTP restart mechanisms only know about
what byte offset to start resending the server based copy of the file
from; they know nothing about the client.

On Windows and Unix, the client byte offset is the same as the server
byte offset so the client just asks for the size of the partial file
as it currently exists on the client and sends that.

On VMS, if that byte stream has been converted to variable length records
during the download, then the client needs to read the _whole_ of the
partial file to determine how much actual file data there is in the
partial file and it needs to add back in the size of any record
terminators which were not written to disk.

Only after you have done all that, can you tell the server which file
position it needs to start resending the file from.

>> Also, IMHO I think the default sequential record type for today's
>> world should be stream (and hence the terminator is also included in
>> the file data.) I do not think it should be variable length records.
>
> I'm still thinking about that one. Perhaps I'm blinded by past expectations.
>

There are some specialist uses I can think of but I don't see any
advantage for normal sequential files such as program source code
and text files to be variable length records instead of stream
records.

One specialist use I can think of is Fortran carriage control. It's
been a very long time since I've written Fortran code, so I don't
know the answer to this question. How is Fortran carriage control
implemented on a stream only filesystem such as on Unix or Windows ?

Are the carriage control characters actually encoded within the file
data itself ?

Can anyone think of any other uses in today's world for variable
length records (and VFC records) for disk files which can't be handled
by a stream record format ?

There's an argument for binary sequential data because the length
bytes are not part of the data stream, and hence cannot be confused
with it. OTOH, the application generally knows the structure of that
data so it can be stored as stream or fixed length records. This last
part is actually _required_ if you are creating a binary sequential
file for interchange with another system (for example an image file).

>> The problem is that today's protocols are simply not designed to
>> handle the case of metadata buried within the file contents
>> themselves; the people who design this stuff have probably never
>> even encountered that case.
>
> Perhaps we need different designers ???
>
>:-)
>

It's a byte stream world out there David, not a record orientated one
and VMS needs to be able to deal with that efficiently.

[Stephen Hoffman]

On 2016-06-19 01:44:35 +0000, lawren...@gmail.com said:

> The funny thing is, rsync can perform mirror backups with deduping of
> unchanged files, without messing around with filesystem integrity; why
> couldn’t Apple have done the same?

rsync is somewhat analogous to Time Machine in that both copy files
around. But the two differ. rsync is a remote synchronization tool.
Time Machine is a backup tool, and maintains months or years of
previous versions of files over one or more local or remote backup
target volumes, as well as providing the ability to restore and boot
the recovered volume(s), among other differences from rsync. TM avoids
storing duplicate copies of a file, though does not provide the file or
sector-level de-dupling available with some storage systems. Having
used both, TM and rsync are different targets and different users and
different implementations.

As for the integrity, Time Machine solves the problem that the vast
majority of its users has. That the vast majority of OpenVMS users
have, but have to write their own DCL procedures for.

Features of APFS provide capabilities that would appear to benefit Time
Machine, so it'll be interesting to see how that's integrated. TM is
one of the last consumers of AFP on macOS systems, and it's very likely
it'll migrate to SMB in some upcoming release. OpenVMS BACKUP, by
contrast, works very well but has limited remote access — other than
via an existing cluster and its storage access or via DECnet FAL, or
via some unsupported DECnet hackery*, and none those lend themselves to
easy restoration and recovery. BACKUP does have compression and Time
Machine does not, but BACKUP lacks de-duping, and the incremental
support within BACKUP is rather more complex to implement than it
should be.

*unsupported DECnet hackery: http://labs.hoffmanlabs.com/node/598

[Stephen Hoffman]

On 2016-06-19 01:46:17 +0000, Kerry Main said:

> I am not talking about just OpenVMS . This is platform agnostic and not
> specific to any OS. It is also a timeless core principle that is not
> likely going to change any time soon.
>
>
> As I stated in my previous reply, by its very nature, write back &
> asynch update strategies increase the risk of a significant event (site
> lost or system halted due to HW issue?) causing one to lose data.
>
>
>
> When you tell an application that the update was completed, and yet no
> form of data persistence has taken place that transcends at least 2
> sites, then there is a risk that data will be lost in the event of a
> site
>
> and or server being lost.

And what do you think server-level replication is? OpenVMS doesn't
have any concept of this — wait, let me rephrase that, OpenVMS fully
supports remote server-level replication, you just have to write and
support and maintain all the necessary code involved — but other
platforms do have this and are working to add this. Remote access to
redundant servers with either fast nonvolatile storage or with a way to
reliably roll volatile storage to disk during an outage are faster than
dinking around with tertiary and quaternary storage such as hard disks
and tape. Or you use the remote server as a replication log. Though
if you're at lower ranges of transactions but have a comparatively
large budget — where most of the OpenVMS servers I've worked with are
used — sure, your long-familiar synchronous-write-to SSD or disk works.
Even in the lower performance ranges, remote server replication may
still be of interest, even there.

[Stephen Hoffman]

On 2016-06-19 02:35:16 +0000, David Froble said:

> I would suggest "what does it matter?"
>
> It's not like some other file is going to use the unused portion of
> some disk block, or, as suggested, unused part of 4096 bytes ....

If I'm mapping the user data into memory, having its size is useful.
Otherwise, I slog through RMS records or such.

If I'm messing with disk allocations and more directly managing storage
— something I'd prefer to avoid, but that can arise on OpenVMS and
other systems — then the total size of the file including all user data
and all of its overhead and/or pre-allocation and/or metadata is of
interest.

This is akin to UTF-8, where you might be looking for the numbers of
glyphs in the string, or for the size of the required memory buffer to
store the string.

Many OpenVMS apps simply don't have the concept of mapping the entire
file into memory, they assume datasets far larger than available memory
and/or comparatively weak and slow paging I/O support, and for those
apps the total data size wasn't really a factor. They're written to
deal with records. (This is also where the overhead of dealing with
zillions of small hunks via RMS — and not a few and larger hunks — can
become a run-time performance-limiting factor.) For apps that map
whole files or that map masses of data into memory, the user dataset
size (modulo any virtual memory allocation granularity in play) tends
to matter more, unless those apps are implementing their own paging —
as various databases do.

Defining data structures and allowing the run-time system to pack and
unpack records for the app is a concept that largely doesn't exist on
OpenVMS. You have to roll your own (glue) code to do that.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@info-vax.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: 19-Jun-16 10:57 AM
> To: info...@info-vax.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [New Info-vax] Re; Spiralog, RMS Journaling (was Re:
> FREESPADRIFT)
>

> On 2016-06-19 01:46:17 +0000, Kerry Main said:
>
> > I am not talking about just OpenVMS . This is platform agnostic and not
> > specific to any OS. It is also a timeless core principle that is not
> > likely going to change any time soon.
> >
> >
> > As I stated in my previous reply, by its very nature, write back &
> > asynch update strategies increase the risk of a significant event (site
> > lost or system halted due to HW issue?) causing one to lose data.
> >
> >
> >
> > When you tell an application that the update was completed, and yet
> no
> > form of data persistence has taken place that transcends at least 2
> > sites, then there is a risk that data will be lost in the event of a
> > site
> >
> > and or server being lost.
>
> And what do you think server-level replication is?

Its asynchronous communications whereby the app thinks the data
has been updated, when in fact, it has not.

Case in point:

- app thinks a multi- $M transaction has been completed. App
continues with post update processing.
- before async replication can complete, plane crashes into DC
- transaction update has not been written to remote site
- result - Bank App thinks transaction was completed, but when
the App is restarted at remote site, there are inconsistencies
and possibly huge legal/tech/regulatory issues to sort out.

>OpenVMS doesn't
> have any concept of this — wait, let me rephrase that, OpenVMS fully
> supports remote server-level replication, you just have to write and
> support and maintain all the necessary code involved — but other
> platforms do have this and are working to add this. Remote access to
> redundant servers with either fast nonvolatile storage or with a way to
> reliably roll volatile storage to disk during an outage are faster than
> dinking around with tertiary and quaternary storage such as hard disks
> and tape. Or you use the remote server as a replication log. Though
> if you're at lower ranges of transactions but have a comparatively
> large budget — where most of the OpenVMS servers I've worked with
> are
> used — sure, your long-familiar synchronous-write-to SSD or disk works.
> Even in the lower performance ranges, remote server replication may
> still be of interest, even there.
>

You keep trying to bring this back to what OpenVMS has or has not.

It’s a platform agnostic issue. Applies to OpenVMS as well as every
other platform on the planet.

You can reduce the time between data replication periods to reduce the
chances of inconsistent data happening, but the bottom line is that for
critical apps, when you tell the app with RPO=0 requirements that the
update is complete, then that update has to be complete in both sites
or it is rolled back. Data has to be consistent - even in the case where
a plane hits the DC.

This is also where 2PC comes in as well.

To address your "grass is greener on the other side" type comments,
I would encourage you to review this recent (Feb 2016) distributed
systems presentation "Architecting Distributed Databases for Failure":
https://www.infoq.com/presentations/data-integrity-distributed-systems?utm_campaign=rightbar_v2&utm_source=infoq&utm_medium=presentations_link&utm_content=link_text
(click on InfoQ video)

The grass on the other side is certainly not as green as you like to
make it out to be.

:-)

[Stephen Hoffman]

On 2016-06-19 02:48:33 +0000, David Froble said:

> lawren...@gmail.com wrote:
>>
>> Hands up all those who think RMS should die already...
>
> Perhaps you are ready to re-write billions of dollars of existing software?
>
> Why cannot the old and new co-exist?

That works for a while, but then you still have to update and test and
maintain the old stuff, and there's a non-trivial and increasing cost
to that effort.

You often end up putting in place workarounds, and which almost always
hamper new adoptions and new development, too.

If there are sufficiently good reasons and sufficient incentives to
migrate, then the old code will migrate across.

If not, then there were problems — whether technical or incentives or
ease-of-migration or otherwise — with the new code, or your customers
are underfunded and/or understaffed and/or are focused elsewhere.

Assuming that you didn't screw up the new code bits somehow and did
deliver substantial improvements, then those folks that aren't moving
forward and aren't updating their tools are a completely different
business and different business model, with different expectations.
This is where (and why) long-term-support products are offered. by some
vendors Though those always age out, too.

But in this case, RMS isn't ever going away on OpenVMS, if for no
reason other than that it's far too entrenched in the operating system
and tools. It'd be nice to see some file system defaults change (e.g.
not defaulting to creating VFC), and better and simpler file system
alternatives becoming available. But RMS works. (If VSI did ever
replace RMS, ODS-2 and ODS-5 and related bits with truly modern
alternatives and with more modern programming interfaces, it'd approach
— or would effectively be — a completely different operating system.)

[Stephen Hoffman]

On 2016-06-19 02:57:05 +0000, David Froble said:

> Stephen Hoffman wrote:
>>
>> Okay. Let me try this a different way. The RMS interface — and many
>> of the other OpenVMS interfaces — has no mechanism for differentiating
>> a data field that is invalid or uninitialized from a field that
>> contains a valid value, and RMS further has no ready means to flag that
>> mere accesses to these fields are invalid at run-time. This is the
>> software equivalent of NaN or NaT that can be implemented in various
>> contexts, or of a database field that's marked as null. This
>> differentiation is analogous to what Itanium and other systems are
>> using NaN, NaT or such to represent. Something which really isn't.
>> This as differentiated from a data field that's empty, zero or some
>> other such explicitly-set value.
>
> Ok, dumb Dave is having a problem with this. And yes, I had problems
> with the concept of Null in Visual Basic.
>
> A data field is a place where data is stored. Even if it is zeros, or
> whatever, it's still data. An application can determine if it doesn't
> like the value of the data. I'm not sure anything else can do so.

The NaT/NaN/null stuff is metadata. It is data that describes the
data. In this case, that there is no data.

Other metadata common on OpenVMS lurks in string descriptors, for
instance — data that describes data.

Having that metadata separate or out-of-band or otherwise
differentiated from the data means you can unambiguously tell when you
have data and when you don't.

The design of the RMS data structures and APIs don't allow that to
happen, which leads long-time users to point to RTFM as "the answer".

This is the "you should have known not to look in that data cell",
rather than having the operating system flag the coding error.

Using a non-sensical constant is an older form of this, but that
assumes the field can never be or can never become {zero, negative,
whatever}, and it assumes that the user is paying attention to those
values, and it's really effectively impossible to add new non-sensical
constants if (when?) changes are required — you've largely pushed the
error detection and error handing into the user code and into the
documentation, after all — and there's no single non-sensical error
constant that works across all APIs because the valid ranges inherently
varies.

Newer systems don't tend to use OpenVMS-style descriptors, but various
systems and frameworks do use objects for this. Objects are an updated
and opaque data structure that's very much a variant of an OpenVMS
descriptor, and usually also "containing" the referenced data. Why
opaque? Because once you've established a set of descriptors, you
can't change those as the user code knows about the contents and format.

The BASIC compiler and run-time largely "hides" the use of descriptors,
so that's somewhat more akin to an object reference than is typical
when working with descriptors in C, C++, Macro32, Bliss or such; in
languages where you often have to work more directly with the
descriptor metadata.

[Stephen Hoffman]

On 2016-06-19 13:04:49 +0000, Kerry Main said:

> "New" technology does not necessarily mean "better".
> What is so often over looked with "new technology" is that there

> isalways a period of bugs, tweaks, unforeseen challenges & workarounds

> that the early adopter has to address.
>
> In addition, techie marketing hype touting the "benefits" of new

> techoften overlooks the challenges of highly customized & rock solid
> businesslogic integrated with the "old" code. That rock solid business

> logic is where the real added value of IT exists & Is often supported

> by more junior resources who do simple tweaks, butin no way would they
> pretend that they could re-write the business logicusing some form of

> "cool" new technology.
>
> The marketing hype will often tout the expense of hiring resources to
> maintain the code so will spend $10M+ in a major redo of the
> application that takes 2+ years (often freezing current state

> enhancements) in orderto save the cost of simply paying their current
> resources a bit more orhiring resources at more than what they would

> get at other companies.
>
> Smarter companies have woken up to the hype and are now looking at
> "upgrade-and-integrate" strategies vs. "rip-and-replace".

Isn't this the flip-side of that "blue ocean" stuff you're fond of pointing at?

[Stephen Hoffman]

On 2016-06-19 15:32:54 +0000, Kerry Main said:

> You keep trying to bring this back to what OpenVMS has or has not.

No, I keep bringing this back to dragging OpenVMS forward, and toward
supporting competitive features.

Because — with the much-vaunted features and benefits you keep pointing
to, and various of which I quite agree with — OpenVMS has ended up
where it is now.

Because OpenVMS is comparatively hard to use, hard to develop for, hard
to configure, hard to install, and such. In five years, tools on
other platforms are only going to be better. Some of those vaunted
features — that wonderfully cautious I/O and the record-oriented design
for file access — makes applications slow. That'll take some thought,
quite possibly involving replicated servers and other
wholly-new-to-OpenVMS approaches. Some of those vaunted and valuable
features — clustering, HBVS — need a configuration and management
overhaul and a re-think — e.g. integrated distributed authentication
client and server, a logical volume manager, vastly easier
configurations, etc. Some features — like the current PKE security
"design" — should be copied wholesale into NLA0: and reimplemented and
integrated into OpenVMS, rather than being ad-hoc and grafted on. This
also includes better tools for your app-stacking and for app isolation
— sandboxes/jails — and better tools for local and hosted deployments —
provisioning, etc. — and a whole host of other areas that are expected
or will be expected over the next five or ten years. Better patch
support, DVCS support, integrated relational database support, etc.

You keep telling me OpenVMS is great. For a number of apps, it is.
But there is very little here that's presently going to gain the
attention of new sites and wholly new applications, and the x86-64 port
is just a down payment on the effort ahead of VSI here. There's a
tremendous amount of work involved here for VSI and also for end-users
to start adopting current and new OpenVMS features, and the competitive
and the comparative operating system platforms and tools are themselves
always being updated. Often substantially.

You keep telling everybody about "blue oceans". Maybe VSI heads that
way. But that blue ocean will inherently have to be far different and
far more advanced than what OpenVMS offers today. Hopefully much
simpler, faster, cheaper, too.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of

> Stephen Hoffman via Info-vax
> Sent: 19-Jun-16 12:21 PM
> To: info...@rbnsn.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [Info-vax] RMS record metadata, was: Re: Re; Spiralog,
RMS
> Journaling (was Re: FREESPADRIFT)
>

Nope - the blue oceans is for new service offerings.

The approach above Is when you have an established service offering
whereby there are SLA's and expectations from your user / business
community.

The upgrade-and-integrate approach also takes into consideration the
Integration of new service offerings with older ones.

[Stephen Hoffman]

On 2016-06-19 18:18:15 +0000, Kerry Main said:

>> Hoff: Isn't this the flip-side of that "blue ocean" stuff you're fond

>> of pointing at?
>
> Nope - the blue oceans is for new service offerings.

And doesn't a "blue ocean" have to put together existing and new tools
in a way that allows those services to be delivered?

That involves both knowing what the folks want before they want it.,
and then getting there in a sustainably-profitable way.

That — in IT — usually means adapting some software and reusing other
software — because nothing in IT is ever entirely "from scratch".
It's always built on existing tools and processes.

If VSI wants to take a product and associated services — in a new
direction — because the existing market for OpenVMS isn't exactly
seeing substantial growth — they're going to have to adapt.

For your case, if Stark is going to have high I/O loads, then you're
likely going to have to implement geographically-dispersed servers both
for redundancy and particularly for lower latency, and then figure out
how to coordinate and synchronize those. This as clustering inevitably
runs into latency. At higher I/O loads, you're either buying piles of
really fast storage hardware — thankfully for folks other than the
major vendors, the prices on that are dropping — and quote possibly
replicating your data in battery-protected servers at the server level,
and not at the "traditional" disk spindle or array level. That's
before looking at contending with what will likely be greatly shifting
I/O loads due to game releases and seasonal variations, and this gets
interesting as hardware and network over-provisioning gets hugely
expensive; that comes right off the bottom line. If you do start
toward replicating data centers, then you're going to be looking at
DVCS and other and newer tools, at scaling deployments and other topics
— most off which I've been pointing you at, too.

But then... Blue Oceans ain't ever "all-new". Nothing in IT is ever
that way. Never all-new. It's all built on existing tools. Bolted
together from old tools, and usually from some of the best of new ideas
and new tools, and with a dollop of insight. As with most other
projects, the idea behind "Blue Ocean" is worthless. It's the
execution and the timing that matters. Then — for a successful
endeavor — dragging those old tools and now-old ideas forward fast
enough that some new competitor doesn't show up and "Blue Ocean" your
business.

But enough of the who-moved-my-cheese.

Please: go learn about server-level replication, about DVCS tools, and
about newer approaches toward deployments and provisioning. The IT
business moves forward quickly, after all.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: 19-Jun-16 12:42 PM
> To: info...@rbnsn.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [Info-vax] Re; Spiralog, RMS Journaling (was Re:
> FREESPADRIFT)
>

> On 2016-06-19 15:32:54 +0000, Kerry Main said:
>
> > You keep trying to bring this back to what OpenVMS has or has not.
>
> No, I keep bringing this back to dragging OpenVMS forward, and toward
> supporting competitive features.
>

I was talking about generic asyn/replication/write back concepts that are
platform agnostic.

> Because — with the much-vaunted features and benefits you keep
> pointing
> to, and various of which I quite agree with — OpenVMS has ended up
> where it is now.
>

Let's not forget that the primary reason why OpenVMS is not as advanced
In some areas (while leading in others), is not the lack of tech Engineering
or strategies.

It is because OpenVMS funding has been slashed and cut back starting
with DEC, then Compaq and then HP. With each acquisition, OpenVMS
became an even smaller fish in a rapidly growing ocean of products.
When I left in 2012, the number of companies that HP acquired since
Compaq was something like 43.

Now, moving to the future, one can continue to look through the rear
view mirror, or alternatively, keep looking through the front window.

OpenVMS now has a new owner 100% focused on OpenVMS. That’s
the good news. From a business perspective, you have to admit when
new owners take over an existing company, they would be crazy to
ignore the current installed base.

X86-64 with binary Alpha/IA64 translators, new file system, new
TCPIP stack, Java8 w new compiler enhancements are all examples of
things that will keep (delight) the installed base.

When the previous things are added to areas like galaxy, increased
scalable clusters using RoCEv2 for low latency DLM, enhanced VM,
Enterprise Directory integration (LDAP, resource mgmt./control)
capabilities are all examples which could (imho) start getting the
attention of new Customers.

Of course, marketing and pricing strategies are key to this as well.

System management features are also needed (e.g. SNMP V3), but
new customers do not buy an OS for how cool the mgmt. tools are.

With all of the above, one needs to remember that VSI does not need
to do everything themselves. As an example, Microsoft has a backup
utility that few med-large shop use. These customers use alternative
commercial offerings from third parties.

I expect there will be third parties that will jump on the OpenVMS
wagon with their offerings as well once things start to get rolling again.

> Because OpenVMS is comparatively hard to use, hard to develop for,
> hard
> to configure, hard to install, and such. In five years, tools on
> other platforms are only going to be better.

Of course - just as I expect OpenVMS mgmt. tools (VSI, 3rd party) to
Advance as well.

As just one example -
Server to Server network latencies (App-DB server, App-App, Server-
Server replication) are by far the biggest source of latency in most
solutions today. While CPU, Memory, and cache technologies have all
increased exponentially, the latency (not speed) associated with all of
this inter node network activity has not.

Reference:
http://www.research.ed.ac.uk/portal/files/14176012/Novakovic_Daglis_et_al_2014_Scale_Out_NUMA.pdf

Red ocean approach to problem -
How can we reduce the latency with all this server App-App-DB
network activity?

Blue ocean approach to problem -
How can we ELIMINATE the latency with all this server App-App-DB
network activity?

[Kerry Main]

> -----Original Message-----
> From: Kerry Main [mailto:kemain...@gmail.com]
> Sent: 19-Jun-16 3:01 PM
> To: 'comp.os.vms to email gateway' <info...@rbnsn.com>
> Subject: RE: [Info-vax] Re; Spiralog, RMS Journaling (was Re:
> FREESPADRIFT)
>

[snip..]

Crap - Typo in following paragraph ..should read:

While CPU, Memory, and cache technologies have all *improved*

exponentially, the latency (not speed) associated with all of this inter
node network activity has not.


>
> As just one example -
> Server to Server network latencies (App-DB server, App-App, Server-
> Server replication) are by far the biggest source of latency in most
> solutions today. While CPU, Memory, and cache technologies have all
> increased exponentially, the latency (not speed) associated with all of
> this inter node network activity has not.
>
> Reference:
> http://www.research.ed.ac.uk/portal/files/14176012/Novakovic_Daglis_
> et_al_2014_Scale_Out_NUMA.pdf
>
> Red ocean approach to problem -
> How can we reduce the latency with all this server App-App-DB
> network activity?
>
> Blue ocean approach to problem -
> How can we ELIMINATE the latency with all this server App-App-DB
> network activity?
>

[Paul Sture]

On 2016-06-18, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
> On 2016-06-18, Paul Sture <nos...@sture.ch> wrote:
>> On 2016-06-18, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:
>>
>>> Do you count that metadata in the file size, or not? Arguably no, but
>>> it's inherently part of the file on OpenVMS and not at all easy to
>>> exclude — whether VFC or the indexed file structures or otherwise.
>>
>> Tricky. If you are looking at the output of TYPE, then the record
>> attributes (e.g. Record attributes Carriage return carriage control)
>> make a difference to the output.
>>
>

> IMHO, it depends on the context. If you are doing block level reads,
> then you count the size of the on-disk metadata in the file size.
>
> If you are doing record level reads, then you do not include the
> metadata IMHO but you _do_ add in any additional terminator bytes to
> the length (which might not actually be stored on disk).

Yes for things like TYPE and the print symbiont, but typical end-user
applications don't need to interrogate those attributes, and the
output format is probably going to be vastly different from the
input format; I'm thinking of something like transactions headed out
of an ordering system into an accounts system here, with a printout
for audit purposes along the way.

> You need the former if you are doing an image copy. You need the latter
> if you want to tell a webserver at which point it should resume the
> download.

In a pure VMS context, at one time I would have been thinking of RFAs
there...

> Also, IMHO I think the default sequential record type for today's
> world should be stream (and hence the terminator is also included in
> the file data.) I do not think it should be variable length records.

Certainly for general text files which are to be presented via a web
interface, but we're quickly back to legacy debt for applications
dealing with transaction files and the like. Can existing COBOL or
Fortran programs cope with stream format files without modification?

(There's a way to find out, but I'm pressed for time right now)

> The problem is that today's protocols are simply not designed to
> handle the case of metadata buried within the file contents
> themselves; the people who design this stuff have probably never
> even encountered that case.
>

> I do wonder how the IBM mainframe people handle this problem.

Throw up a load of Linux instances? I'm only half joking here; IBM
didn't invest in the ability to do that for fun.

--
There are two hard things in computer science, and they are cache invalidation,
naming, and off-by-one errors.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: 19-Jun-16 3:04 PM
> To: info...@rbnsn.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [Info-vax] RMS record metadata, was: Re: Re; Spiralog, RMS
> Journaling (was Re: FREESPADRIFT)
>

> On 2016-06-19 18:18:15 +0000, Kerry Main said:
>
> >> Hoff: Isn't this the flip-side of that "blue ocean" stuff you're fond
> >> of pointing at?
> >
> > Nope - the blue oceans is for new service offerings.
>
> And doesn't a "blue ocean" have to put together existing and new tools
> in a way that allows those services to be delivered?
>

[snip..]

>
> But enough of the who-moved-my-cheese.
>
> Please: go learn about server-level replication, about DVCS tools, and
> about newer approaches toward deployments and provisioning. The IT
> business moves forward quickly, after all.
>

Let's agree to disagree. Each group / company has their own views.

While I never say never, what I am not prepared to do is adopt current
practices which are counter to new requirements which, in general, are
much more centralized in line with where future technologies are now
heading.

[David Froble]

Kerry Main wrote:

> The grass on the other side is certainly not as green as you like to
> make it out to be.

Or, Steve is well aware of such, but feels that some pressure needs to be
maintained in order to see improvements on VMS.

I know nothing, but, that's how I take lots of his posts.

Personally, I don't see green grass on the "other side", more like brown, or
perhaps black, as in scorched ....

But then, I'm a VMS bigot, I'm allowed to have such perspectives ....

[lawren...@gmail.com]

On Monday, June 20, 2016 at 2:47:43 AM UTC+12, Stephen Hoffman wrote:

> rsync is somewhat analogous to Time Machine in that both copy files
> around. But the two differ. rsync is a remote synchronization tool.
> Time Machine is a backup tool, and maintains months or years of
> previous versions of files over one or more local or remote backup
> target volumes, as well as providing the ability to restore and boot
> the recovered volume(s), among other differences from rsync. TM avoids
> storing duplicate copies of a file, though does not provide the file or
> sector-level de-dupling available with some storage systems. Having
> used both, TM and rsync are different targets and different users and
> different implementations.

rsync does all that and more.

[David Froble]

Stephen Hoffman wrote:
> On 2016-06-19 02:35:16 +0000, David Froble said:
>
>> I would suggest "what does it matter?"
>>
>> It's not like some other file is going to use the unused portion of
>> some disk block, or, as suggested, unused part of 4096 bytes ....
>
> If I'm mapping the user data into memory, having its size is useful.
> Otherwise, I slog through RMS records or such.

Agreed. Wasn't thinking about things like that. And I've never done that, so
it's understandable I would not have considered it.

> If I'm messing with disk allocations and more directly managing storage
> — something I'd prefer to avoid, but that can arise on OpenVMS and other
> systems — then the total size of the file including all user data and
> all of its overhead and/or pre-allocation and/or metadata is of interest.

Agreed.

> This is akin to UTF-8, where you might be looking for the numbers of
> glyphs in the string, or for the size of the required memory buffer to
> store the string.

Don't have such experience, but you're correct.

> Many OpenVMS apps simply don't have the concept of mapping the entire
> file into memory, they assume datasets far larger than available memory
> and/or comparatively weak and slow paging I/O support, and for those
> apps the total data size wasn't really a factor. They're written to
> deal with records. (This is also where the overhead of dealing with
> zillions of small hunks via RMS — and not a few and larger hunks — can
> become a run-time performance-limiting factor.) For apps that map whole
> files or that map masses of data into memory, the user dataset size
> (modulo any virtual memory allocation granularity in play) tends to
> matter more, unless those apps are implementing their own paging — as
> various databases do.
>
> Defining data structures and allowing the run-time system to pack and
> unpack records for the app is a concept that largely doesn't exist on
> OpenVMS. You have to roll your own (glue) code to do that.

Yep, did that long ago ....

Still works well, but, I would not recommend it vs today's database products.

You give good examples, I've just never seen any of those examples.

[lawren...@gmail.com]

On Monday, June 20, 2016 at 2:40:36 AM UTC+12, Simon Clubley wrote:
> It's been a very long time since I've written Fortran code, so I don't
> know the answer to this question. How is Fortran carriage control
> implemented on a stream only filesystem such as on Unix or Windows ?

Seems like it no longer exists in current Fortran standards. (See, some legacy technologies know how to adapt to the modern world...)

By the way, VMS had some fun implementing this as well. As I recall, the convention for line termination was supposed to be: line feed before the line, carriage return after. Which meant the screen cursor was left sitting at the start of the line just output.

But to be kind to the user trying to type input, the terminal driver would automatically move down to a new line as soon as the program started a read. Then it would have to remember it had done so, and not move down *another* line on the next output LF...

[David Froble]

Stephen Hoffman wrote:
> On 2016-06-19 02:48:33 +0000, David Froble said:
>
>> lawren...@gmail.com wrote:
>>>
>>> Hands up all those who think RMS should die already...
>>
>> Perhaps you are ready to re-write billions of dollars of existing
>> software?
>>
>> Why cannot the old and new co-exist?
>
> That works for a while, but then you still have to update and test and
> maintain the old stuff, and there's a non-trivial and increasing cost to
> that effort.

I got to jump all over this.

While there are some specifics, such as needing TLS V1.2 because the older stuff
is no longer secure, in general, if software does needed tasks, it will continue
to perform those needed tasks. Why does one need to update, or test? What are
the non-trivial and increasing costs? Labor might cost more, but that's across
the board, not just for old stuff. I just don't agree with the above statement.

> You often end up putting in place workarounds, and which almost always
> hamper new adoptions and new development, too.

Nope. Hasn't happened to me. New development going on all the time.

> If there are sufficiently good reasons and sufficient incentives to
> migrate, then the old code will migrate across.

If there are NOT reasons to migrate, then don't. If it ain't broke, don't fix it.

> If not, then there were problems — whether technical or incentives or
> ease-of-migration or otherwise — with the new code, or your customers
> are underfunded and/or understaffed and/or are focused elsewhere.

Maybe that is the issue. I read about clusters, 50-100 servers, large migration
projects, and such. Perhaps my customers are underfunded?

Back in the day, we didn't have 24 hour operations. We could take the
applications down for backups and such. Today, with the internet, e-commerence,
and such, the is the desire / demand for 24 hour operations. So, customer wants
less down time. Can such be done? Sure, it happens all the time. But the
customer's environment doesn't support such. So, you look at options, and
present customer with options and cost. Quite often the customer's response is
something like, "ya know, the current stuff is doing pretty well, maybe we don't
need all the bells and whistles".

> Assuming that you didn't screw up the new code bits somehow and did
> deliver substantial improvements, then those folks that aren't moving
> forward and aren't updating their tools are a completely different
> business and different business model, with different expectations.
> This is where (and why) long-term-support products are offered. by some
> vendors Though those always age out, too.

Not yet ....

> But in this case, RMS isn't ever going away on OpenVMS, if for no reason
> other than that it's far too entrenched in the operating system and
> tools. It'd be nice to see some file system defaults change (e.g. not
> defaulting to creating VFC), and better and simpler file system
> alternatives becoming available. But RMS works. (If VSI did ever
> replace RMS, ODS-2 and ODS-5 and related bits with truly modern
> alternatives and with more modern programming interfaces, it'd approach
> — or would effectively be — a completely different operating system.)

Selling RDB was a HUGE mistake. If not sold, perhaps it would have found it's
way into more OS things. VMS problems are mistakes and neglect.

But yeah, VMS was written in the 1970s, based upon RSX to some degree which was
even older. There are always going to be concepts that are outdated. There
have been improvements through the years. Mostly good things. But still the
old concepts remain, and while some maybe timeless, some aren't.

A while back we discussed doing I/O directly into and out of memory devices,
such as SSD thingies. Not sure how well such would fit into VMS, but I can sure
see some benefits. And some problems.

Today brings new requirements, but it also has many of the old requirements.
Supporting the old requirements is probably more important than the new
requirements, mainly because the ratio of old to new favors the existing tasks
greatly.

[David Froble]

Dave whines "if you're dealing with data, why would you care about the metadata?"

In general, you don't care. I have to admit that there have been times when
I've played with the descriptors / metadata, which means maybe I'm dealing off
the bottom of the deck.

:-)

Yeah, doing so in Macro32 is easier than in Basic.

[Paul Sture]

Time Machine uses a File System Event store to track changed directories.

See the details here:

http://pondini.org/OSX/Home.html>

and

<http://pondini.org/TM/Works.html>

[lawren...@gmail.com]

On Monday, June 20, 2016 at 5:39:08 PM UTC+12, Paul Sture wrote:

> On 2016-06-20, Lawrence D’Oliveiro wrote:
>> On Monday, June 20, 2016 at 2:47:43 AM UTC+12, Stephen Hoffman wrote:
>>
>>> rsync is somewhat analogous to Time Machine in that both copy files
>>> around. But the two differ. rsync is a remote synchronization tool.
>>> Time Machine is a backup tool, and maintains months or years of
>>> previous versions of files over one or more local or remote backup
>>> target volumes, as well as providing the ability to restore and boot
>>> the recovered volume(s), among other differences from rsync. TM avoids
>>> storing duplicate copies of a file, though does not provide the file or
>>> sector-level de-dupling available with some storage systems. Having
>>> used both, TM and rsync are different targets and different users and
>>> different implementations.
>>
>> rsync does all that and more.
>
> Time Machine uses a File System Event store to track changed directories.

rsync doesn’t need one.

[Paul Sture]

Please explain.

[lawren...@gmail.com]

On Monday, June 20, 2016 at 9:41:36 PM UTC+12, Paul Sture wrote:

> Please explain.

rsync basically answers the question “given a source directory «src» and a destination directory «dst», what is the minimum that needs to be done to the contents of «dst» to turn it into an exact copy of «src»?”

Adding/removing files and subdirectories would seem to be fairly straightforward. The clever part of rsync is that it can also compare two versions of a file residing on different nodes on the network, *without having to copy the entire file across*, to figure out which parts have changed and which parts haven’t. That was the key part of Andrew Tridgell’s PhD thesis on rsync. It was an algorithm he could have patented, but he chose not to.

rsync is a wonderful general file-copying tool. You can use it to do huge copies, that might take hours or days. If a link goes down and the operation aborts, you can simply re-execute the same rsync command after things come back up, and it will resume from where it left off.

It can also create incremental backups that can be browsed and retrieved from as though they were full backups. The procedure goes something like this:

Initial backup:

rsync --archive «src» «backup1»

Next backup:

rsync --archive --link-dest=«backup1» «src» «backup2»

Basically, any file that hasn’t changed in «src» since the creation of «backup1» is hard-linked from that previous backup into its place in «backup2», instead of being copied again. The net result is that «backup2» behaves like a full backup, without the extra storage space (or network traffic) consumption.

And the nice thing is, there is nothing special about the backup format: it is just a regular filesystem volume obeying standard POSIX semantics--no funny tricks like Apple pulls with Time Machine. No special metadata required, no special file-retrieval software required--just the same standard file-manipulation commands you already use every day.

Because restoring from a backup will likely happen in a high-stress situation: the user or the company has lost some important files, and you have to get them back NOW. Screw up, and say goodbye to your customer, or your job, maybe even face legal consequences. The fewer extra mechanisms that are required to access the backups, the less chance there is for something to go wrong.

[Stephen Hoffman]

On 2016-06-20 02:47:55 +0000, lawren...@gmail.com said:

> rsync does all that and more.

No version of rsync I've met or read about maintains a history of and a
delta of the file changes over time within the backup target, nor does
any version allow file or directory or disk restorations based on the
original backup time. rsync is a nice tool for folks using the command
line and that want to synchronize directories. But it's not directly
analogous to Time Machine.

[John E. Malmberg]

I have seen rsync cause cause more WAN overhead because of lack of
understanding what it does.

If you use rsync like a copy command (no remote daemon) over a network
shared drive to do an update, it can actually read every byte of the
remote files over the network, even if all files are up to date.

The problem gets worse if one side is Windows or OpenVMS and the other
Linux, because there seems to be some issues with converting a those 64
bit timestamps to the 32 bit Linux systems. I have not had the time to
chase that down.

Regards,
-John
wb8...@qsl.net_work

[Stephen Hoffman]

On 2016-06-20 03:26:09 +0000, David Froble said:

> Stephen Hoffman wrote:
>> On 2016-06-19 02:48:33 +0000, David Froble said:
>>
>>> Perhaps you are ready to re-write billions of dollars of existing software?
>>>
>>> Why cannot the old and new co-exist?
>>
>> That works for a while, but then you still have to update and test and
>> maintain the old stuff, and there's a non-trivial and increasing cost
>> to that effort.
>
> I got to jump all over this.
>
> While there are some specifics, such as needing TLS V1.2 because the
> older stuff is no longer secure, in general, if software does needed
> tasks, it will continue to perform those needed tasks. Why does one
> need to update, or test? What are the non-trivial and increasing
> costs? Labor might cost more, but that's across the board, not just
> for old stuff. I just don't agree with the above statement.

Look at OpenVMS, David. Really, really look at it. Look at the
password hash. Look at the design of SYSUAF and of clustering. Look
at how PKE and TLS works, and how it is (not) integrated. Look at what
has happened with 64-bit addressing, and the effort involved in using
that. Fixing some of these cases compatibly is certainly possible,
but that will mean leaving the old code around and — for cases such as
TLS or weak hashes or such— and having old security code around is very
far from desirable. Compatibility means accretions of deprecated and
variously under- or undocumented interfaces and tools which then either
have to be tested and maintained and considered whenever working on
related code or risk breaking existing applications. Code accretions
mean the time and effort involved in making future changes increases,
too. Sometimes dramatically. Eventually asymptotically.

The vast majority of OpenVMS works well. But there are areas that
need to be rethought and refactored. Some changes — like going to a
flat address space and away from the current segmented model, or
yanking out RMS and its rather baroque record access processing – will
never happen. Some — such as seeing executable code in 64-bit space –
seem unlikely, but might be possible. But in specific places,
OpenVMS and some of the dependent applications really need to be broken
apart, re-designed, rebuilt, and the old code deprecated. Not
everything. Certainly not on a whim. Just specific spots. For good
and valid and customer-benefiting reasons.

>> You often end up putting in place workarounds, and which almost always
>> hamper new adoptions and new development, too.
>
> Nope. Hasn't happened to me. New development going on all the time.

There are things that fundamentally cannot change in OpenVMS without
breaking applications. Compatibility means never changing those
things. Compatibility means that what would some simple designs —
such as a flat 64-bit address space — are unachievable. Which leaves
us with 32-bit and 64-bit descriptors, $mumble and $mumble64() calls,
and a host of other arcana.

Compatibility is a most wonderful thing, right up until it becomes an
impediment and a millstone.

[Stephen Hoffman]

On 2016-06-20 10:13:41 +0000, lawren...@gmail.com said:

> rsync basically answers the question “given a source directory «src»
> and a destination directory «dst», what is the minimum that needs to be
> done to the contents of «dst» to turn it into an exact copy of «src»?”

Which is only part of what Time Machine does. You're obligated to
maintain multiple rsync targets, if you want any depth to your backups.
Time Machine functions more like a VCS here, in that a single target
contains anything from a single version of the source to months or
years of source file changes, with file, directory or everything
restorable by time. Time Machine also avoids storing multiple copies
of unmodified files, where using multiple rsync targets would require
the storage system to do that de-dup.

Take rsync and combine it with a VCS — and with a much simpler and
backup-targeted user interface — and you'll get closer to Time Machine.

[John Reagan]

On Monday, June 20, 2016 at 9:07:59 AM UTC-4, Stephen Hoffman wrote:

> The vast majority of OpenVMS works well. But there are areas that
> need to be rethought and refactored. Some changes — like going to a
> flat address space and away from the current segmented model, or
> yanking out RMS and its rather baroque record access processing – will
> never happen. Some — such as seeing executable code in 64-bit space –
> seem unlikely, but might be possible.

You can put code into P2 space today on Itanium. Fully supported. Just link with /SEGMENT=CODE=P2. The compilers don't care. The generated code is the same.

Personally, I'd like to see that be the default in the future. No sense cluttering up your P0 space with code especially if you have data structures that are still 32-bit address dependent.

[Stephen Hoffman]

On 2016-06-20 13:47:48 +0000, John Reagan said:

> On Monday, June 20, 2016 at 9:07:59 AM UTC-4, Stephen Hoffman wrote:
>
>> The vast majority of OpenVMS works well. But there are areas that
>> need to be rethought and refactored. Some changes — like going to a
>> flat address space and away from the current segmented model, or
>> yanking out RMS and its rather baroque record access processing – will
>> never happen. Some — such as seeing executable code in 64-bit space –
>> seem unlikely, but might be possible.
> You can put code into P2 space today on Itanium. Fully supported.
> Just link with /SEGMENT=CODE=P2. The compilers don't care. The
> generated code is the same.

Interesting. (Wanders off to see what C and BASIC do with that...)

> Personally, I'd like to see that be the default in the future. No
> sense cluttering up your P0 space with code especially if you have data
> structures that are still 32-bit address dependent.

Makes sense. P0, P1 and P2 aren't anywhere near as bad as TKB. But
it's still the same segmented address space mess.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: 20-Jun-16 9:08 AM
> To: info...@rbnsn.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [Info-vax] RMS record metadata, was: Re: Re; Spiralog, RMS
> Journaling (was Re: FREESPADRIFT)
>

While no one here would state OpenVMS does not have things it
needs to address, you make it sound like all the other platforms
do not have major challenges as well.

We both know these other platforms have their share of significant
challenges as well - and these challenges include how to maintain
compatibility while still moving forward with new technology that could
enhance the underlying platform.

Remember - it's not the technology that Business types like, it’s the
ability of that technology (old OR new) to support their business and
their customers that is important to them.

And it’s the business that funds the IT groups.

[VAXman-]

In article <mailman.11.1466432458.1...@rbnsn.com>, "Kerry Main" <kemain...@gmail.com> writes:
>> -----Original Message-----
>> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
>> Stephen Hoffman via Info-vax
>> Sent: 20-Jun-16 9:08 AM
>> To: info...@rbnsn.com
>> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
>> Subject: Re: [Info-vax] RMS record metadata, was: Re: Re; Spiralog, =
>RMS
>> Journaling (was Re: FREESPADRIFT)
>>=20
>> On 2016-06-20 03:26:09 +0000, David Froble said:

>>=20

>> > Stephen Hoffman wrote:
>> >> On 2016-06-19 02:48:33 +0000, David Froble said:
>> >>
>> >>> Perhaps you are ready to re-write billions of dollars of existing
>> software?
>> >>>
>> >>> Why cannot the old and new co-exist?
>> >>

>> >> That works for a while, but then you still have to update and test =
>and
>> >> maintain the old stuff, and there's a non-trivial and increasing =

>cost
>> >> to that effort.
>> >
>> > I got to jump all over this.
>> >
>> > While there are some specifics, such as needing TLS V1.2 because the
>> > older stuff is no longer secure, in general, if software does needed
>> > tasks, it will continue to perform those needed tasks. Why does one
>> > need to update, or test? What are the non-trivial and increasing
>> > costs? Labor might cost more, but that's across the board, not just
>> > for old stuff. I just don't agree with the above statement.

>>=20

>> Look at OpenVMS, David. Really, really look at it. Look at the
>> password hash. Look at the design of SYSUAF and of clustering. Look

>> at how PKE and TLS works, and how it is (not) integrated. Look at =

>what
>> has happened with 64-bit addressing, and the effort involved in using
>> that. Fixing some of these cases compatibly is certainly possible,

>> but that will mean leaving the old code around and =E2=80=94 for cases =
>such as
>> TLS or weak hashes or such=E2=80=94 and having old security code =

>around is very
>> far from desirable. Compatibility means accretions of deprecated and
>> variously under- or undocumented interfaces and tools which then
>> either
>> have to be tested and maintained and considered whenever working on
>> related code or risk breaking existing applications. Code accretions
>> mean the time and effort involved in making future changes increases,
>> too. Sometimes dramatically. Eventually asymptotically.

>>=20

>> The vast majority of OpenVMS works well. But there are areas that

>> need to be rethought and refactored. Some changes =E2=80=94 like =

>going to a
>> flat address space and away from the current segmented model, or

>> yanking out RMS and its rather baroque record access processing =
>=E2=80=93 will
>> never happen. Some =E2=80=94 such as seeing executable code in =
>64-bit space
>> =E2=80=93

>> seem unlikely, but might be possible. But in specific places,
>> OpenVMS and some of the dependent applications really need to be
>> broken
>> apart, re-designed, rebuilt, and the old code deprecated. Not
>> everything. Certainly not on a whim. Just specific spots. For good
>> and valid and customer-benefiting reasons.

>>=20

>> >> You often end up putting in place workarounds, and which almost
>> always
>> >> hamper new adoptions and new development, too.
>> >
>> > Nope. Hasn't happened to me. New development going on all the
>> time.

>>=20

>> There are things that fundamentally cannot change in OpenVMS without
>> breaking applications. Compatibility means never changing those

>> things. Compatibility means that what would some simple designs =
>=E2=80=94
>> such as a flat 64-bit address space =E2=80=94 are unachievable. =

>Which leaves
>> us with 32-bit and 64-bit descriptors, $mumble and $mumble64() calls,
>> and a host of other arcana.

>>=20

>> Compatibility is a most wonderful thing, right up until it becomes an
>> impediment and a millstone.

>>=20

>
>While no one here would state OpenVMS does not have things it
>needs to address, you make it sound like all the other platforms

>do not have major challenges as well.=20
>
>We both know these other platforms have their share of significant=20
>challenges as well - and these challenges include how to maintain=20
>compatibility while still moving forward with new technology that could=20
>enhance the underlying platform.

Look at what moving forward has done with/for 'El CRAPitan'!

--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.

[Stephen Hoffman]

On 2016-06-20 14:19:56 +0000, Kerry Main said:

> While no one here would state OpenVMS does not have things it needs to
> address, you make it sound like all the other platforms do not have
> major challenges as well.

For not the first time I've had to write this, this is not about right
now. This is about three or five years out. About getting OpenVMS to
be better competitive with what will be. This is a never-ending
process, and it's a process that's only accelerating. VSI doesn't
have the ability to push out updates and patches anywhere near as fast
as other platforms either, because the VSI folks don't yet have the
staff and the infrastructure in place to do that (yet). But I digress.

> We both know these other platforms have their share of

> significantchallenges as well - and these challenges include how to
> maintaincompatibility while still moving forward with new technology

> that couldenhance the underlying platform.

We both know "these other platforms" are very successful. More
successful than OpenVMS, by more than a few measures; on the job
postings and questions and the sorts of discussions around here, and
particularly on the numbers of servers installed. Ponder why that
might be, too. Some of that difference is certainly familiarity and
availability of staff for "these other platforms", some is marketing,
and more than a little of it is tools, designs and software that meet
the needs of the folks using them. That usage means the folks don't
have to go build and maintain their own tool chains for common tasks.

Costs are always a factor for "Business types", too. OpenVMS has no
competitively-priced low-end or entry level hardware and software
configurations, and that's not a new gap in the product offerings.
The x86-64 port will help reduce the hardware costs and likely also
allow folks to standardize on some of the related software tools.

Again: this is where OpenVMS is competing now, and where it will be
competing in five or ten years.

The installed base is a massive advantage for VSI and one that many new
startups and new businesses don't have, but — if the OpenVMS business
doesn't stabilize and start to grow at least enough to offset ports off
of OpenVMS and OpenVMS products being deprecated — those too young to
retire will be porting code elsewhere sooner or later.

Make no mistake, vendors are still deprecating OpenVMS products, too.
Here's one from this month:
https://techsupport.osisoft.com/Troubleshooting/Alerts/AL00304
https://techsupport.osisoft.com/Troubleshooting/KB/KB01114

> Remember - it's not the technology that Business types like, it’s the

> ability of that technology (old OR new) to support their business

> andtheir customers that is important to them.
> And it’s the business that funds the IT groups.

Which right now is Linux and Windows Server, for a great many
businesses. Yes, there are problems with those. There are problems
with all systems. Which is also why I keep suggesting folks here try
new tools and new platforms, too. Think of this as the hobbyist
program for yourself, but in reverse. Learning other stuff, to make
OpenVMS and your own applications better. Which is why moving
OpenVMS substantially forward in features, ease of use and related is
vital. That — combined with astute* marketing — is will make "Business
types" more interested in OpenVMS than they would otherwise be.
Because right now, most "Business types" are clearly not interested in
OpenVMS. In years past, some of the senior IT "Business types" in a
number of organizations would have used OpenVMS in their careers, too.
Now? That's very rare. That lack of interest and the lack of
familiarity is due to much more than just the lack of marketing, too.

TL;DR: I'm going to keep pointing at features and tools and good ideas
from other platforms, and pointing out problems and limits in OpenVMS.

*That's time and thought and consistency of messages and timing and
preparation; far more than running some adverts (somewhere), starting
up the hobbyist and educational programs, and related. Good marketing
is not easy while inept marketing is hugely expensive, as those
familiar will tell you. But I digress.

[hb]

On 06/20/2016 03:47 PM, John Reagan wrote:

> You can put code into P2 space today on Itanium. Fully supported.
> Just link with /SEGMENT=CODE=P2. The compilers don't care. The
> generated code is the same.

As far as I remember, C++ was different: you had (and still may have) to
compile with POINTER_SIZE=64 to run the code from P2. Otherwise you may
experience an unexpected ACCVIO. No, the linker doesn't check this: you
say move code to P2 and the linker moves it. And yes, linker generated
code is the same, as well.

[Paul Sture]

On 2016-06-20, lawren...@gmail.com <lawren...@gmail.com> wrote:

> On Monday, June 20, 2016 at 9:41:36 PM UTC+12, Paul Sture wrote:
>
>> Please explain.
>
> rsync basically answers the question “given a source directory «src»
> and a destination directory «dst», what is the minimum that needs to
> be done to the contents of «dst» to turn it into an exact copy of
> «src»?”

Let's back up to the bit you snipped:

Paul:

>> Time Machine uses a File System Event store to track changed directories.

Lawrence:

> rsync doesn’t need one.

You underestimate the size of the problem. When not much has changed, an hourly
Time Machine run will complete in a minute or two.

rsync cannot possibly scan all the directories on a system for changes in
such a short time, at least not on the spinning rust I have here.

Want timings?

find /Applications | wc -l # get total number of files in /Applications

995716 files, elapsed time 33 seconds

for the directories in there (I'm assuming 'find' is reasonably efficient
here):

find /Applications -type d | wc -l # get total number of directories
# in /Applications

145437 directories, elapsed time 5 minutes 37 seconds

That's just the OS X GUI applications. I've got many more files
spanning several disks. Think half a million directories in total.
not including backups. All to be scanned to detect changes, unless
you have a faster method of targeting modified directories.

The hint provided by the File System Event store cuts that down to a
manageable level. If you read the description at pondini.org, you'll
discover the precautionary measures taken in the event of a system
crash, when the Event store might be incomplete.

> Adding/removing files and subdirectories would seem to be fairly
> straightforward. The clever part of rsync is that it can also compare
> two versions of a file residing on different nodes on the network,
> *without having to copy the entire file across*, to figure out which
> parts have changed and which parts haven’t. That was the key part of
> Andrew Tridgell’s PhD thesis on rsync. It was an algorithm he could
> have patented, but he chose not to.

Ah yes. Do you realise you are sounding like a PR type acting on
Tridgell's behalf?

> rsync is a wonderful general file-copying tool. You can use it to do
> huge copies, that might take hours or days. If a link goes down and
> the operation aborts, you can simply re-execute the same rsync command
> after things come back up, and it will resume from where it left off.

Yes, I'm aware of it's good points thank you, but it is not a direct
replacement for Time Machine.

<snip>

> Because restoring from a backup will likely happen in a high-stress
> situation: the user or the company has lost some important files, and
> you have to get them back NOW. Screw up, and say goodbye to your
> customer, or your job, maybe even face legal consequences. The fewer
> extra mechanisms that are required to access the backups, the less
> chance there is for something to go wrong.

Yes, we know all that stuff thanks. :-)

[David Froble]

Stephen Hoffman wrote:
> On 2016-06-20 03:26:09 +0000, David Froble said:
>
>> Stephen Hoffman wrote:
>>> On 2016-06-19 02:48:33 +0000, David Froble said:
>>>
>>>> Perhaps you are ready to re-write billions of dollars of existing
>>>> software?
>>>>
>>>> Why cannot the old and new co-exist?
>>>
>>> That works for a while, but then you still have to update and test
>>> and maintain the old stuff, and there's a non-trivial and increasing
>>> cost to that effort.
>>
>> I got to jump all over this.
>>
>> While there are some specifics, such as needing TLS V1.2 because the
>> older stuff is no longer secure, in general, if software does needed
>> tasks, it will continue to perform those needed tasks. Why does one
>> need to update, or test? What are the non-trivial and increasing
>> costs? Labor might cost more, but that's across the board, not just
>> for old stuff. I just don't agree with the above statement.
>
> Look at OpenVMS, David.

Ok, here is the lack of communication ...

:-)

I was talking about applications, not the OS.

You are correct, VMS has been neglected, it needs attention, and some things
don't really work so well anymore, the password hash, TCP/IP, SSL, and such.

But my applications do not have such problems. Perhaps because they have not
been neglected? Development is ongoing. New requirements are met.

[David Froble]

Kerry Main wrote:

> While no one here would state OpenVMS does not have things it
> needs to address, you make it sound like all the other platforms
> do not have major challenges as well.
>
> We both know these other platforms have their share of significant
> challenges as well - and these challenges include how to maintain
> compatibility while still moving forward with new technology that could
> enhance the underlying platform.
>
> Remember - it's not the technology that Business types like, it’s the
> ability of that technology (old OR new) to support their business and
> their customers that is important to them.
>
> And it’s the business that funds the IT groups.

Perhaps that is the reason there is an Alpha evaluation kit available from VSI?

:-)

[John Reagan]

I don't see anything in the current release notes or notes files but that isn't 100% proof. The compiler should know that the FD is 32-bit addressable and the FD contains a 64-bit pointer to the code regardless of the current pointer size. It feels like a bug to me, it might have accidentally checked the current pointer size for the fetch from the FD? Or maybe something in the STL had a 32-bit fetch? Dunno.