Shared Stream IO (SSIO) beta kit available for OpenVMS

[BillPedersen]

Shared Stream IO (SSIO) beta kit available for OpenVMS - contact OpenSourc...@hp.com or OpenVMS....@hp.com to participate.

SSIO
====
Shared Stream IO feature provides POSIX compliant read/write to byte stream files.
Hence SSIO feature, the data consistency is guaranteed when mutiple processes are performing a Read/Write to non overlapping byte boundaries with the same block boundary.

Problems Addressed
------------------
- Lost Update Problem
When mutiple WRITERS are concurrently accessing non overlapping byte boundaries within the
same block boundary, one of the WRITERS' update is lost.

- Mix Update Problem
When a WRITER and READER are accessing the data concurrently, READER gets some data
which is before the WRITE is completed and some data which is after the WRITE is completed.

V1 Release
----------

* SSIO Feature release
Mutiple processes will be able to Read/Write data from non overlapping byte boundaries with the
same block boundary successfully.
- No Lost Update Problem
- No Mix Update Problem

* Stand Alone implementation
In this release, SSIO feature is supported only on a stand alone system.

* Write though cache
OpenVMS caching is write through cache.
When a applicaiton issues a IO, data is written to cache and then to disk and only then the IO
operation is considered complete.
SSIO write operation will continue to be a write through operation.


V2 Release
----------
* Next release of SSIO.

* Plans to enable Cluster support.

Bill Pedersen
https://sourceforge.net/p/vms-ports/wiki/Home/

[BillPedersen]

SSIO V1 UPDATE AVAILABLE.

I have been informed my HP OpenVMS Engineering that there are update images available for those interested in testing this new feature on OpenVMS. Anyone interested in testing Serial Stream IO should contact OpenVMS....@hp.com and request a kit. SSIO is an important precursor to the proper implementation of a fully compatible fork on OpenVMS for Unix Portability and Open Source Porting.

Information from HP OpenVMS Engineering:

SSIO IMAGE DETAILS –

1) SSIO Update

The latest set of SSIO images provides the following –

A) Fix to various major defects found internally

B) Fix to known problems mentioned in the previous version of SSIO V1 beta kits

C) Provides functionality to block concurrent SSIO access to the same file in cluster environment

Concurrent SSIO access to the file cluster-wide is handled as follows -

All the nodes would be able to open/close a given file but only one of the node would be able to successfully perform SSIO operations on the file. The node that requests the first SSIO operation will be able to successfully perform the SSIO operation. And until this node closes the file, all other nodes attempt to perform SSIO operation on the same file will be failed. i.e. the CRTL APIs supporting the SSIO functionality will fail with return status of -1.

Thanks for your continued interest and support.

Regards,

Bill Pedersen
CCSS - Computer Consulting System Services, LLC

[Neil Rieck]

This is great news. Thanks for working with HP OpenVMS Engineering to have this developed. (speaking of UNIX portability, any idea if you can get them to implement a true fork?)

Neil Rieck
Kitchener / Waterloo / Cambridge,
Ontario, Canada.
http://www3.sympatico.ca/n.rieck/

[Stephen Hoffman]

On 2013-01-28 12:54:57 +0000, Neil Rieck said:

> This is great news. Thanks for working with HP OpenVMS Engineering to
> have this developed. (speaking of UNIX portability, any idea if you can
> get them to implement a true fork?)

I can save Bill the trouble, here.

Fork is not going to happen on VMS.

Ever.

There are existing DEC/Compaq/HP C fork-related routines available
which make porting ~95%[1] of the C code using fork() fairly easy, but
that last ~5%[1] of the C code around is really using the true, full
capabilities of fork(), and analogs of those capabilities are not
available on VMS due to the low-level architectural differences between
VMS and Unix. The copy-on-write memory management and the differences
in I/O channel contexts being two examples of this.

Could fork() be implemented? Sure. But that means either massive
complexity and untenable overhead, or ripping up the kernel. Or both.

————
[1]I'm quoting those percentages from memory, but those numbers should
get you in the range.


--
Pure Personal Opinion | HoffmanLabs LLC

[BillPedersen]

On Monday, January 28, 2013 7:54:57 AM UTC-5, Neil Rieck wrote:
> On Sunday, January 27, 2013 8:48:16 AM UTC-5, BillPedersen wrote:
>
> > On Monday, October 15, 2012 11:02:18 AM UTC-4, BillPedersen wrote:
>
> >
>
> > > Shared Stream IO (SSIO) beta kit available for OpenVMS - contact OpenSourc...@hp.com or OpenVMS....@hp.com to participate.
>
>

> This is great news. Thanks for working with HP OpenVMS Engineering to have this developed. (speaking of UNIX portability, any idea if you can get them to implement a true fork?)
>
>

The issue of performance will always be there with VMS. As Hoff has pointed out a "true" fork may never really exist. But the work being done with SSIO is important in that it is a pre-cursor to a fork implementation that potentially can then be of the form where a child inherits the parent's environment, something much closer to the Unix/Linux fork. I do not have the details of the implementation and am not even certain if they exist other than in concept - I just try to encourage both HP OpenVMS Engineering as well as the VMS Community.

The timing is probably not something I would even try to hang my hat or name on at this time. I would of course, like many of us, would have liked to have seen it 10 years ago. It is dependent on SSIO. So that hurdle has to be crossed. And at least that effort is now moving forward.

If people want to keep up with discussions and even provide comment and feedback please visit http://sourceforge.net/projects/vms-ports/. There is a wiki environment where we are working to lay out, in one place, helpful hints, tools, techniques as well as a directory of packages and their status. There is also a issue tickets mechanism where we are tracking problems which have been found that directly affect open source porting to VMS.

And there are discussion forums. Here, for instance, we post the fortnightly HP OpenVMS Engineering conference call agendas and notes from the calls. There are in the notes discussions around the various aspects of SSIO, PIPE and FORK including status updates and some discussions on implementation.

If you are interested in attending the calls in future please just drop me a note. Or if you have suggestions or comments we look forward to them as well.

We also welcome participation in the effort. There is a lot to do!

Bill.

[Johnny Billquist]

On 2013-01-28 14:31, Stephen Hoffman wrote:
> On 2013-01-28 12:54:57 +0000, Neil Rieck said:
>
>> This is great news. Thanks for working with HP OpenVMS Engineering to
>> have this developed. (speaking of UNIX portability, any idea if you
>> can get them to implement a true fork?)
>
> I can save Bill the trouble, here.
>
> Fork is not going to happen on VMS.
>
> Ever.
>
> There are existing DEC/Compaq/HP C fork-related routines available which
> make porting ~95%[1] of the C code using fork() fairly easy, but that
> last ~5%[1] of the C code around is really using the true, full
> capabilities of fork(), and analogs of those capabilities are not
> available on VMS due to the low-level architectural differences between
> VMS and Unix. The copy-on-write memory management and the differences
> in I/O channel contexts being two examples of this.
>
> Could fork() be implemented? Sure. But that means either massive
> complexity and untenable overhead, or ripping up the kernel. Or both.

copy on write is not a semantic of fork. It's just a very nice optimization.
However, the file handle duplication of fork would be more or less
impossible to achieve in VMS. (Yes, I suppose everything is possible
with enough work, but this one would probably be enormous.)

Johnny

--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol

[Stephen Hoffman]

On 2013-01-28 21:43:22 +0000, Johnny Billquist said:

> On 2013-01-28 14:31, Stephen Hoffman wrote:
>> Could fork() be implemented? Sure. But that means either massive
>> complexity and untenable overhead, or ripping up the kernel. Or both.
>
> copy on write is not a semantic of fork. It's just a very nice optimization.

With the inverse of "a very nice optimization" being "untenable
overhead". Lacking copy on write means spending a whole lot of time
and processor effort replicating the contents of process address space
around. Sure. Watch your cores all spin up with buffer copies,
transferring the contents of process memory through a window and into
the new process. An effort which would often be followed immediately
by tearing everything down, for the percentage of fork() calls that
don't use that feature, too.

[Johnny Billquist]

Indeed. A lot of copying will happen if when you don't have copy on
write. But that would not by any means mean that a fork wouldn't work.
Trying to behave like unix is already something fraught with inefficiencies.
Unix itself survived without copy on write for a long time, since that
wasn't available in there as well. That's why you had inventions like
vfork...

I'm not saying that it would be very nice or efficient, just that
lacking copy on write is not something that in any way would prevent VMS
from having a fork. File handling on the other hand... The whole concept
of how that is handled in Unix is totally foreign to VMS.