Internals and Data Structures - DLM information

[David Froble]

I'm looking at a half dozen books on Amazon, all VMS internals and such
publications. What I'm looking for is information on the DLM, data structures
and design.

There is a V5.2 from 1987. Not sure if this is what I'm looking for. Can
anyone with the books check in theirs and tell me which has DLM information.

Dave

[VAXman-]

VAX/VMS Internals and Data Structures -- Version 5.2
Ruth Goldenberg/Order Number EY-C171E-DP/ISBN 1-55558-059-9

Part III/Synchronization
Chapter 10 --- Lock Management
10.1 - Lock Management Data Structures
10.2 - Lock Management System Services
10.3 - Handling Deadlocks

--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.

[Stephen Hoffman]

Not exactly sure what sort of information you're looking for, here.
Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
a DLM? Alternatives to the DLM? Etc...

Beyond the two manuals in the doc set and articles in the DEC and VMS
Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
ISBN was 55558-112-9

Less so the IDSM. The DLM gets a fairly general discussion in the
IDSM, but not much in the way of details.

Or the OpenVMS source listings, for the full details.

Or for more general interest, skim through the OpenSSL open-source
variant of clustering software that Compaq released; that was a Unix
DLM similar to the OpenVMS DLM. Or skim the books and resources about
one of the more recent variants; see
https://en.wikipedia.org/wiki/Distributed_lock_manager for some
pointers to those.

If you do want to go for an IDSM for this or other reasons, you'll be
looking at a mix of V6 and the V7 updates to get all of the information
with the "latest" available information. The V6 stuff was the last
full copy of the IDSM, the V7 subset documents added chapters with some
details on specific and then-newer areas.

Closely related to the DLM is the distributed transaction manager;
DECdtm, and also the DEC RTR and the more recent open-source
0MQ/ActiveMQ/RabbitMQ messaging packages.


--
Pure Personal Opinion | HoffmanLabs LLC

[Stephen Hoffman]

On 2017-01-05 14:50:31 +0000, Stephen Hoffman said:

> ...The DLM gets a fairly general {design and usage} discussion in the
> IDSM, but not much in the way of {design and usage} details.

It's... Internals and Data Structures. But if that's what you're
looking for here...

[David Froble]

Stephen Hoffman wrote:
> On 2017-01-05 06:29:20 +0000, David Froble said:
>
>> I'm looking at a half dozen books on Amazon, all VMS internals and
>> such publications. What I'm looking for is information on the DLM,
>> data structures and design.
>>
>> There is a V5.2 from 1987. Not sure if this is what I'm looking for.
>> Can anyone with the books check in theirs and tell me which has DLM
>> information.
>
> Not exactly sure what sort of information you're looking for, here.
> Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
> a DLM? Alternatives to the DLM? Etc...

Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
able to do so, are to come up with a preliminary design for additions to the VMS
DLM to implement byte range locking, then distribute it to those interested to
comment upon, add comments and suggestions, and present the result to VSI as a
potential modification. It would be up to VSI as to whether to include it in VMS.

I just don't have enough information now, and, will need some help, as right
now, I'd come up with a solution looking for a problem. For instance, Postgre
database, how large a range is reasonable, and if talking bytes, how many should
be supported, which would determine the size of integer required. I figure a
database could be rather large.

Don't know that I need exact information on the data structures. Whoever
implements such a modification would have to work from what's current in VMS.
But I need enough information to get close to what might be required.

One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
have to also be implemented to use the new feature. There it gets interesting.
If using I/O buffers, how to determine the size for one thing. Or perhaps
direct I/O into the "on disk" file?

[Stephen Hoffman]

On 2017-01-05 15:23:00 +0000, David Froble said:

> Briefly, I'm looking at the issue of byte range locking. My
> intentions, if I'm able to do so, are to come up with a preliminary
> design for additions to the VMS DLM to implement byte range locking,
> then distribute it to those interested to comment upon, add comments
> and suggestions, and present the result to VSI as a potential
> modification. It would be up to VSI as to whether to include it in VMS.

There's nothing on the byte-range locking implementation in the IDSM or
VAXcluster books. That implementation is entirely too new, and — other
than the C API — largely undocumented. IIRC, somebody did some
reverse-engineering and figured out parts, but there's been little
published on the topic from HP/HPE/VSI.

> I just don't have enough information now, and, will need some help, as
> right now, I'd come up with a solution looking for a problem. For
> instance, Postgre database, how large a range is reasonable, and if
> talking bytes, how many should be supported, which would determine the
> size of integer required. I figure a database could be rather large.
>
> Don't know that I need exact information on the data structures.
> Whoever implements such a modification would have to work from what's
> current in VMS. But I need enough information to get close to what
> might be required.
>
> One thing seems to be clear to me, it's not just the DLM, but I/O stuff
> would have to also be implemented to use the new feature. There it
> gets interesting. If using I/O buffers, how to determine the size for
> one thing. Or perhaps direct I/O into the "on disk" file?

Not sure how you're envisioning I/O would be connected to the DLM.
Something else will usually map storage ranges and caches, and that
mapping and caching might or will be coordinated via DLM, but the I/O
itself is likely going to be isolated from the DLM. In OpenVMS, RMS
coordinates the caches in a cluster, if that's where you're headed
here. There's discussion of that sort of coordination in Davis' book.

If you're thinking of porting and using PostgreSQL on OpenVMS, then the
OpenVMS problems with shared stream I/O (SSIO) and a few other details
will arise. You've been involved in that discussion before, though.
For those that haven't been, there's an overview of SSIO (for
PostgreSQL or otherwise) at
http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf and
the discussions of SSIO-related bugs and problems are buried in the
OpenVMS bug lists, and probably a few postings around the 'net.

If you're using the DLM for your own resource coordination outside of C
(and not of something like PostgreSQL and its locking and I/O
activities via the C library), then you get to roll your own byte-range
locking if that's what you need. AFAIK, what OpenVMS C is doing there
for byte-range locking is specific to C applications, too. File and
record locks are general.

As for application design, I've had success with just mapping the
entire file. That's not a typical OpenVMS design, but it does work
nicely, and it makes some tasks simpler. Reliability and crash
recovery from journals and/or from a shadowed server. Or using code to
unmarshall the data into memory, and then operating there, and
marshalling the data back out for a clean shutdown or such. Again,
with a journal written to storage or to another host or such, for
purposes of crash recovery.

Some general locking information, and not specific to OpenVMS and the
DLM: http://apenwarr.ca/log/?m=201012#13

[johnwa...@yahoo.co.uk]

There is/was a Digital Technical Journal article on
the original architecture and design of the DLM.
It might be a bit, er, basic for your stated needs,
but you are in a better position than I am to make
that judgement.

It's in the VAXcluster Systems special edition of the
DTJ, from September 1987, pages 29-44.

I found it at
http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
but other sources may also be available e.g. Bitsavers.

Are you aware that PATHWORKS did byte range locking
some twenty-odd years ago? Again, there is a DTJ special
issue with some potentially relevant info: Volume 4
Number 1, from 1992. The details have doubtless changed,
but the basics may or may not have changed.

Here are a few words from the piece on Byte Range Locking:
"Byte Range Locking
The MS-NET architecture allows for concurrent
access to server-based files by multiple clients. PC
applications acquire this functionality through the
MS-DOS byte range locking calls. These calls allow
PC applications to lock and unlock ranges of bytes
in a file and to detect conflicts. Conflicts occur
when part or all of a range specified to be locked
has been locked from a previous call. In contrast,
the approach taken by RMS provides locking on a
record basis. RMS uses the VMS distributed lock
manager to implement this functionality. Unfortu-
nately, the lock manager is not well suited to imple-
menting byte range locks because the byte range is
represented in a form that allows the lock manager
to arbitrate access. Therefore, the file server imple-
ments its own lock database and arbitrates access
to shared files. Internally, the server process main-
tains a list of locks for each file the server has open
and arbitrates access based on these lock struc-
tures. Files opened by the file server cannot be
shared with other VMS processes because the file
server has an exclusive mode lock on each file it has
open through the VMS lock manager. The exclusive
mode lock guarantees protection from other VMS
processes.
[etc]"


The DLM article abstract says:
The VMS distributed lock manager provides the
synchronization mechanism needed to ensure
transparent and reliable data sharing between
nodes in a VAXcluster system. The lock
manager provides services for mutual exclusion
and event notification, and achieves high
performance by minimizing the number of messages
sent between nodes. The lock manager also
handles deadlock situations with a minimum
of messages exchanged. Since processors systems
can join or leave a cluster at any time, a
connection manager was developed to handle
reconfigurations in a dynamic, efficient manner.

[David Froble]

Stephen Hoffman wrote:
> On 2017-01-05 14:50:31 +0000, Stephen Hoffman said:
>
>> ...The DLM gets a fairly general {design and usage} discussion in the
>> IDSM, but not much in the way of {design and usage} details.
>
> It's... Internals and Data Structures. But if that's what you're
> looking for here...
>
>

I've used the DLM quite a bit. Usage isn't a problem. But now I'm looking at
internal design, and I need some documentation.

[David Froble]

I'm not. The DLM should be sort of an object that does a job. However, there
is nothing that I'm aware of that does byte range locking, or has that need,
that's currently implemented. Perhaps I should ignore the I/O part of it and
just concentrate on doing locks on byte ranges.

> Something else will usually map storage ranges and caches, and that
> mapping and caching might or will be coordinated via DLM, but the I/O
> itself is likely going to be isolated from the DLM. In OpenVMS, RMS
> coordinates the caches in a cluster, if that's where you're headed
> here. There's discussion of that sort of coordination in Davis' book.
>
> If you're thinking of porting and using PostgreSQL on OpenVMS,

Slow down Steve, that's beyond me. What I'm looking at is a capability that
those wishing to do such a port say they need.

> then the
> OpenVMS problems with shared stream I/O (SSIO) and a few other details
> will arise. You've been involved in that discussion before, though.
> For those that haven't been, there's an overview of SSIO (for PostgreSQL
> or otherwise) at
> http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf and
> the discussions of SSIO-related bugs and problems are buried in the
> OpenVMS bug lists, and probably a few postings around the 'net.
>
> If you're using the DLM for your own resource coordination outside of C
> (and not of something like PostgreSQL and its locking and I/O activities
> via the C library), then you get to roll your own byte-range locking if
> that's what you need. AFAIK, what OpenVMS C is doing there for
> byte-range locking is specific to C applications, too. File and record
> locks are general.

No, nothing for me. Just looking to perhaps help out the people who need such a
capability. One concept I'm solid on is that it should be a generic VMS
solution, not a C solution. But, that's how I am ... :-)

> As for application design, I've had success with just mapping the entire
> file. That's not a typical OpenVMS design, but it does work nicely, and
> it makes some tasks simpler. Reliability and crash recovery from
> journals and/or from a shadowed server. Or using code to unmarshall the
> data into memory, and then operating there, and marshalling the data
> back out for a clean shutdown or such. Again, with a journal written to
> storage or to another host or such, for purposes of crash recovery.

Maybe you'd be interested in contributing to the I/O side of things ???

:-)

[BillPedersen]

It should be noted that the Pathworks implementation works nicely for serving to PCs as it is a single process based implementation.

Samba/CIFS in a multi-process implementation as is PostgreSQL and so there is then the need to coordinate between processes as well as between cluster members. It would also be nice to be able to access the files from the VMS host...

It should also be noted that even Samba, and I expect PostgreSQL as well, implements its own byte range locking due to limitations in the fcntl() function call to allow access to LARGE files. So even there we have to consider some interesting issues. Samba implements an opportunistic locking mechanism which allows a client to have a cached, local copy of the file until such time as a second client comes along... This is not supported by VMS obviously. And is only supported by a few hosts as of now, from what I can tell. Samba also supports the deny-mode locks of Windows.

So, no matter what we do we will need to see how we can accommodate the locking in a way that eventually we can access from VMS as well as the clients.

Bill.

[David Froble]

Thanks John. Read that when it came out, and again recently. Nothing real
helpful there for what I'm looking at.

> I found it at
> http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
> but other sources may also be available e.g. Bitsavers.
>
> Are you aware that PATHWORKS did byte range locking
> some twenty-odd years ago? Again, there is a DTJ special
> issue with some potentially relevant info: Volume 4
> Number 1, from 1992. The details have doubtless changed,
> but the basics may or may not have changed.

No, I wasn't aware of that. However, I'm thinking that it didn'tuse the DLM,
and I'm looking at an enhance to the DLM to provide a generic solution for such
needs.

Which works very well, including in a cluster. I'm just looking at the
possibility of extending the capabilities.

[Chris Scheers]

David Froble wrote:
> Stephen Hoffman wrote:
>> On 2017-01-05 06:29:20 +0000, David Froble said:
>>
>>> I'm looking at a half dozen books on Amazon, all VMS internals and
>>> such publications. What I'm looking for is information on the DLM,
>>> data structures and design.
>>>
>>> There is a V5.2 from 1987. Not sure if this is what I'm looking
>>> for. Can anyone with the books check in theirs and tell me which has
>>> DLM information.
>>
>> Not exactly sure what sort of information you're looking for, here.
>> Use of the DLM on OpenVMS? Interested in the DLM design?
>> Implementing a DLM? Alternatives to the DLM? Etc...
>
> Briefly, I'm looking at the issue of byte range locking. My intentions,
> if I'm able to do so, are to come up with a preliminary design for
> additions to the VMS DLM to implement byte range locking, then
> distribute it to those interested to comment upon, add comments and
> suggestions, and present the result to VSI as a potential modification.
> It would be up to VSI as to whether to include it in VMS.

The DLM does support byte range locking (or at least it used to). But
it is undocumented.

AFAIK, for more information you need to look at the sources.

--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.

Voice: 817-237-3360 Internet: ch...@applied-synergy.com
Fax: 817-237-3074

[Bob Gezelter]

Chris,

With all due respect, a point of architecture.

The DLM manages locks. Locks have limited attributes and names.

How the locks are named and used by other components (e.g., RMS) is what determines what locks mean (e.g., records, files, byte ranges).

For example, one could create a run-time library that manages golf carts, and use the DLM to coordinate allocations (admittedly silly, one would want to use a database in actuality). The DLM is none the wiser, all it manages are lock names and values. The import of those names is outside the scope of the DLM.

- Bob Gezelter, http://www.rlgsc.com

[David Froble]

Correct. The DLM offers locks on "resource names" for cooperating processes.
It actually does no locking of anything. It just allows a user to request a
lock on a resource name, and if granted, will not issue another lock on that
resource until the first is released. Either the second user waits for the lock
to be granted, or, determines that the resource cannot be accessed. In reality,
if it wished to screw up things, it could go ahead and access the resource
anyway. The key word is "cooperating".

Lots of uses. Cause a batch job to wait. Cause user applications to be
synchronized, and such.

[dgordo...@gmail.com]

On Thursday, January 5, 2017 at 2:26:19 PM UTC-5, Chris Scheers wrote:

> The DLM does support byte range locking (or at least it used to). But
> it is undocumented.
>
> AFAIK, for more information you need to look at the sources.
>

.SBTTL EXE$ENQ - Enqueue system service

;++
; FUNCTIONAL DESCRIPTION:
;
; This routine handles the $ENQ system service.
;
; CALLING SEQUENCE:
;
; CALLS/G EXE$ENQ (actually called through the system service
; dispatcher)
;
; INPUT PARAMETERS:
;
; ENQ_EFN(AP) Event flag number
; ENQ_LKMODE(AP) Lock mode
; ENQ_LKSB(AP) Address of lock status block
; ENQ_FLAGS(AP) Flags
; ENQ_RESNAM(AP) Address of descriptor of resource name
; ENQ_PARID(AP) Parent lock id
; ENQ_ASTADR(AP) Address of completion AST routine
; ENQ_ASTPRM(AP) AST parameter
; ENQ_BLKAST(AP) Address of blocking AST routine
; ENQ_ACMODE(AP) Access mode
; ENQ_RSDM_ID(AP) Resource Domain ID
; ENQ_RNGBLK(AP) Range Block
; ENQ_PRIORITY(AP) Priority of request

[Jan-Erik Soderholm]

If we take a database (Rdb in this case) it uses DLM to coordinate
it's locking, but is from the application point of view, not only
"getting access" or "not getting access". Maybe this is only by
the way Rdb interprets the locks in the DLM (?).

CR Concurrent Read:
The current transaction explicitly locks every record it
reads. Other transactions may read from and write to
the same table.

CW Concurrent Write:
The current transaction explicitly locks every record it
reads or writes. Other transactions may read from and
write to the same table.

PR Protected Read:
The current transaction does not need to lock any
records it reads. Other transactions may read from, but
not write to, the same table.

PW Protected Write:
The current transaction does not need to lock any
records it reads, but it locks every record it modifies or
adds. Other transactions may read in CR mode.

EX Exclusive:
The current transaction does not need to lock any
record it reads or modifies because no other access to
the table is allowed. Other transactions must wait
until this transaction finishes before they can have any
access to the table.


Rdb also modifies the locks on-the-fly when needed such as locking
the whole table, 1000 pages, 100 pages, 10 pages and then, if needed,
individual table rows. This gives fewer locking operation in most
cases. The feature is called "Adjustable lock granularity (ALG)".

Anyway, this shows an environment where the locking is not only
of the "yes/no" type but much more complex.

[VAXman-]

In article <o4m6hu$ikr$1...@dont-email.me>, Chris Scheers <ch...@applied-synergy.com> writes:
>David Froble wrote:
>> Stephen Hoffman wrote:
>>> On 2017-01-05 06:29:20 +0000, David Froble said:
>>>
>>>> I'm looking at a half dozen books on Amazon, all VMS internals and
>>>> such publications. What I'm looking for is information on the DLM,
>>>> data structures and design.
>>>>
>>>> There is a V5.2 from 1987. Not sure if this is what I'm looking
>>>> for. Can anyone with the books check in theirs and tell me which has
>>>> DLM information.
>>>
>>> Not exactly sure what sort of information you're looking for, here.
>>> Use of the DLM on OpenVMS? Interested in the DLM design?
>>> Implementing a DLM? Alternatives to the DLM? Etc...
>>
>> Briefly, I'm looking at the issue of byte range locking. My intentions,
>> if I'm able to do so, are to come up with a preliminary design for
>> additions to the VMS DLM to implement byte range locking, then
>> distribute it to those interested to comment upon, add comments and
>> suggestions, and present the result to VSI as a potential modification.
>> It would be up to VSI as to whether to include it in VMS.
>
>The DLM does support byte range locking (or at least it used to). But
>it is undocumented.

Huh?

[Bob Koehler]

In article <a2f0cb16-98e7-435d...@googlegroups.com>, Bob Gezelter <geze...@rlgsc.com> writes:
>
> How the locks are named and used by other components (e.g., RMS) is what de=

> termines what locks mean (e.g., records, files, byte ranges).

In particular, how RMS and XQP use DLM is not what one might naively
expect.

[Simon Clubley]

On 2017-01-05, David Froble <da...@tsoft-inc.com> wrote:

> Stephen Hoffman wrote:
>>
>> Not exactly sure what sort of information you're looking for, here.
>> Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
>> a DLM? Alternatives to the DLM? Etc...
>
> Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
> able to do so, are to come up with a preliminary design for additions to the VMS
> DLM to implement byte range locking, then distribute it to those interested to
> comment upon, add comments and suggestions, and present the result to VSI as a
> potential modification. It would be up to VSI as to whether to include it in VMS.
>

I haven't read the original Gray paper in a _long_ time.

Was there anything in there which might help ?

It seems to me that you are looking more for design papers which
discuss byte range locking than you are detailed implementation
notes of the existing DLM solution.

Have you tried looking further afield for how other locking systems
(distributed or otherwise) handle this ?

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[David Froble]

Simon Clubley wrote:
> On 2017-01-05, David Froble <da...@tsoft-inc.com> wrote:
>> Stephen Hoffman wrote:
>>> Not exactly sure what sort of information you're looking for, here.
>>> Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
>>> a DLM? Alternatives to the DLM? Etc...
>> Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
>> able to do so, are to come up with a preliminary design for additions to the VMS
>> DLM to implement byte range locking, then distribute it to those interested to
>> comment upon, add comments and suggestions, and present the result to VSI as a
>> potential modification. It would be up to VSI as to whether to include it in VMS.
>>
>
> I haven't read the original Gray paper in a _long_ time.
>
> Was there anything in there which might help ?
>
> It seems to me that you are looking more for design papers which
> discuss byte range locking than you are detailed implementation
> notes of the existing DLM solution.
>
> Have you tried looking further afield for how other locking systems
> (distributed or otherwise) handle this ?
>
> Simon.
>

There are a few things about the VMS DLM that need to be understood.

The first, and biggest item, is that the name is misleading. The DLM does NO
LOCKING. It doesn't have that capability.

The VMS DLM is cluster aware, and lots of work has been done to cause it to work
well in a VMS Cluster environment.

I'm using the term "lock" because it's in general use. It might be more
confusing to attempt to use another term.

What the VMS DLM currently does is accept requests for "locks", determine
whether to allow the request, and create an entry in it's database if the
request is allowed. Trying to keep this simple, no sense in getting into
pending locks, null locks, and such. Not really relavent.

The key issue is that a "lock" is granted for a "resource name". Doesn't matter
what the name is, could be "XYZZY", "Dave's a fool", anything. Subsequent
requests for locks on the identical name are considered upon what's already in
the DLM database. Critical key concept, the DLM doesn't care what the name is,
and does nothing, no evaluation, and such with the "resource name".

I know of implementations that take out a separate lock on each block when
attempting to lock multiple blocks in a file. I don't think it's practical to
take out a lock for every byte in a byte range. So, additional capability is
required.

I've been thinking about this for over a month. My best, and only, idea is that
to address byte range locking, and perhaps other types, is for the DLM to be
able to use data supplied with the lock request to determine whether to grant
the lock, and to store such data in all locks of this type. Since "class" is
already in use for some things, and I'm not really sure what those things are,
while I'd like to call it a "class of lock", maybe I should not do that. So
I'll just call it a "type" of lock.

For this, at least two pieces of data are required. First, an indicator of the
"type" of lock being requested. Second, the actual data to use to describe the
"resource" for which a lock is requested.

Type of lock would be arbitrary and up to the implimenter(s). It would perhaps
be another input parameter to be presented to the DLM. This means the current
type of locks would perhaps need to be assigned a "type". It would also most
likely be the "default type".

For range locking, the first and last element of the range would need to be
submitted to the DLM. This would be for the sub locks. Then there would need
to be new code, implemented by type of lock, to do evaluation of the data.
Note, this is a new concept for the DLM.

When looking for conflicts, the new code would need to search for conflicts in
the range. If found, deny or hold the request. If not found, grant the
request, which includes another entry in the DLM database.

I'm thinking the comparison of ranges will require some interesting concepts and
code. Haven't really thought about that. Got to be doable.

Right now I'm looking for documentation that might allow me to do some grunt
work for whoever might be assigned to implement the proposed enhancement.
Current DLM database table design and such. Hopefully enough to cause those
making decisions to consider the proposed enhancement.

My hope is that if the community, that's us, can present VSI with a concept and
plan that they feel they can implement, then perhaps we'd get the enhancement.

You up for it?

[Stephen Hoffman]

On 2017-01-05 17:45:48 +0000, David Froble said:

> Stephen Hoffman wrote:
>>
>> Not sure how you're envisioning I/O would be connected to the DLM.
>
> I'm not. The DLM should be sort of an object that does a job.
> However, there is nothing that I'm aware of that does byte range
> locking, or has that need, that's currently implemented. Perhaps I
> should ignore the I/O part of it and just concentrate on doing locks on
> byte ranges.

Byte-range locking is "just" a convention of resource names added
within and used (solely) within the C RTL. Some details of the
resource names used were posted (somewhere) a while back IIRC, but
there's nothing published about the internal implementation within the
C RTL, AFAIK. RMS and the XQP are where the file system and
volume-level locking is implemented respectively, and the C RTL then
added some of its own lock-based coordination specifically for the
byte-range processing, as well as the buffer shuffling entailed for
that access as OpenVMS doesn't have a byte-range I/O API. The caller
— the C RTL in this case — has to shuffle the data around within its
own context. RMS I/O doesn't know from C byte-range locking, it (RMS)
has its own lock resource names and its own conventions. C uses its
own locks as well as having the usual and typical dependencies of other
RTL and application software on OpenVMS around the coordination
provided by lower-level software including RMS and the XQP and the lock
manager calls in those layers.

Somewhat related, there are some wrinkles here around what I/O
underneath C is atomic and what is not too, and that's what tripped up
the SSIO work.

>> then the OpenVMS problems with shared stream I/O (SSIO) and a few other
>> details will arise. You've been involved in that discussion before,
>> though. For those that haven't been, there's an overview of SSIO (for
>> PostgreSQL or otherwise) at
>> http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf and
>> the discussions of SSIO-related bugs and problems are buried in the
>> OpenVMS bug lists, and probably a few postings around the 'net.
>>
>> If you're using the DLM for your own resource coordination outside of C
>> (and not of something like PostgreSQL and its locking and I/O
>> activities via the C library), then you get to roll your own byte-range
>> locking if that's what you need. AFAIK, what OpenVMS C is doing there
>> for byte-range locking is specific to C applications, too. File and
>> record locks are general.
>
> No, nothing for me. Just looking to perhaps help out the people who
> need such a capability. One concept I'm solid on is that it should be
> a generic VMS solution, not a C solution. But, that's how I am ...
> :-)

OpenVMS I/O knows nothing of reading and writing ranges of bytes. No
support for that. OpenVMS I/O reads and writes records, and blocks.
This is part of why moving to byte-addressable storage — that's
probably two generations past what VSI is working on with VAFS or
whatever they're calling that new file system they're working on.
(First generation past is arguably a file system and I/O optimized for
block-addressable non-volatile storage, then for byte-addressable
non-volatile storage.)

>> As for application design, I've had success with just mapping the
>> entire file. That's not a typical OpenVMS design, but it does work
>> nicely, and it makes some tasks simpler. Reliability and crash
>> recovery from journals and/or from a shadowed server. Or using code to
>> unmarshall the data into memory, and then operating there, and
>> marshalling the data back out for a clean shutdown or such. Again,
>> with a journal written to storage or to another host or such, for
>> purposes of crash recovery.
>
> Maybe you'd be interested in contributing to the I/O side of things ???
>
> :-)

I've already posted open source that does this.

As for digging the C RTL out from its present quagmire, I'm already on
record recommending rethinking and redesigning the whole tamalé, and
relegating the existing C RTL to retirement and eventual removal; of
replacing the current C RTL with a wholly new implementation. One
that does what's needed now and particularly going forward, that's
64-bit throughout, and preferably one that eliminates the workarounds,
hackery and endless compatibility shims and fixes and that replaces the
C RTL logical names as the control interface with something that works
rather more modularly, assuming that mechanism is even still needed.

[Michael Moroney]

Byte range locking appears to be in the DLM, but undocumented.

Are you looking for other internal/undocumented stuff for whatever
you are doing, or just byte range locking.

Perhaps your request should be for VSI to document byte range
locking after cleaning it up, if necessary.

[Craig A. Berry]

What David would like about SSIO is that the CRTL just becomes a
consumer of a new byte range API in the XFC that bypasses all the stuff
described in your preceding paragraph.[1] It mystifies me why he would
want to start over from scratch rather than just encouraging VSI to fix
whatever kinks remain in SSIO and make it ready for prime time. He might
even advocate for the new API to be documented so it's available from
languages other than C :-).


[1] <http://de.openvms.org/TUD2012/opensource_and_unix_portability.pdf>

[Chris Scheers]

>> Chris,
>>
>> With all due respect, a point of architecture.
>>
>> The DLM manages locks. Locks have limited attributes and names.
>> How the locks are named and used by other components (e.g., RMS) is
>> what determines what locks mean (e.g., records, files, byte ranges).
>>
>> For example, one could create a run-time library that manages golf
>> carts, and use the DLM to coordinate allocations (admittedly silly,
>> one would want to use a database in actuality). The DLM is none the
>> wiser, all it manages are lock names and values. The import of those
>> names is outside the scope of the DLM.
>>
>> - Bob Gezelter, http://www.rlgsc.com
>
> Correct. The DLM offers locks on "resource names" for cooperating
> processes. It actually does no locking of anything. It just allows a
> user to request a lock on a resource name, and if granted, will not
> issue another lock on that resource until the first is released. Either
> the second user waits for the lock to be granted, or, determines that
> the resource cannot be accessed. In reality, if it wished to screw up
> things, it could go ahead and access the resource anyway. The key word
> is "cooperating".
>
> Lots of uses. Cause a batch job to wait. Cause user applications to be
> synchronized, and such.

OK, I misstated and you misunderstood.

Yes, I fully understand that the DLM does not enforce anything and use
of it is cooperative.

Granted, the DLM does not actually lock anything. What we are
discussing are resources.

In the "normal" usage of the DLM, a resource is a singular item. You
can lock "X", but that has no effect on a lock on "Y".

What you misunderstood is that the DLM already has the (undocumented)
capability for a resource to be a range. So if you lock the range
1000..2000, and then ask for a lock on 500..1500, a conflict should be
detected.

YMMV: I have not actually tried this as I have not had a need so far.

[Stephen Hoffman]

On 2017-01-06 22:20:47 +0000, Craig A. Berry said:

> What David would like about SSIO is that the CRTL just becomes a

> consumer of a new byte range API in the XFC...

Works for me. (But then I'd like also to see a byte-addressable file
system, too.)

The existing byte-range locking and SSIO and C jackets is something
most of us would rather not have to deal with.

Though I can easily forsee an application where access into the
byte-range DLM API would be handy. This if I'm using the DLM to
coordinate activity in a ginormous memory section. I'm usually using
interlocked queues for that sort of thing now, and that isn't always
the nicest approach for some of the designs.

[David Froble]

I'm not looking at starting over. I'm looking at an enhancement to the DLM.
The DLM is fully cluster aware and works fine in that environment. That is the
place to locate additional locking capabilities.

Maybe others think differently, but, I consider SSIO a language feature, and the
DLM an OS feature. ME, I'd choose the lower level for something that can be so
useful.

If what I'm considering is workable and is implemented, it would be something
that an SSIO implementation could use. And, other things could use also.
Thinking just SSIO is just too narrow of a focus, perhaps except for C bigots.

[David Froble]

An immediate question comes to mind. What is the magnitude of what you're
talking about? One question I've asked myself is, how big of integers might be
needed to define a range? Maybe quadword?

Yes, what I'm considering would be perfect for what you mention, and cluster
aware, if that's a consideration.

Any idea what language(s) were used to implement the DLM?

[David Froble]

Well, if that is there, then job may be already completed. But I'm not aware of
such a capability. It requires the DLM to work on more than just resource
names, as you suggest, "X" and "Y".

You got any pointers to any information on such a capability?

[David Froble]

Well, that's sure a teaser.

Is something implemented?

If so, is there any information about how it's implemented, it's capabilities,
and such.

I'd rather not go spinning my wheels trying to work up some specs on something
that already exists.

[Stephen Hoffman]

On 2017-01-07 02:03:59 +0000, David Froble said:

> Stephen Hoffman wrote:
>> On 2017-01-06 22:20:47 +0000, Craig A. Berry said:
>>
>>> What David would like about SSIO is that the CRTL just becomes a
>>> consumer of a new byte range API in the XFC...
>>
>> Works for me. (But then I'd like also to see a byte-addressable file
>> system, too.)
>>
>> The existing byte-range locking and SSIO and C jackets is something
>> most of us would rather not have to deal with.
>>
>> Though I can easily forsee an application where access into the
>> byte-range DLM API would be handy. This if I'm using the DLM to
>> coordinate activity in a ginormous memory section. I'm usually using
>> interlocked queues for that sort of thing now, and that isn't always
>> the nicest approach for some of the designs.
>
> An immediate question comes to mind. What is the magnitude of what
> you're talking about?

"It depends". Anything from a month or so for a DLM-interlocked
memory design and implementation and testing assuming a small and
well-specified and single-host requirement, on up to scheduling
infinity. Where in this range depending on what is and is not in
scope in the project, performance requirements and related, and how
compatible the design is to any existing source code and available
tools. It's a how-high-is-up right now.

> One question I've asked myself is, how big of integers might be needed
> to define a range? Maybe quadword?

In OpenVMS it'd be a quadword, as that's the size of the addresses
given current virtual addressing limit of OpenVMS. There's a a LCK
data structure pointing to that range block. Based on the definitions,
the relevant data structure appears to be named LKIDEF_BR, though that
structure does not appear to have been exposed in the Bliss or C
definitions for V8.4.

> Yes, what I'm considering would be perfect for what you mention, and
> cluster aware, if that's a consideration.

There'll be overhead shuffling the data around; it's basically
server-level mirroring. Performance is going to come down to the
speed of the host-to-host, and that's going to be a whole lot of
bespoke code for networking traffic quite possibly involving some of
the message queue software and DECdtm. RDMA would be handy here too,
certainly.

If the application requirements around performance permit it, I'd
deploy modern storage and 10 GbE networking and only cache the data in
memory when required. Might be able to brute-force the design and
avoid implementing mirrored memory, in other words. Flash and SSD
are massively faster than HDD, after all. That'd avoid a whole lot of
bespoke code, too. There's also the obvious and necessary discussion
around using one of the available commercial or open-source databases;
whether Oracle or otherwise.

[Stephen Hoffman]

On 2017-01-07 02:14:22 +0000, David Froble said:

> Is something implemented?
>
> If so, is there any information about how it's implemented, it's
> capabilities, and such.

C is using the API, and it's undocumented, which means it's either off
to the source listings for details. Or a discussion directly with
VSI. Or both.

[Chris Scheers]

As I said, AFAIK, you need to look at the VMS sources for information.

[John Reagan]

I've stayed away from this all week, but I'll toss in a few facts.

Please, please, please don't call the range operand on $ENQ as byte-range locking. As many other notes have already said, the DLM doesn't "lock" anything. It is up to the cooperating processes to make it all work.

The CRTL's BRL is in there now via the fcntl() interface.

There essentially creates a bunch of DLM locks with the name C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.

It uses sublocks to keep track of the ranges of bytes is it interested in.

The scheme uses various blocking ASTs so each process gets notified of another process accessing the range or attempting to also lock the range (or part of the range).

The range feature of the DLM simply allows some blocking asts to be skipped since the CRTL would simply decide that the ranges don't overlap and that there is nothing to do.

The code in the CRTL has to worry about overlapping ranges, combining adjacent ranges, splitting large ranges into smaller ones, removing system locks at image rundown, etc.

The code in the CRTL is split between DECC$SHR and DECC$SHRP since the actual locking uses SYSTEM wide locks.

From reading the comments, the routines DECC$$BRL_ENQ, DECC$$BRL_ENQW, DECC$$BRL_DEQ, and DECC$$BRL_GETLKIW were meant to be user-visible. There are various probes and parameter checking. I'm not sure if they were ever described in public and I'm not sure of what sort of application would use them directly.

To push BRL down to RMS or the XQP to provide the functionality to non-C applications, it would have to worry about all the things the CRTL does. It would have to worry about more things like what does byte-range locking mean for non-sequential files? I assume you wouldn't allow that since you can already lock records on indexed files. Do you want BRL on individual fields inside an indexed record?

I counted close to 7000 lines of code across both images.

[Bill Gunshannon]

Hoff,
You skipped the one question I would have also considered
interesting.

"Any idea what language(s) were used to implement the DLM? "

bill

[John Reagan]

I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)

[David Froble]

John Reagan wrote:
> I've stayed away from this all week, but I'll toss in a few facts.

Gee, I wish you would have done that tossing a month ago, I might have got more
work done in the shop. But, you had no idea of what I was considering ....

> Please, please, please don't call the range operand on $ENQ as byte-range
> locking. As many other notes have already said, the DLM doesn't "lock"
> anything. It is up to the cooperating processes to make it all work.

I agree 100%, but, the term "lock" has been in use since the mid 1980s. I think
we're stuck with it.

> The CRTL's BRL is in there now via the fcntl() interface.
>
> There essentially creates a bunch of DLM locks with the name
> C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.
>
> It uses sublocks to keep track of the ranges of bytes is it interested in.

Just as sublocks are used to specify other items of interest, such as record
numbers.

> The scheme uses various blocking ASTs so each process gets notified of
> another process accessing the range or attempting to also lock the range (or
> part of the range).
>
> The range feature of the DLM simply allows some blocking asts to be skipped
> since the CRTL would simply decide that the ranges don't overlap and that
> there is nothing to do.

Just like RMS records, and other things.

> The code in the CRTL has to worry about overlapping ranges, combining
> adjacent ranges, splitting large ranges into smaller ones, removing system
> locks at image rundown, etc.

That I don't understand. The DLM has always (as far as I know) removed locks on
image rundown. Then again, you specified CRTL, not DLM.

> The code in the CRTL is split between DECC$SHR and DECC$SHRP since the actual
> locking uses SYSTEM wide locks.

Not sure what good a non-system wide lock is when multiple processes are
accessing the resources in question.

> From reading the comments, the routines DECC$$BRL_ENQ, DECC$$BRL_ENQW,
> DECC$$BRL_DEQ, and DECC$$BRL_GETLKIW were meant to be user-visible. There
> are various probes and parameter checking. I'm not sure if they were ever
> described in public and I'm not sure of what sort of application would use
> them directly.
>
> To push BRL down to RMS or the XQP to provide the functionality to non-C
> applications, it would have to worry about all the things the CRTL does. It
> would have to worry about more things like what does byte-range locking mean
> for non-sequential files? I assume you wouldn't allow that since you can
> already lock records on indexed files. Do you want BRL on individual fields
> inside an indexed record?
>
> I counted close to 7000 lines of code across both images.

This is beginning to sound like other things that happened when VMS was taken
away from competent developers. But I don't know when all this was done.
Regardless, I don't see what business the CRTL has in the entire scheme. Sure
sounds like someone intended a "patch" for a particular and limited need rather
than a well architectured enhanced capability.

But, thanks for the info ....

[Craig A. Berry]

On 1/7/17 2:22 PM, David Froble wrote:

> John Reagan wrote:
>> Please, please, please don't call the range operand on $ENQ as byte-range
>> locking. As many other notes have already said, the DLM doesn't "lock"
>> anything. It is up to the cooperating processes to make it all work.

>> The code in the CRTL has to worry about overlapping ranges, combining
>> adjacent ranges, splitting large ranges into smaller ones, removing
>> system locks at image rundown, etc.
>
> That I don't understand. The DLM has always (as far as I know) removed
> locks on image rundown. Then again, you specified CRTL, not DLM.

I don't think you were listening when he said the DLM doesn't lock
anything. Something else still has to do the actual synchronization of
I/O, and whatever the DLM does at image run-down is not going to undo
any of that. I suspect this was all implemented in the CRTL because
nothing else provided it, but in principle SSIO, if it works out, should
be a step forward by making it more general and more performant.

[Craig A. Berry]

On 1/7/17 8:29 AM, John Reagan wrote:

> [fcntl] essentially creates a bunch of DLM locks with the name

> C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.

Which may need revision when the components of the FID become 64-bit,
and as far as I can see the resource name argument to $ENQ is limited to
31 bytes. :-(

[Stephen Hoffman]

On 2017-01-07 14:29:05 +0000, John Reagan said:

> I've stayed away from this all week, but I'll toss in a few facts.
>
> Please, please, please don't call the range operand on $ENQ as

> byte-range locking....

I think you're going to lose that one, at least prior to a stable API
and published documentation becoming available.

> As many other notes have already said, the DLM doesn't "lock"
> anything. It is up to the cooperating processes to make it all work.

Ayup, and the L in DLM is for limerick, right? That ship has sailed.

> The CRTL's BRL is in there now via the fcntl() interface.
> There essentially creates a bunch of DLM locks with the name
> C$_deviceAAAABBBBCCCC where ABC is the hex representation of the FID.
> It uses sublocks to keep track of the ranges of bytes is it interested in.
> The scheme uses various blocking ASTs so each process gets notified of
> another process accessing the range or attempting to also lock the
> range (or part of the range).
> The range feature of the DLM simply allows some blocking asts to be
> skipped since the CRTL would simply decide that the ranges don't
> overlap and that there is nothing to do.
>
> The code in the CRTL has to worry about overlapping ranges, combining
> adjacent ranges, splitting large ranges into smaller ones, removing
> system locks at image rundown, etc.
>
> The code in the CRTL is split between DECC$SHR and DECC$SHRP since the
> actual locking uses SYSTEM wide locks.
>
> From reading the comments, the routines DECC$$BRL_ENQ, DECC$$BRL_ENQW,
> DECC$$BRL_DEQ, and DECC$$BRL_GETLKIW were meant to be user-visible.
> There are various probes and parameter checking. I'm not sure if they
> were ever described in public and I'm not sure of what sort of
> application would use them directly.

They're not documented anywhere the search engines can locate. The $$
naming usually indicates a private symbol and not a public one, too.

> To push BRL down to RMS or the XQP to provide the functionality to
> non-C applications, it would have to worry about all the things the
> CRTL does. It would have to worry about more things like what does
> byte-range locking mean for non-sequential files? I assume you
> wouldn't allow that since you can already lock records on indexed
> files. Do you want BRL on individual fields inside an indexed record?
>
> I counted close to 7000 lines of code across both images.

For what I suspect David was seeking — using this to coordinate ranges
of memory, and not ranges wthin files — this project will then either
involve somehow recreating the C RTL portion of the coordinating code
in the app and then call the Distributed Limerick, err, Lock Manager,
or it'll involve the creation of a private lock manager. Having this
scattered between the DLM and the C RTL was probably expedient, but
AFAICT it does not seem particularly reusable.

[Bill Gunshannon]

Thank you.

bill

[Stephen Hoffman]

Yes, I did.

Long answer:

That's because I didn't think the question was interesting enough to
warrant the research, nor sufficiently relevant to the rest of what
David was seeking of the distributed lock manager; of the DLM. Seemed
a good way to distract from what David was seeking, if I had researched
and had addressed it.

The OpenVMS operating system is written largely in a mixture of
Macro32, Bliss and C.

As of twenty years ago, the bulk of the source code was in thirds,
among those three languages. Plus far smaller numbers of a whole slew
of minor languages, of course. Macro32 and Bliss are the oldest, and
C became more prevalent in the 1990s, as a decision was made to create
new and overhaul existing and to migrate development to that language.

I'd expect C to have become the predominant programming language used
in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
doubt that anybody has run the line-counting tools and the statistical
reporting tools in the OpenVMS source code control system — not since I
last did that ~twenty years ago.

What language the DLM is written in is not relevant to calling
programs, as the APIs are classic OpenVMS. Not particularly modern,
requires lots and lots of glue code, but typical among APIs. VSI is
not going to change the implementation languages, either.

As for determining what mixture of languages where used, I don't know
and would have to rummage the source listings. I'd guess a mix of all
three languages here, given both the vintage and the ongoing work in
the DLM.

>> "Any idea what language(s) were used to implement the DLM? "
>

> I see both Macro-32 and C (ie, SYSENQDEQ.MAR and LOCK_UTILS.C)

If it's not in Rust or something else equivalently newer, it's not
really all that interesting (to me). Why? Because it's all been
discussed and debated before, and I well remember the debates inside
OpenVMS development particularly from the folks fond of Bliss too.
Because it's all very familiar. Because there hasn't (yet) been a
shift in the OpenVMS developer mindset and the internal practices.
Either at VSI, or among OpenVMS customers. And even knowing there's
Rust or otherwise in use is not relevant to application use of the DLM.

Macro32, Bliss, C and how all of OpenVMS and most of the OpenVMS
applications are developed are pretty crufty, and the devops tools are
very old and comparatively very problematic. (More than a few OpenVMS
folks don't use IDEs, but many developers on other platforms do.)
They were good for their time certainly and the implementations still
work well, and there are excellent reasons to avoid any wholesale
rewrites of these and of other existing OpenVMS hunks into newer
languages. Incremental rewrite-migration-retirements, on the other
hand...

There are definite considerations around the scale of the operating
system source code base and its builds and the substantial test suite
certainly, but the languages and development tools are poor and —
compared with what's available now — dated. The scale of the OpenVMS
source code alone is something that most folks here don't have to deal
with. It wouldn't surprise me to learn that sometime over the next
decade or so, VSI will look at migrating some of their internal tooling
to git, mercurial or fossil, for instance. Both for their own internal
use, and for community and shared development, They'll also likely
consider implementing easier migrations from CMS to one or more of
those tools, too. But I digress.

The languages and tools available to and used by many of the OpenVMS
customers aren't comparatively very good or very efficient to use or
very capable by contemporary standards, either. Certainly not in
comparison to the languages and frameworks I'm working with elsewhere.
I'd wager those sites haven't looked at this topic in a decade too,
not that there are very good alternatives presently available on
OpenVMS.

VSI is addressing parts of this conundrum with updates to the existing
languages and tools and the migration to LLVM, but — given how busy the
next five years will undoubtedly be in Bolton, and how tight their
budget will be — they're likely not in a position to start working with
more modern tools and API designs themselves, nor to start to introduce
those to their customers, and they'll also want and need to preserve
the installed base. Which means rolling forward the languages and tools
they have is the best of bad choices, and they're probably not soon
going to be introducing Rust or Go or such into OpenVMS, or into the
OpenVMS community. Then there's whether the community is even
interested, and whether enough younger developers can become interested
in OpenVMS as a viable and serious platform for serious applications;
as something other than a candidate for OS tourism or retrocomputing or
schadenfreude.

I'd hope we'll see Python, Lua and a few tools added into the base
install, kits for git and mercurial and such, but then similar hope
holds for fully integrating IP networking and many other necessary
improvements into the platform; the work necessary to make development
and deployment and maintenance on and for OpenVMS less problematic and
less uncompetitive. (n.b. I'm aware of the ports and implementations
of these and other tools on OpenVMS and have looked at or used most of
these, too.)

In summary.... Debating whether two-revisions-of-the-standard-back C,
or Macro32, or Bliss — or any other languages of similar vintage — was
an appropriate choice for the DLM seemed somewhat uninteresting.
Now y'all better know why i expunged that part of the question.

https://www.fossil-scm.org
https://www.rust-lang.org

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: January 8, 2017 9:51 AM
> To: info...@rbnsn.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [Info-vax] Internals and Data Structures - DLM information
>
> On 2017-01-07 17:27:42 +0000, John Reagan said:
>
> > On Saturday, January 7, 2017 at 11:49:51 AM UTC-5, Bill Gunshannon
> wrote:
> >> Hoff,
> >> You skipped the one question I would have also considered
> interesting.
>
> Yes, I did.
>
> Long answer:
>
> That's because I didn't think the question was interesting enough to
> warrant the research, nor sufficiently relevant to the rest of what
> David was seeking of the distributed lock manager; of the DLM.
> Seemed
> a good way to distract from what David was seeking, if I had
> researched and had addressed it.
>

[snip..]

> In summary.... Debating whether two-revisions-of-the-standard-back
> C, or Macro32, or Bliss — or any other languages of similar vintage —
> was
> an appropriate choice for the DLM seemed somewhat uninteresting.
> Now y'all better know why i expunged that part of the question.
>
> https://www.fossil-scm.org
> https://www.rust-lang.org
>
>

Agree - imho, this reimplementation of tools discussion is analogous to someone looking at the Taj Mahal and thinking yes, it’s a great structure, but it would have been so much "better" if they had built it with "modern" tools.

Remember - Todays "modern" tools are tomorrows "fossils".

At some point in the future (X years), there will be references to Rust as a "fossil" in the same way as many today might refer to Bliss as a fossil. The only question is what is that value of "X".

What is important as a differentiator is not the tools something was built with, but rather the beauty (yes, there are always flaws) in what and how something was built.


Regards,

Kerry Main
Kerry dot main at starkgaming dot com

[Stephen Hoffman]

On 2017-01-08 16:02:36 +0000, Kerry Main said:


> Todays "modern" tools are tomorrows "fossils".

People using tools from yesterday with depreciating values today are
excellent competitive targets tomorrow, too.

Good tools and good designs do hold their value longer than lesser
choices, but even good tools and good designs are often replaced by
better ones.

Be aware of what you're doing, and be aware of what others are doing.
You can't assume your tools and designs are still good, if you haven't
looked around.

As for the DLM, if it works for what you need, use it. Or look around
at the alternative approaches, and use one of those. There are many
papers and discussions...

http://www.mcs.anl.gov/~thakur/papers/byte-range.pdf
http://www.mcs.anl.gov/~thakur/papers/rma-fv.pdf
https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html
http://fsfilters.blogspot.com/2011/11/byte-range-locks-and-irpmjcleanup.html
http://wiki.linux-nfs.org/wiki/index.php/Cluster_Coherent_NFS_and_Byte_Range_Locking

https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html
(issues with the POSIX locking APIs, etc. Includes "The Tar Pit:
Backwards Compatibility", too. Anybody need to guess where those share
modes came from?)
Etc...

[clairg...@gmail.com]

Trivia time. In the most recent Itanium system build the breakdown of 8756 modules by language is:

Ada 1%
Assembler 1%
C 54%
BLISS 32%
MACRO 12%

These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of
> Stephen Hoffman via Info-vax
> Sent: January 8, 2017 3:00 PM
> To: info...@rbnsn.com
> Cc: Stephen Hoffman <seao...@hoffmanlabs.invalid>
> Subject: Re: [Info-vax] Internals and Data Structures - DLM
information
>

> On 2017-01-08 16:02:36 +0000, Kerry Main said:
>
>
> > Todays "modern" tools are tomorrows "fossils".
>
> People using tools from yesterday with depreciating values today are
> excellent competitive targets tomorrow, too.
>

It is also true that those breaking trail usually take the first
arrows ...

> Good tools and good designs do hold their value longer than lesser
> choices, but even good tools and good designs are often replaced by
> better ones.
>

Replaced, yes .. but not necessarily "better".

One only has to look at how commodity OS's have, in many areas,
overcome tradition enterprise platforms (Solaris, AIX, OpenVMS,
NonStop etc.). Very few Customers who have migrated to commodity OS
platforms viewed them as "better" than their current platform, but
rather were cheaper (albeit debateable when looking at the big
picture, but that is the power of marketing).

I am sure every med to large company also have examples of business
applications which were replaced by corporate IT in the name of
"standardization", but in fact, provided only a fraction of the
flexibility and benefits of the "older" application.

> Be aware of what you're doing, and be aware of what others are
> doing.
> You can't assume your tools and designs are still good, if you
haven't
> looked around.
>

Agree. Keeping current with trends is always good as long as one
understands that new does not necessarily mean better.

> As for the DLM, if it works for what you need, use it. Or look
around
> at the alternative approaches, and use one of those. There are
many
> papers and discussions...
>
> http://www.mcs.anl.gov/~thakur/papers/byte-range.pdf
> http://www.mcs.anl.gov/~thakur/papers/rma-fv.pdf
> https://www.samba.org/samba/docs/man/Samba-HOWTO-
> Collection/locking.html
> http://fsfilters.blogspot.com/2011/11/byte-range-locks-and-
> irpmjcleanup.html
> http://wiki.linux-
> nfs.org/wiki/index.php/Cluster_Coherent_NFS_and_Byte_Range_Loc
> king
>
> https://www.samba.org/samba/news/articles/low_point/tale_two_st
> ds_os2.html
> (issues with the POSIX locking APIs, etc. Includes "The Tar Pit:
> Backwards Compatibility", too. Anybody need to guess where those
> share modes came from?) Etc...
>

Interesting quote from the last url - another interesting reference to
where Windows "acquired" past technology from.
"However, the Win32 design does some things very well; security, for
instance. Security isn't the number one thing people think of when
considering Windows, but in the Win32 API security is a very great
concern. In Win32, every object can be secured, and a property called
a "Security Descriptor" which contains an access control list (ACL)
can be attached to it. This means objects like processes, files,
directories, even Windows can have ACLs attached. This is much cleaner
than POSIX, where only objects in the file system can have ACLs
attached to them."

As for the DLM in general, one needs to remember there are different
clustering models in play. Windows/UNIX/Linux/NonStop are primarily
shared nothing models whereas OpenVMS, Linux/GFS (file system limit
100TB) and z/OS are shared disk models.

Both clustering models have pro's and con's, but here is a WP which
explains the pro's and con's of each model:

Shortened url:
http://bit.ly/2dScx9k
"Comparing shared-nothing and shared-disk in benchmarks is analogous
to comparing a dragster and a Porsche. The dragster, like the
hand-tuned shared-nothing database, will beat the Porsche in a
straight quarter mile race. However, the Porsche, like a shared-disk
database, will easily beat the dragster on regular roads. If your
selected benchmark is a quarter mile straightaway that tests all out
speed, like Sysbench, a shared-nothing database will win. However,
shared-disk will perform better in real world environments."

Original (will wrap)
http://www.scaledb.com/wp-content/uploads/2015/11/Shared-Nohing-vs-Sha
red-Disk-WP_SDvSN.pdf

So, do you need a dragster or a Porsche?

:-)

[Bill Gunshannon]

Ada? Doing what? :-)

And, I thought MACRO was the VMS Assembler. Is there another?

bill

[clairg...@gmail.com]

Two security-related facilities are written mostly in Ada.

The MACRO compiler compiles the VAX MACRO-32 code for Itanium.

[Jan-Erik Soderholm]

Macro-32 was the "Macro Assembler" on VAX where you wrote VAX
machine code mnemonics.

On Alpha and Itanium Macro-32 are compilers, compiling the VAX
instructions into Alpha or Itanium machine code.

Then there are native assamblers, on Alpha it is called Macro-64.
It was a separate kit and not delivered with the base OS (such as
MACRO32 is). Seems to be available from the freeware V4 CD:
http://www.decuslib.com/freeware/freewarev40/macro64/

The Itanium assembler seems to be a port of the open source Intel
Itanium version called "IAS". I found this:
http://decuslib.com/HP_Open_Source_Tools/ias/kit/
There is a readme file describing the "kit".

[Simon Clubley]

On 2017-01-06, David Froble <da...@tsoft-inc.com> wrote:
> Simon Clubley wrote:
>> On 2017-01-05, David Froble <da...@tsoft-inc.com> wrote:

>>> Stephen Hoffman wrote:
>>>> Not exactly sure what sort of information you're looking for, here.
>>>> Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
>>>> a DLM? Alternatives to the DLM? Etc...
>>> Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
>>> able to do so, are to come up with a preliminary design for additions to the VMS
>>> DLM to implement byte range locking, then distribute it to those interested to
>>> comment upon, add comments and suggestions, and present the result to VSI as a
>>> potential modification. It would be up to VSI as to whether to include it in VMS.
>>>
>>

>> I haven't read the original Gray paper in a _long_ time.
>>
>> Was there anything in there which might help ?
>>
>> It seems to me that you are looking more for design papers which
>> discuss byte range locking than you are detailed implementation
>> notes of the existing DLM solution.
>>
>> Have you tried looking further afield for how other locking systems
>> (distributed or otherwise) handle this ?
>>
>> Simon.
>>
>
> There are a few things about the VMS DLM that need to be understood.
>
> The first, and biggest item, is that the name is misleading. The DLM does NO
> LOCKING. It doesn't have that capability.
>

[snip]

I know all this, but as Stephen has also pointed out, it was called
the Distributed _Lock_ Manager and not the Distributed _Resource_
Manager when it was designed back in the 1980s.

Blame the DEC people who choose the name, not those of us who just use
the name that DEC gave it. :-)

>
> I'm thinking the comparison of ranges will require some interesting concepts and
> code. Haven't really thought about that. Got to be doable.
>

Why ? If it's too difficult to prove the ranges overlap, you can
reverse the check by proving the ranges _don't_ overlap instead
of trying to prove they _do_ overlap.

In some cases, proving something doesn't overlap can be easier then
trying to prove it does.

Besides, AFAICS, it's just the normal intersection of sets that are
at play here.

> Right now I'm looking for documentation that might allow me to do some grunt
> work for whoever might be assigned to implement the proposed enhancement.
> Current DLM database table design and such. Hopefully enough to cause those
> making decisions to consider the proposed enhancement.
>
> My hope is that if the community, that's us, can present VSI with a concept and
> plan that they feel they can implement, then perhaps we'd get the enhancement.
>
> You up for it?

Sorry but I'm currently working on variety of VMS related and
non-VMS projects in my spare time.

Now that you know the DLM supports a generic range locking capability,
I think you should instead now make a formal request to VSI to have
this formally documented (and completed if required).

Send off an email to Clair at the RnD VSI address with a formal
request (including your justification for why the documentation/work
is required) and see what he has to say.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Simon Clubley]

On 2017-01-08, clairg...@gmail.com <clairg...@gmail.com> wrote:
>
> Two security-related facilities are written mostly in Ada.
>

Well done to whoever did that. (I'm being serious here.)

[Simon Clubley]

On 2017-01-08, Kerry Main <kemain...@gmail.com> wrote:
>
> At some point in the future (X years), there will be references to
> Rust as a "fossil" in the same way as many today might refer to Bliss
> as a fossil. The only question is what is that value of "X".
>

And when that happens, there will still be people using Ada in
safety critical applications. Ada was targetted for a specific
type of high reliability requirement and it does that very well.

(Unfortunately for general system security, there will also still
be people writing low level code in C when that happens because of
the massive installed base and because it still satisfies a certain
technology and (low level) portability requirement.)

BTW, I've tried looking at Rust, but it's syntax is _really_ yucky
to work with and I can't help thinking that looking at Rust code
6 months to a year later would not exactly be a great experience.

[Bill Gunshannon]

On 1/8/17 7:53 PM, Simon Clubley wrote:
> On 2017-01-08, Kerry Main <kemain...@gmail.com> wrote:
>>
>> At some point in the future (X years), there will be references to
>> Rust as a "fossil" in the same way as many today might refer to Bliss
>> as a fossil. The only question is what is that value of "X".
>>
>
> And when that happens, there will still be people using Ada in
> safety critical applications. Ada was targetted for a specific
> type of high reliability requirement and it does that very well.

And it missed the mark by many a mile. Ask the Air Force. It was
their idea in the first place but by the time it became a reality
it was so different from what they wanted they refused to use it
even when DOD tried to mandate it. Last I checked they were still
using Jovial out in Dayton.

>
> (Unfortunately for general system security, there will also still
> be people writing low level code in C when that happens because of
> the massive installed base and because it still satisfies a certain
> technology and (low level) portability requirement.)

Back in my government contracting days my employer was big on using
Ada. I was a member of the company's Ada User Group and once mentioned
to one of the major experts that I thought it would be interesting
to re-write the C parts of Unix in Ada. I was informed that the result
would be a totally unusable OS. The code would be too inefficient
to be practical.

>
> BTW, I've tried looking at Rust, but it's syntax is _really_ yucky
> to work with and I can't help thinking that looking at Rust code
> 6 months to a year later would not exactly be a great experience.
>

Sounds like PHP. I think I need to take a look at Rust. I haven't
had a good laugh all week.

bill

[Scott Dorsey]

Bill Gunshannon <bill.gu...@gmail.com> wrote:
>Back in my government contracting days my employer was big on using
>Ada. I was a member of the company's Ada User Group and once mentioned
>to one of the major experts that I thought it would be interesting
>to re-write the C parts of Unix in Ada. I was informed that the result
>would be a totally unusable OS. The code would be too inefficient
>to be practical.

Ada was the PL/1 of the nineties, with every possible feature added to the
language long before people had figured out how to make compilers work with
all those features.

So first you have the issue where every programmer is using his own personal
subset of the language because the language is too big for everyone to know
everything, so people can't necessarily read one another's code.

And secondly you have the issue of the compiler requiring tremendous resources
and optimization being difficult. In 1989 we had a Data General machine that
could support about 100 developers writing Pascal, or two developers writing
Ada.

Fortunately times have changed and the efficiency issues of the runtime code
has been fixed, while the compilation efficiency is less of a problem due to
much faster hardware.

But the issue with programmers not knowing the whole thing remains.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[David Froble]

The Assembler 1% is most likely itanic assembly code.

Looks like John is getting his way, the use of Macro-32 is shrinking. If the
compiler ever goes away I've got a database product that will be going extinct.

[clairg...@gmail.com]

Guess I should have been more precise. This was an Itanium build so

assembler = Itanium assembler

MACRO = VAX MACRO-32

[RobertsonEricW]

On Thursday, January 5, 2017 at 12:49:04 PM UTC-5, BillPedersen wrote:
> On Thursday, January 5, 2017 at 11:21:21 AM UTC-5, johnwa...@yahoo.co.uk wrote:

> > On Thursday, 5 January 2017 15:23:03 UTC, David Froble wrote:
> > > Stephen Hoffman wrote:
> > > > On 2017-01-05 06:29:20 +0000, David Froble said:
> > > >
> > > >> I'm looking at a half dozen books on Amazon, all VMS internals and
> > > >> such publications. What I'm looking for is information on the DLM,
> > > >> data structures and design.
> > > >>
> > > >> There is a V5.2 from 1987. Not sure if this is what I'm looking for.
> > > >> Can anyone with the books check in theirs and tell me which has DLM
> > > >> information.
> > > >

> > > > Not exactly sure what sort of information you're looking for, here.
> > > > Use of the DLM on OpenVMS? Interested in the DLM design? Implementing
> > > > a DLM? Alternatives to the DLM? Etc...
> > >
> > > Briefly, I'm looking at the issue of byte range locking. My intentions, if I'm
> > > able to do so, are to come up with a preliminary design for additions to the VMS
> > > DLM to implement byte range locking, then distribute it to those interested to
> > > comment upon, add comments and suggestions, and present the result to VSI as a
> > > potential modification. It would be up to VSI as to whether to include it in VMS.
> > >

> > > I just don't have enough information now, and, will need some help, as right
> > > now, I'd come up with a solution looking for a problem. For instance, Postgre
> > > database, how large a range is reasonable, and if talking bytes, how many should
> > > be supported, which would determine the size of integer required. I figure a
> > > database could be rather large.
> > >
> > > Don't know that I need exact information on the data structures. Whoever
> > > implements such a modification would have to work from what's current in VMS.
> > > But I need enough information to get close to what might be required.
> > >
> > > One thing seems to be clear to me, it's not just the DLM, but I/O stuff would
> > > have to also be implemented to use the new feature. There it gets interesting.
> > > If using I/O buffers, how to determine the size for one thing. Or perhaps
> > > direct I/O into the "on disk" file?
> > >
> > > > Beyond the two manuals in the doc set and articles in the DEC and VMS
> > > > Technical Journals, Roy Davis' VAXcluster Principles book, most likely.
> > > > ISBN was 55558-112-9
> > > >
> > > > Less so the IDSM. The DLM gets a fairly general discussion in the IDSM,
> > > > but not much in the way of details.
> > > >
> > > > Or the OpenVMS source listings, for the full details.
> > > >
> > > > Or for more general interest, skim through the OpenSSL open-source
> > > > variant of clustering software that Compaq released; that was a Unix DLM
> > > > similar to the OpenVMS DLM. Or skim the books and resources about one
> > > > of the more recent variants; see
> > > > https://en.wikipedia.org/wiki/Distributed_lock_manager for some pointers
> > > > to those.
> > > >
> > > > If you do want to go for an IDSM for this or other reasons, you'll be
> > > > looking at a mix of V6 and the V7 updates to get all of the information
> > > > with the "latest" available information. The V6 stuff was the last
> > > > full copy of the IDSM, the V7 subset documents added chapters with some
> > > > details on specific and then-newer areas.
> > > >
> > > > Closely related to the DLM is the distributed transaction manager;
> > > > DECdtm, and also the DEC RTR and the more recent open-source
> > > > 0MQ/ActiveMQ/RabbitMQ messaging packages.
> > > >
> > > >
> >
> > There is/was a Digital Technical Journal article on
> > the original architecture and design of the DLM.
> > It might be a bit, er, basic for your stated needs,
> > but you are in a better position than I am to make
> > that judgement.
> >
> > It's in the VAXcluster Systems special edition of the
> > DTJ, from September 1987, pages 29-44.
> >
> > I found it at
> > http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v01-05_sep1987.pdf
> > but other sources may also be available e.g. Bitsavers.
> >
> > Are you aware that PATHWORKS did byte range locking
> > some twenty-odd years ago? Again, there is a DTJ special
> > issue with some potentially relevant info: Volume 4
> > Number 1, from 1992. The details have doubtless changed,
> > but the basics may or may not have changed.
> >
> > Here are a few words from the piece on Byte Range Locking:
> > "Byte Range Locking
> > The MS-NET architecture allows for concurrent
> > access to server-based files by multiple clients. PC
> > applications acquire this functionality through the
> > MS-DOS byte range locking calls. These calls allow
> > PC applications to lock and unlock ranges of bytes
> > in a file and to detect conflicts. Conflicts occur
> > when part or all of a range specified to be locked
> > has been locked from a previous call. In contrast,
> > the approach taken by RMS provides locking on a
> > record basis. RMS uses the VMS distributed lock
> > manager to implement this functionality. Unfortu-
> > nately, the lock manager is not well suited to imple-
> > menting byte range locks because the byte range is
> > represented in a form that allows the lock manager
> > to arbitrate access. Therefore, the file server imple-
> > ments its own lock database and arbitrates access
> > to shared files. Internally, the server process main-
> > tains a list of locks for each file the server has open
> > and arbitrates access based on these lock struc-
> > tures. Files opened by the file server cannot be
> > shared with other VMS processes because the file
> > server has an exclusive mode lock on each file it has
> > open through the VMS lock manager. The exclusive
> > mode lock guarantees protection from other VMS
> > processes.
> > [etc]"
> >
> >
> > The DLM article abstract says:
> > The VMS distributed lock manager provides the
> > synchronization mechanism needed to ensure
> > transparent and reliable data sharing between
> > nodes in a VAXcluster system. The lock
> > manager provides services for mutual exclusion
> > and event notification, and achieves high
> > performance by minimizing the number of messages
> > sent between nodes. The lock manager also
> > handles deadlock situations with a minimum
> > of messages exchanged. Since processors systems
> > can join or leave a cluster at any time, a
> > connection manager was developed to handle
> > reconfigurations in a dynamic, efficient manner.
>
> It should be noted that the Pathworks implementation works nicely for serving to PCs as it is a single process based implementation.
>
> Samba/CIFS in a multi-process implementation as is PostgreSQL and so there is then the need to coordinate between processes as well as between cluster members. It would also be nice to be able to access the files from the VMS host...
>
> It should also be noted that even Samba, and I expect PostgreSQL as well, implements its own byte range locking due to limitations in the fcntl() function call to allow access to LARGE files. So even there we have to consider some interesting issues. Samba implements an opportunistic locking mechanism which allows a client to have a cached, local copy of the file until such time as a second client comes along... This is not supported by VMS obviously. And is only supported by a few hosts as of now, from what I can tell. Samba also supports the deny-mode locks of Windows.
>
> So, no matter what we do we will need to see how we can accommodate the locking in a way that eventually we can access from VMS as well as the clients.
>
> Bill.

Do the existing (or newly imagined) Open Source uses of byte-range locking involve multi-threaded process contexts in which multiple, independent user-mode threads within the same process may attempt to access a particular byte range for a particular file?

The reason I ask is that the OpenVMS System Services locking API as represented by SYS$ENQ(W),SYS$DEQ, SYS$GETLKI, and SYS$SET_RESOURCE_DOMAIN grant, convert, snapshot, and destroy OpenVMS Resource Lock and Domain state at the process level. If multi-threading is potentially in play, then there will also need to be code written as part of this hypothetical byte-range locking API that can specifically deal with such multi-threading contexts so that multiple threads can independently (and transparently) manipulate Resource Locks and Domains (when necessary).

Apologies in advance if this complicates any hypothetical implementation. :)

[Stephen Hoffman]

On 2017-01-08 21:54:58 +0000, clairg...@gmail.com said:

> On Sunday, January 8, 2017 at 9:50:37 AM UTC-5, Stephen Hoffman wrote:
>> I'd expect C to have become the predominant programming language used
>> in OpenVMS and is continuing to supplant Bliss and Macro32, but I'd
>> doubt that anybody has run the line-counting tools and the statistical
>> reporting tools in the OpenVMS source code control system — not since I
>> last did that ~twenty years ago.
>

> Trivia time. In the most recent Itanium system build the breakdown of
> 8756 modules by language is:
>
> Ada 1%
> Assembler 1%
> C 54%
> BLISS 32%
> MACRO 12%
>
> These totals do NOT include the loader, layered products, compilers,
> DECnet V, TCPIP, and DECwindows.

Thanks for posting that.

The KLOC tools measured usage based on the numbers of lines of code,
rather than the numbers of source modules.

Every source code control system eventually develops these metadata
tools; the KLOC stuff or git-extras or otherwise.

All sorts of meaningless metrics and statistics are possible here, too.
More than a few places collect and use this data.

Worse — whether lines of code or module counts or bug reports closed or
sales commissions — you tend to get what you measure.

One of my favorite tales of this morass:
http://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt

I'd personally prefer to see less of C, Bliss and Macro32 code solely
because of the problems of each of those three.

And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
efficient system-level coding.

See how we're drifting away from the DLM and byte-range DLM distributed
limerick managment discussion, too?

[David Froble]

RobertsonEricW wrote:

> Do the existing (or newly imagined) Open Source uses of byte-range locking
> involve multi-threaded process contexts in which multiple, independent
> user-mode threads within the same process may attempt to access a particular
> byte range for a particular file?
>
> The reason I ask is that the OpenVMS System Services locking API as
> represented by SYS$ENQ(W),SYS$DEQ, SYS$GETLKI, and SYS$SET_RESOURCE_DOMAIN
> grant, convert, snapshot, and destroy OpenVMS Resource Lock and Domain state
> at the process level. If multi-threading is potentially in play, then there
> will also need to be code written as part of this hypothetical byte-range
> locking API that can specifically deal with such multi-threading contexts so
> that multiple threads can independently (and transparently) manipulate
> Resource Locks and Domains (when necessary).
>
> Apologies in advance if this complicates any hypothetical implementation. :)

I don't see how this can be a problem. When anything requests a lock which is
granted, it's totally up to the user of that lock to manage the lock. The lock
remains in the DLM database until it is released, or image rundown. The latter
case would be, in my opinion, defective code, so normally, the thread that has
the lock will release the lock when finished with it. Additional threads would
obtain their own locks, if required, for any operations. Multiple threads could
not assume the rights of a lock granted to another thread.

At least that's how I understand things ....

[Stephen Hoffman]

>> ...

>> And, I thought MACRO was the VMS Assembler. Is there another?

>> ...
> ...

> The MACRO compiler compiles the VAX MACRO-32 code for Itanium.

I suspect Bill is wondering why "MACRO" — likely the VAX Macro32
compiler — and "Assembler" — likely ias in this context, the Itanium
native assembler — are called out separately in the list.

[RobertsonEricW]

If the existing uses of the byte-range locking API within the existing Open Source under consideration implicitly treats the locking requests on a process basis, then your response may be correct. Otherwise, you would be stuck with dealing with multiple threads whether you liked it or not. However, after having thought just a little bit about this, the only cases that get complicated with multi-threading involve transitions to or from Resource Lock states states of Protected Read, Protected Write, Concurrent Write, and Exclusive. All of these state transition problems could be dealt with using a per-Lock Resource Mutex associated with the Resource Lock request. Each thread would need to acquire this mutex before proceeding with the usual process level Resource Lock usage or state transition.

[Stephen Hoffman]

On 2017-01-09 14:54:51 +0000, RobertsonEricW said:

> Do the existing (or newly imagined) Open Source uses of byte-range
> locking involve multi-threaded process contexts in which multiple,
> independent user-mode threads within the same process may attempt to
> access a particular byte range for a particular file?

Supposition based on existing and documented capabilities and
behaviors: the DLM is and would remain thread-safe.

The POSIX locking APIs have issues around threading, though.

What the OpenVMS C RTL POSIX APIs might provide for thread safety? Donno.

But given the issues around the POSIX APIs and threading, I'm not hopeful.

https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html

[Simon Clubley]

On 2017-01-09, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:
>
> And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
> efficient system-level coding.
>

You are right about the first three, but you are way wrong about Ada.

Ada has various language features for low level access and Ada
compilers allow you to compile for various levels of runtime support
so you can even run Ada at bare metal level without RTL support if
you wish.

The ARG has even been working on a couple of AIs I submitted a couple
of years ago to enhance Ada's abilities in this area so you can more
easily use Ada types with device registers instead of opaque integers
and C style bitmasks.

One of those was ratified by WG9 in October (ironically, the one
which I didn't think stood a chance of being passed :-)) and the
main one is an active ARG work item.

My main complaint with the Ada ecosystem is the lack of freely
available Ada compilers for all the systems that have freely
available C compilers. It is not about the language itself,
although I have sometimes wondered if some kind of intermediate
systems level programming language (such as what DEC tried with
Pillar) might make it easier to ease people into the discipline
required when writing Ada code.

[David Froble]

Honest question here, as I've never used Bliss. Wasn't it developed with system
level code as the main target? If so, why is it un-preferred? Is this a case
of greener grass elsewhere from where one's current location?

> And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
> efficient system-level coding.

And here we have the issue of compiler implementation vs applicability. Basic
for one is not appropriate for system level code because most of it's activity
uses RTL routines, which aren't appropriate for system level code. Or at least
some of it. However, if a Basic compiler were implemented that was targetted at
kernel mode code, threads, and such, perhaps the actual language might be a
decent choice.

Nah, too easy to read ...

Actually, I've seen Basic code that was rather un-readable, for me at least.

Now, if the statement was "Basic, Fortran, and Cobol as implemented on VMS are
not viable solutions for system-level coding", I'll buy that.

> See how we're drifting away from the DLM and byte-range DLM distributed
> limerick managment discussion, too?
>

Ditch the "limerick" joke. I think we've been stuck with "lock" for way too
long to change now. Besides, an accurate description would be way too wordy.

[johnwa...@yahoo.co.uk]

Agreed.

Ada is a decent language to design in. It can even require
people to think before stuff will even compile properly.
Obviously that requirement is very unhelpful in 'IT' in
recent years, which is entirely unrelated to why so many
IT projects *still* fail to meet budgets and/or
timescales, and/or fail in service.

Given a sufficiently skilled developer team and a
sufficiently adequate compilation system (there's more to
Ada than a compiler), it's even a decent language to
develop some kinds of programs in, including the kind of
low level stuff more normally done in C. These dodgier
things can be done, it will look awkward, and to those
with a clue, it shouts out a million miles that "this
code may be high risk".

The language design, and the then-typical state of the art
of compiler development, was such that many Ada compilers
were pretty rubbish at producing error messages (hmmm,
which other 'popular' language's 'popular' compiler(s)
often provokes similar-sounding complaints...).

A couple of Ada users I knew kept their code compatible
with DEC Ada even though the deployment platform was SPARC,
and kept a VMS system on hand with DEC Ada, just because
of the quality of the DEC Ada compiler error messages.

Then someone in HQ decided that there was no money in
updating DEC Ada to something more recent, and that
GreenHills or GNAT would be more than adequate. Oh well.

[Stephen Hoffman]

On 2017-01-09 15:54:40 +0000, David Froble said:

> RobertsonEricW wrote:
>
>> Do the existing (or newly imagined) Open Source uses of byte-range
>> locking involve multi-threaded process contexts in which multiple,
>> independent user-mode threads within the same process may attempt to
>> access a particular byte range for a particular file?
>

> I don't see how this can be a problem.

You're not dealing with threads David, and BASIC has no concept of
threading. Not without using the KP-related system service calls to
explicitly implement that in your code. So not likely a problem for
your BASIC environment, if you're using system services and BASIC and
not implementing threading locally and not using C threading-related
calls...

> When anything requests a lock which is granted, it's totally up to the
> user of that lock to manage the lock.

Threads are routines executing within the context of one application,
and quite possibly fully in parallel. This processing is
differentiated from ASTs which are either active or not, interrupting
the mainline processing while active. The OpenVMS locking-related
system services should be fine with threading — should be, but I'm not
entirely certain anybody's reviewed and confirmed that — but the C
byte-range locking APIs are not necessarily thread safe, and the POSIX
locking APIs are quite possibly inherently broken around threading.
If your BASIC code calls C code with byte-range locks using the POSIX
APIs, things might get a little more "interesting", obviously. And
it's apparently currently only the byte-range locking calls made via
the C API that are supported and documented.

[Michael Moroney]

David Froble <da...@tsoft-inc.com> writes:

>Ditch the "limerick" joke. I think we've been stuck with "lock" for way too
>long to change now. Besides, an accurate description would be way too wordy.

I think a nice short description might be to compare it to a traffic light.
A traffic light tells you when to go or stop, but does nothing to enforce
it (unlike, say, a railroad crossing which will at least try to stop you
with a crossing gate). It's up to the desire not to get into a wreck, and
cops writing tickets, that causes drivers to obey the light.

Yeah, yeah, some traffic lights will take your picture and send you a
ticket, ignore those and just consider the old fashioned ones on a timer.

[David Froble]

I think it's past time to drop the term "byte range locking". That is a
particular need, not a good description of a capability.

Perhaps numeric range locking would be much more appropriate. For sure such a
capability could lend itself to more than just locking a range of bytes in a file.

I can imagine other "types" of locks, though I haven't imagined what they might
be, yet. And that's putting the cart miles ahead of the horse, and it's not
clear the horse is even heading in the same direction ....

[VAXman-]

In article <edckhc...@mid.individual.net>, Bill Gunshannon <bill.gu...@gmail.com> writes:
>On 1/6/17 11:15 PM, Stephen Hoffman wrote:
>> On 2017-01-07 02:03:59 +0000, David Froble said:
>>
>>> Stephen Hoffman wrote:
>>>> On 2017-01-06 22:20:47 +0000, Craig A. Berry said:
>>>>
>>>>> What David would like about SSIO is that the CRTL just becomes a
>>>>> consumer of a new byte range API in the XFC...
>>>>
>>>> Works for me. (But then I'd like also to see a byte-addressable
>>>> file system, too.)
>>>>
>>>> The existing byte-range locking and SSIO and C jackets is something
>>>> most of us would rather not have to deal with.
>>>>
>>>> Though I can easily forsee an application where access into the
>>>> byte-range DLM API would be handy. This if I'm using the DLM to
>>>> coordinate activity in a ginormous memory section. I'm usually
>>>> using interlocked queues for that sort of thing now, and that isn't
>>>> always the nicest approach for some of the designs.
>>>
>>> An immediate question comes to mind. What is the magnitude of what
>>> you're talking about?
>>
>> "It depends". Anything from a month or so for a DLM-interlocked memory
>> design and implementation and testing assuming a small and
>> well-specified and single-host requirement, on up to scheduling
>> infinity. Where in this range depending on what is and is not in scope
>> in the project, performance requirements and related, and how compatible
>> the design is to any existing source code and available tools. It's a
>> how-high-is-up right now.
>>
>>> One question I've asked myself is, how big of integers might be needed
>>> to define a range? Maybe quadword?
>>
>> In OpenVMS it'd be a quadword, as that's the size of the addresses given
>> current virtual addressing limit of OpenVMS. There's a a LCK data
>> structure pointing to that range block. Based on the definitions, the
>> relevant data structure appears to be named LKIDEF_BR, though that
>> structure does not appear to have been exposed in the Bliss or C
>> definitions for V8.4.
>>
>>> Yes, what I'm considering would be perfect for what you mention, and
>>> cluster aware, if that's a consideration.
>>
>> There'll be overhead shuffling the data around; it's basically
>> server-level mirroring. Performance is going to come down to the
>> speed of the host-to-host, and that's going to be a whole lot of bespoke
>> code for networking traffic quite possibly involving some of the message
>> queue software and DECdtm. RDMA would be handy here too, certainly.
>>
>> If the application requirements around performance permit it, I'd deploy
>> modern storage and 10 GbE networking and only cache the data in memory
>> when required. Might be able to brute-force the design and avoid
>> implementing mirrored memory, in other words. Flash and SSD are
>> massively faster than HDD, after all. That'd avoid a whole lot of
>> bespoke code, too. There's also the obvious and necessary discussion
>> around using one of the available commercial or open-source databases;
>> whether Oracle or otherwise.

>>
>
>Hoff,
> You skipped the one question I would have also considered
>interesting.
>

>"Any idea what language(s) were used to implement the DLM? "

Macro
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.

[VAXman-]

In article <edftht...@mid.individual.net>, Bill Gunshannon <bill.gu...@gmail.com> writes:
>On 1/8/17 4:54 PM, clairg...@gmail.com wrote:

>Ada? Doing what? :-)

>
>And, I thought MACRO was the VMS Assembler. Is there another?
>

>bill
>
On Alpha and Itanium, Macro is compiled. Both Alpha and Itanium have native assemblers.

[Stephen Hoffman]

On 2017-01-09 21:13:18 +0000, David Froble said:

> Honest question here, as I've never used Bliss. Wasn't it developed
> with system level code as the main target? If so, why is it
> un-preferred? Is this a case of greener grass elsewhere from where
> one's current location?

The OpenVMS development decision was based on what most of the industry
was using for system code, and that was C and C++. Not Bliss.

Looking forward, and why I'm less than fond of it: Bliss has problems
akin to those of C around pointers and type punning and memory
management and the ease of crashing around unexpected null pointers and
such. This is part of why I'm less than fond of all three; of system
code written in Bliss, C or Macro32.

Looking backward: DEC priced Bliss completely out of the market decades
ago, even among OpenVMS sites. Which had the inevitable result; of
the pool of Bliss programmers and Bliss compilers and tools and open
source and third-party projects using Bliss being... a small one.
Which means you get to create and maintain all your own tooling, and
staff training is more involved. Which adds costs.

Looking at project costs: Adding languages adds incremental costs to
your development efforts, so you either need to reap substantial
benefits from any added language, or you need to slowly migrate your
old code and tooling over — and which isn't a very effective use of
time, and tends to work best when you can deprecate the old hunks — or
you make due and try to make the tools you're using work as well as you
can and hope you don't miss out if (when?) somebody gains an edge from
the efficiency of the tools they're using.

Considerations by and of the staff: the flip side of the relatively
small pool of Bliss developers means that the developers you have and
that you have trained aren't themselves going to be as interesting to
the rest of the industry, and that tends to limit the job choices that
these Bliss developer have available now and in the future. Bliss
just isn't a skill that's particularly transportable nor overly popular
among developers nor among prospective new employers. The pool of
folks that want to learn Bliss isn't large, either.

>> And no, neither BASIC, Fortran, COBOL nor Ada are viable solutions for
>> efficient system-level coding.
>
> And here we have the issue of compiler implementation vs applicability.
> Basic for one is not appropriate for system level code because most of
> it's activity uses RTL routines, which aren't appropriate for system
> level code. Or at least some of it. However, if a Basic compiler were
> implemented that was targetted at kernel mode code, threads, and such,
> perhaps the actual language might be a decent choice.

Sure, but then you've invented your own Bliss. Could you do this?
Sure. But you're left to deal with language syntax design messes, and
differences in how traditional BASIC handles memory allocations and how
system code tends to want to have that work. You'll likely want
multithreading support in the language, too. Among other details.

As for Rust, there are various write-ups that might help understand
some of the motivations.
http://www.oreilly.com/programming/free/files/why-rust.pdf Rust is
also really easy to call into C code, too.

Go is one of the newer languages that's intended for tasks similar to
what you'd use BASIC for. Probably not the best choice for writing
system-level code, though.

But then I don't see VSI adding a fourth language for OpenVMS system
programming to the existing Macro32, C and Bliss usage, and migrating
substantial quantities of the existing code away from one of those
three languages — or translating from Ada or any other existing OpenVMS
code over to Rust for that matter — is vanishingly unlikely.

[Stephen Hoffman]

On 2017-01-09 23:11:06 +0000, David Froble said:

> I think it's past time to drop the term "byte range locking". That is
> a particular need, not a good description of a capability.
>
> Perhaps numeric range locking would be much more appropriate. For sure
> such a capability could lend itself to more than just locking a range
> of bytes in a file.
>
> I can imagine other "types" of locks, though I haven't imagined what
> they might be, yet. And that's putting the cart miles ahead of the
> horse, and it's not clear the horse is even heading in the same
> direction ....

The associated C locking calls — which is presently the only documented
part of this whole OpenVMS implementation — are called the byte-range
locking calls.

That's the nomenclature used in the applicable standards documentation
and the OpenVMS C documentation.

That's what everybody else on other platforms calls those routines and
that capability, too.

[David Froble]

Well, we all know I don't get out much, but at least:

While there is a one to one correspondence between a process and an executing
image (I think), there are still can be multiple processes running under VMS,
and they can all choose to cooperate. So, why cannot multiple threads choose to
cooperate?

You're correct, I don't use threads, and could be unaware of subtle things, but
that admitted, a cooperating process, or image, or thread, either cooperates, or
it doesn't, right?

I'll also admit that at this time, due to the little I've read about the
"special case" +(my terminology) BRL in the CRTL, I'm choosing to ignore it and
just look at the DLM, and how to add additional "types" of locks to what it
already does. Not sure how it will work out, if at all, but, to me it seems the
"right" way to approach the issue.

[David Froble]

And just look at how many words you needed to use. :-)

Best to just stick with "lock", most of us understand what it really does.

:-)

Better too than getting into discussions of running red lights and busting
through crossing gates ....

[David Froble]

>>>> reporting tools in the OpenVMS source code control system — not since I

>>>> certainly, but the languages and development tools are poor and —
>>>> compared with what's available now — dated. The scale of the OpenVMS

>>>> source code alone is something that most folks here don't have to deal
>>>> with. It wouldn't surprise me to learn that sometime over the next
>>>> decade or so, VSI will look at migrating some of their internal tooling
>>>> to git, mercurial or fossil, for instance. Both for their own internal
>>>> use, and for community and shared development, They'll also likely
>>>> consider implementing easier migrations from CMS to one or more of
>>>> those tools, too. But I digress.
>>>>
>>>> The languages and tools available to and used by many of the OpenVMS
>>>> customers aren't comparatively very good or very efficient to use or
>>>> very capable by contemporary standards, either. Certainly not in
>>>> comparison to the languages and frameworks I'm working with elsewhere.
>>>> I'd wager those sites haven't looked at this topic in a decade too,
>>>> not that there are very good alternatives presently available on
>>>> OpenVMS.
>>>>
>>>> VSI is addressing parts of this conundrum with updates to the existing

>>>> languages and tools and the migration to LLVM, but — given how busy the

>>>> next five years will undoubtedly be in Bolton, and how tight their

>>>> budget will be — they're likely not in a position to start working with

>>>> more modern tools and API designs themselves, nor to start to introduce
>>>> those to their customers, and they'll also want and need to preserve
>>>> the installed base. Which means rolling forward the languages and tools
>>>> they have is the best of bad choices, and they're probably not soon
>>>> going to be introducing Rust or Go or such into OpenVMS, or into the
>>>> OpenVMS community. Then there's whether the community is even
>>>> interested, and whether enough younger developers can become interested
>>>> in OpenVMS as a viable and serious platform for serious applications;
>>>> as something other than a candidate for OS tourism or retrocomputing or
>>>> schadenfreude.
>>>>
>>>> I'd hope we'll see Python, Lua and a few tools added into the base
>>>> install, kits for git and mercurial and such, but then similar hope
>>>> holds for fully integrating IP networking and many other necessary
>>>> improvements into the platform; the work necessary to make development
>>>> and deployment and maintenance on and for OpenVMS less problematic and
>>>> less uncompetitive. (n.b. I'm aware of the ports and implementations
>>>> of these and other tools on OpenVMS and have looked at or used most of
>>>> these, too.)
>>>>
>>>> In summary.... Debating whether two-revisions-of-the-standard-back C,

>>>> or Macro32, or Bliss — or any other languages of similar vintage — was

>>>> an appropriate choice for the DLM seemed somewhat uninteresting.
>>>> Now y'all better know why i expunged that part of the question.
>>>>
>>>> https://www.fossil-scm.org
>>>> https://www.rust-lang.org
>>>>
>>>>
>>>> --
>>>> Pure Personal Opinion | HoffmanLabs LLC
>>> Trivia time. In the most recent Itanium system build the breakdown of 8756 modules by language is:
>>>
>>> Ada 1%
>>> Assembler 1%
>>> C 54%
>>> BLISS 32%
>>> MACRO 12%
>>>
>>> These totals do NOT include the loader, layered products, compilers, DECnet V, TCPIP, and DECwindows.
>>>
>> Ada? Doing what? :-)
>>
>> And, I thought MACRO was the VMS Assembler. Is there another?
>>
>> bill
>>
> On Alpha and Itanium, Macro is compiled. Both Alpha and Itanium have native assemblers.
>

Ayep! And, if you got any smarts at all, you'll run, fast, screaming, from
either one.

I sort of liked Macro-10 ....

[David Froble]

Anyone remember Kirbey Altman?

"The final holocast, cause by a bug check" (or something like that)

And

"At least it won't come from Basic"

Fun times ....

[Stephen Hoffman]

On 2017-01-10 00:48:50 +0000, David Froble said:

> While there is a one to one correspondence between a process and an
> executing image (I think),

A multithreaded application is treated as separate processes sharing
the same address space, the same variables, heap and the rest of the
context.

> there are still can be multiple processes running under VMS, and they
> can all choose to cooperate.

Ayup, at least as far as shared memory that's been mapped among the
processes, and files and other such resources.

> So, why cannot multiple threads choose to cooperate?

Threads inherently cooperate, as they're in the same address space,
running the same executable code, using the same variables and the same
heap. In parallel.

Multiple separate processes — the traditional design — only explicitly
cooperates among the processes involved, and only shares what they've
explicitly mapped into memory across the processes or are accessing
file storage and such.

From BASIC, you'd have to use the KP threads calls to implement
multithreading; akin to the behavior within C applications when the
POSIX threading-related calls are used.

> You're correct, I don't use threads, and could be unaware of subtle
> things, but that admitted, a cooperating process, or image, or thread,
> either cooperates, or it doesn't, right?

Yes, but the POSIX C calls — the API design itself — are somewhat of a
dog's breakfast around multi-threading.

> I'll also admit that at this time, due to the little I've read about
> the "special case" +(my terminology) BRL in the CRTL, I'm choosing to
> ignore it and just look at the DLM, and how to add additional "types"
> of locks to what it already does. Not sure how it will work out, if at
> all, but, to me it seems the "right" way to approach the issue.

I'd expect the system services would be thread-safe, but — until that's
documented and supported — we're probably left using the thread-unsafe
POSIX C byte-range locking calls.

[VAXman-]

In article <o51c70$9s7$1...@dont-email.me>, David Froble <da...@tsoft-inc.com> writes:

>Ayep! And, if you got any smarts at all, you'll run, fast, screaming, from
>either one.

I rather enjoyed the Alpha's assembly. I done Itanium too but there's more
"thought" involved to get it right.

[John Reagan]

On Monday, January 9, 2017 at 7:16:23 PM UTC-5, Stephen Hoffman wrote:
> Bliss
> just isn't a skill that's particularly transportable nor overly popular
> among developers nor among prospective new employers. The pool of
> folks that want to learn Bliss isn't large, either.

As VSI's BLISS "artist in residence" I'm devastated. :)

Seriously, at this year's LLVM fall developers conference, somebody saw the "VMS Software" logo on my fleece (Sue gets us good swag) and come up to me to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"

[Scott Dorsey]

John Reagan <xyzz...@gmail.com> wrote:
>On Monday, January 9, 2017 at 7:16:23 PM UTC-5, Stephen Hoffman wrote:

>> Bliss=20
>> just isn't a skill that's particularly transportable nor overly popular=
>=20
>> among developers nor among prospective new employers. The pool of=20

>> folks that want to learn Bliss isn't large, either.
>
>As VSI's BLISS "artist in residence" I'm devastated. :)

I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
at DECUS...

>Seriously, at this year's LLVM fall developers conference, somebody saw the=
> "VMS Software" logo on my fleece (Sue gets us good swag) and come up to me=
> to talk about OpenVMS. As I discussed the plans for using LLVM, his eyes =

>got big and he said "You mean you'll have a BLISS compiler for x86? Cool!"

Now, offer a free BLISS port for Linux and you might get some users. Probably
not enough users to make up for the cost of making a free BLISS port to Linux,
though.
--scott

--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[Bill Gunshannon]

How about someone writing a BLISS to C converter? That's how the Open
Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)

bill

[Scott Dorsey]

Bill Gunshannon <bill.gu...@gmail.com> wrote:
>
>How about someone writing a BLISS to C converter? That's how the Open
>Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)

I can't speak for the others, but in the case of g77, the f2c front end
nonsense delayed getting a decent working fortran compiler by nearly two
decades.

[John Reagan]

On Tuesday, January 10, 2017 at 10:54:20 AM UTC-5, Bill Gunshannon wrote:
> On 1/10/17 10:34 AM, Scott Dorsey wrote:

You mean "readable", "maintainable", and "debuggable" C from BLISS? You'll be standing on top of all the corpses that have tried before you. Easy programs are easy, complex BLISS modules which rely on the BLISS macro facility approach "impossible". Even the Holy Hand Grenade of Antioch won't help.

And a tip-o-the-hat to Rust, it has a very powerful macro language that actually might come close to handling BLISS' feature set.

[John Reagan]

On Tuesday, January 10, 2017 at 10:34:56 AM UTC-5, Scott Dorsey wrote:

>
> Now, offer a free BLISS port for Linux and you might get some users. Probably
> not enough users to make up for the cost of making a free BLISS port to Linux,
> though.

Another reminder that VSI doesn't OWN the BLISS frontend code so I'm unable to create such a compiler even if I wanted to (and I want to) without permission from HPE.

[Bill Gunshannon]

No, doesn't have to be "readable", "maintainable", or "debuggable". I
can't think of anyone who ever worked with the C output from any of
the products I mentioned who wasn't a compiler developer trying to
debug the translator. As it is with GnuCOBOL the C is merely an
intermediate language for the machine.

> You'll be standing on top of
> all the corpses that have tried before you. Easy programs are easy, complex BLISS modules which
> rely on the BLISS macro facility approach "impossible". Even the Holy Hand Grenade of Antioch won't help.

I can remember when people said converting COBOL to C was impossible,
too. And yet, there have been two completely functional examples that
it wasn't.

>
> And a tip-o-the-hat to Rust, it has a very powerful macro language that actually might come close
> to handling BLISS' feature set.
>

I really need to look at this Rust thing. But I doubt I will
be as impressed as some of the people here seem to be.

bill

[Scott Dorsey]

In article <589d6607-bd0a-44f0...@googlegroups.com>,

Is the frontend code even useful any longer, in this modern day of automatic
parser generators and LLVM and the like? I think if I wanted to build a
compiler to make x86 ELF files, I'd just take the gcc code generator and llvm
the rest of it.

[Scott Dorsey]

Bill Gunshannon <bill.gu...@gmail.com> wrote:
>
>No, doesn't have to be "readable", "maintainable", or "debuggable". I
>can't think of anyone who ever worked with the C output from any of
>the products I mentioned who wasn't a compiler developer trying to
>debug the translator. As it is with GnuCOBOL the C is merely an
>intermediate language for the machine.

With g77, I did it all the time. Constantly. People would bring me in
code that didn't give the right answers and after the first step of making
sure it wasn't a user error, I wound up regularly spending time trying to
figure out why the translator was doing what it was.

It was horrible, horrible, horrible. So many different stages for things
to go awry. I tried to get users onto Intel or Portland Group compilers
or even the NagWare compiler as much as I could.

When gfortran came along, the first few releases were pretty full of bugs,
but it was comparatively easy to find and fix them, unlike g77. After the
first year or so of fooling with it, it was working better than g77 ever did.

The whole f2c nonsense was a temporary thing while the real fortran compiler
was developed, except that it turned out there wasn't any money or enough
people to develop the real fortran compiler, leaving us all with g77 until
2007 or so.

>I can remember when people said converting COBOL to C was impossible,
>too. And yet, there have been two completely functional examples that
>it wasn't.

I can say nothing about the COBOL translator, having not used it. The thing
about COBOL is that the difficult parts and the most important parts of typical
COBOL programs have nothing to do with the actual "procedure division" code
itself. How hard that is to deal with I don't know.

[John Reagan]

On Tuesday, January 10, 2017 at 11:57:53 AM UTC-5, Scott Dorsey wrote:
> In article <>,

Why not LLVM's code generator? I see it as an either/or situation and LLVM's license is less restrictive.

As for whether the BLISS frontend contains any value and could be open sourced, that would have to be HPE's decision. Given that there is no support money today for BLISS compilers and we have BLISS compilers on the freeware CDs, it would be an easy choice for me. The BLISS frontend uses GEM shell routines for command line processing, error message generation, listing file management, etc. that would pull in some of GEM, etc. That would also need a Linux version (since GEM ran on Tru64), such things sorta exist down in our CMS libraries if you dig deep enough. We even had a Tru64 version of a portable MESSAGE compiler.

[Simon Clubley]

On 2017-01-09, Stephen Hoffman <seao...@hoffmanlabs.invalid> wrote:
>

> Go is one of the newer languages that's intended for tasks similar to
> what you'd use BASIC for. Probably not the best choice for writing
> system-level code, though.
>

Ah, yes, the language which enforces the K&R brace style. :-)

(I'm a Whitesmiths person myself BTW).

[Simon Clubley]

On 2017-01-09, David Froble <da...@tsoft-inc.com> wrote:
>
> I sort of liked Macro-10 ....

I didn't realise you were a TOPS person. Which TOPS did you use ?

[Simon Clubley]

On 2017-01-09, David Froble <da...@tsoft-inc.com> wrote:
>

> Well, we all know I don't get out much, but at least:
>

$ set response/mode=good_natured

You know, if you keep saying that, I'm going to suggest you get out
for some nice day walks in England's green and pleasant lands (and
there's some very pleasant lands around here) while I educate you
on all the technologies you are missing out on. :-)

[Simon Clubley]

On 2017-01-10, Scott Dorsey <klu...@panix.com> wrote:
>
> Is the frontend code even useful any longer, in this modern day of automatic
> parser generators and LLVM and the like? I think if I wanted to build a
> compiler to make x86 ELF files, I'd just take the gcc code generator and llvm
> the rest of it.
>

Should that be the other way around, because LLVM essentially _is_
the code generator part of the compiler and you supply the frontend
code to LLVM.

The last attempt to use the gcc frontends BTW was called DragonEgg
and it didn't end well as the project fell into disuse and doesn't
work with current LLVM versions.

[Stephen Hoffman]

On 2017-01-10 15:20:44 +0000, John Reagan said:

> On Monday, January 9, 2017 at 7:16:23 PM UTC-5, Stephen Hoffman wrote:
>> Bliss just isn't a skill that's particularly transportable nor overly
>> popular among developers nor among prospective new employers. The pool
>> of folks that want to learn Bliss isn't large, either.
>
> As VSI's BLISS "artist in residence" I'm devastated. :)

Oh but John, you know you must stay true to your own inner Wirth. 😉

> Seriously, at this year's LLVM fall developers conference, somebody saw
> the "VMS Software" logo on my fleece (Sue gets us good swag) and come
> up to me to talk about OpenVMS. As I discussed the plans for using
> LLVM, his eyes got big and he said "You mean you'll have a BLISS
> compiler for x86? Cool!"

For those interested in following their own Bliss:

https://harlanhaskins.com/2017/01/08/building-a-compiler-with-swift-in-llvm-part-1-introduction-and-the-lexer.html

[David Froble]

You just need to be a better negotiator. HP has come to you for some things in
the past, I believe. Should they do so again, add a few more demands onto your
already substantial renumeration.

:-)

I really cannot see what HP would lose by giving up the exclusive rights to
something they probably don't even want.

[Johnny Billquist]

Ask that question again, but substitute "PDP-11 Software" for "BLISS",
and see how far you get... :-/

Johnny

--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol

[VAXman-]

In article <b4024224-c32d-4173...@googlegroups.com>, John Reagan <xyzz...@gmail.com> writes:
>On Tuesday, January 10, 2017 at 10:54:20 AM UTC-5, Bill Gunshannon wrote:
>> On 1/10/17 10:34 AM, Scott Dorsey wrote:
>> > John Reagan <> wrote:
>> >> On Monday, January 9, 2017 at 7:16:23 PM UTC-5, Stephen Hoffman wrote:

>> >>> Bliss=3D20
>> >>> just isn't a skill that's particularly transportable nor overly popul=
>ar=3D
>> >> =3D20
>> >>> among developers nor among prospective new employers. The pool of=3D=

>20
>> >>> folks that want to learn Bliss isn't large, either.
>> >>
>> >> As VSI's BLISS "artist in residence" I'm devastated. :)
>> >
>> > I still have the "BLISS IS IGNORANCE" T-shirt that was being handed out
>> > at DECUS...
>> >

>> >> Seriously, at this year's LLVM fall developers conference, somebody sa=
>w the=3D
>> >> "VMS Software" logo on my fleece (Sue gets us good swag) and come up t=
>o me=3D
>> >> to talk about OpenVMS. As I discussed the plans for using LLVM, his e=
>yes =3D
>> >> got big and he said "You mean you'll have a BLISS compiler for x86? Co=
>ol!"
>> >
>> > Now, offer a free BLISS port for Linux and you might get some users. P=
>robably
>> > not enough users to make up for the cost of making a free BLISS port to=
> Linux,
>> > though.
>> > --scott
>> >
>>=20

>> How about someone writing a BLISS to C converter? That's how the Open
>> Source Compilers for Ada, Fortran, Pascal and COBOL all started. :-)

>>=20
>> bill
>
>You mean "readable", "maintainable", and "debuggable" C from BLISS? You'll=
> be standing on top of all the corpses that have tried before you. Easy pr=
>ograms are easy, complex BLISS modules which rely on the BLISS macro facili=
>ty approach "impossible". Even the Holy Hand Grenade of Antioch won't help=
>..

I'd rather read the Bliss in VMS than the C -- with or without Brother Maynard
by my side.

[David Froble]

Simon Clubley wrote:
> On 2017-01-09, David Froble <da...@tsoft-inc.com> wrote:
>> I sort of liked Macro-10 ....
>
> I didn't realise you were a TOPS person. Which TOPS did you use ?
>
> Simon.
>

It was a rather long time ago. University of Pittsburgh had PDP-10 systems.
TOPS-10. I was just a student, and when I got into some classes that used
assembler, I was a happy camper. Different than most of the people in the
class. Ended up doing much more than the class work, as I liked the language.
Guess I'm just a strange guy.

Never had a chance to use it after I graduated. Personal computers weren't back
then.

[VAXman-]

In article <o53aj5$fj$5...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>On 2017-01-09, David Froble <da...@tsoft-inc.com> wrote:
>>
>> Well, we all know I don't get out much, but at least:
>>
>
>$ set response/mode=good_natured
>
>You know, if you keep saying that, I'm going to suggest you get out
>for some nice day walks in England's green and pleasant lands (and
>there's some very pleasant lands around here) while I educate you
>on all the technologies you are missing out on. :-)

And did those feet in ancient times,
Walk upon England's mountains green?

[David Froble]

Simon Clubley wrote:
> On 2017-01-09, David Froble <da...@tsoft-inc.com> wrote:
>> Well, we all know I don't get out much, but at least:
>>
>
> $ set response/mode=good_natured
>
> You know, if you keep saying that, I'm going to suggest you get out
> for some nice day walks in England's green and pleasant lands (and
> there's some very pleasant lands around here) while I educate you
> on all the technologies you are missing out on. :-)
>
> Simon.
>

Well, I am waiting for some nice days, but to do some flying, not walking.

:-)

[Simon Clubley]

On 2017-01-10, David Froble <da...@tsoft-inc.com> wrote:
>
> It was a rather long time ago. University of Pittsburgh had PDP-10 systems.
> TOPS-10. I was just a student, and when I got into some classes that used
> assembler, I was a happy camper. Different than most of the people in the
> class. Ended up doing much more than the class work, as I liked the language.
> Guess I'm just a strange guy.
>
> Never had a chance to use it after I graduated. Personal computers weren't back
> then.

I don't know if you are aware, but there are various free PDP-10
emulators available so there's nothing to stop you from running
it again if you wish.

[Simon Clubley]

On 2017-01-10, VAXman- @SendSpamHere.ORG <VAX...@SendSpamHere.ORG> wrote:
> In article <o53aj5$fj$5...@dont-email.me>, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> writes:
>>
>>You know, if you keep saying that, I'm going to suggest you get out
>>for some nice day walks in England's green and pleasant lands (and
>>there's some very pleasant lands around here) while I educate you
>>on all the technologies you are missing out on. :-)
>
> And did those feet in ancient times,
> Walk upon England's mountains green?
>

Congratulations. You pass the test. :-)