OpenSSL 1.1.0 released

[Stephen Hoffman]

For those folks here tracking security-related updates...

https://mta.openssl.org/pipermail/openssl-announce/2016-August/000074.html

SSLv2 and the ancient and long-insecure "export" crypto is gone.
scrypt and many other updates.

Whether this ends up as HPE SSL2 or some other construction, and as for
what VSI has planned here?




--
Pure Personal Opinion | HoffmanLabs LLC

[David Froble]

Stephen Hoffman wrote:
>
> For those folks here tracking security-related updates...
>
> https://mta.openssl.org/pipermail/openssl-announce/2016-August/000074.html
>
> SSLv2 and the ancient and long-insecure "export" crypto is gone. scrypt
> and many other updates.
>
> Whether this ends up as HPE SSL2 or some other construction, and as for
> what VSI has planned here?

That's almost funny. But, you know, from what we've seen lately, we just might
see an SSL2.

Pray to the IT gods that VSI gets their product out sooner rather than later.

[John Reagan]

Well since 1.1.n changed the layout of several data structures (promoting some 32-bit integers to 64-bit integers for example), you do end up with different APIs. Just like you can install both openssl.1.0.0 and openssl0.9.8 on your Linux box, I expect to see openssl.1.1.0 alongside of them.

[Stephen Hoffman]

On 2016-08-25 18:47:24 +0000, John Reagan said:

> Well since 1.1.n changed the layout of several data structures
> (promoting some 32-bit integers to 64-bit integers for example), you do
> end up with different APIs. Just like you can install both
> openssl.1.0.0 and openssl0.9.8 on your Linux box, I expect to see
> openssl.1.1.0 alongside of them.

And on OpenVMS, I expect to see a similar migration as happened from
HPE SSL to HPE SSL1, including the same prerequisite product changes
and similar application changes to build procedures and quite possibly
to the application source code. On the plus side, I know where most
of those build changes are. Why the build-related DCL will probably
have to change, I don't know.

BTW: this is why folks have gone toward their own transports. Callers
to the http://labs.hoffmanlabs.com/node/1853 socket library didn't need
to change — the private build procedures underneath that library
certainly did, though— and I wouldn't expect a particular change to
folks with applications using libcurl or such, either.

Whether HPE decides to provide an SSL1 kit with both, an SSL11 kit, or
an SSL2 kit? Donno....

[Steven Schweda]

> Whether HPE decides to provide an SSL1 kit with both, an
> SSL11 kit, or an SSL2 kit? Donno....

I might vote for SSL11 as a name, or else someone should
design some actual scheme to get from an OpenSSL version to
an HPE kit/logical name.

I'll take this opportunity to suggest (again) that someone
at HPE who adapts the OpenSSL code to suit HPE might consider
communicating with the actual OpenSSL folks who have some
interest in VMS. It'd be nice never to see problems like
those caused by SSLROOT being hard-coded in the source, while
HP[E] defines (only) SSL$ROOT in its start-up procedures. If
the HPE changes are actually valuable, then getting them into
the main OpenSSL code might be good for everyone.

[Craig A. Berry]

What a mess. I need to update the Net::SSLeay Perl extension so that it
links against the latest version of OpenSSL available on the system, but
heck if I can see a clean way to do that, especially one that will not
require lots of niggly ongoing maintenance (dare I say "glue code"?).

What I used to do is prefer an OpenSSL built from source, then the HP
SSL kit, and I did that by looking in order for the following files, and
taking the first one I found as an indicator of what's latest and available:

sslroot:[000000]openssl.cnf
ssl$root:[000000]openssl.cnf

HP's SSL1 broke that; if I don't want people linking to an older kit (or
not finding one at all), I need to insert

ssl1$root:[000000]openssl.cnf

in the middle of that list so that the SSL1 libraries are preferred over
the SSL libraries. If that were the end of it, I could live with that.
But I have no idea what the VSI libraries and configuration files are
named or where they live, and odds are even that what fixes this scheme
for HP's SSL1 breaks it for VSI's SSL-whatever.

Yes, I'm aware of the OPENSSL logical name, which is supposed to be a
clue to what the system manager selected. But if the system manager
selected anything, s/he as likely as not has it pointing at the older
kits to keep older software from breaking, and that's no guide to what I
should link new stuff against *now*.

Note that one of the difficulties here is that the actual library names
differ, so selecting which SSL kit to link against is not just a matter
of pointing to the right directory:

$ dir/col=1 sys$share:*ssl*32.exe;

Directory SYS$COMMON:[SYSLIB]

SSL$LIBCRYPTO_SHR32.EXE;1
SSL$LIBSSL_SHR32.EXE;1
SSL1$LIBCRYPTO_SHR32.EXE;1
SSL1$LIBSSL_SHR32.EXE;1

Total of 4 files.

This has got to be a big headache for HP and VSI as well, so it would be
nice to have this sorted out and some guidance made available -- at
least naming conventions for what things will look like going forward
and a documented way to locate and link against "current." Based on the
number of HPE-supplied things on an up-to-date v8.4 system that don't
know about SSL1, they need help here as much as I do.

[David Froble]

I'm just really hoping that VSI has a bit of a clue, and the SSL1 fiasco is a
stopgap for those unfortunate enough to have to use it. And unfortunately, I'm
in that group.

:-(

I'm hoping that VSI can fix this one way or another. My preference, though I'm
doubting it, is they take the Microsoft approach, declare OpenSSL a total
disaster, and produce their product. Then again, that also might be a problem.

The names should be standard (whatever that is) and for whichever product you
choose to use, something, perhaps logicals, would be set for your chosen
product. I have no idea how this will affect your PERL issues.

A big mess, and getting bigger ....

[Richard Levitte]

Den torsdag 25 augusti 2016 kl. 22:05:00 UTC+2 skrev Steven Schweda:
> If the HPE changes are actually valuable, then getting them into
> the main OpenSSL code might be good for everyone.

I am in contact with VSI and they have already provided fixes, most of which went into 1.1.0.

Cheers,
Richard

[Richard Levitte]

Den torsdag 25 augusti 2016 kl. 22:52:31 UTC+2 skrev Craig A. Berry:
> On 8/25/16 2:24 PM, Stephen Hoffman wrote:
> > On 2016-08-25 18:47:24 +0000, John Reagan said:
> >
> >> Well since 1.1.n changed the layout of several data structures
> >> (promoting some 32-bit integers to 64-bit integers for example), you
> >> do end up with different APIs. Just like you can install both
> >> openssl.1.0.0 and openssl0.9.8 on your Linux box, I expect to see
> >> openssl.1.1.0 alongside of them.
> >
> > And on OpenVMS, I expect to see a similar migration as happened from HPE
> > SSL to HPE SSL1, including the same prerequisite product changes and
> > similar application changes to build procedures and quite possibly to
> > the application source code. On the plus side, I know where most of
> > those build changes are. Why the build-related DCL will probably have
> > to change, I don't know.
> >
> > BTW: this is why folks have gone toward their own transports. Callers
> > to the http://labs.hoffmanlabs.com/node/1853 socket library didn't need
> > to change — the private build procedures underneath that library
> > certainly did, though— and I wouldn't expect a particular change to
> > folks with applications using libcurl or such, either.
> >
> > Whether HPE decides to provide an SSL1 kit with both, an SSL11 kit, or
> > an SSL2 kit? Donno....
>
> What a mess.

HPE's mistake was to make the logical names version specific. Formally speaking, SSL and are to be regarded SSL1 are entirely different product rather than different versions of the same.

(note, btw, that we've taken a similar approach for OpenSSL 1.1, but considering the fairly massive changes, it might as well be regarded that way. The changes we made are designed to hold consistently for a while, though)

> I need to update the Net::SSLeay Perl extension so that it
> links against the latest version of OpenSSL available on the system, but
> heck if I can see a clean way to do that, especially one that will not
> require lots of niggly ongoing maintenance (dare I say "glue code"?).

For OpenSSL 1.1.0 and on, you will see this for library names:

OSSL$LIBCRYPTO0101_SHR.EXE
OSSL$LIBSSL0101_SHR.EXE

(or OSSL$LIBCRYPTO0101_SHR32.EXE for a pointer_size 32 build, or OSSL$LIBCRYPTO0101_SHR64.EXE for a pointer size 64 build)

This was a choice made so it would be possible to have several versions of OpenSSL installed at the same time (i.e. solving what HPE tried to solve with different facility names a bit differently, by putting in the version number where it matters). I hope that will help with this particular issue.

Cheers,
Richard

[Richard Levitte]

Den torsdag 25 augusti 2016 kl. 23:37:23 UTC+2 skrev David Froble:
> My preference, though I'm doubting it, is they take the Microsoft approach,
> declare OpenSSL a total disaster, and produce their product. Then again,
> that also might be a problem.

That is, of course, a choice. Mind you, adding *another* library entirely will not make the market any less messy, and comes with security risks (re-inventing the wheel and all that...).

Speaking of TLS libraries, there is a number of free ones to choose from:

https://en.wikipedia.org/wiki/Comparison_of_TLS_implementations

Of course, it's a porting job either way. Take your pick!

Cheers,
Richard

[Stephen Hoffman]

On 2016-08-26 16:31:08 +0000, Richard Levitte said:

> Den torsdag 25 augusti 2016 kl. 22:52:31 UTC+2 skrev Craig A. Berry:
>> On 8/25/16 2:24 PM, Stephen Hoffman wrote:
>> What a mess.
>
> HPE's mistake was to make the logical names version specific. Formally
> speaking, SSL and are to be regarded SSL1 are entirely different
> product rather than different versions of the same.

In simpler terms, HPE's mistake was in deciding to not create a
multi-version product kit; in creating kits that are neither wholly
disparate nor entirely compatible.

> For OpenSSL 1.1.0 and on, you will see this for library names:
>
> OSSL$LIBCRYPTO0101_SHR.EXE
> OSSL$LIBSSL0101_SHR.EXE
>
> (or OSSL$LIBCRYPTO0101_SHR32.EXE for a pointer_size 32 build, or
> OSSL$LIBCRYPTO0101_SHR64.EXE for a pointer size 64 build)
>
> This was a choice made so it would be possible to have several versions
> of OpenSSL installed at the same time (i.e. solving what HPE tried to
> solve with different facility names a bit differently, by putting in
> the version number where it matters). I hope that will help with this
> particular issue.

Using a registered prefix is fine, but embedding version numbers in
shareable images — unless there are logical names to reference those —
wouldn't have been my choice here as that tends to mean build changes.

Rdb solved this multi-version problem a couple of decades ago, and did
so fairly cleanly. If anybody here is trying to do multi-version, go
look at the Rdb kits.

At some point, I'll have to post a tech note on how to create a
multi-version software package akin to what Rdb uses.

The question here is which way VSI will be going with this, and how
much of the existing SSL and SSL1 (and SSL11 or SSL2?) design will be
maintained, how much will be replaced, and what path VSI takes to allow
applications to choose to better isolate themselves from TLS-layer
considerations and changes. Also around what VSI might choose to do
around adding better support and documentation for multi-version kits
and related, too.

I'd consider ignoring the HPE software here and compiling and linking
in my own OpenSSL bits. To stay away from this mess, at least until
VSI sorts out what's going on here. This means the application code
is independent of what HPE or VSI does with the TLS support. Though
there is a downside in doing this. This approach requires the
application provider to ship OpenSSL security patches, and preferably
expeditiously. (HPE tends to kit, test and ship patches roughly a
month or so after OpenSSL releases the patches, unfortunately. That
schedule is relatively fast in terms of the OpenVMS view of patches,
and glacially slow in terms of what can happen to servers during that
month.)

[Richard Levitte]

Den fredag 26 augusti 2016 kl. 20:32:49 UTC+2 skrev Stephen Hoffman:
> On 2016-08-26 16:31:08 +0000, Richard Levitte said:
>
> > Den torsdag 25 augusti 2016 kl. 22:52:31 UTC+2 skrev Craig A. Berry:
> >> On 8/25/16 2:24 PM, Stephen Hoffman wrote:
> >> What a mess.
> >
> > HPE's mistake was to make the logical names version specific. Formally
> > speaking, SSL and are to be regarded SSL1 are entirely different
> > product rather than different versions of the same.
>
> In simpler terms, HPE's mistake was in deciding to not create a
> multi-version product kit; in creating kits that are neither wholly
> disparate nor entirely compatible.

Yeah. That amounts to about the same... (perhaps not philosophically, but...)

> > For OpenSSL 1.1.0 and on, you will see this for library names:
> >
> > OSSL$LIBCRYPTO0101_SHR.EXE
> > OSSL$LIBSSL0101_SHR.EXE
> >
> > (or OSSL$LIBCRYPTO0101_SHR32.EXE for a pointer_size 32 build, or
> > OSSL$LIBCRYPTO0101_SHR64.EXE for a pointer size 64 build)
> >
> > This was a choice made so it would be possible to have several versions
> > of OpenSSL installed at the same time (i.e. solving what HPE tried to
> > solve with different facility names a bit differently, by putting in
> > the version number where it matters). I hope that will help with this
> > particular issue.
>
> Using a registered prefix is fine, but embedding version numbers in
> shareable images — unless there are logical names to reference those —
> wouldn't have been my choice here as that tends to mean build changes.

Note that I'm only embedding the major/minor version number. So "edit" and "letter" updates will keep the same library names. This is on par with our versioning policy, that a changed minor version number implies incompatible API/ABI changes, while "edit" changes only adds to the API/ABI, and "letter" changes don't even do that.
(in OpenSSL terminology, 1.1 is a new major version, which deviates from the norm...)

> Rdb solved this multi-version problem a couple of decades ago, and did
> so fairly cleanly. If anybody here is trying to do multi-version, go
> look at the Rdb kits.

I did look at the manual after someone pointed this out, and if I understand correctly, they do this by fiddling with logicals, correct? I don't have access to a Rdb kit, so can't look for myself. But either way, fiddling with logicals is exactly what's done in the OpenSSL installation on VMS.

I just did the installation on the cluster I play on, and here are the logicals created (on alpha, you can imagine how much/little they differ on ia64):

$ sh log ossl*

(LNM$PROCESS_TABLE)

(LNM$JOB_82A7CD80)

(LNM$GROUP_000200)

(LNM$SYSTEM_TABLE)

"OSSL$DATAROOT" = "DSA30:[OPENSSL.COMMON.]"
"OSSL$ENGINES" = "OSSL$ENGINES0101"
"OSSL$ENGINES0101" = "OSSL$INSTROOT:[ENGINES0101.ALPHA]"
"OSSL$EXE" = "OSSL$INSTROOT:[EXE.ALPHA]"
= "OSSL$INSTROOT:[EXE]"
"OSSL$INCLUDE" = "DSA30:[OPENSSL.INCLUDE.]"
"OSSL$INSTROOT" = "DSA30:[OPENSSL.]"
"OSSL$LIB" = "OSSL$INSTROOT:[LIB.ALPHA]"
"OSSL$LIBCRYPTO" = "OSSL$LIB:OSSL$LIBCRYPTO.OLB"
"OSSL$LIBCRYPTO0101_SHR" = "OSSL$SHARE:OSSL$LIBCRYPTO0101_SHR.EXE"
"OSSL$LIBCRYPTO_SHR" = "OSSL$LIBCRYPTO0101_SHR"
"OSSL$LIBSSL" = "OSSL$LIB:OSSL$LIBSSL.OLB"
"OSSL$LIBSSL0101_SHR" = "OSSL$SHARE:OSSL$LIBSSL0101_SHR.EXE"
"OSSL$LIBSSL_SHR" = "OSSL$LIBSSL0101_SHR"
"OSSL$SHARE" = "OSSL$INSTROOT:[LIB.ALPHA]"

(LNM$SYSCLUSTER_TABLE)

(DECW$LOGICAL_NAMES)

> I'd consider ignoring the HPE software here and compiling and linking
> in my own OpenSSL bits. To stay away from this mess, at least until
> VSI sorts out what's going on here. This means the application code
> is independent of what HPE or VSI does with the TLS support. Though
> there is a downside in doing this. This approach requires the
> application provider to ship OpenSSL security patches, and preferably
> expeditiously. (HPE tends to kit, test and ship patches roughly a
> month or so after OpenSSL releases the patches, unfortunately. That
> schedule is relatively fast in terms of the OpenVMS view of patches,
> and glacially slow in terms of what can happen to servers during that
> month.)

Ok, so here's a question, would you use a third party package? I'm building one for OpenSSL, and plan on releasing a field test next week, mostly to make sure I don't make any gaffe (first time I build a .pcsi), and to collect opinions. If you want to have a look, check here:

https://github.com/levitte/vms-hacks/tree/master/package-openssl

Cheers,
Richard

[Craig A. Berry]

On 8/26/16 11:31 AM, Richard Levitte wrote:
> Den torsdag 25 augusti 2016 kl. 22:52:31 UTC+2 skrev Craig A. Berry:

>> I need to update the Net::SSLeay Perl extension so that it
>> links against the latest version of OpenSSL available on the system, but
>> heck if I can see a clean way to do that, especially one that will not
>> require lots of niggly ongoing maintenance (dare I say "glue code"?).
>
> For OpenSSL 1.1.0 and on, you will see this for library names:
>
> OSSL$LIBCRYPTO0101_SHR.EXE
> OSSL$LIBSSL0101_SHR.EXE
>

> (or OSSL$LIBCRYPTO0101_SHR32.EXE for a pointer_size 32 build, or
> OSSL$LIBCRYPTO0101_SHR64.EXE for a pointer size 64 build)

I just built and installed OpenSSL 1.1.0 on OpenVMS I64 v8.4 with a
default configuration. The configuration process spat out:

. . .
THIRTY_TWO_BIT mode

Configured for vms-ia64.

and there were no pointer size arguments added to the compiler flags.
Yet the libraries I got do not have "32" anywhere in the name:

$ dir/col=1 ossl$lib

Directory OSSL$INSTROOT:[lib.IA64]

ossl$libcrypto.OLB;1
ossl$libcrypto0101_shr.EXE;1
ossl$libssl.OLB;1
ossl$libssl0101_shr.EXE;1

Total of 4 files.

> This was a choice made so it would be possible to have several
> versions of OpenSSL installed at the same time (i.e. solving what HPE
> tried to solve with different facility names a bit differently, by
> putting in the version number where it matters). I hope that will help
> with this particular issue.

It may well have been the least worst way to support multiple
simultaneous versions, but it adds another dimension or two to the
matrix of possibilities, which for me pushes the complexity beyond
anything I'm likely to find the time or energy to maintain. To support
this I would have to hard-code the library names with version numbers,
and then add more options every time there is a release.

Or I could work a bit harder to translate the OSSL$* logical pointing to
the shareable image like so:

$ type findlib.pl
use strict;
use File::Spec::Functions qw(splitpath);

my ($vol, $dir, $lib) =
splitpath(VMS::Filespec::rmsexpand('OSSL$LIBSSL_SHR'));

print " prefix: $vol$dir\n";
print "library: $lib\n";

$ perl findlib.pl
prefix: OSSL$INSTROOT:[lib.IA64]
library: ossl$libssl0101_shr.EXE

but that's going to work only for OpenSSL 1.1.0, not earlier versions of
OpenSSL, not HPE's SSL or SSL1, and gosh only knows how the VSI
libraries are named or located.

And I can also now no longer tell whether I've got a 32-bit or 64-bit
build by looking at the filenames of the shareable images.

If anyone would like to make me look silly and show how easy this is
please do. The version detection code I'm talking about is in the
ssleay_get_build_opts and find_openssl_prefix routines in:

<http://cpansearch.perl.org/src/MIKEM/Net-SSLeay-1.78/inc/Module/Install/PRIVATE/Net/SSLeay.pm>

Search for "VMS."

[David Froble]

Richard Levitte wrote:

> "OSSL$DATAROOT" = "DSA30:[OPENSSL.COMMON.]"
> "OSSL$ENGINES" = "OSSL$ENGINES0101"
> "OSSL$ENGINES0101" = "OSSL$INSTROOT:[ENGINES0101.ALPHA]"
> "OSSL$EXE" = "OSSL$INSTROOT:[EXE.ALPHA]"
> = "OSSL$INSTROOT:[EXE]"
> "OSSL$INCLUDE" = "DSA30:[OPENSSL.INCLUDE.]"
> "OSSL$INSTROOT" = "DSA30:[OPENSSL.]"
> "OSSL$LIB" = "OSSL$INSTROOT:[LIB.ALPHA]"
> "OSSL$LIBCRYPTO" = "OSSL$LIB:OSSL$LIBCRYPTO.OLB"
> "OSSL$LIBCRYPTO0101_SHR" = "OSSL$SHARE:OSSL$LIBCRYPTO0101_SHR.EXE"
> "OSSL$LIBCRYPTO_SHR" = "OSSL$LIBCRYPTO0101_SHR"
> "OSSL$LIBSSL" = "OSSL$LIB:OSSL$LIBSSL.OLB"
> "OSSL$LIBSSL0101_SHR" = "OSSL$SHARE:OSSL$LIBSSL0101_SHR.EXE"
> "OSSL$LIBSSL_SHR" = "OSSL$LIBSSL0101_SHR"
> "OSSL$SHARE" = "OSSL$INSTROOT:[LIB.ALPHA]"

So, it appears that you're both allowing this to reside on other than the system
disk, and, you're not lumping it in with the VMS stuff. I like that.

[Richard Levitte]

Yup. I looked around a bit and ended up choosing the same convention gnv$zlib seems to follow; with no indication of pointer size, the libs get no pointer size number (implied 32-bit size pointers, but someone pointed out that there were subtle differences when you compiler with /POINTER_SIZE=32 and when you compiler with no pointer size qualifier, so I chose the paranoid path). With an indication of 32-bit pointer size, you get 32 added to the library name, and with 64-bit pointer size, you get *tadaaaa* 64.

So how do you get the pointer size indications? If you use config.com, you do one of these:

$ @config ! no pointer size indication
$ @config -32 ! pointer size 32
$ @config -64 ! pointer size 64

If you use Configure directly, you choose the appropriate config target:

$ perl Configure vms-alpha ! no pointer size
$ perl Configure vms-alpha-p32 ! pointer size 32
$ perl Configure vms-alpha-p64 ! pointer size 64

> > This was a choice made so it would be possible to have several
> > versions of OpenSSL installed at the same time (i.e. solving what HPE
> > tried to solve with different facility names a bit differently, by
> > putting in the version number where it matters). I hope that will help
> > with this particular issue.
>
> It may well have been the least worst way to support multiple
> simultaneous versions, but it adds another dimension or two to the
> matrix of possibilities, which for me pushes the complexity beyond
> anything I'm likely to find the time or energy to maintain. To support
> this I would have to hard-code the library names with version numbers,
> and then add more options every time there is a release.

Errrrr, no? As far as I can see from the NET/SSLeay.pm you pointed at, you're linking with the static library. No version number there!

> Or I could work a bit harder to translate the OSSL$* logical pointing to
> the shareable image like so:
>
> $ type findlib.pl
> use strict;
> use File::Spec::Functions qw(splitpath);
>
> my ($vol, $dir, $lib) =
> splitpath(VMS::Filespec::rmsexpand('OSSL$LIBSSL_SHR'));
>
> print " prefix: $vol$dir\n";
> print "library: $lib\n";
>
> $ perl findlib.pl
> prefix: OSSL$INSTROOT:[lib.IA64]
> library: ossl$libssl0101_shr.EXE

Why do you have to do that? Why not simply use them as they are?

> but that's going to work only for OpenSSL 1.1.0, not earlier versions of
> OpenSSL, not HPE's SSL or SSL1, and gosh only knows how the VSI
> libraries are named or located.
>
> And I can also now no longer tell whether I've got a 32-bit or 64-bit
> build by looking at the filenames of the shareable images.

I hope the explanation above helped a bit.

> If anyone would like to make me look silly and show how easy this is
> please do. The version detection code I'm talking about is in the
> ssleay_get_build_opts and find_openssl_prefix routines in:
>
> <http://cpansearch.perl.org/src/MIKEM/Net-SSLeay-1.78/inc/Module/Install/PRIVATE/Net/SSLeay.pm>
>
> Search for "VMS."

Sure:

--- SSLeay.pm.orig 2016-08-26 23:41:16.028465296 +0200
+++ SSLeay.pm 2016-08-26 23:52:07.267103753 +0200
@@ -110,7 +110,15 @@
}
}
elsif ($^O eq 'VMS') {
- if (-r 'sslroot:[000000]openssl.cnf') { # openssl.org source install
+ if (-r 'ossl$lib:ossl$libcrypto.olb') { # openssl.org v1.1 source install
+ @{ $opts->{lib_paths} } = 'OSSL$LIB';
+ @{ $opts->{lib_links} } = qw( ossl$libssl.olb ossl$libcrypto.olb );
+ }
+ elif (-r 'ossl$lib:ossl$libcrypto32.olb') { # openssl.org v1.1 source install
+ @{ $opts->{lib_paths} } = 'OSSL$LIB';
+ @{ $opts->{lib_links} } = qw( ossl$libssl32.olb ossl$libcrypto32.olb );
+ }
+ elif (-r 'sslroot:[000000]openssl.cnf') { # openssl.org source install
@{ $opts->{lib_paths} } = 'SSLLIB';
@{ $opts->{lib_links} } = qw( ssl_libssl32.olb ssl_libcrypto32.olb );
}


Cheers,
Richard

[Richard Levitte]

The installation from source has always allowed that.

Cheers,
Richard

[Stephen Hoffman]

On 2016-08-26 20:11:11 +0000, Richard Levitte said:

>
> Note that I'm only embedding the major/minor version number. So "edit"
> and "letter" updates will keep the same library names. This is on par
> with our versioning policy, that a changed minor version number implies
> incompatible API/ABI changes, while "edit" changes only adds to the
> API/ABI, and "letter" changes don't even do that.
> (in OpenSSL terminology, 1.1 is a new major version, which deviates
> from the norm...)

Ayup; aware of that. That approach is something I prefer to avoid,
but it works.

> I did look at the manual after someone pointed this out, and if I
> understand correctly, they do this by fiddling with logicals, correct?
> I don't have access to a Rdb kit, so can't look for myself. But either
> way, fiddling with logicals is exactly what's done in the OpenSSL
> installation on VMS.

Rdb uses the version in the path, not in the filenames. That version
string is "masked" through the consistent use of logical names for the
path. The provided tool selects which version of the product to use,
and then all of the reference are identical and — depending on whether
the logical names are process or group or system or otherwise, the
default for the respective environment is established.

There's no mechanism nor framework nor dicumentation for this in
OpenVMS, so folks roll their own. PCSI could be better here, too. But
having everybody roll their own just means we repeat the same mistakes.

> Ok, so here's a question, would you use a third party package? I'm
> building one for OpenSSL, and plan on releasing a field test next week,
> mostly to make sure I don't make any gaffe (first time I build a
> .pcsi), and to collect opinions.

Probably not. Not here. I've used third-party distros, though
downloading and building from the canonical distribution site tends to
be preferred by various of the folks I work with. (The general lack
of support for code signing and verification is problematic on OpenVMS.
Even with posted SHA-2 values to verify the integrity of the
download, there are potential considerations around trust and security
of what actually gets downloaded. I can't recall encountering a
cryptographically signed app nor signed source code for OpenVMS,
either. Yes, HPE and VSI have what they call "secure delivery", and
that's a discussion for another day. But I digress.) Once you're up
for building your own bits, then downloading and verifying and building
from the canonical sources isn't a substantial incremental effort.

FWIW... PCSI kits can be multi-architecture, if you've not already
discovered that. The PCSI kits can be signed, too, though I don't know
if VSI has a process in place to offer that just yet.

[Richard Levitte]

Den fredag 26 augusti 2016 kl. 23:59:20 UTC+2 skrev Stephen Hoffman:
> Rdb uses the version in the path, not in the filenames. That version
> string is "masked" through the consistent use of logical names for the
> path. The provided tool selects which version of the product to use,
> and then all of the reference are identical and — depending on whether
> the logical names are process or group or system or otherwise, the
> default for the respective environment is established.

I did consider that for a bit, but chose not to. I can't remember the whole train of thought (there were a number of "what if"s that I wasn't sure I could control, so...). One concrete thing I wanted to be able to support was installation of shared libraries in SYS$SHARE and the like, and in that case, having the version in the library name was the only safe way to go and avoid the dreaded DLL hell. Also, this conforms with how it's done on all other platforms we support.

> There's no mechanism nor framework nor dicumentation for this in
> OpenVMS, so folks roll their own. PCSI could be better here, too. But
> having everybody roll their own just means we repeat the same mistakes.

In the vein of the "All OS suck" song, I imagine all solutions have their own sucking points. What remains is to look at all solutions, take one that's closest to your train of thought, and hope nobody notices the mistakes ;-) (or at least that the mistake in one's choice isn't a total fuck-up)

> > Ok, so here's a question, would you use a third party package? I'm
> > building one for OpenSSL, and plan on releasing a field test next week,
> > mostly to make sure I don't make any gaffe (first time I build a
> > .pcsi), and to collect opinions.
>
> Probably not. Not here. I've used third-party distros, though
> downloading and building from the canonical distribution site tends to
> be preferred by various of the folks I work with.

Okie.

> (The general lack
> of support for code signing and verification is problematic on OpenVMS.
> Even with posted SHA-2 values to verify the integrity of the
> download, there are potential considerations around trust and security
> of what actually gets downloaded. I can't recall encountering a
> cryptographically signed app nor signed source code for OpenVMS,
> either.

Errrr, the OpenSSL release tarballs come with a detached gpg signature. Of course, that's difficult to verify on VMS...

> FWIW... PCSI kits can be multi-architecture, if you've not already
> discovered that.

I haven't even thought of looking. Not sure how to go about it, though. Would it be something like this?

IF (<software DEC AXPVMS OPENVMS> OR
(<software HP AXPVMS OPENVMS> OR
<software VSI AXPVMS OPENVMS>)) ;
FILE [SYSLIB]OSSL$LIBCRYPTO0101_SHR.EXE
source [.AXPVMS]OSSL$LIBCRYPTO0101_SHR.EXE ;
! yada yada yada
END IF ;
IF (<software DEC I64VMS OPENVMS> OR
(<software HP I64VMS OPENVMS> OR
<software VSI I64VMS OPENVMS>)) ;
FILE [SYSLIB]OSSL$LIBCRYPTO0101_SHR.EXE
source [.I64VMS]OSSL$LIBCRYPTO0101_SHR.EXE ;
! yada yada yada
END IF ;

> The PCSI kits can be signed, too, though I don't know
> if VSI has a process in place to offer that just yet.

"Just yet" is a bit amusing... after all, Polycenter has been around for, how many years?

Cheers,
Richard

[Craig A. Berry]

It's currently doing that for older versions of OpenSSL. I believe at
the time I did that, static libraries were the only thing you got with a
default build. Maybe I misremember, but in any case, I was hoping to
make it work for shareable images so that it wouldn't have to be
re-built for every release of OpenSSL.

>> Or I could work a bit harder to translate the OSSL$* logical pointing to
>> the shareable image like so:
>>
>> $ type findlib.pl
>> use strict;
>> use File::Spec::Functions qw(splitpath);
>>
>> my ($vol, $dir, $lib) =
>> splitpath(VMS::Filespec::rmsexpand('OSSL$LIBSSL_SHR'));
>>
>> print " prefix: $vol$dir\n";
>> print "library: $lib\n";
>>
>> $ perl findlib.pl
>> prefix: OSSL$INSTROOT:[lib.IA64]
>> library: ossl$libssl0101_shr.EXE
>
> Why do you have to do that? Why not simply use them as they are?

Because there are version numbers in the names. That forces the
downstream user to get older versions until patches have been produced,
sent upstream, accepted, and released for each and every OpenSSL release
that changes the version number.

That's going to fail to locate openssl.exe, it continues to use the
static libraries, which I'm hoping to get away from, it doesn't handle
all variations of HPE SSL, HPE SSL1, and VSI SSL-whatever, and it
prefers a library whose pointer size is unknown. I was hoping to do
better than that. Adding support for one more variant isn't that hard.
Adding support for all possible variants in the "right" order, whatever
that is, preferably without having to continually add new,
version-specific code every time anybody releases something, is the
problem that needs solving.

[Arne Vajhøj]

My understanding purely based on hearsay is that the OpenSSL
code base is very large - much larger than required to provide
what 99% of users need.

A much smaller code base would be easier to review for
vulnerabilities.

Arne

[Richard Levitte]

Ok. You won't have to rebuild for every OpenSSL release, only for the major ones (such as 1.1.0). Do you remember when 1.0.0 was released? That was the previous major release. It was released 6 years ago.

So, for any 1.1.x version, you will get the exact same shared library name (for the same configuration re pointer size).

> >> Or I could work a bit harder to translate the OSSL$* logical pointing to
> >> the shareable image like so:
> >>
> >> $ type findlib.pl
> >> use strict;
> >> use File::Spec::Functions qw(splitpath);
> >>
> >> my ($vol, $dir, $lib) =
> >> splitpath(VMS::Filespec::rmsexpand('OSSL$LIBSSL_SHR'));
> >>
> >> print " prefix: $vol$dir\n";
> >> print "library: $lib\n";
> >>
> >> $ perl findlib.pl
> >> prefix: OSSL$INSTROOT:[lib.IA64]
> >> library: ossl$libssl0101_shr.EXE
> >
> > Why do you have to do that? Why not simply use them as they are?
>
> Because there are version numbers in the names. That forces the
> downstream user to get older versions until patches have been produced,
> sent upstream, accepted, and released for each and every OpenSSL release
> that changes the version number.

Not sure I understand. If you're linking something with libcrypto and libssl, isn't it enough to check what logical names exist and simply use them? After that, the user should be able to update OpenSSL to his/her heart's content without fear (at least from 1.1 and on), *and* you won't have to care exactly which major OpenSSL version is there (not entirely true, but that's not important from a building point of view).

> >> If anyone would like to make me look silly and show how easy this is
> >> please do. The version detection code I'm talking about is in the
> >> ssleay_get_build_opts and find_openssl_prefix routines in:
> >>
> >>
> <http://cpansearch.perl.org/src/MIKEM/Net-SSLeay-1.78/inc/Module/Install/PRIVATE/Net/SSLeay.pm>
> >>
> >> Search for "VMS."
> >
> > Sure:

...

> That's going to fail to locate openssl.exe,

I would suggest using the DCL symbol 'OPENSSL' to figure that out. That or don't guess at all, just use the verb 'openssl'.

> it continues to use the
> static libraries, which I'm hoping to get away from, it doesn't handle
> all variations of HPE SSL, HPE SSL1, and VSI SSL-whatever, and it
> prefers a library whose pointer size is unknown. I was hoping to do
> better than that. Adding support for one more variant isn't that hard.
> Adding support for all possible variants in the "right" order, whatever
> that is, preferably without having to continually add new,
> version-specific code every time anybody releases something, is the
> problem that needs solving.

Sure. Mind you, I didn't dig very deep into NET::SSLeay, so I can't pretend to understand how it's built. You are triggering my curiousity, though, and given time, I can try to help out.

Cheers,
Richard

[Richard Levitte]

I was curious to see how correct you were, so I had a look at tarball sizes... and it turns out that OpenSSL is comparable to most of the major ones (it's smaller than GNUtls and NSS, it's larger than Botan and LibreSSL). The embedded projects are noticably smaller, but that should not be a surprise.

Of course, your basis for comparison will vary...

Cheers,
Richard

[Craig A. Berry]

The current scheme requires a list of one or more library paths and a
list of one or more library names, which will then be put together in
all possible combinations and used as the list of libraries to be
supplied to the linker. If I use OSSL$LIB as the library path and, e.g.,
OSSL$LIBCRYPTO as the library name, those two won't place nice together
when combined because the latter already includes a path.

That's why I would have to take the library logical and split it into a
path and a filename. But that won't work for anything other than OpenSSL
1.1.0. Bypassing the current scheme for VMS is probably possible, but
someone usually inadvertently breaks such things with unrelated changes.

[Richard Levitte]

Hmmm... I just read through ExtUtils::Liblist::Kid::_vms_ext, which is the function that does this, and it seems that it will start with '' as directory string, and among the library variants it will try is the unmodified library name. The we come to the crucial line:

$fullname = VMS::Filespec::rmsexpand( $name );

So if the library "file" is given as 'OSSL$LIBCRYPTO_SHR', you should get the full file spec of the shared library it points at.

So, uhmmm, I disagree with your conclusion.

> That's why I would have to take the library logical and split it into a
> path and a filename. But that won't work for anything other than OpenSSL
> 1.1.0. Bypassing the current scheme for VMS is probably possible, but
> someone usually inadvertently breaks such things with unrelated changes.

Mm, speaking of broken things, I just tried to cpan install Module::Install::Admin, and, well, that was barf all over... But that's darn off topic for this thread.

Cheers,
Richard

[Craig A. Berry]

On 8/27/16 9:29 AM, Richard Levitte wrote:
> Den lördag 27 augusti 2016 kl. 15:37:44 UTC+2 skrev Craig A. Berry:
>> If I use OSSL$LIB as the library path and, e.g.,
>> OSSL$LIBCRYPTO as the library name, those two won't place nice together
>> when combined because the latter already includes a path.
>

> Hmmm... I just read through ExtUtils::Liblist::Kid::_vms_ext, which

> isthe function that does this, and it seems that it will start with '' as

> directory string, and among the library variants it will try is the
> unmodified library name. The we come to the crucial line:
>
> $fullname = VMS::Filespec::rmsexpand( $name );
>
> So if the library "file" is given as 'OSSL$LIBCRYPTO_SHR', you should get the full file spec of the shared library it points at.
>
> So, uhmmm, I disagree with your conclusion.

If that's the only place the two components are combined, it might work,
though there's enough twisty code involved I wouldn't bet on it without
seeing it work first.

Then there's the (admittedly lame) heuristic of removing "32" from the
filename if Perl was configured for 64-bit pointers. That's not
something that works right now, but if it did, your scheme would break
it because the logical name occludes any pointer size embedded in the
filename.

>> That's why I would have to take the library logical and split it into a
>> path and a filename. But that won't work for anything other than OpenSSL
>> 1.1.0. Bypassing the current scheme for VMS is probably possible, but
>> someone usually inadvertently breaks such things with unrelated changes.
>

> Mm, speaking of broken things, I just tried to cpan install
> Module::Install::Admin, and, well, that was barf all over... But that's
> darn off topic for this thread.

Yes, Module::Install is badly broken on VMS. In the README.vms that
comes with Net::SSLeay, there is an embedded patch that will get it
working well enough to build Net::SSLeay. Or it did at the time I
created it -- it's probably out of date now.

[Stephen Hoffman]

On 2016-08-26 22:41:30 +0000, Richard Levitte said:

> Den fredag 26 augusti 2016 kl. 23:59:20 UTC+2 skrev Stephen Hoffman:
>
>> FWIW... PCSI kits can be multi-architecture, if you've not already>
>> discovered that.
>
> I haven't even thought of looking. Not sure how to go about it,
> though. Would it be something like this?

NEWUSER is a multi-architecture kit. Have a look at that.

http://labs.hoffmanlabs.com/node/1260

That PCSI kit is missing the VSI two-step around software providers and
prerequisite products, though. (Another PCSI enhancement...)

[dgordo...@gmail.com]

On Monday, August 29, 2016 at 10:11:05 AM UTC-4, Stephen Hoffman wrote:

> That PCSI kit is missing the VSI two-step around software providers and
> prerequisite products, though. (Another PCSI enhancement...)
>

That particular ship sailed years ago when the planned support to alias producers wasn't added when CPQ became HP (then-no-E).

New kits wishing to support VSI and HPE versions just need to list both producers. If the best-effort command procedure supplied by VSI fails, one must resort to unpacking and hand-editing older kits.

[Richard Levitte]

Den måndag 29 augusti 2016 kl. 16:11:05 UTC+2 skrev Stephen Hoffman:
> On 2016-08-26 22:41:30 +0000, Richard Levitte said:
>
> > Den fredag 26 augusti 2016 kl. 23:59:20 UTC+2 skrev Stephen Hoffman:
> >
> >> FWIW... PCSI kits can be multi-architecture, if you've not already>
> >> discovered that.
> >
> > I haven't even thought of looking. Not sure how to go about it,
> > though. Would it be something like this?
>
> NEWUSER is a multi-architecture kit. Have a look at that.
>
> http://labs.hoffmanlabs.com/node/1260

Thanks!

Cheers,
Richard

[Stephen Hoffman]

On 2016-08-29 17:48:36 +0000, dgordo...@gmail.com said:

> On Monday, August 29, 2016 at 10:11:05 AM UTC-4, Stephen Hoffman wrote:
>
>> That PCSI kit is missing the VSI two-step around software providers and
>> prerequisite products, though. (Another PCSI enhancement...)
>
> That particular ship sailed years ago when the planned support to alias
> producers wasn't added when CPQ became HP (then-no-E).

The ship allowing one kits to install files with the same filename in
different target directories sailed a whole long time ago, too.
Something about the innards of the PCSI kit format being a single
directory, IIRC.

I like PCSI. But — like other parts of OpenVMS – it needs some help.

Getting an actual relational database into the product (e.g. SQLite)
and migrating away from the morass that inevitably arises with
RMS-based app-specific data files would be nice, too. Various of the
limitations and issues with the designs and implementation of PCSI and
of various other OpenVMS components track back to the limits of RMS.

> New kits wishing to support VSI and HPE versions just need to list both
> producers. If the best-effort command procedure supplied by VSI fails,
> one must resort to unpacking and hand-editing older kits.

Ayup. BTW, is that recommendation — more than a few of us have been
using it, of course — officially documented somewhere?

[Johnny Billquist]

On 2016-08-30 17:26, Stephen Hoffman wrote:
> On 2016-08-29 17:48:36 +0000, dgordo...@gmail.com said:
>
>> On Monday, August 29, 2016 at 10:11:05 AM UTC-4, Stephen Hoffman wrote:
>>
>>> That PCSI kit is missing the VSI two-step around software providers
>>> and prerequisite products, though. (Another PCSI enhancement...)
>>
>> That particular ship sailed years ago when the planned support to
>> alias producers wasn't added when CPQ became HP (then-no-E).
>
> The ship allowing one kits to install files with the same filename in
> different target directories sailed a whole long time ago, too.
> Something about the innards of the PCSI kit format being a single
> directory, IIRC.
>
> I like PCSI. But — like other parts of OpenVMS – it needs some help.
>
> Getting an actual relational database into the product (e.g. SQLite) and
> migrating away from the morass that inevitably arises with RMS-based
> app-specific data files would be nice, too. Various of the limitations
> and issues with the designs and implementation of PCSI and of various
> other OpenVMS components track back to the limits of RMS.

I find that claim a little too much. RMS is just a bunch of code to
manipulate file content. It does not impose any limits on how something
like PCSI can work. Obviously other tools using RMS can solve it, so RMS
is not that kind of a limiter.

Pretty lame to blame RMS for PCSI shortcomings, if you ask me.

Johnny

[Richard Levitte]

Den tisdag 30 augusti 2016 kl. 17:26:23 UTC+2 skrev Stephen Hoffman:
> On 2016-08-29 17:48:36 +0000, dgordo...@gmail.com said:
>
> > On Monday, August 29, 2016 at 10:11:05 AM UTC-4, Stephen Hoffman wrote:
> >
> >> That PCSI kit is missing the VSI two-step around software providers and
> >> prerequisite products, though. (Another PCSI enhancement...)
> >
> > That particular ship sailed years ago when the planned support to alias
> > producers wasn't added when CPQ became HP (then-no-E).
>
> The ship allowing one kits to install files with the same filename in
> different target directories sailed a whole long time ago, too.

Hmmm, and installing the same file from different sources depending on platform as well, apparently. I just tried something easy:

if ( ( (<software DEC AXPVMS OPENVMS>)
or (<software HP AXPVMS OPENVMS>) )
or (<software VSI AXPVMS OPENVMS>) ) ;
information alpha ;
file [RL-TEST]foo.txt source [.AXPVMS]foo.txt ;
else ;
information ia64 ;
file [RL-TEST]foo.txt source [.I64VMS]foo.txt ;
end if ;

And with installation, it did give me the right message according to the information command, but both the alpha and the ia64 installs got the exact same content, [.AXPVMS]foo.txt...

> Something about the innards of the PCSI kit format being a single
> directory, IIRC.

That doesn't sound right:

DSA20:[LEVITTE.TEST.PCSI]LEVITTE-VMS-TEST-T0001--1.PCSI;2
Format: Sequential
Kit Size: 16 blocks
Manifest: Kit indicates that a manifest file was not created for it

--------------------------------------------------------------------------------
CONTENTS OF KIT USING RELATIVE FILE SPECIFICATION
--------------------------------------------------------------------------------
[000000]LEVITTE-VMS-TEST-T0001--1.PCSI$TLB
[.AXPVMS]FOO.TXT
[.I64VMS]FOO.TXT
[000000]LEVITTE-VMS-TEST-T0001--1.PCSI$DESCRIPTION
--------------------------------------------------------------------------------

So, thanks for the tip on multi-platform PCSIs, but with this kind of bugger all crap going on, I pass.

> I like PCSI. But — like other parts of OpenVMS – it needs some help.

*ahem* obviously...

Cheers,
Richard

[Stephen Hoffman]

On 2016-09-02 14:27:09 +0000, Richard Levitte said:

> Den tisdag 30 augusti 2016 kl. 17:26:23 UTC+2 skrev Stephen Hoffman:
>> On 2016-08-29 17:48:36 +0000, dgordo...@gmail.com said:
>>
>>> On Monday, August 29, 2016 at 10:11:05 AM UTC-4, Stephen Hoffman wrote:
>>>
>>>> That PCSI kit is missing the VSI two-step around software providers
>>>> and> >> prerequisite products, though. (Another PCSI enhancement...)
>>>
>>> That particular ship sailed years ago when the planned support to
>>> alias> > producers wasn't added when CPQ became HP (then-no-E).
>>
>> The ship allowing one kits to install files with the same filename in>
>> different target directories sailed a whole long time ago, too.
>
> Hmmm, and installing the same file from different sources depending on
> platform as well, apparently. I just tried something easy:
>

> ...

Ugh, sorry, NEWUSER isn't multi-architecture. My bad. Multi-arch
looks like this:


-- the executables and architecture-specific pieces
if ( < software DEC AXPVMS VMS > );
File [SYSEXE]FOOCP.EXE protection public source [ALPHA_OBJ]FOOCP.EXE;
...
else if ( < software HP I64VMS VMS > );
File [SYSEXE]FOOCP.EXE protection public source [IA64_OBJ]FOOCP.EXE;
...
end if;



Code which unfortunately degrades as vendor names get added.

[Craig A. Berry]

On 9/2/16 12:02 PM, Stephen Hoffman wrote:

>
> Multi-arch looks like this:
>
>
> -- the executables and architecture-specific pieces
> if ( < software DEC AXPVMS VMS > );
> File [SYSEXE]FOOCP.EXE protection public source [ALPHA_OBJ]FOOCP.EXE;
> ...
> else if ( < software HP I64VMS VMS > );
> File [SYSEXE]FOOCP.EXE protection public source [IA64_OBJ]FOOCP.EXE;
> ...
> end if;
>
>
>
> Code which unfortunately degrades as vendor names get added.

...and architectures.

[Richard Levitte]

Den fredag 2 september 2016 kl. 19:02:45 UTC+2 skrev Stephen Hoffman:
> Ugh, sorry, NEWUSER isn't multi-architecture. My bad. Multi-arch
> looks like this:
>
>
> -- the executables and architecture-specific pieces
> if ( < software DEC AXPVMS VMS > );
> File [SYSEXE]FOOCP.EXE protection public source [ALPHA_OBJ]FOOCP.EXE;
> ...
> else if ( < software HP I64VMS VMS > );
> File [SYSEXE]FOOCP.EXE protection public source [IA64_OBJ]FOOCP.EXE;
> ...
> end if;

Except that doesn't quite seem to work... The demonstration I made indicates that [SYSEXE]FOOCP.EXE will be a copy of [ALPHA_OBJ]FOOCP.EXE in any case.

Cheers,
Richard

[Stephen Hoffman]

I've just noticed PCSI rolled its own help file format parser, and
didn't implement it quite the same as the standard help file parser: no
support for bang characters as comment character delimiters. HELP has
accepted exclamation points as comment delimiters for aeons. Not a
huge surprise that, given the description file parser and the entirety
of the PCSI tool has its share of quirks. But I digress.

Since you haven't shown your (non-working) reproducer, I'll show you my
(working) example. Here is a tested, working multi-architecture kit:
http://labs.hoffmanlabs.com/node/1934

[Richard Levitte]

Den måndag 5 september 2016 kl. 16:55:03 UTC+2 skrev Stephen Hoffman:
> Since you haven't shown your (non-working) reproducer

Actually, I did, at least the central part that didn't seem to work...

> I'll show you my (working) example. Here is a tested, working multi-architecture kit:
> http://labs.hoffmanlabs.com/node/1934

I see what differs, and probably why yours works; In my case, the only difference was that the source was in different directories, but same file name:

[.AXPVMS]FOO.TXT
[.I64VMS]FOO.TXT

In your case, the file names are different depending on the architecture. I guess that's workable, although needs a bit more work...

Cheers,
Richard

[Stephen Hoffman]

On 2016-09-05 15:12:45 +0000, Richard Levitte said:

> I see what differs, and probably why yours works; In my case, the only

> difference was that the source was in different directories...

Remember that flat directory comment?

PCSI could be so much better with an actual database underneath and not
the RMS morass, and with some fixes and enhancements, and maybe with a
look around at what PCSI is actually doing with all the disk I/O it
tosses out, and for a two-file install, but there are undoubtedly many
other projects and improvements much higher on the schedule.

[John Reagan]

Not to be insulting, but have you EXTRACTed the files from your kit to see if the [.I64VMS]FOO.TXT file is really the right one?

[Richard Levitte]

I haven't checked, but the extracted [.I64VMS] isn't the right one, the contents is from the Alpha one.

Here are the three files used for this test:

test-package.com:
----------------------------------------
$ cre/dir [.axpvms]
$ cre/dir [.i64vms]
$ cre [.axpvms]foo.txt
$ deck
Alpha FOO
$ eod
$ cre [.i64vms]foo.txt
$ deck
IA64 FOO
$ eod
$
$ PRODUCT PACKAGE TEST -
/BASE=VMS -
/PRODUCER=Levitte -
/SOURCE=[]test -
/DESTINATION=[] -
/MATERIAL=[] -
/FORMAT=SEQUENTIAL
----------------------------------------

test.pcsi$desc:
----------------------------------------
product Levitte VMS TEST T0.1 Full ;

if ( ( (<software DEC AXPVMS OPENVMS>)
or (<software HP AXPVMS OPENVMS>) )
or (<software VSI AXPVMS OPENVMS>) ) ;
information alpha ;
file [RL-TEST]foo.txt source [.AXPVMS]foo.txt ;
else ;
information ia64 ;
file [RL-TEST]foo.txt source [.I64VMS]foo.txt ;
end if ;

end product ;
----------------------------------------

test.pcsi$text:
----------------------------------------
=product Levitte VMS TEST T0.1 Full
1 'PRODUCT
=prompt TEST kit T0.1
1 'NOTICE
=prompt Effing pubdom
1 'LICENSE
=prompt Pubdom, I tell ya
1 'PRODUCER
=prompt Richard Levitte
1 alpha
=prompt Installing on Alpha
1 ia64
=prompt Installing on IA64
----------------------------------------

After running test-package.com, I can verify that the source files are correct:

$ type [...]foo.txt

USER:[LEVITTE.TEST.PCSI.AXPVMS]FOO.TXT;1

Alpha FOO

USER:[LEVITTE.TEST.PCSI.I64VMS]FOO.TXT;1

IA64 FOO
$ sh def
USER:[LEVITTE.TEST.PCSI]

Note that when running PRODUCT PACKAGE, I got this warning:

%PCSI-I-MULOBJ, the file [RL-TEST]FOO.TXT is multiply defined; conflict resolution may occur on installation

Cheers,
Richard

[Bill Pedersen]

I believe John Malmberg has discussed here and also on SourceFORGE and the DECUServe site the fact that there cannot be duplicate file names in PCSI installation kits.

The issue, according to John, is that PCSI looks only at the file name, not the directory information.

This has haunted the Open Source on OpenVMS Community for some time. And we have worked to stay away from such cases.

Bill.

Bill Pedersen
CCSS - Computer Consulting System Services, LLC
211 Ruth Drive
Gaffney, SC 29341
Telephone: +1-864-490-8863
Facsimile: +1-206-984-3068
www: www.ccsscorp.com
Skype Number: +1-864-649-0904
Skype Name: william.a.pedersen
LinkedIn Profile: www.linkedin.com/in/billpedersen

_______________________________________________
Info-vax mailing list
Info...@rbnsn.com
http://rbnsn.com/mailman/listinfo/info-vax_rbnsn.com

[Craig A. Berry]

On 9/5/16 6:05 PM, Bill Pedersen wrote:
> I believe John Malmberg has discussed here and also on SourceFORGE and the DECUServe site the fact that there cannot be duplicate file names in PCSI installation kits.
>
> The issue, according to John, is that PCSI looks only at the file name, not the directory information.

There has to be more to the story than that, at least with current
versions of PCSI. The Perl kits I produce have multiple files with the
same name in different directories and it works just fine.

Do "dir/size perl_root:[000000...]VMS.pm" for an example.

[Bill Pedersen]

I am going from what I have heard from John.

I have not specifically evaluated this issue.

Bill.

Bill Pedersen
CCSS - Computer Consulting System Services, LLC
211 Ruth Drive
Gaffney, SC 29341
Telephone: +1-864-490-8863
Facsimile: +1-206-984-3068
www: www.ccsscorp.com
Skype Number: +1-864-649-0904
Skype Name: william.a.pedersen
LinkedIn Profile: www.linkedin.com/in/billpedersen

-----Original Message-----
From: Info-vax [mailto:info-vax...@rbnsn.com] On Behalf Of Craig A.
Berry via Info-vax
Sent: Monday, September 05, 2016 9:22 PM
To: info...@rbnsn.com
Cc: Craig A. Berry
Subject: Re: [Info-vax] OpenSSL 1.1.0 released

[John E. Malmberg]

I do not recall exactly what versions of VMS and PCSI ECO kits where I
saw the behavior.

I am building PCSI kits in some cases on VAX/VMS 7.3, Alpha VMS 8.3/8.4,
and IA64/VMS 8.4.

What I saw was that in at least some cases, when there were multiple
files with the same name, PCSI picked one of the files and used it
everywhere and ignored the source directory directive.

I am not sure that it picked the first file with that name it found, or
if it did it with all duplicate names in the kit.

Since then I have set up the PCSI kits so that all filenames in the kit
are unique, ODS-2 compliant and have a post-install/pre-uninstall
procedure to make links with the expected names to the installed files.

And for ODS-5 files a post-install procedure that will rename them to
the expected case on an ODS-5 volume, as some versions of PCSI did not
seem to preserve case.

Regards,
-John
wb8...@qsl.net_work

[Craig A. Berry]

The problem may be the line:

/MATERIAL=[]

in the PRODUCT PACKAGE command. I believe [] is going to be considered
a single directory rather than a rooted (foo:[bar.]) or wildcard
([.foo...]) so it may not be descending into your [.axpvms] and
[.i64vms] subdirectories to locate material for the kit.

The docs say:

"Note that when you use either a wildcard directory or a list of path
names, if files in different directories have the same name, only the
first file found in the search path is used."

These docs are from:

http://h41379.www4.hpe.com/doc/732final/6048/6048pro_004.html#heading_18.3

(search for "/MATERIAL")

My guess is you'll have to declare a rooted logical and use that rather
than a wildcard or single path name, though maybe something like
SYS$DISK:[.] would work.

[Paul Sture]

I'm pretty sure duplicate file names caused problems for VMSINSTAL kits.

The easiest way to get those to fall over is to forget to purge the
source directory down to one version before creating the kit; a
VMSINSTAL installation will keel over if it finds more than one file of
the same name in its working directory.

Perhaps that is what is being remembered. Perhaps similar behaviour was
present in earlier versions of PRODUCT INSTALL?

--
It was untidy, so got unplugged.
It was unplugged, so got thrown away.

[Richard Levitte]

That was faulty, actually. I changed that to '/MATERIAL=[...]' while I was writing the post, but forgot to update the post ('/MATERIAL=[]' was a temporary experiment)

> The docs say:
>
> "Note that when you use either a wildcard directory or a list of path
> names, if files in different directories have the same name, only the
> first file found in the search path is used."
>
> These docs are from:
>
> http://h41379.www4.hpe.com/doc/732final/6048/6048pro_004.html#heading_18.3
>
> (search for "/MATERIAL")
>
> My guess is you'll have to declare a rooted logical and use that rather
> than a wildcard or single path name, though maybe something like
> SYS$DISK:[.] would work.

No, that didn't quite work... BUT, you did put me on the right path here, 'cause this seems to work the way I want:

test-package.com:
----------------------------------------
$ cre/dir [.axpvms]
$ cre/dir [.i64vms]
$ cre [.axpvms]foo.txt
$ deck
Alpha FOO
$ eod
$ cre [.i64vms]foo.txt
$ deck
IA64 FOO
$ eod
$

$ HERE = F$ENV("DEFAULT") - "]" + ".]" ! Added
$ DEFINE/TRANS=CONC/USER SOURCE 'HERE' ! Added

$ PRODUCT PACKAGE TEST -
/BASE=VMS -
/PRODUCER=Levitte -
/SOURCE=[]test -
/DESTINATION=[] -

/MATERIAL=SOURCE: -

/FORMAT=SEQUENTIAL
----------------------------------------

test.pcsi$desc:
----------------------------------------
product Levitte VMS TEST T0.1 Full ;
if ( ( (<software DEC AXPVMS OPENVMS>)
or (<software HP AXPVMS OPENVMS>) )
or (<software VSI AXPVMS OPENVMS>) ) ;
information alpha ;

file [RL-TEST]foo.txt source [AXPVMS]foo.txt ;
else ;
information ia64 ;
file [RL-TEST]foo.txt source [I64VMS]foo.txt ;

end if ;
end product ;
----------------------------------------

Cheers,
Richard

[Stephen Hoffman]

On 2016-09-06 11:40:38 +0000, Richard Levitte said:

> No, that didn't quite work... BUT, you did put me on the right path
> here, 'cause this seems to work the way I want:

I'm getting the impression that the following diagnostic means that the
parser and/or the scanner may well have lost its way:

%PCSI-I-MULOBJ, the file {filespec} is multiply defined; conflict

resolution may occur on installation

The PCSI filename parser has already had problems with lowercase and
with <>, so another oddity is certainly not out of the realm of
possibility.

IOW: the above diagnostic looks like it might really be an error, not
an informational.

[Richard Levitte]

It is quirky in this regard for sure...

Cheers,
Richard

[Stephen Hoffman]

On 2016-09-06 14:31:30 +0000, Richard Levitte said:

> It is quirky in this regard for sure...

"OpenVMS! Runs great! Less quirky!"

[Richard Levitte]

Den tisdag 6 september 2016 kl. 17:01:39 UTC+2 skrev Stephen Hoffman:
> On 2016-09-06 14:31:30 +0000, Richard Levitte said:
>
> > It is quirky in this regard for sure...
>
> "OpenVMS! Runs great! Less quirky!"

Oh hush, you ;-)

[Craig A. Berry]

Quirky, but documented. There is a little more info in the v8.4 on-line help under PRODUCT PACKAGE/MATERIAL than there was in in the 7.3-2 era docs I cited earlier:

=======

When the path-name contains a root directory, the utility appends
the relative file specification from the FILE statement in the
PDF to the root directory to determine where to find the file.
However, when either a specific directory or a wildcard directory
is used in the path-name, the relative file specification from
the FILE statement is not used to find the file. Instead, the
directory as specified in the /MATERIAL qualifier is used to
search for the file.

Note that when you use either a wildcard directory or a list of
path names, if files in different directories have the same name,

only the first file in the search path is packaged in the kit.
As a result, the same file is packaged each time a FILE statement
refers to the file name because the relative file specification
is not used to identify the file uniquely. Therefore, if your
product has different files with the same name in different
directories, you must use the root directory form of the path-
name to package these files correctly.

=======

To me the moral of the story is "always use a rooted directory for value of the /MATERIAL qualifier." I've done that, but I think it was more by accident rather than knowing that I needed to.

[Stephen Hoffman]

On 2016-09-06 15:38:46 +0000, Craig A. Berry said:

> Quirky, but documented. There is a little more info in the v8.4
> on-line help under PRODUCT PACKAGE/MATERIAL than there was in in the
> 7.3-2 era docs I cited earlier:

Ayup, quite true.

More folks have become disabused to the "fix it in documentation" notion.

The doc update certainly implies the workaround didn't effectively
resolve the design and/or implementation issue.

This case is fine fodder for the "mistakes" presentation being written
for boot camp.

Not that the presentation doesn't already have more than a few of my
own mistakes, and lists of those of others.

[Richard Levitte]

Den tisdag 6 september 2016 kl. 19:52:29 UTC+2 skrev Stephen Hoffman:
> On 2016-09-06 15:38:46 +0000, Craig A. Berry said:
>
> > Quirky, but documented. There is a little more info in the v8.4
> > on-line help under PRODUCT PACKAGE/MATERIAL than there was in in the
> > 7.3-2 era docs I cited earlier:
>
> Ayup, quite true.
>
> More folks have become disabused to the "fix it in documentation" notion.

Sounds like a feature (see meaning number 6 in http://www.catb.org/jargon/html/F/feature.html) ;-)

Cheers,
Richard

[Stephen Hoffman]

I'd infer either short-sighted or schedule-constrained development
work, here. That being polite phrasing for what amounts to a short
cut or software expediency.

These sorts of cases — the lack of schedule and the willingness to
refractor or to rework or to iterate, even the ability to recognize
that there are design or implementation problems — are unfortunately
common in enterprise software and hardware. Some can be subtle. Ever
notice the I/O ports on some server don't have any semblance of
organization on the I/O bulkhead, or the connectors themselves are
oriented such that you can't easily get at the latches or the labels,
for instance? The very different consoles on Integrity Itanium
(iLO, BMC, EFI)? Software itself either gets rethought and refactored
for new needs and new users and uses, or the accretion simply hardens
into brittleness and cryptic arcana. But I digress.

IMO, this is a bug in PCSI. This is a very nicely documented bug, but
it's still a bug. Barring extenuating circumstances, good software
should not allow users to trigger surprise corruptions.

[IanD]

Even humble sqlite would be a good fit as a db underneath many of the OpenVMS information repositories although getting something like PostgreSQL fully working on OpenVMS might lead to bigger and better things

I think from memory, sqlite is multi-reader but limited to 1 writer at a time, might be a limiting factor in some instances ?

How I have longed for files like sysuaf.dat to be in a DB format beyond a simplistic RMS indexed file format

How about log files stored as xml/JSON or at least every utility having the option to input/export in those formats?

When RDB added xml as a table output / import format , it's usability for interfacing from simple script access when up immensely.

OpenVMS could do itself a lot of favors by having these near universal data exchange methods on tap across it's utilities and interfaces, not only for external world interfacing but also to assist the system manager and automated bots - it's such a PITA to have to continually work with differing formats that you have to parse / filter over and over again to get at what you want.

How many times does one start a new job at a place and then get asked to create some new type of report and then start the whole script re-invention coding again because the log file / data file has some unique layout? Boring and repetitive, it's time for OpenVMS to support people not make people work with native, somewhat stone-age formats and utilities

We have the The Common Log Format for web servers, is there a universal log file format anywhere? Does OpenVMS actually stipulate a guideline for log file formats or is it a free-for-all? It sure seems like a free-for-all, even looking at the OpenVMS internal log file formats

Perhaps we can develop something? or move towards an xml/JSON structured one? We some some proper base classes for OpenVMS from which to build upon

I know these formats can get a bit long winded when reading an extensive log file but that can be worked around by pointing a browser at the log file and letting it deal with the tags and prettying it up and/or once it's in these universal formats you can run some tool over them anyhow and down select the major data types you are after anyhow.
If your a real masochist you still have the option of dealing with the raw data if you want anyhow.

The big take is that OpenVMS needs to move forward and DB's add a hell of a lot of value as does using universal data exchange formats like xml/JSON

[Kerry Main]

> -----Original Message-----
> From: Info-vax [mailto:info-vax...@rbnsn.com] On
> Behalf Of IanD via Info-vax
> Sent: 10-Sep-16 7:55 PM
> To: info...@rbnsn.com
> Cc: IanD <iloveo...@gmail.com>
> Subject: Re: [Info-vax] OpenSSL 1.1.0 released
>

Log files in a formal OpenVMS Prod/dev environment need to
be part of a cluster aware, scalable DB. Hence, if using
an open source DB, it needs to be one that is cluster
aware and R/W from many different sources at the same
time.

Console messages, error logs, audits etc. in a prod
environment are now not only being handled formally on OS
platforms, but being centralized as part of an Operations
Bridge environment. In fact, larger OPS Bridge
environments are now using prods like Splunk (expensive!)
to analyze log files from many sources as part of a Big
Data strategy for proactively identifying and resolving
issues before they impact the business and/or end users.

Proactive security analysis also requires a big data
strategy to log file analysis.

https://www.splunk.com/en_us/products/splunk-enterprise.ht
ml
https://www.splunk.com/en_us/products/splunk-enterprise.ht
ml#E1OW5iMjE6kvoCE7mXcyJLohGEr7J2qJ (Video)

Splunk competition:
https://www.quora.com/Who-are-the-biggest-direct-competito
rs-to-Splunk


Regards,

Kerry Main
Kerry dot main at starkgaming dot com

[Stephen Hoffman]

On 2016-09-10 23:54:36 +0000, IanD said:

> Does OpenVMS actually stipulate a guideline for log file formats or is
> it a free-for-all? It sure seems like a free-for-all, even looking at
> the OpenVMS internal log file formats

It's a mess. Some tools log in documented binary formats with
tool-specific conversion routines, more than a little logs in free form
text, applications are all over the map, and some of the OpenVMS system
data feeds are not readily accessible without using a broadcast mailbox
or tailing the log file. Commands and syntax all differ, too.
There's no distribution logging mechanism or API, and there's no log
aggregator / scanner. There's nothing around uploading application
and system crashes and scanning those. There's no mechanism for push
notifications. There's no SNMPv3, either. There's no concept of
distributed coordination and control (beyond the existing cluster-level
break-in activities, and those don't work across clusters). The
OpenVMS state of the art is not at all what's going to be necessary to
play well together in 2021 or 2026. It's all home-made and/or
third-party or ignored.