OpenVMS Clusters - Grid computing? A combined Virtual machine of X nodes?

[IanD]

I mentioned in another thread OpenVMS clusters needing to differentiate themselves again from other 'clusters'

I was thinking of ways to expand VMS clustering and thought about grid style computing and even concepts like a unified single machine consisting of however many machines one has in a cluster (selectable of course)

Perhaps I'm wrong but the way I look at VMS clusters today is really nothing more than an array of systems with a unified security offering that shares a job control mechanism with disk replication

I thought wouldn't it be nice to have VMS clusters that an application would see as a single virtual machine of infinite size or at least the size of how ever many machines you wish to combine into a virtual machine so that the application can share disks, memory and CPU resources as though it was a single machine

It would sure make applications that can scale up easier to develop as the complexity of the scaling up would be handled by the OS - it might make developing on OpenVMS desirable again? :-)

Items like sorting could then farm out the sort across nodes and then collate the results. Large memory operations could spread out across a cluster and collate results at the end of the operation. CPU intensive crunching the same

Things like Hadoop are already doing certain aspects of this now so we are already lagging in some ways but we have a good cluster framework on top of which we could build

I know there are logistics to overcome around memory operations and that sharing memory across machines is highly latent affected but there are other systems out there doing this I think through MPI etc

VMS clusters are nice but I don't see that they offer anything totally outstanding in a world where downtime is increasingly expensive and Linux clusters that perform application clusters are deemed good enough but good enough is not a goal and I think a business might open up the purse strings for an OS that is expandable upwards easily from an application perspective by simply bolting on additional systems without the need for buying ever bigger and faster machines. CPU speed is already hitting a bottle neck, which is why Intel has been pushing parallel programming and parallel tasking for quite a while now

What about process fault tolerance too, like what Tandem has. It's disruptive to have a node crash and loose all the compute work done to date, not that VMS crashes much

In my workplace, we have one machine that is more powerful than the other. we are having to upgrade the more powerful machine which is really causing the business gripes. It sure would have been nice to have an OS that I could simply pool the lessor powerful machine's resources towards a virtual VMS machine that encompasses both machines - I could have brought real cost savings to the business and gotten VMS a pat on the back at the same time

My 2c worth...

Would this be too difficult a task or are application clusters good enough for large scale business now?

I'm sure some will say is that a vision you have or have you been smoking something you shouldn't have... :-)

[clairg...@gmail.com]

Not radical at all. Hadoop or something like it should be in the future of VMS.

[IanD]

On Monday, September 21, 2015 at 10:33:51 PM UTC+10, clairg...@gmail.com wrote:

<snip>

> Not radical at all. Hadoop or something like it should be in the future of VMS.

Yeah, I post a lot of this stuff to foster discussion, to invoke thought if people have not thought along these lines before and because I frequent a number of different sites, get different technical journals that discuss the future of computing and generally think it's important that these ideas get tossed around

Hadoop is gaining everywhere

I just think that VMS clusters forwarded VMS to a respectable position today, for it survive, it needs to go forward again

I've seen some clusters with 10 nodes in them, no way to pool all those resources together is just a crying shame IMO

Concepts like bitcoin's blockchain could add extra security to VMS and be at the forefront. Banks are already getting together to work on bitcoin technology (I'll leave my dislike of that they actually want to do aside for now). The blockchain could take VMS security one step further than just encryption

[IanD]

grrr, is there no ability to edit a post on this forum?

Anyhow, you could extend the blockchain to a secure, unhackable file system that cannot be forged for example. The worst you can do with the blockchain at present is capture the majority of the compute and stall transactions, you cannot wipe them out, they remain on the ledger

[Simon Clubley]

On 2015-10-04, IanD <iloveo...@gmail.com> wrote:
>
> grrr, is there no ability to edit a post on this forum?
>

It's not a forum; it's Usenet. You are just using a web based interface
to Usenet provided by Google. Other people, such as myself, use a
dedicated Usenet client (in my case, slrn).

In a real Usenet client, you would supersede the article. Whether that
still works with the Usenet servers around these days I have no idea.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Stephen Hoffman]

On 2015-10-04 05:54:21 +0000, IanD said:

> Hadoop is gaining everywhere
>
> I just think that VMS clusters forwarded VMS to a respectable position
> today, for it survive, it needs to go forward again
>
> I've seen some clusters with 10 nodes in them, no way to pool all those
> resources together is just a crying shame IMO

Ten is a trivially-small cluster. Configurations with thousands of
servers are common. Those are not OpenVMS clusters, obviously.

Interconnection speeds are inversely proportional to distances
involved, and adding to this problem, the OpenVMS-supported
interconnects are also comparatively old and slow. This means that
big SMP boxes and big OpenVMS-style DLM,
distributed-write-shared-storage clusters have overhead, and that the
overhead increases faster than the configurations can be scaled up.
Find a way to scale server computing linearly and generically, and
there's at least a PhD waiting for you.

Interestingly, your applications can run faster reading and writing to
servers across the network than to local disk storage, too. This
strictly with the hardware speeds, and before any consideration of the
glacial speed of OpenVMS file I/O in comparison to other platforms.

Hadoop <http://hadoop.apache.org> is one of many packages that allows
better use of resources, and the easier and incremental addition of
servers, etc.

Other Apache projects, and which can be associated with Hadoop configurations:

Kafka distributed messaging
http://kafka.apache.org/

Flume log aggregation
http://flume.apache.org

Mesos distributed computing
http://mesos.apache.org

Spark data mining
http://spark.apache.org

Java 8 support will help with getting these and other Java and
JVM-based bits running on OpenVMS.

There's also OpenMP distributed multiprocessing, which ties into the
compilers and the run-time: http://openmp.org LLVM has hooks, here.

Outside of cases where OpenVMS brings specific additional benefits to
the project, there's very little reason to use OpenVMS for Hadoop or to
add OpenVMS to a Hadoop configuration, as there are pricing and support
and server management disincentives to incorporating OpenVMS boxes into
Hadoop configurations.

My own as well as the more general fondness for technologies and tools
here in the comp.os.vms newsgroup aside, it's the financials and the
staffing costs and the associated efforts to consolidate onto fewer
platforms and fewer tools and fewer applications that drive more than a
few of the decisions. That, and how much work you can get done, given
the available too-low staffing. TCO is nice, but any discussion of
TCO that omits the application costs, the tools, management interfaces
and staffing costs is incomplete. That there aren't enough OpenVMS
folks available have also been reported, but that's arguably a
statement that the necessary staff for the competitive platforms are
cheaper. Maintaining and lowering the fixed costs — of which salaries
are part — is basic business operations, after all. In short, the
finances have to work, or the tech is irrelevant.

> Concepts like bitcoin's blockchain could add extra security to VMS and
> be at the forefront. Banks are already getting together to work on
> bitcoin technology (I'll leave my dislike of that they actually want to
> do aside for now). The blockchain could take VMS security one step
> further than just encryption

Blockchains target distributed consensus-based transactional integrity
and the associated detection of tampering (or however the Blockchain
folks phrase that statement), and not "traditional" data security.

Blockchains can certainly be applicable for certain sorts of
transaction logs and for security logs in computing, which are cases
analogous to what the cryptocurrencies use the technique for.

AFAICT, blockchains also get rather hairy across multiple files — each
file likely ends up with a blockchain, and now you have to figure out
how to roll back across multiple files and across multiple chains, and
quite possibly somebody will decide to hang a blockchain off the
application's own activity.

(Dealing with blockchain rollover is an issue that would have to be
more cleanly handled as otherwise that data grows without bound. This
because a busy transaction log on a busy server can grow extremely
quickly. But I digress.)

(Blockchains and data encryption don't mix all that well, AFAICT, and
particularly with data that must be kept private — re-encrypting
encrypted data with upgraded ciphers would corrupt the blockchain — and
with data that must be expunged for business or legal reasons. But I
digress again.)

Other mechanisms that have been used with OpenVMS for this general goal
are write-once tape drive cartridges and write-once optical media.
Both are readily available, for those that need this.

The general benefits of blockchains for other sorts of files and for
other server environments is rather less clear (to me). That there'll
be added overhead is clear.

Implementing anew or porting libbitcoin or analogous would probably get
folks most of what they need, and (just) for the files that they want
and need blockchains. That, and getting a semi-modern certificate
store and related baggage implemented on OpenVMS, too.

Transactional and data integrity against accidental or intentional data
corruptions is nice and useful, but recoverability and rollback is key
to many operations; to the recovery from these corruptions or hacks.
Blockchains can help identify this, but somebody then has to implement
the rollbacks, and that can and will be implementation- and
application-specific work.

If you're doing banking or financials, then blockchains can help. If
you're after transactional integrity and recovery, there are other
options and alternatives to blockchains. Some generic examples of
alternatives: Apple Time Machine and other tools all target integrity
and the ability to restore, with some definite limits. OpenVMS
expects the system manager to establish archiving locally and entirely
site-specific, often with bespoke, artisanal DCL procedures created
from the finest field-grown organic bits. DVCS packages such as git
and mercurial can also be used for integrity and restoration. The
Rational ClearCase VOBs — a virtual disk volume that instantiates a
particular software baselevel or particular release — are a solution to
working with "dumb" tools.

OpenVMS has a good 1980s-style we-have-a-backup-window backup
mechanism. Many sites lack that window and are either moving to
databases with online backup capabilities, are quiescing and splitting
off HBVS volumes, or (less desirably and less reliably) are betting
that BACKUP /IGNORE=INTERLOCK or the entirely analogous
controller-level data replication mechanisms will be restorable.

OpenVMS itself is also still stupidly* solving the same problems within
its own components and tools using component-specific punched-card
record layouts and not databases, too — 1980s-style RMS record storage
only gets you so far, and the inevitable data changes and additions are
a pain to design and deploy incrementally and — in the absence of RMS
journaling or analogous — transactional integrity (this integrity
against partially-completed update I/O sequences and crashes and data
corruptions, and which is not really what blockchains help with or
recover from) is questionable.

As was stated repeatedly at the 2015 Boot Camp, the VSI priority is the
x86-64 port, and — with very few exceptions — everything else waits for
that work, and that work specifically targets server-oriented
computing. VSI is targeting the installed base. Whether in five or
ten years, and after the port is available and stable and in use, they
can provide new features and tools to then start picking up enough new
applications and new installations to at least offset the retirements
of existing applications and environments, and preferably to increase
the installed base? Some few additions were mentioned as semi-distant
— my words, not VSI — future possibilities, but that's after the x86-64
port is available.

——

*the current strategy is wise in the short term, and utterly foolhardy
in the medium and long term.



--
Pure Personal Opinion | HoffmanLabs LLC

[Paul Sture]

On 2015-10-04, Simon Clubley

<clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
> On 2015-10-04, IanD <iloveo...@gmail.com> wrote:
>>
>> grrr, is there no ability to edit a post on this forum?
>>
>
> It's not a forum; it's Usenet. You are just using a web based interface
> to Usenet provided by Google. Other people, such as myself, use a
> dedicated Usenet client (in my case, slrn).
>
> In a real Usenet client, you would supersede the article. Whether that
> still works with the Usenet servers around these days I have no idea.

The last time I tried using supersede it did work, but the update didn't
get propagated very well, if at all. The net result was that you'd find
folks replying to the original rather than the updated version.

A side effect was that unless you had taken the precaution of saving
your original separately, you wouldn't have access to it via your usual
interface any longer, except the bits others had quoted (and possibly
mis-attributed or otherwise mangled).

--
Mobile phones now have more processing power than the first computers I
worked on. Yet they still cannot store phone numbers using spaces, as
used for phone books, letter heads or business cards. Why not?

[Craig A. Berry]

On 10/4/15 1:00 AM, IanD wrote:

> Anyhow, you could extend the blockchain to a secure, unhackable file
> system that cannot be forged for example. The worst you can do with
> the blockchain at present is capture the majority of the compute and
> stall transactions, you cannot wipe them out, they remain on the
> ledger

Apple's newest offering introduces some file system and kernel integrity
features that look interesting:

<http://arstechnica.com/apple/2015/09/os-x-10-11-el-capitan-the-ars-technica-review/8/#h1>

The analogy for VMS would be that the SYSTEM account would not have
write access to any of the SYS$* directories, execlets and user-written
system services would have cryptographic signatures checked at load
time, and debuggers like DELTA and XDELTA would not work without
disabling the security features entirely, which would require an
alternate boot with physical access and the ability to decrypt the
volume on which you are disabling the feature. Of course you'd have to
have volume encryption in the first place.

[D W]

On 10/4/2015 5:44 AM, Simon Clubley via Info-vax wrote:
> On 2015-10-04, IanD <iloveo...@gmail.com> wrote:
>> grrr, is there no ability to edit a post on this forum?
>
> It's not a forum; it's Usenet. You are just using a web based interface
> to Usenet provided by Google.

Ignoring those of us who read it via email gateway.

[Simon Clubley]

I didn't say anything about how _you_ read cov. :-)

I looked at Ian's headers before posting; he's using the Google web
based interface.