Relative invulnerability of VMS to buffer-overflow attacks

Keith Parris

unread,

Apr 3, 2002, 12:14:30 PM4/3/02

to

Some here have contended that because TCP/IP Services for OpenVMS is
based on Tru64 Unix code, it is thus subject to the same level of risk
of buffer-overflow exploits as any Unix system out there.

After a bit of investigation, I've discovered that VMS on Alpha
appears to be immune to these common smash-the-stack buffer overflow
attacks.

To understand why, first one must understand how common
buffer-overflow attacks work. (A classic paper on such attacks is
"Smashing the Stack for Fun And Profit" written by a hacker who goes
by the name Aleph One. You can read it at
http://www.cse.ogi.edu/DISC/projects/immunix/StackGuard/profit.html)

A buffer-overflow exploit is done by passing an argument that is
larger than the buffer size to code which fails to check its size.
The data is all stored, resulting in overwriting some memory beyond
the end of the buffer. Hackers know that C compilers often allocate
data space for buffers on the stack, and part of the data area
overwritten on the stack includes the return address in the call
frame. The hacker writes a return address that points to his own
code, which is also included in the area overwritten on the stack.
The function then returns to his code instead of to the calling
routine. His malicious code then takes some dastardly action, like
starting the Unix shell and executing an arbitrary command.

For a buffer-overflow exploit to succeed, then, the hacker must know
the address size of the the target architecture (for the return
address), the format of the stack frame used by the architecture and
the compiler (to know where to put that return address), the
instruction set of the target architecture (so he can create his small
program), and enough about the innards of the operating system for his
program to be able to do something useful once it is executing.

Attacks most commonly target Windows, or Solaris or other common Unix
variants, and because the attack must be so specific to a platform,
these wouldn't affect VMS, because it has a different instruction set,
stack frame format, and so forth. While that dramatically reduces the
statistical probability of an attack succeeding on VMS in practice,
that alone doesn't rule out the possibility of an attack tailored
specifically for VMS on Alpha.

Here is where excellent and fortuitous engineering design comes to the
rescue. The Alpha memory management architecture provides some bits
in the Page Table Entries (PTEs) that control what type of access is
allowed to memory. For example, a read-only page can be protected by
having the Fault-On-Write bit set, and any attempt to write the page
causes a fault and results in a memory access violation error. Alpha
also has a Fault-On-Execute bit, designed to prevent an errant program
from jumping off into the weeds and trying to execute data as
instructions. On Alpha/VMS, the user stack is mapped with PTEs that
have the Fault-On-Execute bit set, so any attempt to branch into data
area on the stack results in an access violation, and the process
dies.

So Alpha VMS is immune to common stack-smashing buffer-overflow
attacks.

While any code that fails to check data lengths against buffer sizes
is arguably broken, and needs to be fixed, and Compaq has been doing
this to TCP/IP code as buffer-overflow bugs are identified, such bugs
are much less critical on Alpha VMS compared with less-protected
implementations.
----------------------------------------------
Keith Parris | parris at encompasserve dot org

labadie

unread,

Apr 3, 2002, 2:49:56 PM4/3/02

to

Keith Parris wrote:
...
May I say that I am tempted to save all the posts from Keith Parris, as
they are always highly valuable, as I learn much each time ?
Of course, google does it for me.

Many thanks Keith

Gérard

Main, Kerry

unread,

Apr 3, 2002, 2:47:20 PM4/3/02

to

Keith,

A definate "keeper".

Muchas gracias,

:-)

Kerry Main
Senior Consultant
Compaq Canada Corp.
Professional Services
Voice: 613-592-4660
Fax : 819-772-7036
Email: Kerry...@Compaq.com

Didier Morandi

unread,

Apr 3, 2002, 2:51:22 PM4/3/02

to

May I say that for a piece of executable code embedded in an overflow buffer to
be executed with VMS, it needs to either spawn a subprocess in which it will be
self-image activated, or call a lib$do_command RTL service or other mechanism
for VMS to execute an image which binary code will be partly in memory somewhere.

I personally prefer to believe that this kind of programming is just unfeasible
with VMS. But maybe we could ask Ruth (Goldenberg) or Larry (Kenah).

D.

JF Mezei

unread,

Apr 3, 2002, 3:22:28 PM4/3/02

to

Re: code that can branch to "data".

Mr Parris has stated that on VMS, it is possible to protect memory from being
executed so that one couldn't branch to a buffer for instance.

Is 100% sure that all VMS compilers will generate code that enable those
protections, notably a program's inability to modify its own code or inability
to execute code residing on the stack ?

Fred Kleinsorge

unread,

Apr 3, 2002, 4:43:39 PM4/3/02

to

There are a lot of issues in making data executable. Ask the Java people.
With enough knowledge could you do it? Maybe, but most likely along the way
you'll stumble over problems that require a privledged operation to do it
outside of a formal system interface... or a "can't get there from here"
problem.

JF Mezei wrote in message <3CAB646C...@videotron.ca>...

Bob Ceculski

unread,

Apr 3, 2002, 8:44:03 PM4/3/02

to

KeithParr...@yahoo.com (Keith Parris) wrote in message news:<6ec1251e.02040...@posting.google.com>...

do you understand this Andrew?

Robert Deininger

unread,

Apr 3, 2002, 8:50:59 PM4/3/02

to

In article <d7791aa1.02040...@posting.google.com>,
b...@instantwhip.com (Bob Ceculski) wrote:

>do you understand this Andrew?

Now THAT's the funniest thing I've read here in a long time!

Bob Ceculski

unread,

Apr 3, 2002, 8:54:59 PM4/3/02

to

KeithParr...@yahoo.com (Keith Parris) wrote in message news:<6ec1251e.02040...@posting.google.com>...

> Some here have contended that because TCP/IP Services for OpenVMS is
> based on Tru64 Unix code, it is thus subject to the same level of risk
> of buffer-overflow exploits as any Unix system out there.
>
> After a bit of investigation, I've discovered that VMS on Alpha
> appears to be immune to these common smash-the-stack buffer overflow
> attacks.
>

> Here is where excellent and fortuitous engineering design comes to the
> rescue. The Alpha memory management architecture provides some bits
> in the Page Table Entries (PTEs) that control what type of access is
> allowed to memory. For example, a read-only page can be protected by
> having the Fault-On-Write bit set, and any attempt to write the page
> causes a fault and results in a memory access violation error. Alpha
> also has a Fault-On-Execute bit, designed to prevent an errant program
> from jumping off into the weeds and trying to execute data as
> instructions. On Alpha/VMS, the user stack is mapped with PTEs that
> have the Fault-On-Execute bit set, so any attempt to branch into data
> area on the stack results in an access violation, and the process
> dies.
>
> So Alpha VMS is immune to common stack-smashing buffer-overflow
> attacks.
>

> ----------------------------------------------
> Keith Parris | parris at encompasserve dot org

and I have a perfect example of this ... from process softwares web
site,
versions of tcpware/multinet on vms were either not affected or gave
access
violations ... care to reply to this Andrew?

SNMP Inquiry - Cert Advisory CA-2002-03
--------------------------------------------------------------------------------

Question:

Are either MultiNet or TCPware affected by CERTŽ Advisory CA-2002-03
in Many Implementations of the Simple Network Management Protocol
(SNMP), dated February 12, 2002?

Answer:

These SNMP vulnerabilities do NOT pose security risks for MultiNet and
TCPware. MultiNet V4.4A is not vulnerable to these SNMP issues at all.
MultiNet 4.3A and TCPware have minor problems with access violations
(resulting in the SNMP process dying), but pose no security risk.
Patches for MultiNet 4.3A and TCPware V5.5-3 are available from the
TCPware ECO Database and the MultiNet ECO database. Use the following
kit names:

MultiNet V4.3A: SNMP-020_A043
TCPware V5.5-3: SNMPD_V553P011

Jan C. Vorbrüggen

unread,

Apr 4, 2002, 5:13:19 AM4/4/02

to

> Is 100% sure that all VMS compilers will generate code that enable those
> protections, notably a program's inability to modify its own code or
> inability to execute code residing on the stack ?

Two comments: First, only the C compiler need apply, because that's what the
TCPIP code is presumably written in. Secondly, that doesn't matter: the page
protections are set by the image activitator, and the stack is not language-
specific.

Jan

Jan C. Vorbrüggen

unread,

Apr 4, 2002, 5:14:17 AM4/4/02

to

I thought I had read that current Alpha processors allow, but do not
implement fault-on-execute?

Jan

Jan-Erik Söderholm

unread,

Apr 4, 2002, 5:20:47 AM4/4/02

to

Now, it *is* possible to execute "data" in VMS programs.
This is how Rdb runs it's queries. A dynamic SQL statement
is converted on-the-fly into machine code, which then is
executed to run the query.

Jan-Erik Söderholm.

Paul Repacholi

unread,

Apr 4, 2002, 9:28:38 PM4/4/02

to

KeithParr...@yahoo.com (Keith Parris) writes:

> Here is where excellent and fortuitous engineering design comes to
> the rescue. The Alpha memory management architecture provides some
> bits in the Page Table Entries (PTEs) that control what type of
> access is allowed to memory. For example, a read-only page can be
> protected by having the Fault-On-Write bit set, and any attempt to
> write the page causes a fault and results in a memory access
> violation error. Alpha also has a Fault-On-Execute bit, designed to
> prevent an errant program from jumping off into the weeds and trying
> to execute data as instructions. On Alpha/VMS, the user stack is
> mapped with PTEs that have the Fault-On-Execute bit set, so any
> attempt to branch into data area on the stack results in an access
> violation, and the process dies.

Not on the itanic...

--
Paul Repacholi 1 Crescent Rd.,
+61 (08) 9257-1001 Kalamunda.
West Australia 6076
Raw, Cooked or Well-done, it's all half baked.
EPIC, The Architecture of the future, always has been, always will be.

Fred Kleinsorge

unread,

Apr 5, 2002, 8:46:22 AM4/5/02

to

Paul Repacholi wrote in message <874riqr...@prep.synonet.com>...

>KeithParr...@yahoo.com (Keith Parris) writes:
>
>> Here is where excellent and fortuitous engineering design comes to
>> the rescue. The Alpha memory management architecture provides some
>> bits in the Page Table Entries (PTEs) that control what type of
>> access is allowed to memory. For example, a read-only page can be
>> protected by having the Fault-On-Write bit set, and any attempt to
>> write the page causes a fault and results in a memory access
>> violation error. Alpha also has a Fault-On-Execute bit, designed to
>> prevent an errant program from jumping off into the weeds and trying
>> to execute data as instructions. On Alpha/VMS, the user stack is
>> mapped with PTEs that have the Fault-On-Execute bit set, so any
>> attempt to branch into data area on the stack results in an access
>> violation, and the process dies.
>
>Not on the itanic...
>
>

You know something about the IA64 VMS design that we don't?

Bob Ceculski

unread,

Apr 5, 2002, 2:51:10 PM4/5/02

to

JF Mezei <jfmezei...@videotron.ca> wrote in message news:<3CAB646C...@videotron.ca>...

> > If the hacker is able to determine the stack frame slot with the return
> > address, why couldn't he also modify the stack allocation instruction and
> > insert his trojan horse in code space and execute from there?
>
> It would be difficult to execute instructions to change memory in a code
> space using the described method. Should the cracker (ok, hacker if he's
> doing it on his own machine just for fun) somehow gets code to execute on
> the stack, executable sections are not usually writeable.
>
> Try it with a C program, get a pointer to the entry point of a function and
> try to change that first byte.I haven't tried this but you might make a
> small array or whatever on the stack, put some code in it and try to call
> it. My bet is it won't work.

Just to illustrate my point:

1 #include <stdio.h>
874
875 void xyzzy(void) { printf("xyzzy!\n"); }
876
877 int main( int argc, char *argv[] )
1 878 {
1 879 unsigned char *x;
1 880 x = (unsigned char *)xyzzy;
1 881 ((void(*)(void))x)();
1 882 *x = 0x00;
1 883 return(1);
884 }

when run produces:

$ r x
xyzzy!
%SYSTEM-F-ACCVIO, access violation, reason mask=04, virtual address=00000208,
PC=0000024F, PSL=03C00000
%TRACE-F-TRACEBACK, symbolic stack dump follows
module name routine name line rel PC abs PC

X main 882 00000033 0000024F
$

Check the attributes of the $CODE section. It's NOWRT and the (empty) $DATA
section is NOEXE. Static variables go in $DATA. I don't know offhand what
the stack area looks like but I suspect it's NOEXE too. This boils down to
what's been said earlier, normally code can't be written over and data can't
be executed.

Psect Name Module Name Base End Length Align
Attributes
---------- ----------- ---- --- ------ -----
----------
$DATA 00000200 00000200 00000000 ( 0.) LONG
2 PIC,USR,CON,REL,LCL,NOSHR,NOEXE, RD, WRT,NOVEC
X 00000200 00000200 00000000 ( 0.) LONG 2

$CODE 00000200 00000271 00000072 ( 114.) QUAD
3 PIC,USR,CON,REL,LCL, SHR, EXE, RD,NOWRT,NOVEC
X 00000200 00000271 00000072 ( 114.) QUAD 3

-- Aaron Sliwinski

Bob Ceculski

unread,

Apr 5, 2002, 2:56:20 PM4/5/02

to

JF Mezei <jfmezei...@videotron.ca> wrote in message news:<3CAB646C...@videotron.ca>...

and I have a perfect example of this ... from process softwares web

Warren Spencer

unread,

Apr 8, 2002, 2:51:12 PM4/8/02

to

a...@aaa.com (Jan-Erik Söderholm) wrote in <3CAC28FF...@aaa.com>:

Are you sure about this? I've never bumped into database that compiles an
sql query down to the native machine code of the target machine for
execution. I'm no expert, but this sounds weird to me.

ws

--

Warren Spencer
Senior Software Engineer (not a writer)
The Associated Press

** Time flies like an arrow. Fruit flies like a bananna. **

Jan-Erik Söderholm

unread,

Apr 8, 2002, 3:14:48 PM4/8/02

to

From the Oracle presentation called "Porting Rdb to Itanium":

On slide 5 ("What we learned, In the beginning") :
...
•Generate query specific code at runtime using VAX instruction set
...

On slide 6 ("What we learned, VAX to Alpha") :
...
•Generate query specific code at runtime using Alpha instruction set
...

And then there is some info on how this was done on Rdb/NT and how
it might/will be done on Rdb/IA64. I'll left that out...

Jan-Erik Söderholm

Art Beane

unread,

Apr 8, 2002, 3:22:09 PM4/8/02

to

>Now, it *is* possible to execute "data" in VMS programs.

Indeed, every interpreter has to do it.

Granted that LISP may be a special case, but it does not require a
distinction between data and program. It's very common to write a LISP
program that dynamically generates and executes code. A LISP program
even has runtime access to and control over the compiler.

Art Beane
Petris Technology, Inc.
(713) 403-8423

Jan-Erik Söderholm

unread,

Apr 8, 2002, 3:45:12 PM4/8/02

to

An interpreter do *NOT' run machine code that have been
created on-the-fly. It *ONLY* runs the machine code it was
created and delivered with, nothing else.

Does the LIST program generate *machine code* ?
Or just some other LISP code ? (Any interpreter
could create a file with any interpreted language
and later call the appropriate interpreter to "run" it...)

Jan-Erik Söderholm.

PS.
With "data" in my post below, I mean (of course) some
data structure belonging the the program where *machine*
code is written and later executed.
DS.

RLC

unread,

Apr 8, 2002, 4:36:45 PM4/8/02

to

On the VAX this was a fairly easy thing to do. I once wrote a
SORT package that did just that for the comparisons. On the
Alpha, this is a much more difficult (although not impossible)
task. I opted for another solution on the Alpha when time came
to rewrite the sort.

The nice thing about VMS for buffer overflow attacks is that if
the web application (say HTTP server) is not given privileges
and file system privs are set correctly, it doesn't really matter
that you can execute code. Access woudl be limited so that
nothing significant could be done anyway. Give me a server
running with CMKRNL priv (or just SETPRV) and I can do
a LOT of damage.

Rick Cadruvi...

"Warren Spencer" <wspe...@ap.nospam.org> wrote in message
news:91EA945BCwarr...@216.168.2.110...

Bill Gunshannon

unread,

Apr 9, 2002, 8:31:21 AM4/9/02

to

In article <ub3vqud...@corp.supernews.com>,

"RLC" <r...@magicalsoftware.com> writes:
|>
|> The nice thing about VMS for buffer overflow attacks is that if
|> the web application (say HTTP server) is not given privileges
|> and file system privs are set correctly, it doesn't really matter
|> that you can execute code. Access woudl be limited so that
|> nothing significant could be done anyway.

Which of course is true of a Unix system or any other OS as well.
The general lack of security in most Unix systems is due to mindset
(an old one that sadly is no longer acceptable for social reasons
rather than technical reasons) and not real technical shortcomings.

bill

--
Bill Gunshannon | de-moc-ra-cy (di mok' ra see) n. Three wolves
bi...@cs.scranton.edu | and a sheep voting on what's for dinner.
University of Scranton |
Scranton, Pennsylvania | #include <std.disclaimer.h>

Bob Ceculski

unread,

Apr 9, 2002, 3:48:21 PM4/9/02

to

bi...@triangle.cs.uofs.edu (Bill Gunshannon) wrote in message news:<a8umup$21gl$1...@info.cs.uofs.edu>...

> In article <ub3vqud...@corp.supernews.com>,
> "RLC" <r...@magicalsoftware.com> writes:
> |>
> |> The nice thing about VMS for buffer overflow attacks is that if
> |> the web application (say HTTP server) is not given privileges
> |> and file system privs are set correctly, it doesn't really matter
> |> that you can execute code. Access woudl be limited so that
> |> nothing significant could be done anyway.
>
> Which of course is true of a Unix system or any other OS as well.
> The general lack of security in most Unix systems is due to mindset
> (an old one that sadly is no longer acceptable for social reasons
> rather than technical reasons) and not real technical shortcomings.
>
> bill

any other os? you better head on over to say ... comp.os.linux.security
and take a gander, or better yet windoze ... they have millions of posts
reading "Help, I have been hacked" appearing hourly ... vms remains
"unhackable" for the reasons stated ... buffer overflows from ip are not
common because of vms design ... unix/linux/windoze are insecure, and that
is putting it kindly!

Andrew Harrison SUNUK Consultancy

unread,

Apr 12, 2002, 9:33:19 AM4/12/02

to

Bob Ceculski wrote:

Good tactic Bob, if in doubt try to get people to think that
all UNIX's are insecure because Linux is.

Lets extend this one step further, OpenVMS is an OS, Win32
is an OS, Win32 is insecure therefor so is OpenVMS they
are both OS's so it must be.

Of course this is ridiculous but no more so than your attempts

Regards
Andrew Harrison

unread,

Apr 17, 2002, 2:21:55 PM4/17/02

to

|> except in an ip world, I never heard of an ip stack for rsts/e ...

It has DECNET, isn't that good enough?? But it's OK, you can still
connect them to an IP network, even without a native IP Stack. Been
doing for machines for ages.

Bill Gunshannon

unread,

Apr 17, 2002, 2:23:47 PM4/17/02

to

In article <d7791aa1.0204...@posting.google.com>,

|> and what about 2000 issues ... I don't think rsts/e was made compliant!

Nope, all four of the OSes I mentioned are Y2K compliant. So, what
other excuses can you think of for not using the most secure OSes in
the world (based on number of CERT advisories.) :-)