[Lit.] Buffer overruns
Mok-Kong Shen
Buffer overruns were sometimes mentioned in a number of past
threads. I just came across a recent article that could be
of interest to some people:
J. Pincus, B. Baker, Beyond Stack Smashing: Recent Advances
in Exploiting Buffer Overruns. IEEE Security & Privacy,
July/August 2004, p.20-27.
M. K. Shen
RobRPM2222
>threads.
Buffer *Overflows*!
> I just came across a recent article that could be
>of interest to some people:
>
> J. Pincus, B. Baker, Beyond Stack Smashing: Recent Advances
> in Exploiting Buffer Overruns. IEEE Security & Privacy,
> July/August 2004, p.20-27.
>
>M. K. Shen
if it's the paper I'm thinking of, you might want to read the DailyDave
comments on the paper.
in short, it's about 4 years behind exploit writing in the non-academic
community.
Mok-Kong Shen
RobRPM2222 wrote:
> if it's the paper I'm thinking of, you might want to read the DailyDave
> comments on the paper.
>
> in short, it's about 4 years behind exploit writing in the non-academic
> community.
Thanks for the hint. (I assume the comments are in a more
recent issue of the journal which is not yet in our library.)
BTW, I like to raise a question: The paper discussed C and
C++. Let's consider a hypothetical situation, where the
OS and application programs were all written in a presumably
'safe' programming language, e.g. ADA (would JAVA be as good?),
could one really 'surely' exclude such (or somewhat varied)
exploits? Thanks.
M. K. Shen
Randy Howard
says...
> BTW, I like to raise a question: The paper discussed C and
> C++. Let's consider a hypothetical situation, where the
> OS and application programs were all written in a presumably
> 'safe' programming language, e.g. ADA (would JAVA be as good?),
> could one really 'surely' exclude such (or somewhat varied)
> exploits? Thanks.
IF you are *really* interested in this, which is more about
programming than crypto, take a look at The Shellcoders
Handbook by Koziol, Litchfield, Aitel, Anley, Eren, Mehta
and Hassell, published this year.
--
Randy Howard (2reply remove FOOBAR)
"For some reason most people seem to be born without the part
of the brain that understands pointers." -- Joel Spolsky
Douglas A. Gwyn
> 'safe' programming language, e.g. ADA (would JAVA be as good?),
> could one really 'surely' exclude such (or somewhat varied)
> exploits? Thanks.
Programmer oversights are possible in any PL. While bounds
enforcement would prevent *some* attacks, it cannot stop
them all. For example, the program might use a fixed-length
array to implement a ring buffer, and there could be a bug
that surfaces only when the buffer is full, which might not
happen in normal use.
There is an additional danger, in that some programmers would
become even more careless if they believe that the PL will
catch all such mistakes.
Mok-Kong Shen
Douglas A. Gwyn wrote:
> Mok-Kong Shen wrote:
>
>>'safe' programming language, e.g. ADA (would JAVA be as good?),
>>could one really 'surely' exclude such (or somewhat varied)
>>exploits? Thanks.
>
>
> Programmer oversights are possible in any PL. While bounds
> enforcement would prevent *some* attacks, it cannot stop
> them all. For example, the program might use a fixed-length
> array to implement a ring buffer, and there could be a bug
> that surfaces only when the buffer is full, which might not
> happen in normal use.
You are certainly right that not 'all' errors could be
prevented. On the other hand, stopping a comparatively
common potential source of trouble can surely be valuable.
Checking bounds would affect efficiency. But it is viable
to let bound-checking be a default feature such that arrays
that needn't that check may be explicitly defined to be so.
That way, in environments where security issues are entirely
irrelevant certain specific software could run more
efficiently without the bound-checking.
> There is an additional danger, in that some programmers would
> become even more careless if they believe that the PL will
> catch all such mistakes.
Certainly there are in real-life barely advantages without
some associated disadvantages. On the other hand, safety
measures (e.g. safety belts) are in general worthwhile.
It seems that combining the use of very 'safe' (in some
sense) PLs with an adequate training of programmers should
be a reasonably good goal to be targeted in the long run.
M. K. Shen
karl malbrain
news:cp4gnr$qp1$02$1...@news.t-online.com...
> BTW, I like to raise a question: The paper discussed C and
> C++. Let's consider a hypothetical situation, where the
> OS and application programs were all written in a presumably
> 'safe' programming language, e.g. ADA (would JAVA be as good?),
> could one really 'surely' exclude such (or somewhat varied)
> exploits? Thanks.
There's nothing "unsafe" about the C programming language. I don't know how
anyone could take a facility designed to store constants (character arrays)
and blindly use them as a general purpose string handling facility (with
strcpy, strcat, etc) is an historical matter.
No well-designed C program is going to use strings without also keeping
track of their current and maximum sizes as an abstracted object. The
question of "buffer overruns" becomes an exception-handled event. karl m
Mok-Kong Shen
karl malbrain wrote:
I suppose that a feature that could be 'intentionally' misused
certainly deserves to be considered as unsafe. Even those that
beginners have a high chance to inappropriately use, thus
leading to wrong results, are generally considered unsafe,
if I don't err.
M. K. Shen
karl malbrain
Beginners need to learn how to ABSTRACT pertinent characteristics of what
they're working with. STRCPY, STRCAT, etc, should not even be taught. karl
m
Mok-Kong Shen
karl malbrain wrote:
As I said in another post, proper education should be
omplementary to safety provided by PL design, i.e. both
would be needed.
M. K. Shen
karl malbrain
>
>
> karl malbrain wrote:
>
> > Beginners need to learn how to ABSTRACT pertinent characteristics of
what
> > they're working with. STRCPY, STRCAT, etc, should not even be taught.
karl
> > m
>
> As I said in another post, proper education should be
> omplementary to safety provided by PL design, i.e. both
> would be needed.
And the point remains that the design of C is just fine. You build safety
in from the ground up, from the first day on the job. karl m
Mok-Kong Shen
karl malbrain wrote:
With your logic any PL, including assembler, would be just fine
as another, I am afriad. There should be some measures of
comparison in my view.
M. K. Shen
karl malbrain
> >>As I said in another post, proper education should be
> >>omplementary to safety provided by PL design, i.e. both
> >>would be needed.
> >
> >
> > And the point remains that the design of C is just fine. You build
safety
> > in from the ground up, from the first day on the job. karl m
>
> With your logic any PL, including assembler, would be just fine
> as another, I am afriad. There should be some measures of
> comparison in my view.
For the record, assembler is just fine. The current solution for the AES
timing attack uses Dr. Gladman's assembly version. karl m
Mok-Kong Shen
karl malbrain wrote:
That certainly doesn't corresponds to the general opinions
about the comparative merits of PLs, anyway not to the
majority view in academics, as far as I am aware.
M. K. Shen
>
>
Xcott Craver
>Mok-Kong Shen wrote:
>> 'safe' programming language, e.g. ADA (would JAVA be as good?), could
>> one really 'surely' exclude such (or somewhat varied) exploits? Thanks.
>
> Programmer oversights are possible in any PL. While bounds enforcement
> would prevent *some* attacks, it cannot stop them all.
vulnerabilities can not occur because they are type mismatches.
Buffer overruns, for example, are type mismatches because the array
size is part of its type. Code containing a buffer overrun issue
simply cannot compile.
This does not prevent all sorts of other vulnerabilities due to
improperly implemented algorithms. But a huge number of exploited
security flaws are mere typing errors, which wouldn't happen if you
used a language that disallowed them.
But I agree that "C is unsafe." Sure it's safer if you make sure to
check everything, but the point is that people don't. Likewise
you can secure a Windows box, but a lot of people don't. In the
end what matters is the likelihood of widespread vulnerabilities
in the real world, and the damage caused by them.
--X
Douglas A. Gwyn
> You are certainly right that not 'all' errors could be
> prevented. On the other hand, stopping a comparatively
> common potential source of trouble can surely be valuable.
I forgot to mention that there is a problem with that,
namely that since the programmer is assumed to be "out
of the loop", whatever steps are taken automatically
when a bounds violation is detected are almost certainly
not appropriate in the context of the actual application.
There really is no substitute for careful programming.
David Wagner
> In a "type-safe" language like ML, a great many security
> vulnerabilities can not occur because they are type mismatches.
> Buffer overruns, for example, are type mismatches because the array
> size is part of its type. Code containing a buffer overrun issue
> simply cannot compile.
Does ML even have arrays?
I got the impression that arrays are definitely not the 'ML style';
ML style strongly encourages you to use lists everywhere, and similar
tree-based data structures where lists are inapplicable.
I agree with your general comments about strong static typing.
Anne & Lynn Wheeler
> Programmer oversights are possible in any PL. While bounds
> enforcement would prevent *some* attacks, it cannot stop
> them all. For example, the program might use a fixed-length
> array to implement a ring buffer, and there could be a bug
> that surfaces only when the buffer is full, which might not
> happen in normal use.
>
> There is an additional danger, in that some programmers would
> become even more careless if they believe that the PL will
> catch all such mistakes.
the claim is that there is something like a hundred fold increase in
buffer-overflows because of the semantics of string library in C
... compared to other infrastructures. That doesn't mean that it isn't
impossible to do length related errors in other infrastructures ...
it is just the frequency is significantly less often.
as of approx. 1999, the majority of (programming exploits) were
C-related buffer overflows.
as of approx. two years ago ... the exploits were something like
1/3rd c-related buffer overlows
1/3rd automatic scripting
1/3rd social engineering
(not so much that the c-related buffer overflows declined ... but that
the other exploits increased significantly).
there is the multics security review paper .... which cliams that
multics had no known cases of length related exploits. part of this is
the different length related semantics in its implementation language
PLI; PLI implementation typically had buffers with max & current
lengths in a header field. copy/move/io library routines honored the
explicit lengths. It wasn't impossible to write bad code with
length-related problems ... but you had to work much, much harder at
doing it (that is typical is in c).
this isn't a situation of the PLI catching mistakes ... it is that c
library semantics provide more opportunities to make mistakes compared
to other languages where the semantics make it much less likely to
make mistakes.
past reference to the multics review
http://www.garlic.com/~lynn/2002l.html#42 Thirty Years Later: Lessons from the Multics Security Evaluation
http://www.garlic.com/~lynn/2002l.html#45 Thirty Years Later: Lessons from the Multics Security Evaluation
lots of past posts discussing buffer overflows
http://www.garlic.com/~lynn/subpubkey.html#overflow
--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/
Randy Howard
> the claim is that there is something like a hundred fold increase in
> buffer-overflows because of the semantics of string library in C
> ... compared to other infrastructures.
And 90% of those are probably related to poor legacy code that still
uses gets(). That one library function is the hackers best friend.
If libc vendors had any balls at all they'd remove it, or make it
generate a segfault every time it is called until code that depends on
it gets overhauled.
David Wagner
>In article <m3sm6h4...@lhwlinux.garlic.com>, ly...@garlic.com says...
>> the claim is that there is something like a hundred fold increase in
>> buffer-overflows because of the semantics of string library in C
>> ... compared to other infrastructures.
>
>And 90% of those are probably related to poor legacy code that still
>uses gets(). That one library function is the hackers best friend.
I very much doubt it. In fact, I'm quite certain that is not so.
It is true that many (perhaps the majority) of buffer overruns are
related to unsafe string library functions. Some buffer overruns are
related to unsafe pointer handling (e.g., walking a string with *p++
stuff and inadvertently overruning bounds). But very few, these days,
are due to gets().
>If libc vendors had any balls at all they'd remove it, or make it
>generate a segfault every time it is called until code that depends on
>it gets overhauled.
Some platforms have already done so. But gets() is only a tiny piece
of the problem.
Arnaud Carré
> about the comparative merits of PLs, anyway not to the
> majority view in academics, as far as I am aware.
I agree with Karl. Anyway our opinion could be "agressive" for academic
people. Let's try to discuss it in detail.
I agree with Karl when he said there is no more special "security weak" in C
langage. A proove is simple: take a langage the academic world consider
"safe" : ADA. When compiled, ADA is just a binary block of assembly code.
Strangely, "assembly" is considered as "not secure" by academic. Trying to
know if a langage as security weak is not pertinent. ALL langages are
compiled into native assembly code so all langage have the same "atomic"
security level . ( And even byte-code langage such JAVA are executed by VM,
which run using native code too !
BUT, the truth is : "It's easyest to write poor security code in one langage
than antoher". It's terribly true for C langage, and that's the why C has
bad reputation in academic. It's easyest to write a weak code in C than ADA.
To resume: As karl said, a "warrior" programmer which know perfectly C can
make a program with same security level than with any other lagage. Simply
it's harder.
So in practice, for standard programmers, better use high level langage if
you're not a C specialist.
Hope it help
Douglas A. Gwyn
> the claim is that there is something like a hundred fold increase in
> buffer-overflows because of the semantics of string library in C
Anybody can make a nonsensical claim; that doesn't make it true.
The string library isn't even used in buffer management.
Mok-Kong Shen
Arnaud Carré wrote:
>>That certainly doesn't corresponds to the general opinions
>>about the comparative merits of PLs, anyway not to the
>>majority view in academics, as far as I am aware.
>
>
> I agree with Karl. Anyway our opinion could be "agressive" for academic
> people. Let's try to discuss it in detail.
>
> I agree with Karl when he said there is no more special "security weak" in C
> langage. A proove is simple: take a langage the academic world consider
> "safe" : ADA. When compiled, ADA is just a binary block of assembly code.
> Strangely, "assembly" is considered as "not secure" by academic. Trying to
> know if a langage as security weak is not pertinent. ALL langages are
> compiled into native assembly code so all langage have the same "atomic"
> security level . ( And even byte-code langage such JAVA are executed by VM,
> which run using native code too !
You distorted the context. When one says that a PL is not safe,
that means it is error-prone when one uses it 'directly'. Even
assembler is transformed to machine code before a progran is
run. Now, would you prefer writing your program directly in
machine code? Wouldn't that be generally more error-prone than
even assembler?
> BUT, the truth is : "It's easyest to write poor security code in one langage
> than antoher". It's terribly true for C langage, and that's the why C has
> bad reputation in academic. It's easyest to write a weak code in C than ADA.
Ah, here you reverted to the proper context.
> To resume: As karl said, a "warrior" programmer which know perfectly C can
> make a program with same security level than with any other lagage. Simply
> it's harder.
>
> So in practice, for standard programmers, better use high level langage if
> you're not a C specialist.
However, C IS generally recognized as a high level programming
language.
M. K. Shen
Arnaud Carré
> that means it is error-prone when one uses it 'directly'. Even
> assembler is transformed to machine code before a progran is
> run. Now, would you prefer writing your program directly in
> machine code? Wouldn't that be generally more error-prone than
> even assembler?
Oh no sorry it's just a sample for my proove that every langage have the
same theorical "atomic" security level when writing code. In my previous
post, I consider "assembly" as "machine code" but you can make the
distinction. ( Only valid for "macro assembler". If you use a classic
assembler only (never used today :-)) you can consider assembly = machine
code.
But, to be perfectly "academic", just forget my "assembly" exemple and think
about a Turing machine.
> Ah, here you reverted to the proper context.
yep :-) That's why sometimes when someone said "C langage is as secure as
other", you can't really know if the guy is a poor C coder and have a bad
opinion, or if the guy is a C power user and know that's possible to write
solid code in C ( as in assembly langage), but you have to spend a lot of
efforts, wich is not realistic in real life for real project. That's why
there is other langages such as ADA , etc... I just pointed out that it's
theorically "possible" to get the same level of security with ANY langage,
even assembly.
> However, C IS generally recognized as a high level programming
> language.
Well, definition of "high level " langage is out of topic here (and
certainly out of my knowledge :-)). To me, writing code since a long time, C
langage is a universal assembly langage without registers contraints. so to
my opinion C is quite a low level langage (so it's very hard to write
"secure" code, but *never* impossible.
If I should ordering langages in term of "time spent to write secure code",
I'll put:
1) Assembly
2) C
3) C++
4) ADA
Phil Carmody
Exactly. It's great for buffer overflows, but utterly useless at the
management part.
Phil
--
God was my co-pilot but we crashed in the mountains and I had to eat him.
Anne & Lynn Wheeler
> The string library isn't even used in buffer management.
there can be two totally different issues here ... buffer
overruns/overflows typically have to do with length management and
moving things into (or out of) buffers. various string libraries do
move things into buffers.
another kind of buffer overrun not mentioned (as frequently) is
incoming characters from some hardware device ... where the rate of
the incoming characters exceed the capacity of the system to allocate
space for them. this buffer overrun/overflow situation usually results
in dropped data ... as opposed to move/copy of data past the end of an
allocated buffer. this kind of buffer overrun strays into the area of
windowing algorithms and rate-based pacing
another buffer management problem can be allocation/deallocation of
the buffers. this is frequently an infrastructure serialization
problem ... with things like dangling pointers still being in use
after dynamic buffer had been de-allocated (or serialization process
trying to play and safe and creating zombie process type problems
trying to make sure a process doesn't go away since there might be
some orphan activity left around which wakes up in the future and
crashes the kernel).
long ago and far away when i was doing kernel stuff ... i got to
release the resource manager
http://www.garlic.com/~lynn/subtopic.html#fairshare
as part of that, developed some sophisticated testing and benchmarking
tools. besides using the benchmarking to validate extremely fine-grain
deterministic scheduling for the fair share scheduler ... i also used
it for severe stress testing ... which, when i started was guarenteed
to crash the kernel. before the release of the resource manager ... i
redesigned and rewrote the kernel serialization infrastructure
eliminating all known cases of kernel crashes because of
dangling/orphan buffer pointers as well as all cases of zombie/hung
processs.
I then went on to do a automated kernel problem/crash analysis and
determination tool ... which at one time was used by all corporate
PSRs responsible for analysis of customer kernel problems.
http://www.garlic.com/~lynn/subtopic.html#dumprx
as part of doing this tool ... i gathered extensive data on all
customer reported problems.
About the same time, i also got involved with the disk engineering lab
.. responsible for developing new disks ... at the time, they were
operating with stand-alone computers ... because attempting to operate
operating system with engineering disks had a MTBF of 15 minutes.. I
redesigned and rewrote the io subsystem so that disk engineering could
concurrently operate with multiple engineering disks in an operating
system environment w/o system crashes.
http://www.garlic.com/~lynn/subtopic.html#disk
so when my wife and I got around to starting the HA/CMP project ...
we did a detailed vulnerability analysis of the environment ....
http://www.garlic.com/~lynn/subtopic.html#hacmp
one of the conclusions was that there would be a hundred fold increase
in the incidents of buffer length related problems and exploits
... that what we had been familiar with in other environments (because
of the common length handling paradigm in C).
minor topic drift post related to ha/cmp, parallel oracle
http://www.garlic.com/~lynn/95.html#13
and the relationship to ssl and electronic commerce
http://www.garlic.com/~lynn/aadsm5.htm#asrn2
http://www.garlic.com/~lynn/aadsm5.htm#asrn3
and recent thread on what is necessary for industrial and business
strength programming and applications
http://www.garlic.com/~lynn/2004p.html#20 Systems software versus applications software definitions
http://www.garlic.com/~lynn/2004p.html#23 Systems software versus applications software definitions
http://www.garlic.com/~lynn/2004p.html#24 Systems software versus applications software definitions
http://www.garlic.com/~lynn/2004p.html#63 Systems software versus applications software definitions
http://www.garlic.com/~lynn/2004p.html#64 Systems software versus applications software definitions
http://www.garlic.com/~lynn/2004q.html#1 Systems software versus applications software definitions
in any case, our resulting experience was that there was, in fact,
something like a hundred fold increase in buffer length related
problems (compared to other environments and paradigms that we were
familiar with, based on some experience having looked in some detail
at customer and other reported operating system failures and problems
and done detail analysis over the years on the causes) ... previous
reference to collected postings on buffer length problems
http://www.garlic.com/~lynn/subpubkey.html#overflow
one specific posting from 1999, referencing a published buffer overflow
study
http://www.garlic.com/~lynn/99.html#219 Study says "buffer overflow" is most common security bug
I also have done some analysis of the cve vulnerability & exploit
database ... some summary of the analysis
http://www.garlic.com/~lynn/2004j.html#58
from prior posting
http://www.garlic.com/~lynn/2004q.html#2 [Lit.] Buffer overruns
last year, i was on panel discussion with somebody from one of the
anti-virus companies and a somebody heading up fbi cyber forensic
... he presented the 1/3rd, 1/3rd, 1/3rd statistics. you can actually
see our picture buried some place on this page:
http://www.w3w3.com/CSSB.htm
and mention of the air force security audit and evaluation of multics
(from section "2.3 No Buffer Overflows") that there were no buffer
overflows.
random past references to the air force multics security evaluation:
http://www.garlic.com/~lynn/aadsm14.htm#32 An attack on paypal
http://www.garlic.com/~lynn/aadsm15.htm#23 NCipher Takes Hardware Security To Network Level
http://www.garlic.com/~lynn/aadsm16.htm#1 FAQ: e-Signatures and Payments
http://www.garlic.com/~lynn/aadsm16.htm#8 example: secure computing kernel needed
http://www.garlic.com/~lynn/2002l.html#42 Thirty Years Later: Lessons from the Multics Security Evaluation
http://www.garlic.com/~lynn/2002l.html#44 Thirty Years Later: Lessons from the Multics Security Evaluation
http://www.garlic.com/~lynn/2002l.html#45 Thirty Years Later: Lessons from the Multics Security Evaluation
http://www.garlic.com/~lynn/2002m.html#8 Backdoor in AES ?
http://www.garlic.com/~lynn/2002m.html#10 Backdoor in AES ?
http://www.garlic.com/~lynn/2002m.html#58 The next big things that weren't
http://www.garlic.com/~lynn/2002o.html#78 Newsgroup cliques?
http://www.garlic.com/~lynn/2002p.html#6 unix permissions
http://www.garlic.com/~lynn/2003b.html#0 Disk drives as commodities. Was Re: Yamhill
http://www.garlic.com/~lynn/2003i.html#59 grey-haired assembler programmers (Ritchie's C)
http://www.garlic.com/~lynn/2003j.html#4 A Dark Day
http://www.garlic.com/~lynn/2003k.html#3 Ping: Anne & Lynn Wheeler
http://www.garlic.com/~lynn/2003k.html#48 Who said DAT?
http://www.garlic.com/~lynn/2003l.html#19 Secure OS Thoughts
http://www.garlic.com/~lynn/2003m.html#1 Password / access rights check
http://www.garlic.com/~lynn/2003o.html#5 perfomance vs. key size
http://www.garlic.com/~lynn/2004b.html#51 Using Old OS for Security
http://www.garlic.com/~lynn/2004f.html#20 Why does Windows allow Worms?
http://www.garlic.com/~lynn/2004h.html#2 Adventure game (was:PL/? History (was Hercules))
http://www.garlic.com/~lynn/2004j.html#29 Vintage computers are better than modern crap !
http://www.garlic.com/~lynn/2004j.html#41 Vintage computers are better than modern crap !
http://www.garlic.com/~lynn/2004l.html#21 "Perfect" or "Provable" security both crypto and non-crypto?
http://www.garlic.com/~lynn/2004m.html#25 Shipwrecks
http://www.garlic.com/~lynn/2004o.html#20 RISCs too close to hardware?
http://www.garlic.com/~lynn/2004q.html#2 [Lit.] Buffer overruns
some random topic drift regarding the other kind of buffer
overrun/overflow having to do with pacing algorithms:
http://www.garlic.com/~lynn/93.html#28 Log Structured filesystems -- think twice
http://www.garlic.com/~lynn/94.html#22 CP spooling & programming technology
http://www.garlic.com/~lynn/99.html#33 why is there an "@" key?
http://www.garlic.com/~lynn/2000b.html#11 "Mainframe" Usage
http://www.garlic.com/~lynn/2001h.html#44 Wired News :The Grid: The Next-Gen Internet?
http://www.garlic.com/~lynn/2002.html#38 Buffer overflow
http://www.garlic.com/~lynn/2002i.html#45 CDC6600 - just how powerful a machine was it?
http://www.garlic.com/~lynn/2002i.html#57 CDC6600 - just how powerful a machine was it?
http://www.garlic.com/~lynn/2002k.html#56 Moore law
http://www.garlic.com/~lynn/2002p.html#28 Western Union data communications?
http://www.garlic.com/~lynn/2002p.html#31 Western Union data communications?
http://www.garlic.com/~lynn/2003b.html#44 filesystem structure, was tape format (long post)
http://www.garlic.com/~lynn/2003g.html#54 Rewrite TCP/IP
http://www.garlic.com/~lynn/2003g.html#64 UT200 (CDC RJE) Software for TOPS-10?
http://www.garlic.com/~lynn/2003.html#55 Cluster and I/O Interconnect: Infiniband, PCI-Express, Gibat
http://www.garlic.com/~lynn/2003.html#59 Cluster and I/O Interconnect: Infiniband, PCI-Express, Gibat
http://www.garlic.com/~lynn/2003j.html#1 FAST - Shame On You Caltech!!!
http://www.garlic.com/~lynn/2003j.html#19 tcp time out for idle sessions
http://www.garlic.com/~lynn/2003j.html#46 Fast TCP
http://www.garlic.com/~lynn/2003p.html#15 packetloss bad for sliding window protocol ?
http://www.garlic.com/~lynn/2004f.html#37 Why doesn't Infiniband supports RDMA multicast
http://www.garlic.com/~lynn/2004k.html#8 FAST TCP makes dialup faster than broadband?
http://www.garlic.com/~lynn/2004k.html#12 FAST TCP makes dialup faster than broadband?
http://www.garlic.com/~lynn/2004k.html#13 FAST TCP makes dialup faster than broadband?
http://www.garlic.com/~lynn/2004k.html#16 FAST TCP makes dialup faster than broadband?
http://www.garlic.com/~lynn/2004k.html#29 CDC STAR-100
http://www.garlic.com/~lynn/2004n.html#35 Shipwrecks
http://www.garlic.com/~lynn/2004o.html#62 360 longevity, was RISCs too close to hardware?
Anne & Lynn Wheeler
> yep :-) That's why sometimes when someone said "C langage is as
> secure as other", you can't really know if the guy is a poor C coder
> and have a bad opinion, or if the guy is a C power user and know
> that's possible to write solid code in C ( as in assembly langage),
> but you have to spend a lot of efforts, wich is not realistic in
> real life for real project. That's why there is other langages such
> as ADA , etc... I just pointed out that it's theorically "possible"
> to get the same level of security with ANY langage, even assembly.
it isn't just the high/low level of the language that helps/aids a
person in making mistakes but also had the length paradigm/semantics
are implemented.
there are a number of operating system examples that use the same
buffer length conventions as mentioned for PLI ... i.e. buffers have
headers with max/current lengths and strings have headers with current
lengths ... and the various string libraries that manipulate buffers
and strings ... honor the header lengths.
the combination of NULL-terminated strings w/o explicit lengths and
the string library implementation that frequently assumes implied
lengths or that the programmer must know what he is doing ... result
in the majority of buffer overrun/overflow problems.
In the past, I dealt extensively with assembler kernal code that used
the same buffer & string conventions as mentioned for PLI ... and the
environment had far lower incidence of buffer overflow/overrun
problems as C-language environment.
misc. past buffer related posts
http://www.garlic.com/~lynn/subpubkey.html#overrun
karl malbrain
news:m3fz2g4...@lhwlinux.garlic.com...
> "Arnaud Carré" <arnaud.ca...@freesurf.fr> writes:
> > yep :-) That's why sometimes when someone said "C langage is as
> > secure as other", you can't really know if the guy is a poor C coder
> > and have a bad opinion, or if the guy is a C power user and know
> > that's possible to write solid code in C ( as in assembly langage),
> > but you have to spend a lot of efforts, wich is not realistic in
> > real life for real project. That's why there is other langages such
> > as ADA , etc... I just pointed out that it's theorically "possible"
> > to get the same level of security with ANY langage, even assembly.
>
> it isn't just the high/low level of the language that helps/aids a
> are implemented.
And C is perfect for this.
> there are a number of operating system examples that use the same
> buffer length conventions as mentioned for PLI ... i.e. buffers have
> headers with max/current lengths and strings have headers with current
> lengths ... and the various string libraries that manipulate buffers
> and strings ... honor the header lengths.
Right. Operating systems are generally examples of well-designed C programs.
> the combination of NULL-terminated strings w/o explicit lengths and
> the string library implementation that frequently assumes implied
> lengths or that the programmer must know what he is doing ... result
> in the majority of buffer overrun/overflow problems.
NULL terminated strings are a facility for entering constants into programs
as character arrays and for passing and working constant character arrays
between/in functions. They are not designed for "buffers."
> In the past, I dealt extensively with assembler kernal code that used
> the same buffer & string conventions as mentioned for PLI ... and the
> environment had far lower incidence of buffer overflow/overrun
> problems as C-language environment.
But you are responsible for your own "C-language environment" and your own
discipline. It's very easy in C to add an object to a character pointer
that enables "buffer management" for non-constant character arrays.
karl m
Anne & Lynn Wheeler
> The string library isn't even used in buffer management.
I apologize for the misunderstanding there was in the original
post
http://www.garlic.com/~lynn/2004q.html#2 {Lit.] Buffer overruns
buffer overruns/overflows with respect to discarding characters when
things are arriving to fast ... and typically involve characters
arriving too fast
buffer overrungs/overflows involving buffer length management
issues and common exploits/vulnerabilities ... frequently associated
with the c programming language environment
dangling/orphan pointers involving buffer allocation managerment
issues and frequently system failures .... or hung/zombie processes
associated with over zeolous attempts to avoid danglin/orphan
pointers.
http://www.garlic.com/~lynn/2004q.html#3 {Lit.] Buffer overruns
I guess that i was hoping that the context of the post would provide
the ability to distrinquish buffer management as being buffer length
management as opposed to buffer allocation managerment.
the issue in the original post and mentioned in the subsequent post
http://www.garlic.com/~lynn/2004q.html#4 {Lit.] Buffer overruns
was default c programming conventions compared to some other
environments. all the PLI language implementations that i'm aware of
have explicit headers with max & current lengths ... and all the
library routines honor and maintain these header fields consistently.
in an environment where buffers don't carry their own explicit
lengths, then it is frequently pushed on to the programmer's
responsbility to perform the administrative-like tasks associated with
buffer length management operation. for infrastructures where the
length is managed as part of the infrastructure ... it is one less
mistake for the programmer to make.
note that infrastructures that maintain such explicit length paradigms
are not just limited to PLI language environment. There are some
number of system infrastructures where the default buffer length
management is with explicit headers ... and all the library rourtines
tend to conform to the system convention ... regardless of the
language ... even low-level assembler and machine languages. For
these environments, when dropping below the library level ... the
recommended coding conventions also explicitly specify managing the
buffer header length fields (if nothing else to maintain compatibility
with the rest of the environment). again, while it is possible to make
coding mistakes in such environments ... length mistakes are
significantly less common (compared to typical c language coding).
there is one other genre not previously mentioned ... the apl/lisp type
environment where the (language&operational) environment manages both
the allocation and the lengths.
long ago and far away apl\360 had real small workstaces (16k-32k
bytes) in real memory. On every allocation, new storage was allocated
from unallocated storage (but previous allocation was not
discarded). when all unallocated storage ran out, garbage collection
was run to reclaim storage not currently in use by assigned variable.
the amount of actual storage touched was proportional to the number
of assignments and the aggregate size of all variables (where it
was possible for the number of assignments to dominate over the
actual aggregate size of all variables).
when the science center ported apl\360 to cms for cms\apl ...
http://www.garlic.com/~lynn/subtopic.html#545tech
it moved it into a (relatively) large virtual memory environment
(1mbyte to 16mbytes ... typically running on 512kbytes to 1mbyte real
machines). the original apl\360 buffer allocation management tended
to touch all available virtual memory pages ... which could cause
severe virtual memory paging characteristics (even for relatively
small programs that otherwise did a large number of assignments).
random past postings, some including apl references:
http://www.garlic.com/~lynn/subtopic.html#hone
so i have an example of a buffer length coding error. Besides
inventing fair share scheduling as an undergraduate (and getting it
deployed in commercial products), i had also done tty/ascii terminal
support which was also shipped in commercial operating system.
Here is tale from somebody that modified the code to support
a non-standard tty/ascii device ... about middle of the page,
referencing the system crashing 27 times in a single day.
http://www.multicians.org/thvv/360-67.html
The way i had remembered what had happened was since tty/ascii
terminal hardware was limited, i had used one-byte arithmatic to
calculate size of incoming data (and all max lengths were well under
255). Somebody at the MIT urban lab(?) had changed their system to
support a non-standard tty/ascii terminal located some place over at
harvard ... which involved changing max. allowed length to something
like 1200. Since the base implementation calculation was using
one-byte arithmatic (0..255) and was not changed ... the length
calculations got messed up.
random other references to the even ...
http://www.garlic.com/~lynn/99.html#44 Internet and/or ARPANET?
http://www.garlic.com/~lynn/99.html#53 Internet and/or ARPANET?
http://www.garlic.com/~lynn/99.html#207 Life-Advancing Work of Timothy Berners-Lee
http://www.garlic.com/~lynn/2000.html#30 Computer of the century
http://www.garlic.com/~lynn/2000b.html#77 write rings
http://www.garlic.com/~lynn/2000f.html#60 360 Architecture, Multics, ... was (Re: X86 ultimate CISC? No.)
http://www.garlic.com/~lynn/2000g.html#2 TSS ancient history, was X86 ultimate CISC? designs)
http://www.garlic.com/~lynn/2000g.html#4 virtualizable 360, was TSS ancient history
http://www.garlic.com/~lynn/2001c.html#36 How Commercial-Off-The-Shelf Systems make society vulnerable
http://www.garlic.com/~lynn/2001f.html#78 HMC . . . does anyone out there like it ?
http://www.garlic.com/~lynn/2001g.html#52 Compaq kills Alpha
http://www.garlic.com/~lynn/2001i.html#32 IBM OS Timeline?
http://www.garlic.com/~lynn/2002f.html#38 Playing Cards was Re: looking for information on the IBM
http://www.garlic.com/~lynn/2002i.html#5 DCAS [Was: Re: 'atomic' memops?]
http://www.garlic.com/~lynn/2002l.html#56 10 choices that were critical to the Net's success
http://www.garlic.com/~lynn/2003.html#73 Card Columns
http://www.garlic.com/~lynn/2003g.html#5 Any DEC 340 Display System Doco ?
http://www.garlic.com/~lynn/2003k.html#55 S/360 IPL from 7 track tape
http://www.garlic.com/~lynn/2003p.html#23 1960s images of IBM 360 mainframes
http://www.garlic.com/~lynn/2004j.html#47 Vintage computers are better than modern crap !
http://www.garlic.com/~lynn/2004k.html#43 Vintage computers are better than modern crap !
http://www.garlic.com/~lynn/2004l.html#18 FW: Looking for Disk Calc program/Exec
http://www.garlic.com/~lynn/2004m.html#26 Shipwrecks
Anne & Lynn Wheeler
> But you are responsible for your own "C-language environment" and
> your own discipline. It's very easy in C to add an object to a
> character pointer that enables "buffer management" for non-constant
> character arrays.
the issue isn't about what might be capable of using C-language
... anymore than what might be capable of using assembler language.
In theory both C and assembler have enuf low-level constructs to
implement almost any paradigm.
the comment was that the standard C-language environment doesn't (as a
default), carry explicit length fields for all strings and buffers
... and that the standard c-language library routines don't (by
default) check the maximum length in the header of a buffer target
... and make sure that it doesn't move more data than is allowed by
what is specified in the buffer header (and therefor overflow the
buffer).
The issue was that all of the PLI language implementations (that i'm
aware of) implemented buffers with length headers and the library
routines all made sure that they didn't violate the buffer lengths
(aka an explicit attribute of a buffer ... in the header of the buffer
... is the maximum length of that buffer). While it is not impossible
to violate the buffer length ... it is much hardware to accidentially
violate such buffer lengths compared to the standard C-language
environemnt.
Futhermore, there are a number of systems where the default system
infrastructure (regardless of the language used) have paradigm that
implements buffer lengths in header fields ... and that all languages
and library routines that exist in that system environment tend to
have coding conventions that consistently use and maintain such buffer
header length fields. Again, it isn't impossibly to write assembler
code in such environments that violate buffer lengths ... but since
the default system coding conventions and operations observes the
buffer header length fields ... it tends to be a significantly lower
frequency mistake than occurs in most typical c-language environments
(where it isn't common to find all coding conventions and all library
routines that involve buffers ... implementing, maintaining, and
consistently observing buffer lengths specified in buffer header
fields).
past pieces of this thread
http://www.garlic.com/~lynn/2004q.html#2 [Lib.] Buffer overruns
http://www.garlic.com/~lynn/2004q.html#3 [Lib.] Buffer overruns
http://www.garlic.com/~lynn/2004q.html#4 [Lib.] Buffer overruns
http://www.garlic.com/~lynn/2004q.html#5 [Lib.] Buffer overruns
karl malbrain
> "karl malbrain" <kar...@acm.org> writes:
> > But you are responsible for your own "C-language environment" and
> > your own discipline. It's very easy in C to add an object to a
> > character pointer that enables "buffer management" for non-constant
> > character arrays.
>
> the issue isn't about what might be capable of using C-language
> ... anymore than what might be capable of using assembler language.
> In theory both C and assembler have enuf low-level constructs to
> implement almost any paradigm.
Thank you.
> the comment was that the standard C-language environment doesn't (as a
> default), carry explicit length fields for all strings and buffers
Since you are responsible for your own environment, YOU get to set the
default for what gets carried. Here's a simple ABSTRACTION:
typedef struct {
int len, max;
char *array;
} String;
> ... and that the standard c-language library routines don't (by
> default) check the maximum length in the header of a buffer target
Again, the standard library doesn't implement a buffer ABSTRACTION. It
implements a few supporting routines for constant character arrays. From
where are you getting other notions?
karl m
Anne & Lynn Wheeler
i've heard various stories about the null terminating by unix (say
compared to multics pli and buffer headers with lengths that were
common at the time).
one was a minimum string header tended to be two bytes (two byte fixed
length) or possibly four bytes (variable length buffer, two byte max
length, two byte current buffer contents length). Having null
termination saved one byte (compared to two byte length header on
fixed string) and saved three bytes compared by keeping track of every
buffers maximum length (aka do nothing and push it up to the
programmer and hopes he does it correctly).
this is the sort of thing from the period of saving every byte
possible in constrained real storage and resource limited
environment. This type of approach also contributed heavily to the y2k
problem ... where years were only implemented as two digit numbers
(there was actually scenario in the past where there one digit years
and problems showed up on decade roll-over ... there was situation
from rolling from the 60s to the 70s).
The other scenario is the addresses/pointers in a string processing
loop becomes a little more expensive. With null-termination ... you
pick up the start of the string and keep processing bytes until you
find a null characters (and only need the pointer to the current
character). In the byte-header scenario ... you have to have both the
current character address and the last character address (and loop
compares whether it has moved past the last character address). It can
also be done with a current character pointer and counter of remaining
characters to process (in either case, the generated machine language
tends to require an additional register).
The startup tends to be slightly more expensive ... say when copying
data or even appending data. If this is a string library appending
data to a buffer ... it has to pick up the source length from the
source string header, it has to pick up the length of the current
destination buffer contents (from the destination buffer header) and
the destination buffer maximum length (from the destination buffer
header). The append library routine then has to have an api semantics
giving either the number of characters actually appended ... or the
inverse ... the number of characters that it was unable to append.
If the api semantics is purely defined as returning characters not
copied/appended ... then the calling code has to
1) specify the origin buffer (the origin string length is an attribute
of the origin buffer, kept in the origin buffer header),
2) specify the destination buffer (the current string length in the
destiniation buffer is an attribute of the destination buffer, kept in
the destination buffer header, and the maximum length of the
destination buffer is an attribute of the destination buffer, kept in
the destination buffer header).
3) call the library append routine,
$) check for non-zero return (which would indicate some characters not
copied/appended).
So, if i'm using a standard C-programming library routine to append
one string to another ... what is the fail-safe programming required?
What is the maximum length of the destination buffer?
What is the current length of a string in the destination buffer?
What is the current length of the origin string (to be appended)?
How do i know whether all of the origin string was appended or only
part of it?
karl malbrain
> "karl malbrain" <kar...@acm.org> writes:
> > But you are responsible for your own "C-language environment" and
> > your own discipline. It's very easy in C to add an object to a
> > character pointer that enables "buffer management" for non-constant
> > character arrays.
>
> The issue was that all of the PLI language implementations (that i'm
> aware of) implemented buffers with length headers and the library
> routines all made sure that they didn't violate the buffer lengths
> (aka an explicit attribute of a buffer ... in the header of the buffer
> ... is the maximum length of that buffer). While it is not impossible
> to violate the buffer length ... it is much harder to accidentially
> violate such buffer lengths compared to the standard C-language
> environemnt.
Guess what? You get to implement your own library in C. I'm responsible
for a 130K lines of code Windows/unix application engine and we don't even
link with libc. The standard c library is of no interest to us.
> Futhermore, there are a number of systems where the default system
> infrastructure (regardless of the language used) have paradigm that
> implements buffer lengths in header fields ...
You also get to set your own "defaults" in C. Here's a simple circular
buffer abstraction:
typedef struct {
int in, out, size;
} Buffer;
that lives below (inherits) a character array of size characters above it.
> and that all languages
> and library routines that exist in that system environment tend to
> have coding conventions that consistently use and maintain such buffer
> header length fields. Again, it isn't impossible to write assembler
> code in such environments that violate buffer lengths ... but since
> the default system coding conventions
Exactly!!! Who sets the standard coding conventions where you're from?
> and operations observes the
> buffer header length fields ...
You are responsible for your own OPERATIONS from your ABSTRACTIONS.
> it tends to be a significantly lower
> frequency mistake than occurs in most typical c-language environments
> (where it isn't common to find all coding conventions and all library
> routines that involve buffers ... implementing, maintaining, and
> consistently observing buffer lengths specified in buffer header
> fields).
Making mistakes is a matter of lack of experience to hold to discipline.
karl m
Anne & Lynn Wheeler
that wasn't the issue ... in theory, i could be faced with the same
scenario in many common assembler language implementations. however,
many of the standard system infrastructures actually implemented
buffer headers as part of the default system infrastructure
... regardless of the language used in that infrastructure.
the tendancy at the time of multics pli ... and some number of other
operating systems of the era ... was more like a 16bit max, a 16bit
length followed by the actual data (predating unix and c). You got a
pointer to the actual data ... and could backup two bytes to get the
length of the current data ... or backup two more bytes and get the
maximum length of the buffer.
There were some fixed length constant strings that were only used as
source (and never destination) ... and so you only needed the actual
length (and the implementation didn't have to waste the maximum buffer
length ... because it was a constant string the current and maximum
were known to be the same).
Variable length strings/buffers had a four byte header. You got a
pointer to the buffer/string ... and could backup two bytes and get
the length of the current string or back up two more bytes and get the
maximum length of the buffer.
the four byte headers were frequently the default infrastructure
implementation for allocated and variable length buffers (two byte
maximum length followed by two byte current length). You had to go to
a different type to get larger lengths.
lots of standard libraries and infrastructures have supported this
paradigm before either unix or c were created.
karl malbrain
> "karl malbrain" <kar...@acm.org> writes:
> > typedef struct {
> > int len, max;
> > char *array;
> > } String;
>
> Variable length strings/buffers had a four byte header. You got a
> pointer to the buffer/string ... and could backup two bytes and get
> the length of the current string or back up two more bytes and get the
> maximum length of the buffer.
The same idea is still used today to create an inheritance environment.
You've got the idea -- why don't you think other programmers do also?
> the four byte headers were frequently the default infrastructure
> implementation for allocated and variable length buffers (two byte
> maximum length followed by two byte current length). You had to go to
> a different type to get larger lengths.
>
> lots of standard libraries and infrastructures have supported this
> paradigm before either unix or c were created.
I think you're reading way to much into the idea of "standard library."
karl m
Anne & Lynn Wheeler
> Guess what? You get to implement your own library in C. I'm
> responsible for a 130K lines of code Windows/unix application engine
> and we don't even link with libc. The standard c library is of no
> interest to us.
so that gets back to my comment about having done detailed vulnerability
and exploit investigation when we started ha/cmp
http://www.garlic.com/~lynn/subtopic.html#hacmp
and predicted that the standard environment would have something like
two orders of magnitude increase in buffer related problems that what
we were used to.
part of ha/cmp was writting a core of code to manage assurance,
availability, fall-over ... and to also provide high performance
distributed lock manager. An objective was to be able to run on a
standard platform and be able to offer fall-over services to a variety
of applications, including off-the-shelf applications that might run
on such platforms. As a result we didn't have control over all the
code that might run on the machine ... either at the point in time
when the code was originally shipped ... or possibly 10-15 years later
when the customer might install any arbitrary application in the
environment.
a big part of the detailed vulnerability and exploit investigation was
to identify possible failure-modes over which we had little or no
control. it identified things that we had to tightly control in our
own code implementation ... but also identified vulnerability/exploits
possibilities where there was going to be little, if any control.
for instance in this scenario ...
http://www.garlic.com/~lynn/95.html#13
we weren't going to be able to control every line of database code.
Bryan Olson
I guess that second sentence is meant to be a case in point for
the first. It's both false and irrelevant.
--
--Bryan
Anne & Lynn Wheeler
> I think you're reading way to much into the idea of "standard library."
is this "standard library" ... as in common use by the large portion
of people writing C code ... or possibly other hypothetical "standard
library" ... like in the pli/multics scnario ... and some other system
infrastructures that predate unix & c?
in the pli/multics scenario ... if all of the kernel is implemented in
pli and all of the kernel uses the pli header strings convention for
both internal kernel constructs as well as constructs that cross the
kernel/application api boundary ... if the standard kernel libraries
all support/assume the standard kernel construct, if all the system
libraries supplied for application support/assume the same standard
header construct .... then this is one form of standard library.
another form of standard library ... is what is the default use by the
largest number of programmers in the c language environment? ... and
possibly a main source of reported buffer length exploits and
vulnerabilities.
karl malbrain
> "karl malbrain" <kar...@acm.org> writes:
> > I think you're reading way to much into the idea of "standard library."
>
> is this "standard library" ... as in common use by the large portion
> of people writing C code ... or possibly other hypothetical "standard
> library" ... like in the pli/multics scnario ... and some other system
> infrastructures that predate unix & c?
You have to extend the "standard library" to support your particular
ABSTRACTIONS, in effect creating your own STANDARDS. Going back to the
String typedef example -- you write appropriate functions in support of that
data type. You design it so it's not going to "overflow".
(...)
> another form of standard library ... is what is the default use by the
> largest number of programmers in the c language environment? ... and
> possibly a main source of reported buffer length exploits and
> vulnerabilities.
The NULL terminated character array functions that are part of the standard
C library go just as far as you let them. They're not used to implement
buffers where I come from. karl m
Douglas A. Gwyn
> However, C IS generally recognized as a high level programming
> language.
Not really. It is a systems implementation language
(that can be highly portable when correctly used),
which puts it only one notch above assembly language.
Douglas A. Gwyn
> the same buffer & string conventions as mentioned for PLI ... and the
> environment had far lower incidence of buffer overflow/overrun
> problems as C-language environment.
You can use simialr conventions in your C programming.
If you don't, don't blame the language. It was
designed intentionally to leave almost all decisions
other than code generation up to the programmer.
Douglas A. Gwyn
plausible designs for handling character strings
and arbitrary data buffers. (For example, counted
strings have more space overhead than terminated
strings, an upper limit to length, and no sharing
of constant identical tails (unless even more
space and time overhead is added).) In C, you get
to choose the design you use, which might match an
existing design or be custom-tailored as you see
fit; nobody is forcing you to use a pre-existing
set of library functions if it happens not to be
appropriate for the task at hand. You can use
that freedom to duplicate any of the facilities
that you have described as more suitable; with C++
you can also use more convenient operator syntax
than is available with C.
There is also an important point that this whole
line of discussion keeps missing, namely: if the
programmer's assumptions are violated at run time,
something *unplanned* is going to happen, which is
bad from the security perspective. That is as true
with boundary-enforced buffer mechanisms as it is
for the sloppy UCB undergraduate hacks that so many
systems "borrowed" for their IP suite. At the very
least, you have a DoS vulnerabililty, but it could
be a lot worse -- since the program will execute
some "error" code that the programmer did not mean
to be executed. Imagine a medical control device
or an automotive or flight control device that
traps to a stack-trace abort when a boundary is
violated.
Douglas A. Gwyn
> largest number of programmers in the c language environment? ... and
> possibly a main source of reported buffer length exploits and
> vulnerabilities.
No. There is an International Standard for the C
programming language, part of which specifies a
standard set of library facilities that every
conforming hosted implementation of C is obliged
to provide; that is generally called "the Standard
C Library". It is by no means the only possible C
library; every specific hosted platform tends to
have its own additional system library functions,
there are many libraries that implement access to
graphics devices or other third-party equipment,
and of course the application programmer can write
his own set of library functions, which if he
knows his business will be largely portable to all
the different platforms he develops for.
No library is responsible for any abuses of it
made by the calling program. If a library function
is specified as inputting from some port and the
program tries to output using that function, it is
not the fault of the library function nor of its
design. Similarly, if a library function is
specified as traversing a data structure until a
specific terminating condition is encountered, it
is the caller's responsibility to provide it with a
properly-terminated input structure. "Garbage in,
garbage out."
Douglas A. Gwyn
> compared to multics pli and buffer headers with lengths that were
> common at the time).
Actually, Unix borrowed a widespread DEC convention.
DEC's MACRO assemblers even supported the type
directly via the .ASCIZ (ASCII terminated by Zero
byte) directive. There are advantages and
disadvantages to this design. Reasonably careful
coding always ensures that a null terminator is
present, within the allocated space for the data.
> this is the sort of thing from the period of saving every byte
> possible ...
No, that's just one of several advantages. If you
code complete string libraries for both approaches,
you will find that the terminated-string approach
is considerably faster for many common operations
(not for concatenation, however), and that it is
simpler to implement correctly.
> environment. This type of approach also contributed heavily to the y2k
> problem ... where years were only implemented as two digit numbers
> (there was actually scenario in the past where there one digit years
> and problems showed up on decade roll-over ... there was situation
> from rolling from the 60s to the 70s).
The Y2K problems were entirely unrelated to whether
terminated strings were used. In fact with self-
delimiting strings there is less inclination for
the programmer to allocate a fixed-width field.
> So, if i'm using a standard C-programming library routine to append
> one string to another ... what is the fail-safe programming required?
You are presupposing that those are arbitrary
strings about whose lengths you know nothing in
advance. If your program has produced such a
situationm it is already at fault. Anyway, no
matter what, space must be allocated to hold
the result of the concatenation, either in one
of the input arrays or in a separate output array.
So, just allocate enough space (simple arithmetic
based on the lengths of the inputs, which are
easily obtained using the standard function strlen).
> What is the maximum length of the destination buffer?
> What is the current length of a string in the destination buffer?
> What is the current length of the origin string (to be appended)?
> How do i know whether all of the origin string was appended or only
> part of it?
There is no problem in obtaining suitable answers
to such questions in coding C programs.
The discussion was originally about buffers, not
strings, and any sensible implementation of buffer
management is not going to write beyond the end of
the available space. The buffer-overrun problems
stem from careless implementations, and would
occur one way or another in any PL, although the
effects differ depending on the PL and other
specific details.
Douglas A. Gwyn
> issues and common exploits/vulnerabilities ... frequently associated
> with the c programming language environment
That is what I (and Karl M in an associated thread)
was referring to, and you're wrong about it. Buffer
length can be managed perfectly well using C, and if
it isn't, blame the programmer for not doing his job.
It isn't C's job to impose assumptions about your
application upon you; if you try to use it to do
harmful things, then harmful things will occur, much
like using a sharp knife (which expert knife users
natually much prefer over safety-enforced knives).
Douglas A. Gwyn
> I guess that second sentence is meant to be a case in point for
> the first. It's both false and irrelevant.
It was both true and relevant (check upthread
to see the string library specifically mentioned).
Ben Pfaff
> Anne & Lynn Wheeler wrote:
>> What is the maximum length of the destination buffer?
>
> There is no problem in obtaining suitable answers
> to such questions in coding C programs.
It is (normally) easy to obtain a suitable answer to this
question in the caller of a string function. It is not possible
to do so in the callee, in a "normal" C implementation, if the
caller does not provide the answer. This is a big source of
trouble.
--
Ben Pfaff
email: b...@cs.stanford.edu
web: http://benpfaff.org
David Wagner
>No library is responsible for any abuses of it
>made by the calling program.
Not even gets(), where the only safe way to use gets() is not to use it?
David Wagner
>There are advantages and disadvantages to all the
>plausible designs for handling character strings
>and arbitrary data buffers. (For example, counted
>strings have more space overhead than terminated
>strings, an upper limit to length, and no sharing
>of constant identical tails (unless even more
>space and time overhead is added).)
Counted strings can share constant identical tails.
The upper limit on the length of a counted string can be made
essentially identical to the upper limit on (uncounted) C strings
-- namely, the amount of the virtual memory allocated to the process.
>In C, you get to choose the design you use, [..]
Yes, except that the C standard libraries -- which, I note, are part
of the C language definition -- exert a strong pressure to use (uncounted,
NUL-terminated) C strings.
Nobody is forcing you to use the C standard library, but then again,
nobody is forcing you to use C, either.
David Wagner
>No, that's just one of several advantages. If you
>code complete string libraries for both approaches,
>you will find that the terminated-string approach
>is considerably faster for many common operations
>(not for concatenation, however), and that it is
>simpler to implement correctly.
Ease of implementation of the *library* is not a particularly compelling
reason to choose one kind of string termination method over another.
C-style (terminated) strings might make the library programmer's life
easier, but it makes the life of everyone who uses that library harder.
When you consider that a string library only needs to be written once,
but is used many times, this starts to sound like a bad tradeoff.
David Wagner
>You can use simialr conventions in your C programming.
>If you don't, don't blame the language.
I don't blame the language. I blame the library.
Douglas A. Gwyn
There are safe-enough uses for gets,
but of course not for processing *arbitrary*
input files. If you stupidly use gets in a
security-critical application to input a line
of text that could be arbitrarily long, then
it's your own damn fault. There are other
widely available facilities available for use
in such circumstances, or you could write
your own (it's not hard) and use that.
Douglas A. Gwyn
> Counted strings can share constant identical tails.
Not if the count field is adjacent to the data (which
is the usual implementation). You'd need to use
string "descriptors" containing (length,pointer) as
well as the data allocations themselves. I alluded
to this (without giving details) in my previous posting.
> Yes, except that the C standard libraries -- which, I note, are part
> of the C language definition -- exert a strong pressure to use (uncounted,
> NUL-terminated) C strings.
Not for inappropriate uses. If you feel such pressure
then something is radically wrong with your environment.
Douglas A. Gwyn
> Ease of implementation of the *library* is not ...
I gave that as an obvious example of detailed programming
dependent on the representation. It also is directly
reflected in the code being smaller and faster, which
*can* be important, depending on the circumstances.
> C-style (terminated) strings might make the library programmer's life
> easier, but it makes the life of everyone who uses that library harder.
It is no harder to use the terminated-string library
than a similar counted-string library. There are
interface details that could be made better, but that
is not a function of the representation.
Of course in C++ one can do a better job, but that
would be equally true for both designs.
David Wagner
>David Wagner wrote:
>> Counted strings can share constant identical tails.
>
>Not if the count field is adjacent to the data
So don't put the count field adjacent to the data, then.
Patient: "Doctor, doctor, it hurts when I hit my knee!"
Doctor: "Don't hit your knee, then."
It is trivial to come up with a string type that doesn't have the "can't
share tails" disadvantage you list.
struct { size_t count; char *string; } counted_string;
The problem is not with the concept of counted strings; the problem
is with poor implementations of counted strings. In short, the "can't
share constant identical tails" criticism was a criticism of a strawman.
David Wagner
>It is no harder to use the terminated-string library
>than a similar counted-string library.
I agree.
It's too bad the C language specified a terminated-string library
as part of the language specification, rather than a counted-string
library. We might have seen fewer buffer overruns if the latter had
been the case.
Anne & Lynn Wheeler
> was referring to, and you're wrong about it. Buffer
> length can be managed perfectly well using C, and if
> it isn't, blame the programmer for not doing his job.
> It isn't C's job to impose assumptions about your
> application upon you; if you try to use it to do
> harmful things, then harmful things will occur, much
> like using a sharp knife (which expert knife users
> natually much prefer over safety-enforced knives).
which is my question from
http://www.garlic.com/~lynn/2004q.html#8
given either copy from source to destination buffer ... or append source
to destination buffer ... the buffer header paraigm has the programmer
specifying
1) the source string/buffer ... where the source length is an attribute
of the source string/buffer in the header field
2) the destinatiion buffer ... where the destination buffer currently
occupied length (in the case of append) is an attribute of the destination
buffer head and the maximum buffer length is an attribute of the destination
buffer
3) call the library append/copy/move routine
4) check for non-zero return (which would indicate some characters
not copied/appended).
the observation is that
1) some number of systems and languages prior to creation of unix & c
implemented such paradigms ...
2) some of these implementations continue to exist today
3) these implementations have tended to have two orders magnitude
fewer of the common buffer overflow vulnerabilities/exploits seen
when using C language implementations.
the prediction was that because of the common C language programming
conventions ... that C language inplemented infrastuctures would tend
to have to have two orders greater buffer related problems.
there was no claim that c language couldn't be used to implement buffer
safe implementations ... the claim was that buffer unsafe implementations
were so much easier in C ... that it would contribute to a significant
increase in buffer length related vulnerabilities/exploits.
I'm only observing that the air force pli/multics security study claimed
that there were no buffer length related problems .... while there are
significant number of buffer length related problems in C implementations.
My contention was that default/standard C programming conventiosn
contribute to this significant number of buffer length related
problems and by comparison there are other infrastructures with
default/standard probramming conventions with signifantly lower buffer
length related problems.
furthermore the default/standard library and programming conventions
can be independent of the programming language ... where at least
some environments where default/standard library and programming conventions
are the same for assembler, pli. and a number of other languages
(being more a characteristic of the infrastructure rather than any
specific language).
Anne & Lynn Wheeler
> terminated strings were used. In fact with self-
> delimiting strings there is less inclination for
> the programmer to allocate a fixed-width field.
I never intended to imply that Y2K issues were related in any way to
null terminated strings. The example I heard was that null terminated
strings conserved some bytes compared to the explicit header length
paradigm ... and that Y2k issues arose from similar issues regarding
conservation bytes.
Using null terminated string might save 1-3 bytes compared to an
explicit length implementation .... using two (or one) digit for year
could conserv 2(-3) bytes compared to an implementation using 4 digits
for years.
In the 60s and at least early 70s, there was a lot more effort to
do implementations that conserved bytes ... potentially at the
expensve of something else
the original statement from
http://www.garlic.com/~lynn/2004q.html#8
wasn't intended to claim that all things might use null terminated
string ... but that null terminated string was (possibly) justified
because it used less storage (conserved 1-3 bytes possibly compared
to explicit buffer header ... where the length of the buffer is
now carried as an explicit attribute of the buffer) ... and that
many of the Y2k problems arose from efforts in the same era attempting
to conserve real storage.
the statements had more to do with trade-offs made in one era with
conserving/optimizing some resource ... could have significant later
repercusions. I would claim that the a lot of the efforts in the 60s
to use two-byte year fields (rather than 4byte) were done to conserve
storage ... and that optimization led to many of the y2k problems.
I would by analogy argue that the performance/conservation trade-off
of using null terminated strings (based on trade-off choice that it
used less storage than explicit length headers) contributes to the
signficant difference in the number of buffer length vulnerabilities
found in standard c language coding ... compared to the
number/frequency of buffer length vulnerabilities found in
infrastructures that utilize buffer headers with explicit length as
standard coding convention
Again, the statement isn't about what the buffer length qfail-safe way
would be to write c language code ... but whether comparing two
default, coding conventions ... one pervasis C coding conventions and
say PLI coding conventions (with explicit lengths attributes carried
with buffers) and the sigificant difference in the number of buffer
length problems ... could the diffence between explicit buffer length
convention compared to the null-termination convention account for most
of the difference
Anne & Lynn Wheeler
> made by the calling program. If a library function
> is specified as inputting from some port and the
> program tries to output using that function, it is
> not the fault of the library function nor of its
> design. Similarly, if a library function is
> specified as traversing a data structure until a
> specific terminating condition is encountered, it
> is the caller's responsibility to provide it with a
> properly-terminated input structure. "Garbage in,
> garbage out."
the original assertion was about the common coding conventions
that occur using the null terminated string convention versis
the explicit buffer length attribute convention.
i would assert that the convention i gave for buffers with explicit
lengths and buffer with buffer length attributes leads to fewer
programming mistakes ... because the programmer has less code to write
... whether the mistakes are classified as notoriously prevalanent and
common abuses ... or just simple mistakes.
again a repeat from the buffer header scenario
http://www.garlic.com/~lynn/2004q.html#8
the default standard infrastructures and libraries result in
the programmer specifying (for copy/move/append)
1) the source string/buffer .... the header field of the string/buffer
carries the length as an integral attribute of the source
2) the destination string/buffer .... the header filed of the string/buffer
has any current occupied length was well as maximum length
3) calls the appropriate copy/move/append routine
4) checks the return for non-zero indicating the number of bytes not
copy/move/append
the destination buffer has two header length fields ... one the
current size (if any) and the other the maximum size. in the case of
an append ... the append operation starts at the end of whatever is
currently in the buffer and continues to the end of the source buffer
or the end of the destination buffer (which ever comes first). If it
is a move/copy, the library routine can start at the beginning of the
destination buffer and continue until the end of the source buffer or
the end of the destination buffer ... which ever comes first. The
return code can indicate the number of source bytes not moved to the
destination buffer ... if any.... or zero if all source bytes are moved
to the destination buffer.
the library routine won't move characters passed the end of the
destination buffer because the maximum length of the destination
buffer is an explicit attribute of the destination buffer ... carried
in the header of the destination buffer.
so i would claim, that at a minimum the default, standard, prevalent
coding conventions in used for C ... would require a number of additional
operations to achieve fail-safe buffer length operations and management
... and at the very least ... if each individual programmer has to do
more work ... then there is an increase in the number of mistakes.
The buffer header paradigm infrasture and their associated standard
library routines doesn't leave it up to the programmer to make the
mistake or abuse.
Anne & Lynn Wheeler
> line of discussion keeps missing, namely: if the
> programmer's assumptions are violated at run time,
> something *unplanned* is going to happen, which is
> bad from the security perspective. That is as true
> with boundary-enforced buffer mechanisms as it is
> for the sloppy UCB undergraduate hacks that so many
> systems "borrowed" for their IP suite. At the very
> least, you have a DoS vulnerabililty, but it could
> be a lot worse -- since the program will execute
> some "error" code that the programmer did not mean
> to be executed. Imagine a medical control device
> or an automotive or flight control device that
> traps to a stack-trace abort when a boundary is
> violated.
my statements weren't intended to replect theoritical conditions
... my statements were intended to relfect that there has been a
signficant difference in the actual observed occurances of buffer
related vulnerabilities/exploits in implementations done with standard
C language as compared to implementations done using buffer header
paradigm.
the assertion is that programmers make signficantly fewer buffer
length abuses/mistakes in environments where there are explicit buffer
length headers ... compared to the frequency of buffer length
abuses/mistakes using standard C language environments.
It is more than a theoritical mistake/abuse/vulnerability/exploit. It
is like if you were getting 1000 fatalities per million miles driven
in one specific kind of vehical ... and 10 fatalities per million
miles driven in another specific kind of vehical ... that it might be
worth consider changing vehicles. This is dispite somebody observing
that it was posssible for either vohicle could still go off a mountain
road and kill everybody.
Bryan Olson
> That is what I (and Karl M in an associated thread)
> was referring to, and you're wrong about it. Buffer
> length can be managed perfectly well using C, and if
> it isn't, blame the programmer for not doing his job.
I saw a PBS documentary on automotive safety in which they explained
that in the early days of the industry, cars were far more dangerous but
people didn't think much about making them safer. The root cause of the
vast majority of the casualties was "driver error". Blame the drivers,
not the machines.
I had to laugh, realizing that the thinking in my own profession is the
better part of a century behind the auto industry. What kind of
engineer would knowingly select a design where common human errors can
easily slip by, and such slips frequently cause disaster?
> It isn't C's job to impose assumptions about your
> application upon you; if you try to use it to do
> harmful things, then harmful things will occur, much
> like using a sharp knife (which expert knife users
> natually much prefer over safety-enforced knives).
Professionals prefer, and demand, sharp tools with practical and
effective safety features. If serious injuries are common, we don't
just blame people for their incompetence; we re-engineer the system so
that accidents are less frequent and less severe.
--
--Bryan
Bryan Olson
> There is also an important point that this whole
> line of discussion keeps missing, namely: if the
> programmer's assumptions are violated at run time,
> something *unplanned* is going to happen, which is
> bad from the security perspective.
And no one is worse about missing this point than Mr. Gwyn. Compared to
C, many languages both reduce the chance of such a violation and
mitigate the damage if it does happen. C is type-weak, unchecked, and
in many common situations punts to undefined and arbitrarily bad run-time
behavior.
--
--Bryan
Phil Carmody
What about when you've freopened stdin from a pipe shared with your own fork?
Unless you're claiming that your program is not in control of its own
_output_ too - in which case you have bigger problems than the use of
gets().
Phil
--
God was my co-pilot but we crashed in the mountains and I had to eat him.
Mok-Kong Shen
Douglas A. Gwyn wrote:
If you loop up any textbook on programming languages
(or perhaps better on history of programming languages)
you would find that your esteemed opinion is not shared.
A number of programming languages have extensive system
implementation usage, e.g. Modula, as far as I am aware.
M. K. Shen
Mok-Kong Shen
Douglas A. Gwyn wrote:
The last sentence seems to be an excellently subtle way of
expressing the non-safety of C, as the term is commonly
understood ;-)
M. K. Shen
Mok-Kong Shen
Douglas A. Gwyn wrote:
> ...... Imagine a medical control device
> or an automotive or flight control device that
> traps to a stack-trace abort when a boundary is
> violated.
I suppose that the project ADA was started because DOD
was in quest of a 'safe' programming language for embedded
systems.
M. K. Shen
Mok-Kong Shen
Douglas A. Gwyn wrote:
[snip]
> It isn't C's job to impose assumptions about your
> application upon you; if you try to use it to do
> harmful things, then harmful things will occur, much
> like using a sharp knife (which expert knife users
> natually much prefer over safety-enforced knives).
It is interesting to recall here an analogy in an apparent
difference in thinking between Europeans and Americans:
In Europe (and many other parts of the world) the possession
of guns is very much severely restricted than in the US.
M. K. Shen
Arnaud Carré
> > designed intentionally to leave almost all decisions
> > other than code generation up to the programmer.
>
> The last sentence seems to be an excellently subtle way of
> expressing the non-safety of C, as the term is commonly
> understood ;-)
Agree, that's why I said in previous post that C is (to my opinion) a low
level langage ( but I really love C !). To me, it's a universal
macro-assembler without registers contrainst. You have to take care of
everything but the code generation. ( as with a macro assembler !)
( don't fight me and read my previous post, I've not said "C == assembler"
LOL )
Mok-Kong Shen
Nobody in our group would be concerned enough by your view
points to hurt you physically, but many in the language
committee for C would feel highly offended, I am afraid.
M. K. Shen
Anne & Lynn Wheeler
> Agree, that's why I said in previous post that C is (to my opinion)
> a low level langage ( but I really love C !). To me, it's a
> universal macro-assembler without registers contrainst. You have to
> take care of everything but the code generation. ( as with a macro
> assembler !)
some of this is coding conventions ... not necessarily regardless
of whether or not you can violate rules if you need to.
the claim is that null terminated convention is open to larger number
of programmer mistakes (as opposed to purposeful abuses) compared to
buffer explicit lengths (both current length and maximum buffer
length).
i claim that there are infrastructures implemented in assembler (at a
lower level than C) where there are significant fewer buffer overrun
... not because of the language characteristics but because of the
standard infrastructure environment and coding conventions.
--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/
karl malbrain
news:cp9b21$jjf$02$1...@news.t-online.com...
1. It's a reflection of the groundwork you get.
2. You are responsible for your own ABSTRACTIONS that you build on that
groundwork.
3. The "string" standard C library functions don't implement "buffers."
4. Safety is built-in from the bottom up from day one as a discipline.
karl m
karl malbrain
news:m31xdz7...@lynnvaio.garlic.com...
> my statements weren't intended to replect theoritical conditions
> ... my statements were intended to relfect that there has been a
> signficant difference in the actual observed occurances of buffer
> related vulnerabilities/exploits in implementations done with standard
> C language as compared to implementations done using buffer header
> paradigm.
And that's why we have DIRECTORS, SUPERVISORS, and O-J-T APPRENTICE programs
to teach METHODS and STANDARDS in the trade.
> the assertion is that programmers make signficantly fewer buffer
> length abuses/mistakes in environments where there are explicit buffer
> length headers ... compared to the frequency of buffer length
> abuses/mistakes using standard C language environments.
Bzzt. There is no standard "buffer" C language environment.
karl m
karl malbrain
> "Douglas A. Gwyn" <DAG...@null.net> writes:
> > No library is responsible for any abuses of it
> > made by the calling program. If a library function
> > is specified as inputting from some port and the
> > program tries to output using that function, it is
> > not the fault of the library function nor of its
> > design. Similarly, if a library function is
> > specified as traversing a data structure until a
> > specific terminating condition is encountered, it
> > is the caller's responsibility to provide it with a
> > properly-terminated input structure. "Garbage in,
> > garbage out."
>
> the original assertion was about the common coding conventions
> the explicit buffer length attribute convention.
There is no direct connection between "NULL terminated string convention"
and "buffer length attribute convention." They are different ABSTRACTIONS
with different attributes.
> I would assert that the convention I gave for buffers with explicit
> lengths and buffer with buffer length attributes leads to fewer
> programming mistakes ... because the programmer has less code to write
> ... whether the mistakes are classified as notoriously prevalanent and
> common abuses ... or just simple mistakes.
Exactly! C is your best language to do just that. You can create your own
STANDARD buffer typedef and the methods that go with it, just like you do
below:
> Again a repeat from the buffer header scenario
> http://www.garlic.com/~lynn/2004q.html#8
>
> The default standard infrastructures and libraries result in
> the programmer specifying (for copy/move/append):
>
> 1) the source string/buffer .... the header field of the string/buffer
> carries the length as an integral attribute of the source;
>
> 2) the destination string/buffer .... the header filed of the
string/buffer
> has any current occupied length was well as maximum length;
>
> 3) calls the appropriate copy/move/append routine;
>
> 4) checks the return for non-zero indicating the number of bytes not
> copy/move/append;
>
> The destination buffer has two header length fields ... one the
> current size (if any) and the other the maximum size. In the case of
> an append ... the append operation starts at the end of whatever is
> currently in the buffer and continues to the end of the source buffer
> or the end of the destination buffer (which ever comes first). If it
> is a move/copy, the library routine can start at the beginning of the
> destination buffer and continue until the end of the source buffer or
> the end of the destination buffer ... which ever comes first. The
> return code can indicate the number of source bytes not moved to the
> destination buffer ... if any.... or zero if all source bytes are moved
> to the destination buffer.
>
> The library routine won't move characters passed the end of the
> destination buffer because the maximum length of the destination
> buffer is an explicit attribute of the destination buffer ... carried
> in the header of the destination buffer.
>
> So i would claim, that at a minimum the default, standard, prevalent
> coding conventions in used for C ... would require a number of additional
> operations to achieve fail-safe buffer length operations and management
> ... and at the very least ... if each individual programmer has to do
> more work ... then there is an increase in the number of mistakes.
>
> The buffer header paradigm infrasture and their associated standard
> library routines doesn't leave it up to the programmer to make the
> mistake or abuse.
Oh, but it did: YOU did it! It's trivial in C to turn your descriptions into
a STANDARD PACKAGE. karl m
karl malbrain
>
> Using null terminated string might save 1-3 bytes compared to an
> explicit length implementation .... using two (or one) digit for year
> for years.
And, on an 80 column holerith coded punch card, 2-3 bytes are 4%.
>
> In the 60s and at least early 70s, there was a lot more effort to
> do implementations that conserved bytes ... potentially at the
> expensve of something else
Right. Changing times, changing technologies, changing ABSTRACTIONS.
> the original statement from
> http://www.garlic.com/~lynn/2004q.html#8
>
> wasn't intended to claim that all things might use null terminated
> strings ... but that a null terminated string was (possibly) justified
> because it used less storage (conserved 1-3 bytes possibly compared
> to explicit buffer header ... where the length of the buffer is
> now carried as an explicit attribute of the buffer) ... and that
> many of the Y2k problems arose from efforts in the same era attempting
> to conserve real storage.
Again, punch card technology was still in use.
> the statements had more to do with trade-offs made in one era with
> conserving/optimizing some resource ... could have significant later
> repercusions. I would claim that the a lot of the efforts in the 60s
> to use two-byte year fields (rather than 4byte) were done to conserve
> storage ... and that optimization led to many of the Y2k problems.
And this relates to you criticism of the C language environment, exactly
how....
> I would by analogy argue that the performance/conservation trade-off
> of using null terminated strings (based on trade-off choice that it
> used less storage than explicit length headers) contributes to the
> signficant difference in the number of buffer length vulnerabilities
> found in standard c language coding ... compared to the
> number/frequency of buffer length vulnerabilities found in
> infrastructures that utilize buffer headers with explicit length as
> standard coding convention
But, NULL terminated strings are not a valid ABSTRACTION for buffers.
karl m
karl malbrain
But we agreed elsewhere that it's not the C language, as thing-in-itself,
that is responsible. What's missing is people like you SETTING STANDARDS!
> The prediction was that because of the common C language programming
> conventions ... that C language inplemented infrastuctures would tend
> to have to have two orders greater buffer related problems.
So, the question becomes, why can't people like you be involved in setting
STANDARDS for programmers to follow?
> There was no claim that the c language couldn't be used to implement
buffer
> safe implementations ... the claim was that buffer unsafe implementations
> were so much easier in C ... that it would contribute to a significant
> increase in buffer length related vulnerabilities/exploits.
But, EVERYTHING'S EASIER IN C, that's why we use it!
>
> I'm only observing that the Air Force PL1/Multics security study claimed
> that there were no buffer length related problems .... while there are
> significant number of buffer length related problems in C implementations.
Due to people with the experience and capable of setting STANDARDS being
isolated from the software production process.
> My contention was that default/standard C programming conventions
> contribute to this significant number of buffer length related
> problems and by comparison there are other infrastructures with
> default/standard probramming conventions with signifantly lower buffer
> length related problems.
Bzzt. There is no "standard C" ABSTRACTION for buffers.
> furthermore the default/standard library and programming conventions
> can be independent of the programming language ... where at least
> some environments where default/standard library and programming
conventions
> are the same for assembler, PL1. and a number of other languages
> (being more a characteristic of the infrastructure rather than any
> specific language).
P.s. I once worked along side Karen who was implementing PL1 for the Air
Force on the CDC 7600 in Sunnyvale. karl m
karl malbrain
"David Wagner" <d...@taverner.cs.berkeley.edu> wrote in message
news:cp8k8o$1v67$4...@agate.berkeley.edu...
But there is no "the library." karl m
karl malbrain
> Douglas A. Gwyn wrote:
> >No, that's just one of several advantages. If you
> >code complete string libraries for both approaches,
> >you will find that the terminated-string approach
> >is considerably faster for many common operations
> >(not for concatenation, however), and that it is
> >simpler to implement correctly.
>
> Ease of implementation of the *library* is not a particularly compelling
> reason to choose one kind of string termination method over another.
> C-style (terminated) strings might make the library programmer's life
> easier, but it makes the life of everyone who uses that library harder.
> but is used many times, this starts to sound like a bad tradeoff.
Of course it is. Take the C language crypto libraries from several
sci.crypt contributors for example. They get to extend the "standard C"
library. karl m
Anne & Lynn Wheeler
> I saw a PBS documentary on automotive safety in which they explained
> that in the early days of the industry, cars were far more dangerous
> but people didn't think much about making them safer. The root
> cause of the vast majority of the casualties was "driver error".
> Blame the drivers, not the machines.
>
> I had to laugh, realizing that the thinking in my own profession is
> the better part of a century behind the auto industry. What kind of
> engineer would knowingly select a design where common human errors
> can easily slip by, and such slips frequently cause disaster?
i've periodically used the analogy to after-market seatbelts
... everybody could install their own seatbelts if they needed ... so
why was it necessary to have manufactors put seatbelts in cars ... or
school buses, or a variety of other vehicles.
Mok-Kong Shen
karl malbrain wrote:
I was here commenting in a 'general' way on Gwyn's sentence
that expressed his general view on C. If one compares C e.g.
with ADA, the safety that can be incorporated into a language
should be apparent. Safety often has a non-trivial cost and
so some people would do a trade-off in perferring less cost
(because one is very sure of one's competence in programming
or for other reasons). Yet the fact remains that safety is
a 'characteristic' that frequently distinguishes one PL from
another.
M. K. Shen
karl malbrain
There is nothing inherently unsafe about the C language.
I think you're looking for a programming language you can turn someone lose
with and they can figure out programming all on their own, with no
"pre-regard" for safety, or any other guidance from someone who knows what
they're doing. You wouldn't advocate letting Medical Doctors loose on that
basis, would you? Why do you think programming is all that different?
karl m
Mok-Kong Shen
karl malbrain wrote:
> There is nothing inherently unsafe about the C language.
>
> I think you're looking for a programming language you can turn someone lose
> with and they can figure out programming all on their own, with no
> "pre-regard" for safety, or any other guidance from someone who knows what
> they're doing. You wouldn't advocate letting Medical Doctors loose on that
> basis, would you? Why do you think programming is all that different?
Since you mention medicine, I recall that decades ago (I am
not informed of current status) in certain regions in Asia
one could get all kinds of medicine without presenting any
prescription from doctors. Do you think that's a good thing?
(In Europe and I assume also in America there are laws
forbidding pharmacies selling medicaments like certain
antibiotics freely to anybody who want them.)
M. K. Shen
karl malbrain
>
>
> karl malbrain wrote:
>
>
> > There is nothing inherently unsafe about the C language.
> >
> > I think you're looking for a programming language you can turn someone
lose
> > with and they can figure out programming all on their own, with no
> > "pre-regard" for safety, or any other guidance from someone who knows
what
> > they're doing. You wouldn't advocate letting Medical Doctors loose on
that
> > basis, would you? Why do you think programming is all that different?
>
> Since you mention medicine, I recall that decades ago (I am
> not informed of current status) in certain regions in Asia
> one could get all kinds of medicine without presenting any
> prescription from doctors. Do you think that's a good thing?
Of course not -- the root of the problem being that there aren't enough
trained doctors to go around -- a temporary/emergency/historical situation,
hopefully. karl m
Douglas A. Gwyn
> It is trivial to come up with a string type that doesn't have the "can't
> share tails" disadvantage you list.
I already covered that, and mentioned again that I had,
and you keep ignoring it. There is an additional
penalty paid in order to obtain that property. As
I have said many times now, there are trade-offs, and
the best design choice depends on the requirements.
Douglas A. Gwyn
> It's too bad the C language specified a terminated-string library
> as part of the language specification, rather than a counted-string
> library. We might have seen fewer buffer overruns if the latter had
> been the case.
Buffer overruns are not due to the design of the str*
functions, but rather to programmer oversight and/or
laziness. Most buffer overrun problems have nothing
to do with the str* functions. This noise about the
str* functions distracts from addressing the *real*
problem.
Douglas A. Gwyn
> implemented such paradigms ...
> 2) some of these implementations continue to exist today
And many of them are coded in C on Unix systems.
> 3) these implementations have tended to have two orders magnitude
> fewer of the common buffer overflow vulnerabilities/exploits seen
> when using C language implementations.
They also tend to have at least two (decimal) orders
of magnitude less use than C implementations.
> the prediction was that because of the common C language programming
> conventions ... that C language inplemented infrastuctures would tend
> to have to have two orders greater buffer related problems.
And that is wrong. To what "common C language programming
conventions" are you referring? It is not a "convention"
to incorrectly code an algorithm.
> there was no claim that c language couldn't be used to implement buffer
> safe implementations ... the claim was that buffer unsafe implementations
> were so much easier in C ...
Most systems programming is easier in C. Think of it
as an amplifier for the programmer's intentions; if the
programmer is incompetent then that is also amplified.
> My contention was that default/standard C programming conventiosn
> contribute to this significant number of buffer length related
> problems ...
No, they don't.
Douglas A. Gwyn
> coding conventions in used for C ... would require a number of additional
> operations to achieve fail-safe buffer length operations and management
No, not unless you are aware of some utterly bad prevalent
coding conventions that I have not heard of. All the
experienced C programmers I know of, and many novice ones,
would naturally encapsulate the buffer data type and use
the associated functions to perform operations on it,
quite safely. My own implementation is tied into the
nested exception handler facility, which makes buffer use
even easier, since there is no need to explicitly test for
exceptional conditions at each operation.
Douglas A. Gwyn
> Professionals prefer, and demand, sharp tools with practical and
> effective safety features. If serious injuries are common, we don't
> just blame people for their incompetence; we re-engineer the system so
> that accidents are less frequent and less severe.
If you find that surgeons are butchering people,
it is *inappropriate* to replace their scalpels
with something "safer". That is *not* where the
problem lies.
What is this tendency to blame tools rather than
hold people responsible for doing things right in
a way that the tools fully support?
C lets you build and use safe buffer managers.
If you don't do so, it isn't C's fault.
Douglas A. Gwyn
> Douglas A. Gwyn wrote:
>> Mok-Kong Shen wrote:
>>> However, C IS generally recognized as a high level programming
>>> language.
>> Not really. It is a systems implementation language
>> (that can be highly portable when correctly used),
>> which puts it only one notch above assembly language.
> If you loop up any textbook on programming languages
> (or perhaps better on history of programming languages)
> you would find that your esteemed opinion is not shared.
I'm not responsible for erroneous ideas you might
glean from some textbooks. I am, however, an expert
in the C programming language, and it is as I have
described it.
Mok-Kong Shen
karl malbrain wrote:
>>
>>Since you mention medicine, I recall that decades ago (I am
>>not informed of current status) in certain regions in Asia
>>one could get all kinds of medicine without presenting any
>>prescription from doctors. Do you think that's a good thing?
>
>
> Of course not -- the root of the problem being that there aren't enough
> trained doctors to go around -- a temporary/emergency/historical situation,
> hopefully. karl m
In the case I mentioned it was because that the politicians
for some political reasons didn't want to face up with (deal
with) that matter. (There were sufficient doctors around.)
M. K. Shen
Douglas A. Gwyn
> Douglas A. Gwyn wrote:
>> If you don't, don't blame the language. It was
>> designed intentionally to leave almost all decisions
>> other than code generation up to the programmer.
> The last sentence seems to be an excellently subtle way of
> expressing the non-safety of C, as the term is commonly
> understood ;-)
It reflects the low level (systems implementation)
of the language. One can use C to build other tools,
and indeed almost all other PLs in use these days are
built using C to a greater or lesser extent. If you
try to construct a huge system without building any
tools to help (like "jigs" in tooling), you aren't
proceeding very intelligently. If you are clever,
you design these extra tools so that many of them can
be used later in other projects as well.
Douglas A. Gwyn
> Nobody in our group would be concerned enough by your view
> points to hurt you physically, but many in the language
> committee for C would feel highly offended, I am afraid.
Actually I am on that committee, and Arnaud is right.
Mok-Kong Shen
Douglas A. Gwyn wrote:
> .... All the
> experienced C programmers I know of, and many novice ones,
> would naturally encapsulate the buffer data type and use
> the associated functions to perform operations on it,
> quite safely.
[snip]
So you think that the problems indicated e.g. in the paper
I referred to stemmed from certain software manufacturers
having the exceptional bad luck to have picked the entirely
wrong programmers (those even much poorer than some novices
you know of)?
M. K. Shen
Douglas A. Gwyn
> of programmer mistakes (as opposed to purposeful abuses) compared to
> buffer explicit lengths (both current length and maximum buffer
> length).
But to handle buffers of the kind found in nearly all
of the IP suite, one would never use the null-terminated
string support functions to handle the buffers, for the
obvious reason that many of the data octets in such
buffers can have zero values.
The objections I keep hearing apparently come from
people who don't know enough about C programming to
make such judgments. Perhaps whatever problems have
occurred are at least partly due to using programmers
who have such a level of (non)understanding of their
tools.
Mok-Kong Shen
Douglas A. Gwyn wrote:
> I'm not responsible for erroneous ideas you might
> glean from some textbooks. I am, however, an expert
> in the C programming language, and it is as I have
> described it.
I see. You are an expert in C and so everything you say
IS true.
M. K. Shen
karl malbrain
Then perhaps the problem was a lack of control over the doctors; or rather,
a lack of money to exert control over where their medicine was practiced --
hence the "shortcut".
In my opinion, the problem with training programmers is that it's not being
done as a trade, with an apprenticeship. Companies like MS can just hire
temporary workers for the immediate task-at-hand, and cut them loose. karl
m
Douglas A. Gwyn
> I suppose that the project ADA was started because DOD
> was in quest of a 'safe' programming language for embedded
> systems.
Actually its main motivation was to replace the
plethora of special PLs that had been used for
such systems, JOVIAL being perhaps the best known.
The idea was that using a single language would
make it easier for "random" DoD programmers to
understand and maintain somebody else's code.
In fact, using Ada by no means ensures that such
errors as overrunning buffers will not occur.
The belief that safety can be guaranteed by choice
of PL is worse than misguided -- it can encourage
thoughtlessness.
Mok-Kong Shen
Douglas A. Gwyn wrote:
He said C is sort of macro-assembler. I rather doubt 'many'
of your colleagues would agree with that comparison, having
worked hard to define the standard for C thru the years.
M. K. Shen
Douglas A. Gwyn
> It is interesting to recall here an analogy in an apparent
> difference in thinking between Europeans and Americans:
> In Europe (and many other parts of the world) the possession
> of guns is very much severely restricted than in the US.
Yes, there is a tremendous difference between the
principles underlying the United States' notion of
the proper role of government and that of most
other nations. The understanding of this by US
citizens has been eroded over the decades through
the influence of European philosophers, but enough
remains that, for example, a majority of US voters
did not support the European preference for who
would be their chief executive.
This is not a sci.crypt issue, so if you want to
discuss it, take it elsewhere.
Mok-Kong Shen
Douglas A. Gwyn wrote:
[snip]
> In fact, using Ada by no means ensures that such
> errors as overrunning buffers will not occur.
> The belief that safety can be guaranteed by choice
> of PL is worse than misguided -- it can encourage
> thoughtlessness.
As others explained, safety could be 'enhanced' by
appropriate language features in general. Also in
everyday life one employs safety measures. Flu vaccine
doesn't ensure that one wouldn't get flu but enhances
the chance of either remaining in health or having
comparatively minor consequences from flu. Certainly
nothing can be perfect, but one attempts to get the
best, if feasible.
M. K. Shen
karl malbrain
>
>
> Douglas A. Gwyn wrote:
>
> [snip]
> > In fact, using Ada by no means ensures that such
> > errors as overrunning buffers will not occur.
> > The belief that safety can be guaranteed by choice
> > of PL is worse than misguided -- it can encourage
> > thoughtlessness.
>
> As others explained, safety could be 'enhanced' by
> appropriate language features in general.
And C provides an excellent way to "enhance" it's standard library: the
typedef. You are free to add whatever appropriate language features you
want with it. karl m