Audit
Jonathan Lee
To be a good little coder I want to ensure all of my functions pass
a checklist of "robustness". To keep things simple, I want to document
each function with a string that will indicate which of the checklist
items the function has been audited for. Something like
abcdefghiJklMnopqRsTuvwxyz
which would show that items J, M, R, and T have been checked. Off the
top of my head I came up with the list below. I wonder if anyone has
items they think should be added to the list. Any advice welcome,
--Jonathan
Audit list (an implicit "where applicable" should be assumed)
A - Arguments checked against domain
B - Arrays have bounded access
C - No C style casts, other casts as appropriate. Avoid
reinterpret_cast<>
D - No #define's - use static const, enum, or function
E - Exception safe
F - Floating point comparisons are safe (eg., don't check against 0.0)
I - Use initialization lists in constructors
L - Loops always terminate
M - Const qualify member functions that need it
N - "new" memory is not leaked, esp., in light of exceptions
O - Integer overflow
P - Wrap non-portable code in "#if"s and warn user with #else
R - Reentrant
Q - Const Qualify object arguments
T - Thread safe
V - Virtual destructor
Victor Bazarov
How are 'D', 'V', applicable to a function? Are you going to make all
d-tors virtual? That's an overkill. 'D' is module-centric, not
function-centric.
E - add "multiple-return-statement safe", IOW, use RAII; but that
probably goes without saying.
F - Comparison of floats to 0.0 is OK in some circumstances.
Q - not always necessary or feasible.
T - sounds good. What does it mean?
V
--
Please remove capital 'A's when replying by e-mail
I do not respond to top-posted replies, please don't ask
Jonathan Lee
> How are 'D', 'V', applicable to a function?
Well, they don't really. I'm just breaking up my auditing function by
function. I don't always have the time to audit the whole module. But
if all of the functions pass 'D', for example, then the module pretty
much will, too.
Admittedly, 'V' doesn't really apply to any function but the
destructor "function". Maybe I can make a module checklist, too.
> Are you going to make all d-tors virtual? That's an overkill.
Understood. I don't mean the list to *insist* on anything -- just that
I should check it. If it turns out I don't need a virtual destructor,
I won't use one. But I should look.
> E - add "multiple-return-statement safe", IOW, use RAII; but that
> probably goes without saying.
Noted.
> F - Comparison of floats to 0.0 is OK in some circumstances.
> Q - not always necessary or feasible.
As with 'V', I think it's good practice to check if these apply. Then
act appropriately.
> T - sounds good. What does it mean?
Heh. Vague, I know. But I mean if the class is going to be accessed by
multiple threads, will that "be a problem". Do I need to guard RW
access to data with mutexes, for example. I think it will depend a lot
on context.
Thanks for your input, Victor.
--Jonathan
Richard Herring
<fe63b3c3-2c69-4f66...@k1g2000yqf.googlegroups.com>,
Jonathan Lee <cho...@shaw.ca> writes
>Hello all,
> To be a good little coder I want to ensure all of my functions pass
>a checklist of "robustness". To keep things simple, I want to document
>each function with a string that will indicate which of the checklist
>items the function has been audited for. Something like
>
> abcdefghiJklMnopqRsTuvwxyz
>
>which would show that items J, M, R, and T have been checked. Off the
>top of my head I came up with the list below. I wonder if anyone has
>items they think should be added to the list. Any advice welcome,
>
>--Jonathan
>
>Audit list (an implicit "where applicable" should be assumed)
>A - Arguments checked against domain
>B - Arrays have bounded access
>C - No C style casts, other casts as appropriate. Avoid
>reinterpret_cast<>
>D - No #define's - use static const, enum, or function
>E - Exception safe
>F - Floating point comparisons are safe (eg., don't check against 0.0)
There's nothing intrinsically "unsafe" about comparing floating-point
values with 0.0, if that's what your algorithm requires. What's unsafe
is programming floating-point arithmetic if you don't understand the
floating-point data model or the algorithm.
>I - Use initialization lists in constructors
>L - Loops always terminate
>M - Const qualify member functions that need it
>N - "new" memory is not leaked, esp., in light of exceptions
>O - Integer overflow
>P - Wrap non-portable code in "#if"s and warn user with #else
>R - Reentrant
>Q - Const Qualify object arguments
>T - Thread safe
>V - Virtual destructor
In a _function_?
--
Richard Herring
Jonathan Lee
> There's nothing intrinsically "unsafe" about comparing floating-point
> values with 0.0, if that's what your algorithm requires. What's unsafe
> is programming floating-point arithmetic if you don't understand the
> floating-point data model or the algorithm.
Perhaps I should extend it to say: ensure floating point is rigorous.
Consider normalizing a vector. The naive way does a bad job of this:
void normalize(double v[3]) { // An example
double x = sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2])
v[0] /= x;
v[1] /= x;
v[2] /= x;
}
I don't want to overlook code like this, which has flaws that are less
obvious than the "if (x == 0.0){}" construction.
--Jonathan
Andrew Tomazos
> To be a good little coder I want to ensure all of my functions pass
> a checklist of "robustness".
Its impractical to double check every function for all of these. It
will slow down development too much. Someone else will be able to
produce equally robust code in fewer manhours. If you must make rigid
rules for yourself, than you need to apply them as you write the
code. The best cooks don't weigh ingredients to the microgram.
-Andrew.
Jonathan Lee
> If you must make rigid rules for yourself, than you need
> to apply them as you write the code.
In order to write better code in the future, I would need to know what
my mistakes in the past have been, no?
--Jonathan
Andrew Tomazos
You have to distinguish between two different kinds of mistakes.
There are logical bugs which result in faulty behavior of the
program. The "rule list" will only protect you from a tiny fraction
of them. The time spent in testing/debugging is far less than the
cost of doggedly applying these rules. Everytime you find a logic bug
in debugging, spend time thinking about the root cause. Most of the
time you'll find that one of the rules wouldn't have saved you.
The other kind of mistake is "bad design". Bad design is subjective.
It is like talking about good and bad art. It is as much to do with
the reader of the code as it is to do with the writer. You have to
negotiate and form an agreement with the people that will read your
code what the definition of "good design" is. Programmers will not
agree on this, anymore than people agree on what constitutes good art.
Furthermore, the rule list you have presented are in fact only rough
guidelines. There are exceptions to the majority of your list.
For example, the one about using inline functions rather than
#defines. Rather than just memorizing that rule, understand the
mechanics of the preprocessor, the mechanics of type/parameter binding
and multipass text processing. Write a compiler for some toy language
with a toy preprocessor. This is good programming practice, as well
as understanding language design. Once you do that you won't need to
think about the "rule", as you will have something better. You will
never accidentally use a #define when you meant to use an inline
function. As a note, there is no way to write...
#define f1(x, y) f2(x, #x, x##y)
...with an inline function. In at least some cases a #define is
preferable. There are similar exceptions to other rules in your list.
-Andrew.
Jonathan Lee
> the rule list you have presented are in fact only rough
> guidelines.
Of course. The fact that *you think* that *I think* it is anything
more than that is causing you to be very condescending toward me.
Seriously, read that paragraph about bad design as if someone else
had sent it to you.
It's meant to be a simple mnemonic. You introduced the words "rule",
"doggedly" and "rigid", but I never used them. See? Just a list.
The letters even sorta match up with the item. Like "A" for
argument, "B" for "bounds", "C" for "C-style cast", "D" for
"define, "E" for "exceptions".
Cute, no?
--Jonathan
Stuart Golodetz
> Hello all,
> To be a good little coder I want to ensure all of my functions pass
> a checklist of "robustness". To keep things simple, I want to document
> each function with a string that will indicate which of the checklist
> items the function has been audited for. Something like
>
> abcdefghiJklMnopqRsTuvwxyz
>
> which would show that items J, M, R, and T have been checked. Off the
> top of my head I came up with the list below. I wonder if anyone has
> items they think should be added to the list. Any advice welcome,
You have good intentions, but this strikes me as a maintenance nightmare
in the making, so I'm going to do my best to dissuade you for a moment :-)
Consider:
- What happens if you decide to modify the audit list? Are you going to
go through every function in your codebase and change the relevant
string? Is your codebase small enough? Are you and your coworkers
patient enough? Will you have time to sit down and update it all? What
happens if you get sidetracked by your boss half-way through? (Are
out-of-date/erroneous comments - of the audit variety or indeed
otherwise - worse than none at all?)
- Assume the audit list is fixed for all time (best-case scenario). Are
all your coworkers rigorous enough to recheck everything on the audit
list every single time they modify a function? What happens if somebody
forgets? (What happens if somebody accidentally has CAPS LOCK on and
doesn't spot it, for that matter?)
Maintenance-aside, you might want to consider the following:
- Are the things you're considering auditing (accepting that it's a
hypothetical list at present) best checked in this manner? For an
example of what I mean by this, checking casts on a per-function basis
doesn't seem sensible. If you want to ensure there are no
reinterpret_casts project-wide, search for them and kill them off. What
benefit do you derive from documenting the lack of them in each
function? What if an intern adds one to an audited function and doesn't
update the comment? Which approach gives you greater confidence that
your project is free of such casts? (I do realise that I'm in some
senses unfairly picking on an off-the-cuff suggestion here, but I do
feel the general point is a valid one.)
- What happens when you make calls to external library functions to
which you don't have the source? They will not have the same auditing
that your functions do, and without the source you can't check them
yourself. What guarantees can you make about your own functions in such
circumstances?
My overall suggestion would be: if you really want to make your programs
robust, look into formal methods and how they develop safety-critical
systems etc. The scheme you're suggesting is in truth a bit ad-hoc and
seems unlikely to make a serious difference to the robustness of your
code. Moreover it'll be a pain in the *** to maintain (and I've come up
with lots of unmaintainable schemes myself in the past, so it's more or
less a case of 'experience is the ability to recognise a mistake when
you make it again').
Cheers,
Stu
Andrew Tomazos
> On Jul 7, 4:00 pm, Andrew Tomazos <and...@tomazos.com> wrote:
>
> > the rule list you have presented are in fact only rough
> > guidelines.
>
> Of course. The fact that *you think* that *I think* it is anything
> more than that is causing you to be very condescending toward me.
> Seriously, read that paragraph about bad design as if someone else
> had sent it to you.
It wasn't my intention to be condescending.
> It's meant to be a simple mnemonic. You introduced the words "rule",
> "doggedly" and "rigid", but I never used them. See? Just a list.
> The letters even sorta match up with the item. Like "A" for
> argument, "B" for "bounds", "C" for "C-style cast", "D" for
> "define, "E" for "exceptions".
The source of my belief that you were holding this rule list sacred
was that you wanted to go through it for every function that you
write.
-Andrew.
James Kanze
> In message
> <fe63b3c3-2c69-4f66-a929-4c389e34b...@k1g2000yqf.googlegroups.com>,
> Jonathan Lee <cho...@shaw.ca> writes
> >To be a good little coder I want to ensure all of my
> >functions pass a checklist of "robustness". To keep things
> >simple, I want to document each function with a string that
> >will indicate which of the checklist items the function has
> >been audited for. Something like
> > abcdefghiJklMnopqRsTuvwxyz
> >which would show that items J, M, R, and T have been checked.
> >Off the top of my head I came up with the list below. I
> >wonder if anyone has items they think should be added to the
> >list. Any advice welcome,
[...]
> >F - Floating point comparisons are safe (eg., don't check against 0.0)
> There's nothing intrinsically "unsafe" about comparing
> floating-point values with 0.0, if that's what your algorithm
> requires.
And there's nothing intrinsically "safe" about any of the
alternatives.
> What's unsafe is programming floating-point arithmetic if you
> don't understand the floating-point data model or the
> algorithm.
Which is generally (not just with regards to floating point) an
important issue: did the author understand the techniques he
used? Not too sure how to make that a check point, however:-).
(One important point with regards to floating point: if any code
uses floating point, one of the code reviewers should be an
expert in numeric processing.)
[...]
> >P - Wrap non-portable code in "#if"s and warn user with #else
The only #if's that code should contain are include guards
(which should normally automatically be inserted by the editor).
Non-portable code belongs in a separate file, in a target
dependent directory.
> >R - Reentrant
> >Q - Const Qualify object arguments
> >T - Thread safe
> >V - Virtual destructor
For the rest, there is some value in having a list of
checkpoints---in many cases, one could even arrange to check
them automatically. But they won't cover everything, and more
or less vague statements like "thread safe" don't belong in
them. On the other hand, he seems to have missed one or two
important ones:
-- All variables are initialized before use (preferrably in the
definition).
-- Functions which return a value have a return statement.
-- Don't return a reference or a pointer to a local variable or
a temporary.
(Good compilers will warn about these. Sometimes incorrectly,
however.)
I'd also set a fairly low limit to the cyclometric complexity
(with an exception for functions which consist of a single
switch with a lot of entries).
--
James Kanze (GABI Software) email:james...@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
James Kanze
> On Jul 7, 4:23 pm, Jonathan Lee <cho...@shaw.ca> wrote:
> > To be a good little coder I want to ensure all of my
> > functions pass a checklist of "robustness".
> Its impractical to double check every function for all of
> these. It will slow down development too much.
It is impractical NOT to check every function for many of these.
Not doing so will slow down development too much.
> Someone else will be able to produce equally robust code in
> fewer manhours.
Equally robust?
> If you must make rigid rules for yourself, than you need to
> apply them as you write the code.
The problem is that code is written by human beings, who are
imperfect, and make mistakes. One of the most effective ways of
reducing total development time is code review. It also
improves the quality of the final product.
James Kanze
<sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
> Jonathan Lee wrote:
[...]
> You have good intentions, but this strikes me as a maintenance
> nightmare in the making, so I'm going to do my best to
> dissuade you for a moment :-)
[...]
> My overall suggestion would be: if you really want to make
> your programs robust, look into formal methods and how they
> develop safety-critical systems etc.
It's usual practice for such formal methods to have such a check
list, and apply it. Not necessarily one exactly like his, and
in many cases, they actually use mechanical means of applying
it, but the list exists and is used.
> The scheme you're suggesting is in truth a bit ad-hoc and
> seems unlikely to make a serious difference to the robustness
> of your code. Moreover it'll be a pain in the *** to maintain
> (and I've come up with lots of unmaintainable schemes myself
> in the past, so it's more or less a case of 'experience is the
> ability to recognise a mistake when you make it again').
Having a check list for common, easily identified errors can
effectively help reduce the occurance of those errors, which
improves maintainability. Obviously, such a list isn't
sufficient. You also need thoughtful code review, which does
more than just go through a check list. But the check list can
help.
Jonathan Lee
> It wasn't my intention to be condescending.
Yeah, I know. Sorry for being a jerk about it yesterday. I just
see a lot of posts in this newsgroup that assume someone
else doesn't possess common sense. Did I write "I want to ensure
all my functions"? Yes. Do I mean literally every function
I've ever written since I was 17? No, of course not. I'm a
reasonable human being. Moreover, it is also true that "I want
to go to the moon". Does it mean I'm going to? No.
Or "Google is your friend". I see that a lot but honestly, how
many people get that as a reply and then think "ZOMG i totaly
didnt search teh Google!".
So sorry for snapping at you. It was just one too many posts
where I found myself shaking my head.
--Jonathan
REH
> There's nothing intrinsically "unsafe" about comparing floating-point
> values with 0.0, if that's what your algorithm requires.
There is if your platform allows for denormalized values. For example:
if (x != 0.0)
a = b / x;
Assume x, a, and b are all doubles. If x contains a denormalized
number, the comparison will be true (x is not 0.0). When the
expression is evaluated, x will be normalized (x becomes 0.0) and thus
cause a divide-by-zero. I usually write:
if (x + 1.0 != 1.0)
a = b / x;
This will force x to be normalized in the condition expression.
REH
Stuart Golodetz
> On Jul 8, 2:15 am, Stuart Golodetz
> <sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
>> Jonathan Lee wrote:
>
> [...]
>> You have good intentions, but this strikes me as a maintenance
>> nightmare in the making, so I'm going to do my best to
>> dissuade you for a moment :-)
>
> [...]
>> My overall suggestion would be: if you really want to make
>> your programs robust, look into formal methods and how they
>> develop safety-critical systems etc.
>
> It's usual practice for such formal methods to have such a check
> list, and apply it. Not necessarily one exactly like his, and
> in many cases, they actually use mechanical means of applying
> it, but the list exists and is used.
If it's applied mechanically, isn't that's an entirely different kettle
of fish though? The maintenance problem is to do with doing it manually.
On a separate note, I admit I don't know all that much about the way
they go about these things (I'm basing what I'm saying on a vague
recollection of a talk we were given by someone at Praxis - and I hope
not misrepresenting what was said!), but I was under the impression that
safety-critical programs were developed based on a rigorous
specification (e.g. using notations like Z) and a load of formal proofs?
>> The scheme you're suggesting is in truth a bit ad-hoc and
>> seems unlikely to make a serious difference to the robustness
>> of your code. Moreover it'll be a pain in the *** to maintain
>> (and I've come up with lots of unmaintainable schemes myself
>> in the past, so it's more or less a case of 'experience is the
>> ability to recognise a mistake when you make it again').
>
> Having a check list for common, easily identified errors can
> effectively help reduce the occurance of those errors, which
> improves maintainability. Obviously, such a list isn't
> sufficient. You also need thoughtful code review, which does
> more than just go through a check list. But the check list can
> help.
If applied rigorously, having a check list might be a good idea, agreed.
But the particular way being suggested still seems like a maintenance
headache to me. Using an automated tool to perform static analysis on
your code and check for common problems is sensible; manually commenting
each function with the results of manual checks which have to be
repeated at every change seems unwise. All it takes is a few typos, or a
bit of laziness, or a misunderstanding of one of the criteria, for the
comments to become wrong or out-of-date. On a code base of any
significant size, this seems likely to cause unnecessary problems. The
intentions are good, as I said, but IMHO the scheme itself could do with
a rethink, as it doesn't scale well.
In particular, automating the checks wherever possible and separating
the results of the checks from the source files seem like good ways to
improve the scheme. The sort of thing I have in mind would essentially
be in the same vein as Lint. If it's not possible to automate certain
checks, then it would at least be sensible to have a tool which keeps
track of which functions have been checked since they were last modified
- which is possible. One which prevents code being checked into the
repository until it's been rechecked would be ideal.
Regards,
Stu
> --
> James Kanze (GABI Software) email:james...@gmail.com
> Conseils en informatique orient�e objet/
> Beratung in objektorientierter Datenverarbeitung
> 9 place S�mard, 78210 St.-Cyr-l'�cole, France, +33 (0)1 30 23 00 34
Pete Becker
> On Jul 7, 11:16 am, Richard Herring <junk@[127.0.0.1]> wrote:
>> There's nothing intrinsically "unsafe" about comparing floating-point
>> values with 0.0, if that's what your algorithm requires.
>
> There is if your platform allows for denormalized values. For example:
>
> if (x != 0.0)
> a = b / x;
>
> Assume x, a, and b are all doubles. If x contains a denormalized
> number, the comparison will be true (x is not 0.0). When the
> expression is evaluated, x will be normalized (x becomes 0.0) and thus
> cause a divide-by-zero.
If that's correct (I'm skeptical about this forced normalization), the
problem isn't comparing to 0.0, but not understanding the rules for
division.
--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of
"The Standard C++ Library Extensions: a Tutorial and Reference"
(www.petebecker.com/tr1book)
Richard Herring
<80cf83a9-5b42-456e...@b14g2000yqd.googlegroups.com>, REH
<spam...@stny.rr.com> writes
>On Jul 7, 11:16�am, Richard Herring <junk@[127.0.0.1]> wrote:
>> There's nothing intrinsically "unsafe" about comparing floating-point
>> values with 0.0, if that's what your algorithm requires.
>
>There is if your platform allows for denormalized values. For example:
>
>if (x != 0.0)
> a = b / x;
>
>Assume x, a, and b are all doubles. If x contains a denormalized
>number, the comparison will be true (x is not 0.0). When the
>expression is evaluated, x will be normalized (x becomes 0.0) and thus
>cause a divide-by-zero.
One could argue that that's "safer" than accepting the wildly inaccurate
value you'll get by continuing to calculate with a denormalized value
;-/.
> I usually write:
>
>if (x + 1.0 != 1.0)
> a = b / x;
>
>This will force x to be normalized in the condition expression.
And will treat *any* number less than numeric_limits<double>::epsilon()
as if it were zero, which may not be at all what you wanted.
--
Richard Herring
Jonathan Lee
On Jul 7, 8:15 pm, Stuart Golodetz
<sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
> You have good intentions, but this strikes me as a maintenance nightmare
> in the making, so I'm going to do my best to dissuade you for a moment :-)
>
> Consider:
Thanks for your comments. I'll consider those points. But in turn can
I ask you
a question? Suppose your boss asked you to check over someone else's
code and
bring it up to snuff. Would you not run through a similar sort of list
in your
head as you examined the code?
To be honest I'm more interested in what such a list might look like
rather
than how to apply it. I asked in the newsgroup because I think people
with
more experience than me would have valuable input about common coding
pitfalls. I'm perfectly willing to abandon a documentation scheme that
becomes
a nightmare, but I'm sure everybody has some version of the above list
in
their heads.
Below you mention mechanically checking code. FWIW, I typically
compile my
code (with g++) using
-ansi -pedantic -Wall -W -Wshadow -Wpointer-arith -Wcast-qual
-Wcast-align -Wwrite-strings -Wconversion -Wold-style-cast
and I never get warnings (well, OK, there's ONE warning I can't get
rid
of in one project). I also run my source through cppcheck, and the
binary
through the valgrind suite. In other words, in practice I do much more
than rely that list.
(Actually, if anyone thinks I'm not doing due diligence with the above
feel free to add)
--Jonathan
Jonathan Lee
thanks for your contribution
On Jul 8, 5:26 am, James Kanze <james.ka...@gmail.com> wrote:
> The only #if's that code should contain are include guards
> (which should normally automatically be inserted by the editor).
> Non-portable code belongs in a separate file, in a target
> dependent directory.
What do you recommend for this scenario: I have a project
which uses the Qt libraries. From version 3 to version 4
of Qt there were many changes to the API, so my code has
a lot of this:
#ifndef QT_V3
statusbar->addPermanentWidget(ageLabel);
#else
statusbar->addWidget(ageLabel, true);
#endif
I would like to support both versions, but having two
separate cpp files for these little one liners seems
difficult to maintain. Having one file, I figure,
allows me to keep the two versions in step.
--Jonathan
Andrew Tomazos
> > On Jul 7, 4:23 pm, Jonathan Lee <cho...@shaw.ca> wrote:
> > > To be a good little coder I want to ensure all of my
> > > functions pass a checklist of "robustness".
>
> > Its impractical to double check every function for all of
> > these. It will slow down development too much.
>
> It is impractical NOT to check every function for many of these.
> Not doing so will slow down development too much.
That is not necessarily in contradiction with what I said. Clearly
there is a BALANCE between (A) just quickly hacking away whatever
works; and (B) going through checklists and formal process at every
step. Checking every function for a list of 20 points is too far
toward B, and will slow down development.
> > Someone else will be able to produce equally robust code in
> > fewer manhours.
>
> Equally robust?
What is your question? You don't understand what "equally robust"
means?
> > If you must make rigid rules for yourself, than you need to
> > apply them as you write the code.
>
> The problem is that code is written by human beings, who are
> imperfect, and make mistakes. One of the most effective ways of
> reducing total development time is code review. It also
> improves the quality of the final product.
There is a difference between a 1 hour code review of a manweek's
worth of code, and double checking every function for 20 things.
-Andrew.
REH
> And will treat *any* number less than numeric_limits<double>::epsilon()
> as if it were zero, which may not be at all what you wanted.
>
I believe IEEE rules state that anytime a denormalized value is used
with a normalized value (e.g., arithmetic), the denormalized value is
normalized (becomes 0.0).
REH
REH
>
> If that's correct (I'm skeptical about this forced normalization), the
> problem isn't comparing to 0.0, but not understanding the rules for
> division.
>
Well, I could be wrong (I usually am!). My understand is that it is
not just division. Any time a denormalized number is used
arithmetically with a normalized number, the former is "flushed" to
zero.
REH
Pete Becker
Here's a quick-and-dirty program to demonstrate the use of denormals
with normalized values:
#include <iostream>
#include <float.h>
#include <math.h>
int main()
{
float base = ldexp(1.0f, FLT_MIN_EXP);
float fl = base;
while (fl != 0.0f)
{
fl /= 2;
std::cout << fl << ", " << (base + fl) << std::endl;
}
return 0;
}
The loop starts with fl having the smallest normalized value (base), and
generates successively smaller denormal values by dividing the previous
value by 2. The output shows that value and the result of adding it to
base. The output I get is this:
[work]$ g++ test.cpp
[work]$ ./a.out
1.17549e-38, 3.52648e-38
5.87747e-39, 2.93874e-38
2.93874e-39, 2.64486e-38
1.46937e-39, 2.49793e-38
7.34684e-40, 2.42446e-38
3.67342e-40, 2.38772e-38
1.83671e-40, 2.36936e-38
9.18355e-41, 2.36017e-38
4.59177e-41, 2.35558e-38
2.29589e-41, 2.35328e-38
1.14794e-41, 2.35214e-38
5.73972e-42, 2.35156e-38
2.86986e-42, 2.35128e-38
1.43493e-42, 2.35113e-38
7.17465e-43, 2.35106e-38
3.58732e-43, 2.35102e-38
1.79366e-43, 2.35101e-38
8.96831e-44, 2.351e-38
4.48416e-44, 2.35099e-38
2.24208e-44, 2.35099e-38
1.12104e-44, 2.35099e-38
5.60519e-45, 2.35099e-38
2.8026e-45, 2.35099e-38
1.4013e-45, 2.35099e-38
0, 2.35099e-38
[work]$
The first dozen or so lines show that adding the denormal value to base
produces a value that's different from base, so the denormal value is
not being flushed to zero. The last several lines show no difference,
because the denormal value is too small to affect the result.
Granted, those last lines could be described as "the denormal is flushed
to zero". But that's not because the value is a denormal; it's simply
because the other value in the sum is much larger, so the denormal value
gets lost in rounding. The same thing happens with 1.0e20 + 1.0e1.
Richard Herring
<9af1604f-9398-41f0...@n11g2000yqb.googlegroups.com>, REH
<spam...@stny.rr.com> writes
>On Jul 8, 9:22�am, Richard Herring <junk@[127.0.0.1]> wrote:
[restoring context]
>>>if (x + 1.0 != 1.0)
>>> a = b / x;
>> And will treat *any* number less than numeric_limits<double>::epsilon()
>> as if it were zero, which may not be at all what you wanted.
>>
>
>I believe IEEE rules state that anytime a denormalized value is used
>with a normalized value (e.g., arithmetic), the denormalized value is
>normalized (becomes 0.0).
I'm talking about when x is *not* denormalized, just less than
epsilon()..
--
Richard Herring
Stuart Golodetz
> Hi Stuart
>
> On Jul 7, 8:15 pm, Stuart Golodetz
> <sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
>> You have good intentions, but this strikes me as a maintenance nightmare
>> in the making, so I'm going to do my best to dissuade you for a moment :-)
>>
>> Consider:
>
> Thanks for your comments. I'll consider those points. But in turn can
> I ask you
> a question? Suppose your boss asked you to check over someone else's
> code and
> bring it up to snuff. Would you not run through a similar sort of list
> in your
> head as you examined the code?
Yes, I would. My objection wasn't to making a list of things to watch
out for, which is evidently a sensible enough plan :-) At the same time,
I would certainly be thinking about getting as decent a mechanical tool
as I could to flag up things I should be looking at before getting 'down
and dirty' with the entire codebase.
> To be honest I'm more interested in what such a list might look like
> rather
> than how to apply it. I asked in the newsgroup because I think people
> with
> more experience than me would have valuable input about common coding
> pitfalls. I'm perfectly willing to abandon a documentation scheme that
> becomes
> a nightmare, but I'm sure everybody has some version of the above list
> in
> their heads.
Ok, that's fair enough in that case. I assumed you were looking for
advice on both the list contents and the scheme itself - mea culpa.
> Below you mention mechanically checking code. FWIW, I typically
> compile my
> code (with g++) using
>
> -ansi -pedantic -Wall -W -Wshadow -Wpointer-arith -Wcast-qual
> -Wcast-align -Wwrite-strings -Wconversion -Wold-style-cast
>
> and I never get warnings (well, OK, there's ONE warning I can't get
> rid
> of in one project). I also run my source through cppcheck, and the
> binary
> through the valgrind suite. In other words, in practice I do much more
> than rely that list.
>
> (Actually, if anyone thinks I'm not doing due diligence with the above
> feel free to add)
You seem to be doing your fair share of mechanical checking and I
certainly wasn't intending to suggest that you don't do due diligence
when writing code :-) My comments were entirely based on the
implementation scheme you had in mind, which struck me as problematic
(and I stand by that). I hope that if nothing else I've raised a couple
of issues which might be worth considering.
Best wishes,
Stu
> --Jonathan
REH
I guess I'll have to try and locate where I read that info about
denormals. I thought it was in "What Every Computer Scientist Should
Know About Floating-Point Arithmetic," by David Goldberg. But I can't
seem to find it in there.
REH
Jonathan Lee
<sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
> My comments were entirely based on the
> implementation scheme you had in mind, which struck me as problematic
> (and I stand by that). I hope that if nothing else I've raised a couple
> of issues which might be worth considering.
Fair enough :) I appreciate the advice
--Jonathan
Stuart Golodetz
> On Jul 8, 11:30 am, James Kanze <james.ka...@gmail.com> wrote:
>>> On Jul 7, 4:23 pm, Jonathan Lee <cho...@shaw.ca> wrote:
>>>> To be a good little coder I want to ensure all of my
>>>> functions pass a checklist of "robustness".
>>> Its impractical to double check every function for all of
>>> these. It will slow down development too much.
>> It is impractical NOT to check every function for many of these.
>> Not doing so will slow down development too much.
>
> That is not necessarily in contradiction with what I said. Clearly
> there is a BALANCE between (A) just quickly hacking away whatever
> works; and (B) going through checklists and formal process at every
> step. Checking every function for a list of 20 points is too far
> toward B, and will slow down development.
The balance depends on the application. If your program going wrong
could potentially kill me (think code to control an X-ray scanner, or a
large, heavy robot arm in a factory, or...), I'd want you to be far
closer to (B) than (A). Equally, though, I'd want to make sure that your
being closer to (B) would actually have the desired effect of making my
demise less likely. Checklists of *irrelevant* points could distract you
from making your code safer.
>>> Someone else will be able to produce equally robust code in
>>> fewer manhours.
>> Equally robust?
>
> What is your question? You don't understand what "equally robust"
> means?
I suspect James might doubt that it will be equally robust. But thinking
about it, I suppose there is an issue of how you quantify the robustness
of your code.
>>> If you must make rigid rules for yourself, than you need to
>>> apply them as you write the code.
>> The problem is that code is written by human beings, who are
>> imperfect, and make mistakes. One of the most effective ways of
>> reducing total development time is code review. It also
>> improves the quality of the final product.
>
> There is a difference between a 1 hour code review of a manweek's
> worth of code, and double checking every function for 20 things.
Agreed - and the sword cuts both ways on that one :-) I'm all for
avoiding onerous processes, except when they're necessary (which is
sometimes the case).
Stu
> -Andrew.
Jerry Coffin
@b15g2000yqd.googlegroups.com>, spam...@stny.rr.com says...
[ ... ]
> Well, I could be wrong (I usually am!). My understand is that it is
> not just division. Any time a denormalized number is used
> arithmetically with a normalized number, the former is "flushed" to
> zero.
That would be a rather strange thing to do. The whole point of having
denormals in the first place is to _avoid_ flushing to zero if at all
possible. If using it in an expression with a normalized number
resulted in its being flushed to zero anyway, you'd render denormals
almost entirely useless.
--
Later,
Jerry.
Andrew Tomazos
<sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
> > That is not necessarily in contradiction with what I said. Clearly
> > there is a BALANCE between (A) just quickly hacking away whatever
> > works; and (B) going through checklists and formal process at every
> > step. Checking every function for a list of 20 points is too far
> > toward B, and will slow down development.
>
> The balance depends on the application. If your program going wrong
> could potentially kill me (think code to control an X-ray scanner, or a
> large, heavy robot arm in a factory, or...), I'd want you to be far
> closer to (B) than (A). Equally, though, I'd want to make sure that your
> being closer to (B) would actually have the desired effect of making my
> demise less likely. Checklists of *irrelevant* points could distract you
> from making your code safer.
>
> about it, I suppose there is an issue of how you quantify the robustness
> of your code.
>
> avoiding onerous processes, except when they're necessary (which is
> sometimes the case).
Robustness is quantified by the degree to which software meets its
requirements.
What you are missing is that time saved on onerous process can be
spent on runtime testing/debugging. Perfect software is practically
impossible for a nontrivial requirement set. You have to look at how
much robustness you get per unit development time. This is true
whether or not the software is mission-critical and/or potentially
life-threatening. That is what I mean when I say that moving from the
balance point toward (B) doesn't get you more robustness in the same
amount of time.
If a software application could potentially injure people when it
fails to meet its requirements, than it should be treated in the same
way that a new medical drug is treated. After feature freeze, a crazy
amount of runtime testing is conducted in a safe controlled
environment. Any new changes to the code must be put through full
regression (ie the testing process much be restarted from scratch).
Let's take a specific example. Suppose there is a function:
void f(int x);
and that f is not exception safe for all values of x. In fact when f
(3) is called, an exception is thrown in an unexpected place, and the
system misbehaves - zapping the patient with a deadly amount of X-ray
radiation.
But suppose that f(3) can never be called at runtime in the final
application. ie There is no calling site of f where x can possibly be
the parameter 3.
If we were to check f for exception safety, we might find this f(3)
problem and fix it. This will however be a waste of time, and
achieves zero improvement on robustness.
Therefore someone that did not bother to check f for exception safety
would have an equally robust application in less time.
This person can then spend this extra time on something else. (For
example runtime testing/debugging)
-Andrew.
Stuart Golodetz
> On Jul 8, 7:16 pm, Stuart Golodetz
> <sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
>>> That is not necessarily in contradiction with what I said. Clearly
>>> there is a BALANCE between (A) just quickly hacking away whatever
>>> works; and (B) going through checklists and formal process at every
>>> step. Checking every function for a list of 20 points is too far
>>> toward B, and will slow down development.
>> The balance depends on the application. If your program going wrong
>> could potentially kill me (think code to control an X-ray scanner, or a
>> large, heavy robot arm in a factory, or...), I'd want you to be far
>> closer to (B) than (A). Equally, though, I'd want to make sure that your
>> being closer to (B) would actually have the desired effect of making my
>> demise less likely. Checklists of *irrelevant* points could distract you
>> from making your code safer.
>>
>> I suspect James might doubt that it will be equally robust. But thinking
>> about it, I suppose there is an issue of how you quantify the robustness
>> of your code.
>>
>> I'm all for
>> avoiding onerous processes, except when they're necessary (which is
>> sometimes the case).
>
> Robustness is quantified by the degree to which software meets its
> requirements.
Not sure I agree that that's what robustness is (or what quantification
is, for that matter): to my mind, robust software is software that
doesn't fall over in the face of unanticipated inputs/scenarios. And
judging by a quick Google search for "quantify robustness of software",
*measuring* robustness is still a research issue.
Software meeting its specification means that it's correct, not that
it's robust. The two are different.
> What you are missing is that time saved on onerous process can be
> spent on runtime testing/debugging. Perfect software is practically
> impossible for a nontrivial requirement set. You have to look at how
> much robustness you get per unit development time. This is true
> whether or not the software is mission-critical and/or potentially
> life-threatening. That is what I mean when I say that moving from the
> balance point toward (B) doesn't get you more robustness in the same
> amount of time.
Mm. Testing can't prove the absence of bugs; formal proof methods in
some cases can. To eschew formal proof methods in safety-critical
situations to save time to conduct more testing is daft - to achieve the
same level of confidence in your system, you'd have to hang around for a
VERY long time (the lifetime of the universe has nothing on your testing
process). I'm not arguing in favour of pointless process, but in some
cases formal methods are the way to go. Incidentally, the goal is to
avoid the need to debug your software, not spend more time doing it. And
you can sit down and prove the correctness of bits of your system while
you're testing it - the two can be done concurrently.
None of this is to say that I think that applying formal methods is
sensible across the board. In particular, they may not be applicable in
your situation (or my situation). They are, however, in certain
circumstances.
> If a software application could potentially injure people when it
> fails to meet its requirements, than it should be treated in the same
> way that a new medical drug is treated. After feature freeze, a crazy
> amount of runtime testing is conducted in a safe controlled
> environment. Any new changes to the code must be put through full
> regression (ie the testing process much be restarted from scratch).
There aren't any formal proof methods for medical testing - if there
were, they'd be used. Not sure I'm convinced by the analogy.
> Let's take a specific example. Suppose there is a function:
>
> void f(int x);
>
> and that f is not exception safe for all values of x. In fact when f
> (3) is called, an exception is thrown in an unexpected place, and the
> system misbehaves - zapping the patient with a deadly amount of X-ray
> radiation.
>
> But suppose that f(3) can never be called at runtime in the final
> application. ie There is no calling site of f where x can possibly be
> the parameter 3.
>
> If we were to check f for exception safety, we might find this f(3)
> problem and fix it. This will however be a waste of time, and
> achieves zero improvement on robustness.
On the contrary: robustness is to do with how well the program behaves
for *unexpected* inputs. Since 3 is unexpected, and you've improved the
behaviour for it, you've likely improved the robustness of the program
(assuming your change didn't mess up something else). Whether it's a
waste of time or not in this particular instance is another matter.
> Therefore someone that did not bother to check f for exception safety
> would have an equally robust application in less time.
They'd have an equally *correct* application (by random luck), but not
an equally robust one.
> This person can then spend this extra time on something else. (For
> example runtime testing/debugging)
All the testing in the world won't be enough in most cases: you'll never
have enough time to test your code for all possible inputs. For
safety-critical systems, it's a choice between something that can
potentially give you the confidence levels you require in your system,
and something that can't. It's a no-brainer really.
Regards,
Stu
> -Andrew.
Andrew Tomazos
Whether you call it robustness, correctness, reliability, stability,
solidness or just plain old quality - isn't very interesting. What I
mean is how well the software works as expected (in all possible
situations). If you don't know what you expect than none of the
methods we have discussed can save you.
Formal Methods are different than general Software Engineering
Process. We started out this conversation talking about a Software
Engineering Process related activity (checking each function for 20
rules). Formal Methods are quite different. They use first-order
logic to prove that a given algorithm is correct. See the book:
Software Reliability Methods, Peled/Springer.
Formal Methods *cannot* be applied to anything other than a trivially
small program. They break down quickly with complexity and large
ranges of chaotic input. Certainly anything as complex as the
software running an X-ray machine cannot and is not verified using
Formal Methods. I have colleagues that work on a line of one of the
most common medical robots that are used in surgeries. They
specifically do not use these type of methods as they are
impractical. You may be surprised to learn that they work pretty much
the same way as most professional developers work, they are just given
more time to get it right, and a hell of a lot bigger QA team.
Nothing can prove the absence of bugs in a nontrivial realworld
application. As I said, perfect software is virtually impossible.
Yes, you can't test every possible set of inputs. Testing isn't
everything. Neither is anything else. None of the methods discussed
are a magic bullet. It is a question of balance. If you want more
reliable software than you need to spend more time on it. Moving off
balance isn't going to help.
> Since 3 is unexpected, and you've improved the
> behaviour for it, you've likely improved the robustness of the program
> (assuming your change didn't mess up something else). Whether it's a
> waste of time or not in this particular instance is another matter.
You didn't understand my example. It is impossible (not just
unexpected) that f is called with a parameter of 3. Go and study my
example again.
-Andrew.
James Kanze
You don't necessarily need two separate source files. Just two
headers, with something like:
inline void
addPermanentWidget(
QT_StatusBar* statusBar,
std::string const& label )
{
statusBar->addWidget( label, true ) ;
}
in one, and:
inline void
addPermanentWidget(
QT_StatusBar* statusBar,
std::string const& label )
{
statusBar->addPermanentWidget( label ) ;
}
in the other. Put them in separate directories, and choose
which one by means of a -I option at compile time. Or wrap the
interface any other way you like.
(Of course, one might also consider changing suppliers
completely. Breaking client code in this manner is pretty
irresponsible, and I'd wonder about a supplier who cared so
little for his customers.)
James Kanze
> On Jul 8, 11:30 am, James Kanze <james.ka...@gmail.com> wrote:
[...]
> > > If you must make rigid rules for yourself, than you need to
> > > apply them as you write the code.
> > The problem is that code is written by human beings, who are
> > imperfect, and make mistakes. One of the most effective
> > ways of reducing total development time is code review. It
> > also improves the quality of the final product.
> There is a difference between a 1 hour code review of a
> manweek's worth of code, and double checking every function
> for 20 things.
The code reviewer's job is to double check that the code works
(and is understandable). Checking against a list, for things
which can be checked against a list, makes the job easier and
quicker.
Note that that doesn't mean I agree with all of the things on
his list. "Thread safety" just doesn't work as a check
point---you have to actually analyse the code, and the context
in which it is designed to be used, to determine that. What
I've seen in experience is that the list should be short and
evolutive: a list with a couple hundred points won't work,
because it will either be ignored, or it will cost too much time
to check each point separately. On the other hand, if you keep
the list short, and have it reflect actual errors which slipped
through code review, then it can be very effective. (I.e. if
you find errors after code review because variables haven't been
initialized, you add a point to the list "check that all
variables have been initialized".)
James Kanze
> On Jul 8, 7:16 pm, Stuart Golodetz
> <sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
> > > That is not necessarily in contradiction with what I said.
> > > Clearly there is a BALANCE between (A) just quickly
> > > hacking away whatever works; and (B) going through
> > > checklists and formal process at every step. Checking
> > > every function for a list of 20 points is too far toward
> > > B, and will slow down development.
> > The balance depends on the application. If your program
> > going wrong could potentially kill me (think code to control
> > an X-ray scanner, or a large, heavy robot arm in a factory,
> > or...), I'd want you to be far closer to (B) than (A).
> > Equally, though, I'd want to make sure that your being
> > closer to (B) would actually have the desired effect of
> > making my demise less likely. Checklists of *irrelevant*
> > points could distract you from making your code safer.
> > I suspect James might doubt that it will be equally robust.
> > But thinking about it, I suppose there is an issue of how
> > you quantify the robustness of your code.
> > I'm all for avoiding onerous processes, except when they're
> > necessary (which is sometimes the case).
> Robustness is quantified by the degree to which software meets
> its requirements.
Amongst other things. (Or are you including "implicit
requirements"? I've never seen a requirements specification
which said that the program shouldn't core dump, but of course,
programs which core dump aren't robust.)
> What you are missing is that time saved on onerous process can
> be spent on runtime testing/debugging.
Whose talking about an "onerous process"? We're talking about a
means of simplifying, and perhaps even mechanising, part of an
essential step.
> Perfect software is practically impossible for a nontrivial
> requirement set. You have to look at how much robustness you
> get per unit development time. This is true whether or not
> the software is mission-critical and/or potentially
> life-threatening. That is what I mean when I say that moving
> from the balance point toward (B) doesn't get you more
> robustness in the same amount of time.
No. It gets you more robustness in less time, because the
errors are found earlier. Finding an error which only shows up
in a test takes significantly more time than finding one where
the error is pointed out to you immediately in the source code.
In general, the further upstream the error is detected, the less
it costs to find and fix it: an error found in code review costs
less to find and fix than one found in unit tests; an error
found in unit tests costs less to find an fix than one found in
integration tests; and an error found in integration tests costs
less than one found in the field.
Using a check list is simply a way of making some aspects of the
code review more efficient (or even automatic).
> If a software application could potentially injure people when
> it fails to meet its requirements, than it should be treated
> in the same way that a new medical drug is treated. After
> feature freeze, a crazy amount of runtime testing is conducted
> in a safe controlled environment. Any new changes to the code
> must be put through full regression (ie the testing process
> much be restarted from scratch).
Don't forget that testing can only prove that a program is
incorrect; it can't prove that a program is correct. For life
critical software, it's usual to require some sort of formal
proofs. And that can be expensive. Generally, judging from my
own experience, anything you do getting the error rate down to
about one error per 100KLoc, going into integration, also
reduces total development costs. Beyond that, I'm not sure.
Formal proofs aren't necessary for that level of quality, but
that level of quality probably isn't acceptable for life
critical software. (But the issue isn't cut and dried. For the
example mentionned, one could argue that the system should be
physically designed so that no matter what the software did, it
couldn't generate an overdose of radiation. I know that when I
worked on a locamotive brake system, the emergency brake
position acted physically, in a way that nothing in the rest of
the system could possibly prevent the brakes from being
applied.)
> Let's take a specific example. Suppose there is a function:
> void f(int x);
> and that f is not exception safe for all values of x. In fact
> when f (3) is called, an exception is thrown in an unexpected
> place, and the system misbehaves - zapping the patient with a
> deadly amount of X-ray radiation.
> But suppose that f(3) can never be called at runtime in the
> final application. ie There is no calling site of f where x
> can possibly be the parameter 3.
> If we were to check f for exception safety, we might find this
> f(3) problem and fix it. This will however be a waste of
> time, and achieves zero improvement on robustness.
Until some modification elsewhere causes f to be called with 3.
Either f's contract allows it to be called with 3, or it
doesn't. If f's contract allows it to be called with 3, then f
must handle 3 correctly. If it doesn't there should be an
assert at the top of the function, so that the system will fail
(and of course, all calling code should be checked that it
respects the contract).
Nick Keighley
<sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
> There aren't any formal proof methods for medical testing - if there
> were, they'd be used. Not sure I'm convinced by the analogy.
Stuart Golodetz
I remember this happening - not sure what you're saying though. I
thought this was a straightforward case of testing gone wrong?
Stuart Golodetz
> On Jul 8, 11:18 pm, Stuart Golodetz
> <sgolod...@NdOiSaPlA.pMiPpLeExA.ScEom> wrote:
>> All the testing in the world won't be enough in most cases: you'll never
>> have enough time to test your code for all possible inputs. For
>> safety-critical systems, it's a choice between something that can
>> potentially give you the confidence levels you require in your system,
>> and something that can't. It's a no-brainer really.
>
> Whether you call it robustness, correctness, reliability, stability,
> solidness or just plain old quality - isn't very interesting.
I'm not sure that interest really comes into it. The distinction is
important because you're talking about different properties of the system.
> What I
> mean is how well the software works as expected (in all possible
> situations). If you don't know what you expect than none of the
> methods we have discussed can save you.
If you expect wrong, you'd prefer that your mistake doesn't result in
your program launching a nuke at somebody. If I write a piece of
software that works correctly for all specified inputs, and I document
those for the end-user, then the program is "correct". If I don't
sensibly handle inputs that shouldn't be given by the end-user, then the
program still isn't "robust" - if the end-user accidentally provides
such an input regardless, bad things can happen (and we don't want
that). Two entirely different properties there.
> Formal Methods are different than general Software Engineering
> Process. We started out this conversation talking about a Software
> Engineering Process related activity (checking each function for 20
> rules). Formal Methods are quite different. They use first-order
> logic to prove that a given algorithm is correct. See the book:
> Software Reliability Methods, Peled/Springer.
I appreciate the distinction - I'm talking about the use of formal proof
methods as one of the more hardcore ways of going about ensuring system
quality. The original process suggested by the OP was a different kettle
of fish, but the basic goal - to try and build a good system - was the same.
> Formal Methods *cannot* be applied to anything other than a trivially
> small program.
I guess that slightly depends on your definition of "trivially small".
See the paper entitled "Practitioners' views on the use of formal
methods: an industrial survey by structured interview" if you've got
access to it. Just in case, here's a quote (emphasis mine):
"A guide to the size of the systems developed using formal methods is
shown in Table 2. The figures should be taken as a rough guide only due
to possible variations in the measurement of a line of code and in the
programming languages used. However, *they indicate that formal methods
were used on systems typically in the region of tens of Kloc*. The
interviewees were asked if large systems were a problem when using
formal methods (compared with any other method). Answers varied somewhat
but, generally, the impression was that size is not a major obstacle any
more than other methods. Marconi and Praxis indicated that proving
becomes problematic with large systems and that the proof checkers and,
to a lesser extent, model checkers may not scale up very well. For
formal specification, though, IBM said that large systems are dealt with
by breaking the system down into 'encapsulated' sub-components that
could be dealt with separately."
Evidently there are issues with applying formal methods to larger
systems, but it does seem to be practical for programs of non-trivial size.
> They break down quickly with complexity and large
> ranges of chaotic input. Certainly anything as complex as the
> software running an X-ray machine cannot and is not verified using
> Formal Methods. I have colleagues that work on a line of one of the
> most common medical robots that are used in surgeries. They
> specifically do not use these type of methods as they are
> impractical.
That may well be the case in their problem domain - it doesn't mean that
formal methods aren't applicable elsewhere.
> You may be surprised to learn that they work pretty much
> the same way as most professional developers work, they are just given
> more time to get it right, and a hell of a lot bigger QA team.
I am surprised to learn that, yes. But if it works for them, then fair
enough - as I said, I'm only trying to make the point that formal
methods are applicable in some settings, not all of them trivial. Since
you're trying to supply an argument of the form "for all, not
applicable", I think you have a significantly higher burden of proof
than I do here.
> Nothing can prove the absence of bugs in a nontrivial realworld
> application. As I said, perfect software is virtually impossible.
Of course - but you can get a lot closer to perfect software if you go
about it the right way. Formal proof methods are not IMHO applicable to
general software development because of the costs, but they are
applicable in some scenarios. Here's another quote from the above survey
paper, for the sake of interest:
"Formal methods are worthwhile in terms of improved quality of software
with little or no additional lifecycle costs, but only when compared to
a rigorous development lifecycle where the cost of software errors is
high. If the market does not demand high quality software, then it is
more difficult to justify their use."
> Yes, you can't test every possible set of inputs. Testing isn't
> everything. Neither is anything else. None of the methods discussed
> are a magic bullet. It is a question of balance. If you want more
> reliable software than you need to spend more time on it. Moving off
> balance isn't going to help.
Of course formal methods aren't a magic bullet - but you can achieve
levels of confidence in your software using them that are unattainable
just by testing. In some domains that's important.
>> Since 3 is unexpected, and you've improved the
>> behaviour for it, you've likely improved the robustness of the program
>> (assuming your change didn't mess up something else). Whether it's a
>> waste of time or not in this particular instance is another matter.
>
> You didn't understand my example. It is impossible (not just
> unexpected) that f is called with a parameter of 3. Go and study my
> example again.
I did understand your example, but there's no reason to suppose that
f(3) can never happen. Just because it doesn't happen in the present
iteration of the code doesn't mean that it never will. Software changes.
If you can assume that bad inputs will never happen, then software
robustness is a non-issue - but in the real world, you can't.
Cheers,
Stu
> -Andrew.
Andrew Tomazos
> > Robustness is quantified by the degree to which software meets
> > its requirements.
>
> Amongst other things. (Or are you including "implicit
> requirements"? I've never seen a requirements specification
> which said that the program shouldn't core dump, but of course,
> programs which core dump aren't robust.)
I'm talking about all requirements. Everything you expect the
software to do in all situations. I'm not necessarily talking about
formal written requirements that are part of some engineering
process. Clearly most software is expected not to core dump.
> > > I'm all for avoiding onerous processes, except when they're
> > > necessary (which is sometimes the case).
>
> > What you are missing is that time saved on onerous process can
> > be spent on runtime testing/debugging.
>
> Whose talking about an "onerous process"? We're talking about a
> means of simplifying, and perhaps even mechanising, part of an
> essential step.
Stuart Golodetz was talking about onerous process, in the paragraph I
was replying to. Consider reading a thread before chiming in.
> > Perfect software is practically impossible for a nontrivial
> > requirement set. You have to look at how much robustness you
> > get per unit development time. This is true whether or not
> > the software is mission-critical and/or potentially
> > life-threatening. That is what I mean when I say that moving
> > from the balance point toward (B) doesn't get you more
> > robustness in the same amount of time.
>
> No. It gets you more robustness in less time, because the
> errors are found earlier. Finding an error which only shows up
> in a test takes significantly more time [snip] integration tests costs
> less than one found in the field.
Your equation assumes that the number of errors found with each
process per unit time is constant. I agree that repairing a runtime
bug takes longer than repairing one found statically. However there
is a diminishing return from static analysis - eventually you reach a
BALANCE point at which looking for bugs at runtime is more useful.
> Using a check list is simply a way of making some aspects of the
> code review more efficient (or even automatic).
I am not against this. My position is that there is a balance.
> Until some modification elsewhere causes f to be called with 3.
I said that in the *final* system, f cannot possibly be called with a
parameter of 3. There are no further modifications. Please study the
example more closely.
-Andrew.
Andrew Tomazos
> > What I
> > mean is how well the software works as expected (in all possible
> > situations). If you don't know what you expect than none of the
> > methods we have discussed can save you.
>
> If you expect wrong, you'd prefer that your mistake doesn't result in
> your program launching a nuke at somebody.
> If I write a piece of software that works correctly for all specified inputs, and I document
> those for the end-user, then the program is "correct". If I don't
> sensibly handle inputs that shouldn't be given by the end-user, then the
> program still isn't "robust" - if the end-user accidentally provides
> such an input regardless, bad things can happen (and we don't want
> that). Two entirely different properties there.
You're quibbling about trivial differences in definition. Under *my*
definition of robustness (the one you are to use when I write the
word) there is no distinction between what you would prefer the
software to do and what you expect it to do. Your expectation covers
all possible inputs and states of the program. Think about it as a
function of the set of all possible input I to the set of all possible
behaviors B:
expect: I -> B
You can generate this function by answering the question:
for all i element of I:
expect(i) = "Given input i what behaviour should the program
exhibit?"
The robustness of the program is then measured by some metric (count
()) over I, where the actual performance:
actual: I -> B
is compared to the expected performance.
Let the set C be generated by:
for all i element of I
i is an element of C iff (actual(i) = expect(i))
And finally:
robustness = ( count(C) / count(I) )
This should be rigorous enough for you.
> > Formal Methods *cannot* be applied to anything other than a trivially
> > small program.
>
> I guess that slightly depends on your definition of "trivially small".
> See the paper entitled "Practitioners' views on the use of formal
> methods: an industrial survey by structured interview" if you've got
> access to it. Just in case, here's a quote (emphasis mine):
>
> *they indicate that formal methods
> were used on systems typically in the region of tens of Kloc*.
Yeah - the paper you have quoted is bullshitting you basically.
Nobody has applied a formal method to any realworld program of 20,000
lines. Perhaps some self-contained mathematical simulation that
doesn't actually do anything. It just can't be done. Anytime you hit
the network, or the filesystem, or any input or output device, or the
user interface, or a system call, or use a library - it just does not
fit into a formal proof.
> "Formal methods are worthwhile in terms of improved quality of software
> with little or no additional lifecycle costs, but only when compared to
> a rigorous development lifecycle where the cost of software errors is
> high. If the market does not demand high quality software, then it is
> more difficult to justify their use."
>
> Of course formal methods aren't a magic bullet - but you can achieve
> levels of confidence in your software using them that are unattainable
> just by testing. In some domains that's important.
Listen carefully. I've studied and used formal methods first hand.
They are infeasible to apply to realworld applications. A project
must be small, self-contained and have rigorous mathematically defined
requirements under all possible input. Go and try to use formal
methods to prove a tiny toy program is correct, and you will see quite
quickly what I am talking about.
> >> Since 3 is unexpected, and you've improved the
> >> behaviour for it, you've likely improved the robustness of the program
> >> (assuming your change didn't mess up something else). Whether it's a
> >> waste of time or not in this particular instance is another matter.
>
> > You didn't understand my example. It is impossible (not just
> > unexpected) that f is called with a parameter of 3. Go and study my
> > example again.
>
> I did understand your example, but there's no reason to suppose that
> f(3) can never happen. Just because it doesn't happen in the present
> iteration of the code doesn't mean that it never will. Software changes.
> If you can assume that bad inputs will never happen, then software
> robustness is a non-issue - but in the real world, you can't.
No, you have not understood my example. I said that in the *final*
application (ie there is no future iteration - software does not
magically change by itself) there is no calling site where f can
possibly be called with a parameter of 3. Therefore it is completely
impossible (even for bad input to the application) that f is called
with a parameter of 3. Look at it again until you understand.
-Andrew.
Jonathan Lee
> No, you have not understood my example. I said that in the *final*
> application (ie there is no future iteration - software does not
> magically change by itself) there is no calling site where f can
> possibly be called with a parameter of 3. Therefore it is completely
> impossible (even for bad input to the application) that f is called
> with a parameter of 3. Look at it again until you understand.
> -Andrew.
You seem to haven simply moved the proof from the function to the
calling system. You claim that f() doesn't need to be checked for an
input of 3 because the caller _cannot_ produce as an argument. But to
make a claim that "it is completely impossible [...] that f is called
with a parameter of 3" you must have *proved* that.
Your conclusion is built into your hypothesis.
--Jonathan
Andrew Tomazos
> > No, you have not understood my example. I said that in the *final*
> > application (ie there is no future iteration - software does not
> > magically change by itself) there is no calling site where f can
> > possibly be called with a parameter of 3. Therefore it is completely
> > impossible (even for bad input to the application) that f is called
> > with a parameter of 3. Look at it again until you understand.
>
> calling system. You claim that f() doesn't need to be checked for an
> input of 3 because the caller _cannot_ produce as an argument. But to
> make a claim that "it is completely impossible [...] that f is called
> with a parameter of 3" you must have *proved* that.
>
> Your conclusion is built into your hypothesis.
No, it is not a circular argument. Consider the following
application:
void f(unsigned int x)
{
if (x == 3)
CRASH_HORRIBLY;
else
OKAY;
}
void main_program(unsigned int i)
{
unsigned int x = i | 7;
f(x);
}
No matter what the application input value i is, f(3) can never by
called. It's impossible. This is true whether or not I prove it to
be true.
Fixing the function f for the case of x=3 will have zero impact on the
robustness of the application, and simply be a waste of time.
Do you understand now?
-Andrew.
Jonathan Lee
> No matter what the application input value i is, f(3) can never by
> called. It's impossible. This is true whether or not I prove it to
> be true.
Yes, but you are using your knowledge that it is true to make the
claim that the check is unnecessary. You can't know that the check is
unnecessary unless, well, you know that the check is unnecessary. See,
in saying that _you know_ it is impossible you must have _proved_ it
somehow otherwise you just _believe_ it.
Consider a different example with the same f() you just provided, but
a different main_program(). One that I make up and won't tell you
about. You don't know if main_program() calls f() with 3 or not. Does
your argument still stand? Should you not handle 3 in f?
Now if I were to post the contents of main_program() you could
consider it "finalized" and determine if 3 was ever passed to f().
But starting from a position of ignorance about main_program() (as you
must) does not allow you to form your hypothesis. You have to check
main_program() in order to do that.
--Jonathan
Andrew Tomazos
> Yes, but you are using your knowledge that it is true to make the
> claim that the check is unnecessary. You can't know that the check is
> unnecessary unless, well, you know that the check is unnecessary. See,
> in saying that _you know_ it is impossible you must have _proved_ it
> somehow otherwise you just _believe_ it.
In this particular example the check was unnecessary. It doesn't
matter whether we know it or not. With this application, someone that
did not make the check would have an equally robust application in the
same time. Therefore we can conclude that there exists (at least one)
case(s) where checking every function for the "checklist" is a waste
of time.
> Consider a different example with the same f() you just provided, but
> a different main_program(). One that I make up and won't tell you
> about. You don't know if main_program() calls f() with 3 or not. Does
> your argument still stand? Should you not handle 3 in f?
No, in your modified example, the same argument cannot be made,
because you have changed the conditions. This does not affect the
existence of my example.
> Now if I were to post the contents of main_program() you could
> consider it "finalized" and determine if 3 was ever passed to f().
> But starting from a position of ignorance about main_program() (as you
> must) does not allow you to form your hypothesis. You have to check
> main_program() in order to do that.
At this point I suspect you are being intentionally obtuse. My
purpose was to provide a concrete example of a situation where the
"checklist every function" process was a waste of time. I have
succeeded in providing that example. I was not suggesting this one
example was a proof that my overall position was correct - simply
trying to motivate your imagination to understand why there is a
balance between (A) and (B).
-Andrew.
Michael Tsang
> Hello all,
> To be a good little coder I want to ensure all of my functions pass
> a checklist of "robustness". To keep things simple, I want to document
> each function with a string that will indicate which of the checklist
> items the function has been audited for. Something like
>
> abcdefghiJklMnopqRsTuvwxyz
>
> which would show that items J, M, R, and T have been checked. Off the
> top of my head I came up with the list below. I wonder if anyone has
> items they think should be added to the list. Any advice welcome,
>
> --Jonathan
>
> Audit list (an implicit "where applicable" should be assumed)
> A - Arguments checked against domain
> B - Arrays have bounded access
> C - No C style casts, other casts as appropriate. Avoid
> reinterpret_cast<>
> D - No #define's - use static const, enum, or function
> E - Exception safe
> F - Floating point comparisons are safe (eg., don't check against 0.0)
> I - Use initialization lists in constructors
> L - Loops always terminate
> M - Const qualify member functions that need it
> N - "new" memory is not leaked, esp., in light of exceptions
> O - Integer overflow
> P - Wrap non-portable code in "#if"s and warn user with #else
> R - Reentrant
> Q - Const Qualify object arguments
> T - Thread safe
> V - Virtual destructor
I usually use the following compiler flags when compiling C/C++ programs:
CFLAGS="-pipe -pedantic -Wall -Wextra -Wformat=2 -Winit-self -Wunused -
Wfloat-equal -Wundef -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings -Wno-
empty-body -Wsign-conversion -Wlogical-op -Wno-missing-field-initializers -
Wredundant-decls -ggdb3 -O0 -Wconversion -std=c99 -Wbad-function-cast -Wc++-
compat -Wstrict-prototypes -Wold-style-definition -Wnested-externs"
CXXFLAGS="-pipe -pedantic -Wall -Wextra -Wformat=2 -Winit-self -Wunused -
Wfloat-equal -Wundef -Wshadow -Wcast-qual -Wcast-align -Wwrite-strings -Wno-
empty-body -Wsign-conversion -Wlogical-op -Wno-missing-field-initializers -
Wredundant-decls -ggdb3 -O0 -Wconversion -std=c++0x -Wctor-dtor-privacy -
Weffc++ -Wstrict-null-sentinel -Wold-style-cast -Woverloaded-virtual -Wsign-
promo -Wno-vla"
However, I can't avoid some warning about virtual destructors because I
derive classes from the STL, as they don't have virtual destructors.
Andrew Tomazos
> However, I can't avoid some warning about virtual destructors because I
> derive classes from the STL, as they don't have virtual destructors.
That's not recommended practice you know. You're supposed to use
containment. ( Actually, we took the third option: and threw the
entire STL out the window. :) )
-Andrew.
Nick Keighley
It was an example where if there had been a better way of validation
than testing then it might have paid to use it. The human test
subjects
were given a dose that was supposed to be far below that needed to
produce any clinical symptoms. And it damn nearly killed them.