Why I Dislike Synthetic Tests

[Andrey Karpov]

I don't like it when people use artificial code examples to evaluate the diagnostic capabilities of static code analyzers. There is one particular example I'm going to discuss to explain my negative attitude to synthetic tests.

Bill Torpey recently wrote a blog post entitled "Even Mo' Static", where he shared his view on the results of testing Cppcheck and PVS-Studio analyzers on the itc-benchmarks project, which is a set of static analysis benchmarks by Toyota ITC.

That post upset me because it would leave you with an impression that Cppcheck's and PVS-Studio's capabilities were very similar. What follows from the article is that one analyzer is better at diagnosing some types of errors and the other, at diagnosing other types of errors, but their capabilities are generally the same.

I think it's a wrong conclusion. My opinion is that our analyzer, PVS-Studio, is several times more powerful than Cppcheck. Well, it's not even an "opinion" - it's what I know for sure!

Full text: http://www.viva64.com/en/b/0471/

[Öö Tiib]

On Monday, 6 February 2017 10:56:58 UTC+2, Andrey Karpov wrote:
>
> I think it's a wrong conclusion. My opinion is that our analyzer,
> PVS-Studio, is several times more powerful than Cppcheck. Well,
> it's not even an "opinion" - it's what I know for sure!

Everybody expects you to have biased opinion about your product.
As for neutral party ... all the text in article felt nonsensical and
religious.

How your product detects if null pointer dereference was designed
maliciously to crash, or test of something else contains accidental
defect that may crash or crash was really part of requirements?
Are you making psychic software? :-D :-P Whatever it does, I don't
expect it to do that. So for me your program misbehaves, sorry.

[Andrey Karpov]

On Monday, 6 February 2017 14:43:35 UTC+3, Öö Tiib wrote:
> Everybody expects you to have biased opinion about your product.
> As for neutral party ... all the text in article felt nonsensical and
> religious.
>
> How your product detects if null pointer dereference was designed
> maliciously to crash, or test of something else contains accidental
> defect that may crash or crash was really part of requirements?
> Are you making psychic software? :-D :-P Whatever it does, I don't
> expect it to do that. So for me your program misbehaves, sorry.

Everything is based on our experience of checking hundreds of projects: http://www.viva64.com/en/examples/

I suggest having a closer look at the description in the article. The criterion is the presence of AND in the title and strange code. When we say strange code we mean when NULL is assigned IMMEDIATELY and is dereferenced right after it. The statement itself

T *x = 0;

*x = 0;


doesn't look natural. Please, understand that people don’t make such mistakes. If the assignment is in different places in the body of the function, then yes, it can be a mistake. And we will find such a case. But you can't go wrong, dereferencing a pointer on the next line.

However, the analyzer is ready for such cases as well and just to be sure, we analyze the name of the function too.

I can change my mind only in case I see a real example of such an error, where there is an assignment and immediate dereference, although it wasn’t meant to be so. And in this case the name of the function is let’s say, crash. But there is no such a thing in the universe.

[Öö Tiib]

On Wednesday, 8 February 2017 08:52:32 UTC+2, Andrey Karpov wrote:
> On Monday, 6 February 2017 14:43:35 UTC+3, Öö Tiib wrote:
> > Everybody expects you to have biased opinion about your product.
> > As for neutral party ... all the text in article felt nonsensical and
> > religious.
> >
> > How your product detects if null pointer dereference was designed
> > maliciously to crash, or test of something else contains accidental
> > defect that may crash or crash was really part of requirements?
> > Are you making psychic software? :-D :-P Whatever it does, I don't
> > expect it to do that. So for me your program misbehaves, sorry.
>
> Everything is based on our experience of checking hundreds of projects: http://www.viva64.com/en/examples/

The problem may be that you checked only with works of rehearsal
and/or beneficence.

> I suggest having a closer look at the description in the article. The
> criterion is the presence of AND in the title and strange code. When
> we say strange code we mean when NULL is assigned IMMEDIATELY
> and is dereferenced right after it. The statement itself
>
> T *x = 0;
>
> *x = 0;
>
> doesn't look natural.

I did read the article. Is it self-irony? Repeating same points
again and again do not make those any more correct.

Q: Does your software have say diagnostic #413 about "accidental-
looking null pointer dereference" and #417 about "willful-looking
null pointer dereference"?
A: No.
Q: So why you detect that willful-lookingness?
A: To silently filter out.
Q: Why? That is bad feature, how to turn it off?
A: There's no way.
Q: Wtf?
A:

> Please, understand that people don’t make such mistakes.

Q: Who told about mistakes? Everybody I know would *love*
to find out willful sabotage *even* *sooner* than accidental
mistakes.

I already wrote that. It is still quoted above. You just are with
fixed mindset and so can't read nor can reason about it.

<snip rest of the same repeated over and over>

[Rick C. Hodgin]

On Wednesday, February 8, 2017 at 1:52:32 AM UTC-5, Andrey Karpov wrote:
> On Monday, 6 February 2017 14:43:35 UTC+3, Öö Tiib wrote:

> > Everybody expects you to have biased opinion about your product...
> Everything is based on our experience of checking hundreds of projects...

Your product may be fantastic. But, you're going about pushing it the
wrong way. You come across as combative and hostile toward those things
many of us have relied upon for years.

If you want to get us to use your product, you'll need to teach us why
your way is better.

There are some excellent videos with Kostya Serebryany from Google on
his use of sanitizers to track down real bugs in code. He explains
what his product does, and the types of bugs it finds, along with
giving some numbers for how many bugs are found at various companies.

2013: https://www.youtube.com/watch?v=Q2C2lP8_tNE
2014: https://www.youtube.com/watch?v=V2_80g0eOMc
2015: https://www.youtube.com/watch?v=qTkYDA0En6U

I haven't found him speaking similarly in 2016 or 2017 yet. If
anyone knows additional links, please post them.

Thank you,
Rick C. Hodgin

[Rick C. Hodgin]

A video from this year at CppCon 2017 entitled "Fuzz or lose...":

2017: https://www.youtube.com/watch?v=k-Cv8Q3zWNQ