C++ tests/testing

[Robert Hutchings]

I decided to start a new thread regarding the "vetting" of C++ job
candidates.

When you interview prospective C++ developers, do you give them tests?
For example, show them a small C++ program that has incorrect or
improper statements, and ask the candidate to identify those flaws. I
have had a couple of these.

Another "test" is to ask the candidate to write a small-ish program that
will calculate some value or perhaps store and retrieve data.

I'd be interested to hear what kind of tests or interview questions you
use to qualify candidates...

[Wouter van Ooijen]

Robert Hutchings schreef op 26-Sep-14 4:17 PM:

If you do this on paper, remember to ignore all problems that would be
diagnosed by a compiler.

IMO the danger of such 'small' tests is that you don't test what is
realy important: whether the candidate gets the 'big architecture
picture' correct.

My approach was to get the candidate to talk. Give him/her a piece of
code to lak about, or ask her/him for his greatest achievement so far.
Not for the achievement itself, but for the talking.

Wouter van Ooijen

[Scott Lurndal]

Indeed. Asking about language features or STL is pretty silly; one
should be determining abstract problem solving skills and, when interviewing
for an architect, how the candidate would architect a solution.

[Wouter van Ooijen]

Scott Lurndal schreef op 26-Sep-14 5:31 PM:

Even for such a question (which is IMO much better than asking for nitty
gritty details) I would be more interested in range of solutions such a
person would propose, and his arguments for and against each solution.

Today I had to compose (multiple-choice) questions for the C++ exam for
my course. I visted a number of sites that show such questions. To my
surprise I could use only a very limited number (maybe 5%), because the
rest was about wording, syntax details, or just plain wrong.

Note that some detail questions *are* good questions. Like: what is
probably wrong with this function?

T & f();

(answer: what does that returned reference refer to? A global -> yuk! A
local -> worse! An allocated heap object -> who is responsible for
managing this object?)

Wouter van Ooijen

[Richard]

[Please do not mail me a copy of your followup]

Robert Hutchings <rm.hut...@gmail.com> spake the secret code
<m03sij$6fk$1...@dont-email.me> thusly:

>When you interview prospective C++ developers, do you give them tests?
>For example, show them a small C++ program that has incorrect or
>improper statements, and ask the candidate to identify those flaws. I
>have had a couple of these.
>
>Another "test" is to ask the candidate to write a small-ish program that
>will calculate some value or perhaps store and retrieve data.

We have done both of these, but some people choke up when you ask
them to write code in front of you in an interview situation. We
supply them a programming problem in advance (make sure they can't
google the answer!) and give them as much time as they'd like to
complete the problem. This makes the programming less a
time-stressor activity and more of a natural programming activity.

I also build a list of questions. I tell them that I don't expect
them to know the answers to all my questions, but that the questions
are designed to find the edge of their knowledge. Then I'll ask more
questions to try and walk along that edge. This tells me the extent
of their knowledge about the programming language, design techniques,
problem domain, whatever. I get a better insight into the depth of
their knowledge this way rather than building a list that every
candidate must answer correctly.
--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
The Computer Graphics Museum <http://computergraphicsmuseum.org>
The Terminals Wiki <http://terminals.classiccmp.org>
Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

[Richard]

[Please do not mail me a copy of your followup]

sl...@pacbell.net spake the secret code
<DFfVv.301660$y33.1...@fx28.iad> thusly:

>Indeed. Asking about language features or STL is pretty silly; [...]

I find value in asking about such things. For one, I don't want to
hire a C++ programmer that is adding code that is full of C-isms[1].
I want a C++ programmer that understands const and references and
RAII, among other things.

I also want a C++ programmer that is familiar with the three main
abstractions in the standard library: iterators, containers and
algorithms. I don't mean to say that they should have every detail
memorized, but that they should know the broad strokes so that they
are leveraging the standard library to its full potential.

I also want a programmer that understands principles of
object-oriented design like SOLID[2] and DRY[3]. If I had to choose
between someone who knew C++ features but didn't know good design
principles and someone who knew good design principles but didn't
know C++, I'd choose the latter. But rarely is it such a cut-and-dried
choice when interviewing candidates. Usually they know a mixture of
some of both.

[1] <http://www.slideshare.net/LegalizeAdulthood/c-traps-and-pitfalls-for-c-programmers>
[2] <http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)>
[3] <http://en.wikipedia.org/wiki/Don%27t_repeat_yourself>

[Scott Lurndal]

legaliz...@mail.xmission.com (Richard) writes:
>[Please do not mail me a copy of your followup]
>
>sl...@pacbell.net spake the secret code
><DFfVv.301660$y33.1...@fx28.iad> thusly:
>
>>Indeed. Asking about language features or STL is pretty silly; [...]
>
>I find value in asking about such things. For one, I don't want to
>hire a C++ programmer that is adding code that is full of C-isms[1].

Misguided attempt at language purity.

>I want a C++ programmer that understands const and references and
>RAII, among other things.

I want programmers that can solve problems effiently and effectively.

I don't particularly
care whether they use "pure C++" (in fact, I have no problem with using
C-isms in C++ programs (e.g. snprintf instead of i/o streams). In several
of the projects I've hired for, STL wasn't even allowed (bare-metal C++ code).

There is much benefit in "C + Classes"-style of C++. As much as the
C++14 purists, if not much, much more.

[Christopher Pisz]

It really depends what you are looking for. I don't think there is one
be all test. It is usually a bit catered to the domain we are working on
rather than just the language itself.

Example:
Write me a method to convert a position on a 600X800 screen to the same
position on a 1600X1080 screen. If you get stuck, explain your reasoning.

There are also those questions that you don't expect anyone to get when
you are looking for someone more senior

Example:
What is the initialization fiasco?

If you just need a grunt medium level C++ guy, just ask the standard
virtual, pure virtual, inheritance, exceptions questions.

Then there are question to weed out the C guys applying for C++
positions (but I have a certain hatred for them):

Example:

Write me a struct,... why do you need that typedef?
How would you perform error handling here?
What are your thoughts on this global pointer in this dll?
Write this to file...you don't like fstream? what is a stream?
How would you do logging in this scenario? Write me some stubs...why do
you have variable argument lists?

[Richard Damon]

On 9/26/14, 1:38 PM, Wouter van Ooijen wrote:
>
> Even for such a question (which is IMO much better than asking for nitty
> gritty details) I would be more interested in range of solutions such a
> person would propose, and his arguments for and against each solution.
>
> Today I had to compose (multiple-choice) questions for the C++ exam for
> my course. I visted a number of sites that show such questions. To my
> surprise I could use only a very limited number (maybe 5%), because the
> rest was about wording, syntax details, or just plain wrong.
>
> Note that some detail questions *are* good questions. Like: what is
> probably wrong with this function?
>
> T & f();
>
> (answer: what does that returned reference refer to? A global -> yuk! A
> local -> worse! An allocated heap object -> who is responsible for
> managing this object?)
>
> Wouter van Ooijen
>

The first thing I would say is probably wrong with the code is that "f"
is too short of a name for a function, unless, maybe, it is being used
in the prototype of a function declaring a function pointer to be passed in.

Second, while maybe it is odd enough to raise a review question, I don't
think that "probably" wrong is right.

Some very correct applications of this sort of function (I presume this
is intended to be a global function that returns a reference):

1) f() is a maker function and is returning the created object. When the
user is done they are expected to remove the object either with a member
function or another free function, think of it sort of like new, but
returning a reference rather than a pointer.

2) f() is returning a singleton like object. f() will check if the
singleton is already created, and if not create it and return a
reference to it. Presumably some link has been made somewhere to
properly dispose of the object at program end, or perhaps it is
reference counted and there is another function to indicate that we are
done, and if no one needs it still, it can be disposed of.

3) f() could also return a "global" object, whose exact type might not
be known at the time of the compiling of the caller, but it is known to
be a subtype of some base "T". This allows the caller to get access to
the object, and the final type might not be known to link time, when an
appropriate module is linked in, defining the object and the access
function. With dynamic linking, it might even be at run time.

Returning an "auto" local to f() is bad, as such an object is gone by
the time you get to the caller.

[Wouter van Ooijen]

Richard Damon schreef op 27-Sep-14 3:42 PM:

> On 9/26/14, 1:38 PM, Wouter van Ooijen wrote:
>>
>> Even for such a question (which is IMO much better than asking for nitty
>> gritty details) I would be more interested in range of solutions such a
>> person would propose, and his arguments for and against each solution.
>>
>> Today I had to compose (multiple-choice) questions for the C++ exam for
>> my course. I visted a number of sites that show such questions. To my
>> surprise I could use only a very limited number (maybe 5%), because the
>> rest was about wording, syntax details, or just plain wrong.
>>
>> Note that some detail questions *are* good questions. Like: what is
>> probably wrong with this function?
>>
>> T & f();
>>
>> (answer: what does that returned reference refer to? A global -> yuk! A
>> local -> worse! An allocated heap object -> who is responsible for
>> managing this object?)
>>
>> Wouter van Ooijen
>>
>
>
> The first thing I would say is probably wrong with the code is that "f"
> is too short of a name for a function, unless, maybe, it is being used
> in the prototype of a function declaring a function pointer to be passed
> in.

I agree, but as the name was not the point I probably should have used
foo but I prefer f.

> Second, while maybe it is odd enough to raise a review question, I don't
> think that "probably" wrong is right.
>
> Some very correct applications of this sort of function (I presume this
> is intended to be a global function that returns a reference):
>
> 1) f() is a maker function and is returning the created object. When the
> user is done they are expected to remove the object either with a member
> function or another free function, think of it sort of like new, but
> returning a reference rather than a pointer.

It is indeed meant as a free function. IMO returning a heap object for
which the receiver has 'lifetime responsibility' as a reference rather
than as a pointer is (at least) an attempt to confuse the reader.

> 2) f() is returning a singleton like object. f() will check if the
> singleton is already created, and if not create it and return a
> reference to it. Presumably some link has been made somewhere to
> properly dispose of the object at program end, or perhaps it is
> reference counted and there is another function to indicate that we are
> done, and if no one needs it still, it can be disposed of.

If you think a singleton is a pattern this is a valid use. IMO singleton
is an anti-pattern.

> 3) f() could also return a "global" object, whose exact type might not
> be known at the time of the compiling of the caller, but it is known to
> be a subtype of some base "T". This allows the caller to get access to
> the object, and the final type might not be known to link time, when an
> appropriate module is linked in, defining the object and the access
> function. With dynamic linking, it might even be at run time.

IMO a global object is a bad thing, one step worse than a singelton.

> Returning an "auto" local to f() is bad, as such an object is gone by
> the time you get to the caller.

That we can agree on.

But as a candidate you would definitely pass, because you understand the
(possible, probable) implications of a function that retruns a
reference. Whether you agree with me on the details is not the point, I
think I would prefer a co-worker with a sharp opinion that differs
somewhat from mine over a clone of myself. One Wouter is enough in most
situations.

Wouter van Ooijen

[Öö Tiib]

On Friday, 26 September 2014 17:18:05 UTC+3, Robert Hutchings wrote:
> I decided to start a new thread regarding the "vetting" of C++ job
> candidates.
>
> When you interview prospective C++ developers, do you give them tests?

No.

> For example, show them a small C++ program that has incorrect or
> improper statements, and ask the candidate to identify those flaws. I
> have had a couple of these.

Why? C++ is such a language that expert can easily write (but
usually avoids to) a piece that rare can tell if it is valid C++
and what it does. Typically we do not need language lawyers but
programmers who can produce working code in certain environment.

Skills with certain tools, libraries or technologies are more
important than familiarity with all the subtle nuances of C++
language.

> Another "test" is to ask the candidate to write a small-ish program that
> will calculate some value or perhaps store and retrieve data.

Yes, but that is hard to do during interview. So we give them
to solve one programming puzzle as a homework. Some simpler
one that takes few hours for skilled developer. Then we
can see if there's point for candidate to come to interview.

For me it doesn't matter if it is written in C with classes or
in C++14 ... I can read both. If it is correct, and does not
hang, crash or leak then it is good. Actually most works that
I have seen do fail with some input but even that can be OK.
Rare quality is efficiency of algorithm ... most people can't
put together efficient algorithms. Even that is OK when there
is at least one in team who can write good algorithms.

> I'd be interested to hear what kind of tests or interview questions you
> use to qualify candidates...

Most important is what, where and with whom the candidate has
been working. Different masters have different quirks of their
school. ;)

[Öö Tiib]

On Sunday, 28 September 2014 04:56:19 UTC+3, Stefan Ram wrote:

> Öö Tiib <oot...@hot.ee> writes:
> >and what it does. Typically we do not need language lawyers but
> >programmers who can produce working code in certain environment.
>

> Sometimes one wants a little bit more than this, like
> to be sure that the code will behave according to some
> specification under a certain range of possible environments.

Sometimes, yes, typically, no. In actual practice the implementers
of all environments are also usual humans. Humans make mistakes as
rule. Typically by dozens as minimum. Everything in our industry
has "ABSOLUTELY NO WARRANTIES WHATSOEVER" disclaimer attached to it.
I know of nothing that lacks it. That enforces us to test
everything we only can. Full mathematical proof that it works
does just add tiny bit of more optimism. ;)

> Knowledge of the programming language used, in many cases,
> does not hinder but help to create such code.

Yes, but programming language is evolving subject and so the
knowledge about it is moving target. Capability to efficiently gain
(and also to forget) such knowledge is on many cases even more
important.

> »The fact that the program works has no relevance.«
>
> Bartosz Milewski

I have found it easier to ask money for the programs that work.
That has sometimes some relevance to something that Bartosz Milewski
apparently did not have in mind when writing that.

[woodb...@gmail.com]

On Saturday, September 27, 2014 11:44:17 PM UTC-5, Öö Tiib wrote:
> On Sunday, 28 September 2014 04:56:19 UTC+3, Stefan Ram wrote:
>
> > Öö Tiib <oot...@hot.ee> writes:
>
> > >and what it does. Typically we do not need language lawyers but
> > >programmers who can produce working code in certain environment.
> >
> > Sometimes one wants a little bit more than this, like
> > to be sure that the code will behave according to some
> > specification under a certain range of possible environments.
>
> Sometimes, yes, typically, no. In actual practice the implementers
> of all environments are also usual humans. Humans make mistakes as
> rule. Typically by dozens as minimum. Everything in our industry
> has "ABSOLUTELY NO WARRANTIES WHATSOEVER" disclaimer attached to it.
> I know of nothing that lacks it.

I don't include that anywhere. Lots of organizations
seem to repeat that in every file and that's no fun.

http://www.metrolyrics.com/fearless-lyrics-dc-talk.html


Brian
Ebenezer Enterprises
http://webEbenezer.net

[Balwinder S Dheeman]

+1

IMHO, A programming language is just a skill, can be easily acquired
with some practice, and same is true for standard or third party
libraries. What I'll look for in good programer as a candidate is
his/her approach, promptness and, or logic at solving problems; the
programming is an art and it's neither easy to test nor acquire *the art
of programming* ;)

--
Balwinder S "bdheeman" Dheeman (http://bdheeman.BlogSpot.in/)
"Working together, works! The proof is GNU/Linux and F/LOSS Projects;
Do you too voluntarily work on or contribute to making any difference?"