duck-type-checking?
Joe Strout
typing.
However, I still want to sprinkle my code with assertions that, for
example, my parameters are what they're supposed to be -- too often I
mistakenly pass in something I didn't intend, and when that happens, I
want the code to fail as early as possible, so I have the shortest
possible path to track down the real bug. Also, a sufficiently clever
IDE could use my assertions to know the type of my identifiers, and so
support me better with autocompletion and method tips.
So I need functions to assert that a given identifier quacks like a
string, or a number, or a sequence, or a mutable sequence, or a
certain class, or so on. (On the class check: I know about
isinstance, but that's contrary to duck-typing -- what I would want
that check to do instead is verify that whatever object I have, it has
the same public (non-underscore) methods as the class I'm claiming.)
Are there any standard methods or idioms for doing that?
Thanks,
- Joe
J Kenneth King
I generally use the 'assert' keyword when I'm in a rush otherwise unit
tests generally catch this kind of thing.
Matimus
The only thing that comes to mind is to use explicit checking
(isinstance). The new 'abc' module in 2.6 is worth a look. It seems
like this sort of thing might become a _little_ more popular.
I think duck-typing is great, but there are instances where you really
want the code to be better at documenting your intention about what
interface you expect an object to have, and also to catch the problems
that it might cause early and in a place where you might actually be
able to catch a meaningful exception. I've been looking and haven't
found any clear rules here. It is a trade off. The more complex the
interface, the more you will likely want to do an explicit check. On
the other hand, the more complex the interface the more likely it is
that you are violating the 'Interface Segregation Principle'. That is,
you probably want to consider breaking the functionality down into
smaller interfaces, or even separate classes.
IMO explicit checking, like global variables or using a 'goto' in C,
are evil. That isn't to say that you should _never_ use them. Heck,
there are plenty of gotos in CPython. You want to minimize, and
clearly document the places where your code is evil. There are times
where it is necessary.
Matt
Cristina Yenyxe González García
>
> So I need functions to assert that a given identifier quacks like a string,
> or a number, or a sequence, or a mutable sequence, or a certain class, or so
> on. (On the class check: I know about isinstance, but that's contrary to
> duck-typing -- what I would want that check to do instead is verify that
> whatever object I have, it has the same public (non-underscore) methods as
> the class I'm claiming.)
>
> Are there any standard methods or idioms for doing that?
You can use hasattr(object, name), with 'name' as the name of the
public method to check. It returns True if the object responds to that
method, False otherwise.
> Thanks,
> - Joe
Hope it helps :)
Tim Rowe
> Let me preface this by saying that I think I "get" the concept of
> duck-typing.
>
> However, I still want to sprinkle my code with assertions that, for example,
> my parameters are what they're supposed to be -- too often I mistakenly pass
> in something I didn't intend, and when that happens, I want the code to fail
> as early as possible, so I have the shortest possible path to track down the
> real bug. Also, a sufficiently clever IDE could use my assertions to know
> the type of my identifiers, and so support me better with autocompletion and
> method tips.
>
> or a number, or a sequence, or a mutable sequence, or a certain class, or so
> on. (On the class check: I know about isinstance, but that's contrary to
> duck-typing -- what I would want that check to do instead is verify that
> whatever object I have, it has the same public (non-underscore) methods as
> the class I'm claiming.)
>
> Are there any standard methods or idioms for doing that?
What do you actually mean by "Quacks like a string"? Supports the
'count()' method? Then you find out if it doesn't when you try to
apply the 'count()' method. Supports some method that you don't
actually use? Then why do you care?
The point about duck typing is that something might quack like a duck
but not walk like a duck -- one of those duck calls that hunters use,
for instance. Quacking like a duck doesn't actually mean it /is/ a
duck, it means that it will do instead of a duck if the quack is all
you want.
If all you care about is that 'foo' quacks like a duck then all you
need to know is that
hasattr(foo, "quack")
and (if so)
callable(foo.quack)
If you need to know that it walks like a duck, mates like a duck and
tastes like a duck when roasted, you probably want it to really /be/ a
duck and should go back to inheritance.
--
Tim Rowe
Joe Strout
> What do you actually mean by "Quacks like a string"? Supports the
> 'count()' method? Then you find out if it doesn't when you try to
> apply the 'count()' method. Supports some method that you don't
> actually use? Then why do you care?
Because if I write a method with the intention of treating the
arguments like strings in various ways (slicing, combining with other
strings, printing to stdout or writing to a file, etc. etc.), and some
idiot (i.e. me six weeks later or long after I should have gone to
bed) manages to accidentally pass in something else, then I want my
program to blow up right away, not plant a roadside bomb and
cheerfully wait for me to drive by.
This is not hypothetical -- just last week I had a hard-to-track-down
abend that ultimately turned out to be an NLTK.Tree object stored
someplace that I expected to only contain strings. I found it by
littering my code with assertions of the form
isinstance(foo,basestring). If I'd had those in there in the first
place, not only documenting my assumptions but letting the computer
check them for me, it would have saved me a lot of grief.
But in the spirit of duck-typing, I shouldn't actually check that foo
is a basestring. I should instead check that foo quacks like a
basestring. I'd define that is:
"x quacks like a basestring if it implements all the public methods of
basestring, and can be used in pretty much any context that a
basestring can."
I have to say "pretty much" since obviously there may be some evil
context that actually checks isinstance. But that's the pathological
case, and we shouldn't let it prevent us from neatly handling the
typical case.
> The point about duck typing is that something might quack like a duck
> but not walk like a duck -- one of those duck calls that hunters use,
> for instance. Quacking like a duck doesn't actually mean it /is/ a
> duck, it means that it will do instead of a duck if the quack is all
> you want.
Well, that's one point, but it's not the only point. If I have code
that expects to be working with strings, and I want to purposely give
it something else, then it's reasonable to expect that the something-
else will act like a string in every way that a string is likely to be
exercised. My string wrapper or doppleganger_string or whatever
should implement all the methods, and support all the operators and
type conversions, that basestring does.
> If you need to know that it walks like a duck, mates like a duck and
> tastes like a duck when roasted, you probably want it to really /be/ a
> duck and should go back to inheritance.
I can't agree; there are times when inheritance just won't do, for
example when you don't have control over the object creation, because
they come from some factory method you can't change. In that case you
may need to make a wrapper instead of a subclass, but if you've
faithfully implemented the interface of the original class, you should
be able to use it wherever the original class could be used (within
reason).
So, since it's pretty clear by now that there's no standard idiom for
this, I'll try to cook up something myself. For classes, I think we
could do a two-stage test:
1. If the given object isinstance of the specified class, then all is
good and return immediately.
2. Otherwise, check each of the public attributes of the specified
class, and make sure that the given object has corresponding callable
attributes.
For case 2, we might be able to cache the result so that we don't do
all that work again the next time the same type comparison is done.
Anyway, I'll evolve something in our shop here and live with it a
while, and in a few months I'll either share what we develop for this
purpose, or admit it was a horrible idea all along. :)
Cheers,
- Joe
Terry Reedy
> Let me preface this by saying that I think I "get" the concept of
> duck-typing.
>
> However, I still want to sprinkle my code with assertions that, for
> example, my parameters are what they're supposed to be -- too often I
> mistakenly pass in something I didn't intend, and when that happens, I
> want the code to fail as early as possible, so I have the shortest
> possible path to track down the real bug. Also, a sufficiently clever
> IDE could use my assertions to know the type of my identifiers, and so
> support me better with autocompletion and method tips.
>
> So I need functions to assert that a given identifier quacks like a
> string, or a number, or a sequence, or a mutable sequence, or a certain
> class, or so on. (On the class check: I know about isinstance, but
> that's contrary to duck-typing -- what I would want that check to do
> instead is verify that whatever object I have, it has the same public
> (non-underscore) methods as the class I'm claiming.)
>
> Are there any standard methods or idioms for doing that?
Test for one on more of the methods that the parameter must have in that
function, that constitute 'quacking like a duck'
>>> ob = [1,2,3]
>>> assert ob.__getitem__ and ob.__setitem__
>>> ob = 3
>>> assert ob.__getitem__ and ob.__setitem__
Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
assert ob.__getitem__ and ob.__setitem__
AttributeError: 'int' object has no attribute '__getitem__'
Focus on testing ones that differentiate the species you want. Lists
have __add__ and __mul__ but do not have
>>> assert ob.__sub__ and ob.__floordiv__
Since methods are always True, no need for hasattr().
Terry Jan Reedy
Terry Reedy
> 2008/11/12 Joe Strout <j...@strout.net>:
>> So I need functions to assert that a given identifier quacks like a string,
>> or a number, or a sequence, or a mutable sequence, or a certain class, or so
>> on. (On the class check: I know about isinstance, but that's contrary to
>> duck-typing -- what I would want that check to do instead is verify that
>> whatever object I have, it has the same public (non-underscore) methods as
>> the class I'm claiming.)
>>
>> Are there any standard methods or idioms for doing that?
>
> public method to check. It returns True if the object responds to that
> method, False otherwise.
Too hard. For methods, which are what define duck species, and any
attribute guaranteed to not be null, "assert ob.name" is equivalent to
"assert hasattr(ob, 'name')".
tjr
prue...@latinmail.com
It seems to me that what you are describing is exactly what abcs were
added for in 2.6, in particular registration:
class AnotherClass(metaclass=ABCMeta):
pass
AnotherClass.register(basestring)
assert isinstance(str, AnotherClass)
Please read this first:
http://www.python.org/dev/peps/pep-3119/
and tell us why that would not work.
George Sakkis
As you just showed they're not; one raises AttributeError and the
other AssertionError. Not that it matters much but these are typically
reported differently by testing frameworks.
George
Joe Strout
> It seems to me that what you are describing is exactly what abcs were
> added for in 2.6, in particular registration:
>
> class AnotherClass(metaclass=ABCMeta):
> pass
> AnotherClass.register(basestring)
>
> assert isinstance(str, AnotherClass)
>
> Please read this first:
> http://www.python.org/dev/peps/pep-3119/
>
> and tell us why that would not work.
You're right, that is exactly the need I was looking to fill. Thanks
for pointing it out!
Now I have only two regrets: first, that our shop is still using 2.5
and this functionality is new in 2.6; and second, that it does not
appear to include an easy way to check the types of elements of a
sequence or mapping.
Still, I can see that this is a very carefully considered PEP and is
the best solution for the future, so I will study it carefully and
incorporate it into our practices ASAP.
Thanks,
- Joe
Tim Rowe
>> If you need to know that it walks like a duck, mates like a duck and
>> tastes like a duck when roasted, you probably want it to really /be/ a
>> duck and should go back to inheritance.
>
> I can't agree; there are times when inheritance just won't do, for example
> when you don't have control over the object creation, because they come from
> some factory method you can't change. In that case you may need to make a
> wrapper instead of a subclass, but if you've faithfully implemented the
> interface of the original class, you should be able to use it wherever the
> original class could be used (within reason).
And then the original class definition changes -- new members added --
but the ones from the factory class don't change, and so are no longer
compliant with the the factory class (which doesn't support the new
"form_pun_relating_to_avoiding_a_high_hazard()" method) . Fine. If you
checked for all the members of the class that your code uses, it makes
no difference. ABCs give you a way of doing this, but in their absence
it's a long list of checks. If you /can/ use inheritance then it saves
having to do those checks.
--
Tim Rowe
Joe Strout
> And then the original class definition changes -- new members added --
> but the ones from the factory class don't change, and so are no longer
> compliant with the the factory class (which doesn't support the new
> "form_pun_relating_to_avoiding_a_high_hazard()" method) .
Yes, that's certainly a risk. But I'd rather risk something that
breaks the code in an obvious way during development, than risk
something that breaks it in a subtle way and is more likely to be
discovered by the end-user.
> Fine. If you checked for all the members of the class that your code
> uses, it makes
> no difference. ABCs give you a way of doing this, but in their absence
> it's a long list of checks.
True. I love the ABC approach; now I just have to figure out whether
I love it enough to move our entire company over to 2.6 (despite 2.5's
great advantage that it's pre-installed on every Mac by default), or
whether instead I'll come up with some sort of ABC-compatible interim
solution I can use to hobble along until we do switch.
> If you /can/ use inheritance then it saves having to do those checks.
Well, sure. And any sensible checking system would first check
isinstance and issubclass before painfully checking each of the
methods and properties in the declared interface (or ABC), for
performance reasons at the very least.
But the point is to have a system that will easily and efficiently
handle the common cases, while still being able to handle the uncommon
ones.
Best,
- Joe
Terry Reedy
> On Nov 12, 2008, at 2:42 PM, Tim Rowe wrote:
> True. I love the ABC approach; now I just have to figure out whether I
> love it enough to move our entire company over to 2.6 (despite 2.5's
> great advantage that it's pre-installed on every Mac by default), or
> whether instead I'll come up with some sort of ABC-compatible interim
> solution I can use to hobble along until we do switch.
Grab _abcoll.py, abc.py, numbers.py and put in application search path.
Modify collections.py to import _abcoll.py.
See if they word 'as is' or if they use new 2.6 features, and if so, if
they can be backported. (If they have to be changed, others might
appreciate the revised version too if made available via pypi.)
Steven D'Aprano
> Let me preface this by saying that I think I "get" the concept of duck-
> typing.
>
> However, I still want to sprinkle my code with assertions that, for
> example, my parameters are what they're supposed to be -- too often I
> mistakenly pass in something I didn't intend, and when that happens, I
> want the code to fail as early as possible, so I have the shortest
> possible path to track down the real bug.
I'm surprised nobody has pointed you at Alex Martelli's recipe here:
http://code.activestate.com/recipes/52291/
While the recipe is great, it can be tiresome to apply all the time. I
would factor out the checks into a function, something like this:
def isstringlike(obj, methods=None):
"""Return True if obj is sufficiently string-like."""
if isinstance(obj, basestring):
return True
if methods is None:
methods = ['upper', 'lower', '__len__', '__getitem__']
for method in methods:
if not hasattr(obj, method):
return False
# To really be string-like, the following test should pass.
if len(obj) > 0:
s = obj[0]
if s[0] != s:
return False
return True
--
Steven
greg
> This is not hypothetical -- just last week I had a hard-to-track-down
> abend that ultimately turned out to be an NLTK.Tree object stored
> someplace that I expected to only contain strings. I found it by
> littering my code with assertions of the form
> isinstance(foo,basestring).
But you have to ask yourself whether the time taken to
write and maintain all these assertions is really
cost-effective. In my experience, occurrences like this
are actually extremely rare -- the vast majority of the
time, you do get a failure very quickly, and it's fairly
obvious from the traceback where to look for the problem.
It's as annoying as hell when something like this does
happen, but averaged over the lifetime of the project,
I find it doesn't cost all that much time.
--
Greg
greg
> I'd rather risk something that
> breaks the code in an obvious way during development, than risk
> something that breaks it in a subtle way and is more likely to be
> discovered by the end-user.
Seems to me that putting in these kinds of assertions isn't
going to make much difference to the proportion of bugs that
slip through testing.
Your test suite still has to be comprehensive enough to
trigger the assignment of the erroneous object. If it's
not, your assertions won't catch anything. If it is, then
you would have found out during testing in any case.
It might be more difficult to *fix* the problem without
the assertions, but you will still catch it.
--
Greg
Ben Finney
> Because if I write a method with the intention of treating the
> arguments like strings in various ways (slicing, combining with
> other strings, printing to stdout or writing to a file, etc. etc.),
> and some idiot (i.e. me six weeks later or long after I should have
> gone to bed) manages to accidentally pass in something else, then I
> want my program to blow up right away, not plant a roadside bomb and
> cheerfully wait for me to drive by.
This is better achived, not by littering the functional code unit with
numerous assertions that obscure the normal function of the code, but
rather by employing comprehensive unit tests *separate from* the code
unit.
This suite of unit tests is then employed and executed any time the
code changes, and indivates any regressions in functionality from what
is asserted in the tests. Meanwhile, the code unit itself is clear and
free from the masses of error checking that one often finds in code
written without such unit test suites.
> This is not hypothetical -- just last week I had a
> hard-to-track-down abend that ultimately turned out to be an
> NLTK.Tree object stored someplace that I expected to only contain
> strings. I found it by littering my code with assertions of the form
> isinstance(foo,basestring). If I'd had those in there in the first
> place, not only documenting my assumptions but letting the computer
> check them for me, it would have saved me a lot of grief.
Rather than littering these assertions *within* the code unit, and
thereby making that code harder to read later, those assertions would
have been better placed in a unit test module that tests the behaviour
of the unit in response to environment and input.
This has the additional benefit of highlighting parts of one's code
that do not have well-defined interfaces: if the code's behaviour in
response to a simple, narrow specification of inputs and environment
is not easy to describe in a simple true-or-false testable assertion,
then the interface and/or the internals of the code unit is very
likely too complex to be maintained well.
> But in the spirit of duck-typing, I shouldn't actually check that
> foo is a basestring. I should instead check that foo quacks like a
> basestring. I'd define that is:
>
> "x quacks like a basestring if it implements all the public methods
> of basestring, and can be used in pretty much any context that a
> basestring can."
That is not duck typing. Rather than checking what foo does in
response to prodding that, by your admission, is only designed to find
out what type it is, duck typing instead advocates that you should use
foo *as though it is known to be* the type of object you want. If it
is not suitable, then appropriate exceptions will be raised and either
caught by some code that knows how to handle them, or crash the
program.
Unit tests then come into play by feeding inputs to your code unit and
asserting that the code behaves appropriately whether fed correct or
incorrect inputs: correct inputs, even of wholly unanticipated types,
should cause correct behaviour (i.e. if the code is designed to handle
ducks it should be content with a goose also); and incorrect inputs
are incorrect by definition *only if* they cause the code unit to be
unable to behave correctly (in which case an appropriate exception
should be raised).
--
\ “Pinky, are you pondering what I'm pondering?” “I think so, |
`\ Brain, but Zero Mostel times anything will still give you Zero |
_o__) Mostel.” —_Pinky and The Brain_ |
Ben Finney
paul
> Joe Strout <j...@strout.net> writes:
>> "x quacks like a basestring if it implements all the public methods
>> of basestring, and can be used in pretty much any context that a
>> basestring can."
>
> That is not duck typing. Rather than checking what foo does in
> response to prodding that, by your admission, is only designed to find
> out what type it is, duck typing instead advocates that you should use
> foo *as though it is known to be* the type of object you want. If it
> is not suitable, then appropriate exceptions will be raised and either
> caught by some code that knows how to handle them, or crash the
> program.
Warning, rant ;)
This whole theory breaks down quickly if you're writing library code.
How do your unittests help the user of your library to use it correctly?
How do you communicate incorrect usage of your interfaces to the user?
If you are able to specify the type of the arguments as part of the
interface the compiler/interpreter will help you. Types are used to
describe behaviour (if thats a good thing, I don't know). While python
has strong types, there could be used better (instead it gets worse, see
the suddently-not-sortable-list-type diskussion and the endless
repetition of the greatest of all after-the-fact theories ever "duck
typing".
cheers
Paul
BTW: Back to Java? No, not really.
Tim Rowe
> That is not duck typing.
Oh, I'm pretty sure it is. It just isn't /using/ the duck typing in
the way you'd like.
--
Tim Rowe
Joe Strout
> I'm surprised nobody has pointed you at Alex Martelli's recipe here:
>
> http://code.activestate.com/recipes/52291/
Thanks for that -- it's clever how he combines binding the methods
he'll use with doing the checking.
> While the recipe is great, it can be tiresome to apply all the time. I
> would factor out the checks into a function, something like this:
>
> def isstringlike(obj, methods=None):
> """Return True if obj is sufficiently string-like."""
> if isinstance(obj, basestring):
> return True
> if methods is None:
> methods = ['upper', 'lower', '__len__', '__getitem__']
> for method in methods:
> if not hasattr(obj, method):
> return False
> # To really be string-like, the following test should pass.
> if len(obj) > 0:
> s = obj[0]
> if s[0] != s:
> return False
> return True
Thanks for this, too; that's the sort of method I had in mind. That
last test for string-likeness is particularly clever. I'll need to
think more deeply about the implications.
Best,
- Joe
Steve Holden
objects when they are passed to functions and methods that use them.
Unfortunately the assignment of the "wrong type of object" may well have
taken place aeons ago, and it's the assignments you really need to catch
(and it's also the assignments that static languages cause compile
errors on).
So it hardly matters whether the code blows up with
Exception: my expensive type checks sounded an alarm
or
Attribute error: foo has no method bar()
since both are equally informative when it comes to tracing the faulty
assignment.
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/
Steve Holden
Craig Allen
> numerous assertions that obscure the normal function of the code, but
> rather by employing comprehensive unit tests *separate from* the code
> unit.
that doesn't seem to work too well when shipping a library for someone
else to use... we don't have access to the caller's code that needs to
be checked. I suppose if the intent is to have a true assert, that
does nothing in shipped code, then you can argue that testing
addresses some of the issues, but one, not all of them, specifically,
not the part where the problem is ably reported, and two, I don't
think we can assume assert meant that sort of assert macro in C which
compiles away in release versions.
Asserts also do not litter code, they communicate the assumptions of
the code. I like the idea of a general duck-type assertion and would
probably use that, especially since I also have arguments which can be
multiple objects, each with their own interface but similar meaning...
i.e. lower level file objects can be passed in, or my higher level
abstraction of the same file.
Craig Allen
> since both are equally informative when it comes to tracing the faulty
> assignment.
>
steve, they are not equally informative, the assertion is designed to
fire earlier in the process, and therefore before much mischief and
corruption can be done compared to later, when you happen to hit the
missing attribute.
George Sakkis
To me this seems it combines the worst of both worlds: the
explicitness of LBYL with the heuristic nature of duck typing.. might
as well call it "doyoufeellucky typing". If you are going to Look
Before You Leap, try to stick to isinstance/issubclass checks
(especially in 2.6+ that they can be overriden) instead of crafting ad-
hoc rules of what makes an object be X-like.
George
Steve Holden
I disagree. The assertion may fire a few lines before the non-existent
attribute access, but typically the damage is done in an assignment
before the function that raises the error is even called.
And I therefore suspect that a whole load of heavyweight type-checking
will be done for no very good reason.
Steven D'Aprano
That's crazy talk. Duck-typing is, at it's very nature, ad-hoc. You want
something that is just duck-like enough for your application, without
caring if it is an honest-to-goodness duck. "Duck-like" depends on the
specific application, in fact the specific *function*. You can't get any
more ad-hoc than that.
What I posted, taken from Alex Martelli, is duck-typing. It's just that
the duck-type checks are performed before any other work is done. The
isinstance check at the start of the function was merely an optimization.
I didn't think I needed to say so explicitly, it should have been obvious.
Take this example:
def foo(alist):
alist.sort()
alist.append(5)
The argument can be any object with sort and append methods (assumed to
act in place). But what happens if you pass it an object with a sort
method but no append? The exception doesn't occur until *after* the
object is sorted, which leaves it in an inconsistent state. This can be
undesirable: you might need the function foo to be atomic, either the
entire function succeeds, or none of it.
Duck-typing is great, but sometimes "if it walks like a duck and quacks
like a duck it might as well be a duck" is not enough. Once you've built
an expensive gold-plated duck pond, you *don't* want your "duck" to sink
straight to the bottom of the pond and drown the first time you put it on
the water. You want to find out that it can swim like a duck *before*
building the pond.
--
Steven
Steven D'Aprano
> Ben Finney schrieb:
>> Joe Strout <j...@strout.net> writes:
>>> "x quacks like a basestring if it implements all the public methods of
>>> basestring, and can be used in pretty much any context that a
>>> basestring can."
>>
>> That is not duck typing. Rather than checking what foo does in response
>> to prodding that, by your admission, is only designed to find out what
>> type it is, duck typing instead advocates that you should use foo *as
>> though it is known to be* the type of object you want. If it is not
>> suitable, then appropriate exceptions will be raised and either caught
>> by some code that knows how to handle them, or crash the program.
>
> Warning, rant ;)
>
> This whole theory breaks down quickly if you're writing library code.
> How do your unittests help the user of your library to use it correctly?
They don't. Unittests don't help you get better fuel economy for your car
either. Neither of those things are the purpose of unittests.
> How do you communicate incorrect usage of your interfaces to the user?
With documentation and exceptions.
> If you are able to specify the type of the arguments as part of the
> interface the compiler/interpreter will help you.
Not in Python it won't. You can argue that this is a weakness of Python,
and you might even be correct (arguably, for some definition of
"weakness"), but it is by design and won't change anytime soon, not even
with ABCs.
> Types are used to
> describe behaviour (if thats a good thing, I don't know).
Types are one way of specifying behaviour, although they are subject to
false negatives and false positives. For example, type Spam may have the
exact same behaviour and interface as type Ham, but if the compiler has
been instructed to reject anything that is not Ham, it will wrongly
reject Spam. This is one of the problems duck-typing is meant to
ameliorate.
[snip rest of rant, which seemed incoherent to me]
--
Steven
George Sakkis
cybersource.com.au> wrote:
> Take this example:
>
> def foo(alist):
> alist.sort()
> alist.append(5)
>
> The argument can be any object with sort and append methods (assumed to
> act in place). But what happens if you pass it an object with a sort
> method but no append? The exception doesn't occur until *after* the
> object is sorted, which leaves it in an inconsistent state. This can be
> undesirable: you might need the function foo to be atomic, either the
> entire function succeeds, or none of it.
In this example atomicity is not guaranteed even if alist is a builtin
list (if it contains a complex number or other unorderable object),
let alone if not isistance(alist, list). It gets worse: false
positives are less likely for full-spelled methods with well-known
names such as "sort" and "append" (i.e. if hasattr(alist, 'append'),
alist.append *probably* does what you think it does), but good luck
with that when testing for __getitem__, __iter__ for more than one
pass, __call__, and other special methods with ambiguous or undefined
semantics.
Let's face it, duck typing is great for small to medium complexity
projects but it doesn't scale without additional support in the form
of ABCs/interfaces, explicit type checking (at compile and/or run
time), design by contract, etc. It must not be a coincidence that both
Zope and Twisted had to come up with interfaces to manage their
massive (for Python at least) complexity.
George
prue...@latinmail.com
What would be actually interesting would be an switch to the python
interpreter that internally annotated function parameters with how
they are used in the function and raised an exception as soon as the
function is called instead of later. Failing earlier rather than
later. Example:
def sub(x,y):
...run some stuff
...print x[2]
...return y.strip().replace('a','b')
internally python generates:
def sub(x: must have getitem, y: must have strip and replace)
sub([1,2,3,4],5)
Error calling sub(x,y): y has to have strip() method.
Joe Strout
> What would be actually interesting would be an switch to the python
> interpreter that internally annotated function parameters with how
> they are used in the function and raised an exception as soon as the
> function is called instead of later. Failing earlier rather than
> later.
That would be interesting, but it wouldn't have helped in the case I
had last week, where the method being called does little more than
stuff the argument into a container inside the class -- only to blow
up much later, when that data was accessed in a certain way.
The basic problem was that the data being stored was violating the
assumptions of the class itself. Sometimes in the past I've used a
"check invariants" method on a class with complex data, and call this
after mutating operations to ensure that all the class invariants are
still true. But this class wasn't really that complex; it's just that
it assumed all the stuff it's being fed were strings (or could be
treated as strings), and I inadvertently fed it an NLTK.Tree node
instead (not realizing that a library method I was calling could
return such a thing sometimes).
So, in this case, the simplest solution was to have the method that
initially accepts and stores the data check to make sure that data
satisfies the assumptions of the class.
Best,
- Joe
Paul McGuire
>
> - Show quoted text -
No, this would mean:
def sub(x: must have getitem, y: must have strip, and y.strip must
return something that has replace)
Or to be even more thorough:
def sub(x: must have getitem, y: must have strip and strip must be
callable, and y.strip must return something that has replace and
replace must be callable)
So even this simple example gets nasty in a hurry, let alone the OP's
case where he stuffs y into a list in order to access it much later,
in a completely different chunk of code, only to find out that y
doesn't support the complete string interface as he expected.
-- Paul
Joe Strout
> Or to be even more thorough:
> def sub(x: must have getitem, y: must have strip and strip must be
> callable, and y.strip must return something that has replace and
> replace must be callable)
>
> So even this simple example gets nasty in a hurry, let alone the OP's
> case where he stuffs y into a list in order to access it much later,
> in a completely different chunk of code, only to find out that y
> doesn't support the complete string interface as he expected.
Very true. That's why I think it's not worth trying to be too pure
about it. Most of the time, if a method wants a Duck, you're going to
just give it a Duck.
However, I would like to also handle the occasional case where I can't
give it a Duck, but I can give it something that is a drop-in
substitute for a Duck (really, truly, I promise, and if it blows up
I'll take responsibility for it).
A real-world example from another language (sorry for that, I've been
away from Python for ten years): in REALbasic, there is a Database
base class, and a subclass for each particular database backend
(Postgres, MySQL, whatever). This works fine most of the time, in
that you can write general code that takes a Database object and Does
Stuff with it.
However, all of those database backends are shipped by the vendor, or
by plugin authors -- you can't create a useful Database subclass
yourself, in RB code, because it has a private constructor. So you
end up making your own database class, but that can't be used with all
the code that expects a real Database object.
Of course, the framework design there is seriously flawed (Database
should have been an interface, or at the very least, had a protected
rather than private constructor). And in Python, there's no way to
prevent subclassing AFAIK, so this particular issue wouldn't come up.
But I still suspect that there may be times when I don't want to
subclass for some reason (maybe I'm using the Decorator or Adapter or
Bridge pattern). Yet I'm willing to guarantee that I've adhered to
the interface of another class, and will behave like it in any way
that matters.
So, the level of assertion that I want to make in a method that
expects a Duck is just that its parameter is either a Duck, or
something that the caller is claiming is just as good as a Duck. I'm
not trying to prevent any possible error; I'm trying to catch the
stupid errors where I inadvertently pass in something completely
different, not duck-like at all (probably because some other method
gave me a result I didn't realize it could produce).
So things like this should suffice:
# simple element
assert(is_stringlike(foo))
assert(is_numeric(foo))
assert(is_like(foo, Duck))
# sequence of elements
assert(seqof_stringlike(foo))
assert(seqof_numeric(foo))
assert(seqof_like(foo, Duck))
# (also "listof_" variants for asserting mutable sequence of whatever)
# dictionary of elements
assert(dictof_like(foo, str, int))
Hmm, I was already forced to change my approach by the time I got to
checking dictionaries. Perhaps a better formalism would be a "like"
method that takes an argument, and something that indicates the
desired type. This could be a tree if you want to check deeper into a
container. Maybe something like:
assert(fits(foo, dictlike(strlike, seqlike(intlike))))
which asserts that foo is something dictionary-like that maps string-
like things to something like a sequence of integer-like things. Most
cases would not be this complex, of course, but would be closer to
assert(fits(foo, strlike))
But this is still pretty ugly. Hmm. Maybe I'd better wait for
ABCs. :)
Cheers,
- Joe
George Sakkis
You might also be interested in the typecheck module whose syntax
looks nicer, at least for the common cases: http://oakwinter.com/code/typecheck/dev/
George
Steve Holden
> On Nov 14, 2008, at 2:07 PM, Paul McGuire wrote:
>
>> def sub(x: must have getitem, y: must have strip and strip must be
>> callable, and y.strip must return something that has replace and
>> replace must be callable)
>>
>> So even this simple example gets nasty in a hurry, let alone the OP's
>> case where he stuffs y into a list in order to access it much later,
>> in a completely different chunk of code, only to find out that y
>> doesn't support the complete string interface as he expected.
>
> about it. Most of the time, if a method wants a Duck, you're going to
> just give it a Duck.
>
> However, I would like to also handle the occasional case where I can't
> give it a Duck, but I can give it something that is a drop-in substitute
> for a Duck (really, truly, I promise, and if it blows up I'll take
> responsibility for it).
I suspect this reduces our difference to a disagreement about the
meaning of the word "occasional" ...
Steve Holden
> On Nov 14, 2008, at 2:07 PM, Paul McGuire wrote:
>
>> def sub(x: must have getitem, y: must have strip and strip must be
>> callable, and y.strip must return something that has replace and
>> replace must be callable)
>>
>> So even this simple example gets nasty in a hurry, let alone the OP's
>> case where he stuffs y into a list in order to access it much later,
>> in a completely different chunk of code, only to find out that y
>> doesn't support the complete string interface as he expected.
>
Steven D'Aprano
> But this class wasn't really that complex; it's just that it assumed all
> the stuff it's being fed were strings (or could be treated as strings),
> and I inadvertently fed it an NLTK.Tree node instead (not realizing that
> a library method I was calling could return such a thing sometimes).
Guido has published a couple of metaclasses to get Eiffel-style pre- and
post-condition tests that may be useful for you:
http://www.python.org/doc/essays/metaclasses/
If you're interested in reading more about metaclasses, this is more
current:
http://www.python.org/download/releases/2.2.3/descrintro/
By the way, even Guido himself isn't immune to the tendency among Python
users to flame at anyone suggesting change to Python's model:
http://www.artima.com/weblogs/viewpost.jsp?thread=87182
(And that's a good thing. It would be really bad if the Python community
were slavishly and mindlessly Guido-fanboys as we're sometimes accused of
being.)
--
Steven
greg
> So, in this case, the simplest solution was to have the method that
> initially accepts and stores the data check to make sure that data
> satisfies the assumptions of the class.
In hindsight, yes, but the trouble is that you can't
tell ahead of time which of the gazillion places in the
code that where you store things away in containers are
likely to cause a problem later.
I can't imagine myself writing code to check every
argument to every method to guard against this sort of
thing. If you're willing to do that, it's up to you,
but it's far from common practice in Python programming.
--
Greg
greg
> So, the level of assertion that I want to make in a method that expects
> a Duck is just that its parameter is either a Duck, or something that
> the caller is claiming is just as good as a Duck.
I'm not sure, but I think the new ABC stuff in Py3 is
going to provide something like this, in that there will
be a way to declare that a class conforms to the Duck
interface even if it doesn't inherit from it. Then
you can just test isinstance(x, Duck).
--
Greg
Arnaud Delobelle
Duck typing...
For a while I thought the word _duck_ was used in the sense of _dodge_.
--
Arnaud
Steven D'Aprano
> Joe Strout wrote:
>
>> So, in this case, the simplest solution was to have the method that
>> initially accepts and stores the data check to make sure that data
>> satisfies the assumptions of the class.
>
> In hindsight, yes, but the trouble is that you can't tell ahead of time
> which of the gazillion places in the code that where you store things
> away in containers are likely to cause a problem later.
>
> I can't imagine myself writing code to check every argument to every
> method to guard against this sort of thing.
Which is, of course, the weakness of dynamic typed languages like Python.
With statically typed languages like Pascal and C, you can get the
compiler to check that for you (often at compile time), but at the cost
of a lot more effort up front. And with languages like Haskell, the type
inference engine can do much of that type checking without you needing to
make explicit type declarations.
> If you're willing to do
> that, it's up to you, but it's far from common practice in Python
> programming.
True. It's generally more efficient for the programmer's time to let the
function or method fail where ever it happens to fail, rather than trying
to get it to fail up front. But the cost of this is that sometimes it's
*less* efficient for the programmer, because he has no idea where the
offending object was injected into the code.
I wonder whether the best solution is to include all your type checks
(isinstance or duck-typing) in the unit tests, so you can avoid paying
the cost of those tests at runtime? If your code passes the unit tests,
but fails with real data, then your unit tests aren't extensive enough.
--
Steven
Paul McGuire
cybersource.com.au> wrote:
> I wonder whether the best solution is to include all your type checks
> (isinstance or duck-typing) in the unit tests, so you can avoid paying
> the cost of those tests at runtime? If your code passes the unit tests,
> but fails with real data, then your unit tests aren't extensive enough.
>
> --
> Steven
The real penalties that you pay for static typing are not at compile
time, but at design time. Because of duck-typing, Python programmers
can skip a lot of the extra hoops/cruft that Java and C++ developers
must jump through, usually to *get around* the restrictions of static
typing. Imagine how much code is needed in those languages to
implement this simple generic Proxy class:
class Proxy(object):
def __init__(self,other):
self.obj = other
def __getattr__(self,attr):
print "Get attribute '%s'" % attr
return getattr(self.obj,attr)
x = "slfj"
xproxy = Proxy(x)
print xproxy.upper()
# prints:
# Get attribute 'upper'
# SLFJ
I used very similar code to write a CORBA interceptor for a client in
about 15 minutes, which would have taken *much* longer in C++ or Java.
-- Paul