Best practise implementation for equal by value objects
Slaunger
I am new here and relatively new to Python, so be gentle:
Is there a recommended generic implementation of __repr__ for objects
equal by value to assure that eval(repr(x)) == x independet of which
module the call is made from?
Example:
class Age:
def __init__(self, an_age):
self.age = an_age
def __eq__(self, obj):
self.age == obj.age
def __repr__(self):
return self.__class__.__name__ + \
"(%r)" % self.age
age_ten = Age(10)
print repr(age_ten)
print eval(repr(age_ten))
print eval(repr(age_ten)).age
Running this gives
Age(10)
Age(10)
10
Exactly as I want to.
The problem arises when the Age class is iomported into another module
in another package as then there is a package prefix and the above
implementation of __repr__ does not work.
I have then experimented with doing somthing like
def __repr__(self):
return self.__module__ + '.' + self.__class__.__name__ +
"(%r)" % self.age
This seems to work when called from the outside, but not from the
inside of the module. That is, if I rerun the script above the the
module name prefixed to the representation I get the following error
Traceback (most recent call last):
File "valuetest.py", line 15, in <module>
print eval(repr(age_ten))
__main__.Age(10)
File "<string>", line 1, in <module>
NameError: name '__main__' is not defined
This is pretty annoying.
My question is: Is there a robust generic type of implementation of
__repr__ which I can use instead?
This is something I plan to reuse for many different Value classes, so
I would like to get it robust.
Thanks,
Slaunger
Terry Reedy
Slaunger wrote:
> Hi,
>
> I am new here and relatively new to Python, so be gentle:
>
> Is there a recommended generic implementation of __repr__ for objects
> equal by value to assure that eval(repr(x)) == x independet of which
> module the call is made from?
The CPython implementation gives up on that goal and simply prints
<modname.classname object at address> for at least two reasons ;-).
1. In general, it require fairly sophisticated analysis of __init__ to
decide what representation of what attributes to include and decide if
the goal is even possible. If an attribute is an instance of a user
class, then *its* __init__ needs to be analyzed. If an attribute is a
module, class, or function, there is no generic evaluable representation.
2. Whether eval(repr(x)) even works (returns an answer) depends on
whether the name bindings in the globals and locals passed to eval
(which by default are the globals and locals of the context of the eval
call) match the names used in the repr. You discovered that to a first
approximation, this depends on whether the call to repr comes from
within or without the module containing the class definition. But the
situation is far worse. Consider 'import somemod as m'. Even if you
were able to introspect the call and determine that it did not come from
somemod**, prepending 'somemod.' to the repr *still* would not work.
Or, the call to repr could come from one context, the result saved and
passed to another context with different name bindings, and the eval
call made there. So an repr that can be eval'ed in any context is hopeless.
If this is a practical rather than theoretical question, then use your
first repr version that uses the classes definition name and only eval
the result in a context that has that name bound to the class object.
from mymod import Age
#or
import mymod
Age = mymod.Age
#in either case
eval(repr(Age(10))) == Age(10)
> class Age:
>
> def __init__(self, an_age):
> self.age = an_age
>
> def __eq__(self, obj):
> self.age == obj.age
>
> def __repr__(self):
> return self.__class__.__name__ + \
> "(%r)" % self.age
**
While such introspection is not part of the language, I believe one
could do it in CPython, but I forgot the details. There have been
threads like 'How do I determine the caller function' with answers to
that question, and I presume the module of the caller is available also.
Terry Jan Reedy
John Krukoff
> http://mail.python.org/mailman/listinfo/python-list
Are you really sure this is what you want to do, and that a less tricky
serialization format such as that provided by the pickle module wouldn't
work for you?
--
John Krukoff <jkru...@ltgc.com>
Land Title Guarantee Company
Steven D'Aprano
> Hi,
>
> I am new here and relatively new to Python, so be gentle:
>
> Is there a recommended generic implementation of __repr__ for objects
> equal by value to assure that eval(repr(x)) == x independet of which
> module the call is made from?
In general, no.
...
> My question is: Is there a robust generic type of implementation of
> __repr__ which I can use instead?
>
> This is something I plan to reuse for many different Value classes, so I
> would like to get it robust.
I doubt you could get it that robust, nor is it intended to be.
eval(repr(obj)) giving obj is meant as a guideline, not an invariant --
there are many things that can break it. For example, here's a module
with a simple class:
# Parrot module
class Parrot(object):
def __repr__(self):
return "parrot.Parrot()"
def __eq__(self, other):
# all parrots are equal
return isinstance(other, Parrot)
Now let's use it:
>>> import parrot
>>> p = parrot.Parrot()
>>> s = repr(p)
>>> assert eval(s) == p
>>> del parrot
>>> assert eval(s) == p
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in <module>
NameError: name 'parrot' is not defined
If you look at classes in the standard library, they often have reprs
like this:
>>> repr(timeit.Timer())
'<timeit.Timer instance at 0xb7f14bcc>'
Certainly you can't expect to successfully eval that!
I believe the recommendation for eval(repr(obj)) to give obj again is
meant as a convenience for the interactive interpreter, and even there
only for simple types like int or list. If you can do it, great, but if
it doesn't work, so be it. You're not supposed to rely on it, and it's
not meant as a general way to serialize classes.
--
Steven
Slaunger
> Slaunger wrote:
> > Hi,
>
> > I am new here and relatively new to Python, so be gentle:
>
> > Is there a recommended generic implementation of __repr__ for objects
> > equal by value to assure that eval(repr(x)) == x independet of which
> > module the call is made from?
>
> The CPython implementation gives up on that goal and simply prints
> <modname.classname object at address> for at least two reasons ;-).
>
> 1. In general, it require fairly sophisticated analysis of __init__ to
> decide what representation of what attributes to include and decide if
> the goal is even possible. If an attribute is an instance of a user
> class, then *its* __init__ needs to be analyzed. If an attribute is a
> module, class, or function, there is no generic evaluable representation.
OK, the situation is more complicated than that then. In the case here
though,
the attributes would always be sinmple bulit-in types, where
eval(repr(x))==x
or, where the attribute is a user-defined equal-by-value class, that I
have
control over.
The classes I am making as struct type classes with some added
functionlity for
human readable string representation, packing into a stream or
unpacking from a stream
using a "private" class Struct.
I come from a Java and JUnit world, where, if I am used to always
overriding the default reference based implementations of the
equals(), toString(),
and hashCode() methods for "equals-by-value" objects such that they
work well
and efficient in, e.g., hash maps.
With my swich-over to Python, I looked for equivalent features and
stumbled over the
eval(repr(x))==x recommendation. It is not that I actually (yet) need
the repr implementations,
but mostly because I find the condition very useful in PyUnit to check
in a test that I have remembered
to initialize all instance fields in __init__ and that I have
remembered to include all relevant
attributes in the __eq__ implementation.
Whereas this worked fine in a unit test module dedicated to only test
the specific module, the test failed
when called from other test package modules, wrapping the unit tests
from several unit test modules.
>
> 2. Whether eval(repr(x)) even works (returns an answer) depends on
> whether the name bindings in the globals and locals passed to eval
> (which by default are the globals and locals of the context of the eval
> call) match the names used in the repr. You discovered that to a first
> approximation, this depends on whether the call to repr comes from
> within or without the module containing the class definition. But the
> situation is far worse. Consider 'import somemod as m'. Even if you
> were able to introspect the call and determine that it did not come from
> somemod**, prepending 'somemod.' to the repr *still* would not work.
> Or, the call to repr could come from one context, the result saved and
> passed to another context with different name bindings, and the eval
> call made there. So an repr that can be eval'ed in any context is hopeless.
>
Ok, nasty stuff
> If this is a practical rather than theoretical question, then use your
> first repr version that uses the classes definition name and only eval
> the result in a context that has that name bound to the class object.
>
> from mymod import Age
> #or
> import mymod
> Age = mymod.Age
>
> #in either case
> eval(repr(Age(10))) == Age(10)
>
> > class Age:
>
> > def __init__(self, an_age):
> > self.age = an_age
>
> > def __eq__(self, obj):
> > self.age == obj.age
>
> > def __repr__(self):
> > return self.__class__.__name__ + \
> > "(%r)" % self.age
>
Yes, it is most from a practicl point of view, altough I was surprised
that I could not find more material on it in the Python documentation
or mailing groups, and I moight just do what you suggest in the unit
test modules to at least make it robust in that context.
Hmm... a bit of a dissapointment for me that this cannot be done
cleaner
> **
> While such introspection is not part of the language, I believe one
> could do it in CPython, but I forgot the details. There have been
> threads like 'How do I determine the caller function' with answers to
> that question, and I presume the module of the caller is available also.
OK, I think CPython, for the moment, is too much new stuff to dig into
right now.
Just grasping some of all the possibilities in the API, and how to do
things the right way
is giving me enough challenges for now...
>
> Terry Jan Reedy
Again, thank you for your thorough answer,
Slaunger
Slaunger
Well, it is not so much yet for serialization (although i have not yet
fully understood the implications), it is more because
I think the eval(repr(x))==x is a nice unit test to make sure my
constructor and equals method is implemented correctly (that I have
rememebered all attributes in their implementations).
As mentioned above, I may go for a more pragmatic approach, where i
only use repr if it "standard" imported
Cheers,
Slaunger
>
> --
> John Krukoff <jkruk...@ltgc.com>
> Land Title Guarantee Company- Skjul tekst i anførselstegn -
>
> - Vis tekst i anførselstegn -
Slaunger
cybersource.com.au> wrote:
> On Wed, 06 Aug 2008 05:50:35 -0700, Slaunger wrote:
> > Hi,
>
> > I am new here and relatively new to Python, so be gentle:
>
> > Is there a recommended generic implementation of __repr__ for objects
> > equal by value to assure that eval(repr(x)) == x independet of which
> > module the call is made from?
>
> In general, no.
>
> ...
>
> > My question is: Is there a robust generic type of implementation of
> > __repr__ which I can use instead?
>
> > This is something I plan to reuse for many different Value classes, so I
> > would like to get it robust.
>
> I doubt you could get it that robust, nor is it intended to be.
>
> eval(repr(obj)) giving obj is meant as a guideline, not an invariant --
> there are many things that can break it. For example, here's a module
> with a simple class:
OK, I had not fully understood the implications of 'not' implementing
__repr__
such that eval(repr(x)) == x, so I just tried to make it work to make
sure
life would be easy for me and my object as I went further into the
Python jungle
As mentioned above, i also find the eval(repr(x))==x condition
convenient from
a unit test point of view.
>
> # Parrot module
> class Parrot(object):
> def __repr__(self):
> return "parrot.Parrot()"
> def __eq__(self, other):
> # all parrots are equal
> return isinstance(other, Parrot)
>
> Now let's use it:
>
> >>> import parrot
> >>> p = parrot.Parrot()
> >>> s = repr(p)
> >>> assert eval(s) == p
> >>> del parrot
> >>> assert eval(s) == p
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "<string>", line 1, in <module>
> NameError: name 'parrot' is not defined
>
OK, I see, but this isn't exactly eval(repr(x))==x but
s = repr(x)
eval(s) == x
so, of course, is s is deleted in between it won't work.
In my implementation I only expect this should work as a one-liner.
> If you look at classes in the standard library, they often have reprs
> like this:
>
> >>> repr(timeit.Timer())
>
> '<timeit.Timer instance at 0xb7f14bcc>'
>
Yes, I noticed that. But the example here is also an object, which is
equal by reference, not value. And for these
it does not make so much sense to evaluate the representation.
> Certainly you can't expect to successfully eval that!
>
> I believe the recommendation for eval(repr(obj)) to give obj again is
> meant as a convenience for the interactive interpreter, and even there
> only for simple types like int or list. If you can do it, great, but if
> it doesn't work, so be it. You're not supposed to rely on it, and it's
> not meant as a general way to serialize classes.
>
> --
> Steven
OK, I will put less emphasis on it in the future.
Thank you for taking your time to answer.
Slaunger
Duncan Booth
> I believe the recommendation for eval(repr(obj)) to give obj again is
> meant as a convenience for the interactive interpreter, and even there
> only for simple types like int or list.
and even with types as simple as list and int the round tripping can easily
fail:
>>> lst = [1, 2, 3]
>>> lst[1] = lst
>>> eval(repr(lst))
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
eval(repr(lst))
File "<string>", line 1
[1, [...], 3]
^
SyntaxError: invalid syntax
--
Duncan Booth http://kupuguy.blogspot.com
Terry Reedy
Slaunger wrote:
> On 6 Aug., 21:36, Terry Reedy <tjre...@udel.edu> wrote:
> OK, the situation is more complicated than that then. In the case here
> though,
> the attributes would always be sinmple bulit-in types, where
> eval(repr(x))==x
> or, where the attribute is a user-defined equal-by-value class, that I
> have
> control over.
I think most would agree that a more accurate and informative
representation is better than a general representation like Pythons
default. For instance,
>>> a=range(2,10,2) # 3.0
>>> a
range(2, 10, 2)
is nicer than <class 'range' object at ######>.
So when the initializers for instances are all 'nice' (as for range), go
for it (as in 'Age(10)'). And test it as you are by eval'ing the rep.
Just accept that the eval will only work in contexts with the class name
bound to the class. For built-in like range, it always is, by default
-- unless masked by another assignment!
Terry Jan Reedy
Paul Rubin
> So when the initializers for instances are all 'nice' (as for range),
> go for it (as in 'Age(10)'). And test it as you are by eval'ing the
> rep. Just accept that the eval will only work in contexts with the
> class name bound to the class. For built-in like range, it always is,
> by default -- unless masked by another assignment!
Eval is extremely dangerous. Think of data from untrusted sources,
then ask yourself how well you really know where ALL your data came
from. It's preferable to avoid using it that way. There have been a
few "safe eval" recipes posted here and at ASPN. It would be good if
one of them made it into the standard library. Note that pickle
(which would otherwise be an obious choice for this) has the same
problems, though not as severely as flat-out evalling something.
Slaunger
Thank you for pointing out the dangers of eval. I think you are right
to
caution about it. In my particular case it is a closed-loop system, so
no
danger there, but that certainly could have been an issue.
That caution should perhaps be mentioned in
http://docs.python.org/lib/built-in-funcs.html
Slaunger
'nice' classes.
I just revisited the documentation for eval and noticed that there are
optional globals
and locals name space variables, that one could specify:
http://docs.python.org/lib/built-in-funcs.html
Quite frankly I do not understand how to make use of these parameters,
but it is my feeling
that if I enforce a convention of always specifying the globals/locals
parameter in a specific
manner:
assert eval(repr(x), globals, locals) == x
would work independent of how I have imported the module under test.
Now, I just need to figure out if this is right and how to specify the
globals and locals if that is not too cumbersome...
or maybe I am just over-engineering...
Steven D'Aprano
> OK, i am encouraged to carry on my quest with the eval(repr)) for my
> 'nice' classes.
> I just revisited the documentation for eval and noticed that there are
> optional globals
> and locals name space variables, that one could specify:
>
> http://docs.python.org/lib/built-in-funcs.html
>
> Quite frankly I do not understand how to make use of these parameters,
> but it is my feeling
> that if I enforce a convention of always specifying the globals/locals
> parameter in a specific
> manner:
> assert eval(repr(x), globals, locals) == x would work independent of how
> I have imported the module under test.
>
> Now, I just need to figure out if this is right and how to specify the
> globals and locals if that is not too cumbersome... or maybe I am just
> over-engineering...
I think it is infeasible for the repr() of an object to know where it was
imported from. Including the globals and locals in the call to eval()
won't help you, because they can have changed between the time you
created the instance and the time you call repr(). Consider:
>>> import datetime
>>> x = datetime.time(20, 21, 22)
>>> x
datetime.time(20, 21, 22)
>>> eval(repr(x)) == x
True
So far so good! But now watch this, starting in a fresh session:
>>> import datetime as timedate
>>> t = timedate.time
>>> timedate.tttime = timedate.time
>>> del timedate.time
>>> assert t is timedate.tttime
>>>
>>> x1 = t(20, 21, 22)
>>> x2 = timedate.tttime(20, 21, 22)
>>> assert x1 == x2
What should repr(x1) and repr(x2) be, for your invariant eval(repr(x))==x
to hold?
It gets better (or worse):
>>> alist = [None, t, None]
>>> del t, timedate
>>> x3 = alist[1](20, 21, 22)
>>> assert x1 == x2 == x3
What should the repr() of x1, x2, x3 be now?
Bringing this back to the unittests... as I see it, there's an easy way
to solve your problem. In the unittest, just do something like the
following:
# repr(x) looks like "module.class(arg)",
# but we actually import it as package.module.submodule.class
module = package.module.submodule
assert eval(repr(x)) == x
I think that is all you need to do.
--
Steven