what's unsafe to do in a __getattr__?

[Mats Wichmann]

Have run into a problem on a "mature" project I work on (there are many
years of history before I joined), there are a combination of factors
that combine to trigger a KeyError when using copy.copy().

I don't want to write a massive essay here but hoping to give enough to
set the context.

There's a class that's a kind of proxy, so there's some "magic" that
could be present. The magic is detected by looking for a kind of memo
annotation, so the __getattr__ starts with this:

# Methods that make this class act like a proxy.
def __getattr__(self, name):
attr = getattr(self.__dict__['__subject'], name)

and that's what blows up. It happens for a user doing something we...
ahem... don't expect. They just picked up the Py3-only version of the
project and now they're getting the issue.

Nothing in the project defined a __reduce__ex__ function, but one is
picked up from the base "object" type, so copy.copy generates some
pickle information and passes it to copy._reconstruct as the state
parameter. This stanza:

if state is not None:
...
if hasattr(y, '__setstate__'):
y.__setstate__(state)

so our class's __getattr__ is called to look for __setstate__. But at
this stage, the copy's instance has only been created, the operations
that will fill in the details haven't happened yet, so we take a KeyError.

So apparently the attempt in the __getattr__ to go fishing in our own
dict for something we set ourselves is unsafe. Is there a guideline for
what you can / cannot expect to be safe to do? My naiive expectations
would be that when __getattr__ is called, you can expect an instance to
have been already initialized, but if I'm not reading the copy module
wrong, that's not always true.

Is a better answer for this class to provide a __copy__ method to more
precisely control how copying happens?

[Mats Wichmann]

On 10/25/21 10:48, Dieter Maurer wrote:

> Mats Wichmann wrote at 2021-10-24 19:10 -0600:
>> Have run into a problem on a "mature" project I work on (there are many
>> years of history before I joined), there are a combination of factors
>> that combine to trigger a KeyError when using copy.copy().

>> ...
>> There's a class that's a kind of proxy ...
>
> "Proxying" has become very difficult since the introduction
> of Python's "new style classes" and looking up "special methods"
> on the type rather then the type instance.
>
> This essentially means that the proxy must implement all
> "special methods" that may be relevant to the application.
>
> All special methods start and end with `"__"`.
> Your `__getattr__` could raise an `AttributeError` as soon as
> it is called with such a name.
>

Thanks. Raising an AttributeError here was what I was going to propose
so maybe I'll take your reply as validation I was on the right track.

This stuff was definitely defined in an era before the new-style
classes, might have to examine some other proxying classes to make sure
there aren't other holes...

thanks again,

-- mats

[Dieter Maurer]