Make __reduce__ to correspond to __getnewargs_ex__

[Neil Girdhar]

Currently __reduce__ returns up to five things:

(1) self.__new__ (or a substitute)

(2) the result of __getnewargs__, which returns a tuple of positional arguments for __new__,

(3) the result of __getstate__, which returns an object to be passed to __setstate__

(4) an iterator of values for appending to a sequence

(5) an iterator of key-value pairs for setting on a string.

Python 3.4 added the very useful (for me) __getnewargs_ex__, which returns a pair:

(1) a tuple of positional arguments for __new__

(2) a dict of keyword arguments for __new__

Therefore, I am proposing that __reduce__ return somehow these keyword arguments for __new__.

Best,

Neil

[Neil Girdhar]

Any comments on this?  I ended up making reduce work using a metaclass:

class KwargsNewMetaclass(type):

    """

    This metaclass reimplements __reduce__ so that it tries to call

    __getnewargs_ex__.  If that doesn't work, it falls back to __getnewargs__.

    In the first case, it will pass the keyword arguments to object.__new__.

    It also exposes a kwargs_new static method that can be overridden for

    use by __reduce__.

    """

    @staticmethod

    def kwargs_new(cls, new_kwargs, *new_args):

        retval = cls.__new__(cls, *new_args, **new_kwargs)

        retval.__init__(*new_args, **new_kwargs)

        return retval

    def __new__(cls, name, bases, classdict):

        result = super().__new__(cls, name, bases, classdict)

        def __reduce__(self):

            try:

                getnewargs_ex = self.__getnewargs_ex__

            except AttributeError:

                new_args, new_kwargs = (self.__getnewargs__(), {})

            else:

                new_args, new_kwargs = getnewargs_ex()

            return (self.kwargs_new(cls),

                    (type(self), new_kwargs,) + tuple(new_args),

                    self.__getstate__())

        result.__reduce__ = __reduce__

        return result

[Ethan Furman]

On 06/07/2014 12:39 PM, Neil Girdhar wrote:
>
> Any comments on this?

__reduce__ is already a well-defined part of pickle.

We also have __reduce_ex__, __getnewargs__, and now __getnewargs_ex__ -- why do we need __reduce__ to do the same thing
as __getnewargs_ex__?

--
~Ethan~
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Neil Girdhar]

Was my proposal clear?  If reduce isn't updated to return the keyword arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell.

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/zohH2BCtYzY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Coghlan]

On 8 Jun 2014 16:53, "Neil Girdhar" <miste...@gmail.com> wrote:
>
> Was my proposal clear?  If reduce isn't updated to return the keyword arguments then it is not compatible with classes that require keyword arguments without some serious metaclass fiddling as far as I can tell.

Those classes shouldn't use reduce for their pickling support.

Cheers,
Nick.

[Neil Girdhar]

Of course they should?  What should they use?

[Ethan Furman]

On 06/08/2014 10:59 AM, Neil Girdhar wrote:

> On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:
>>
>> Those classes shouldn't use reduce for their pickling support.
>

> Of course they should? What should they use?

They should use the methods that make sense. Pickling is a protocol. It will use (in, I believe, this order):

__getnewargs_ex__

__getnewargs__

__reduce_ex__

__reduce__

I'm not sure where __getstate__ and __setstate__ fit in, and happily I don't need to unless I'm subclassing something
that makes use of them.

Anyway, back to the story.

When you call pickle.dump, that code will look for the most advanced method available on the object you are trying to
pickle, and use it. (Well, the most advanced method for the protocol version you have selected, that's available on the
object.) So, if you selected protocol 2, then the pickle code will look for __getnewargs__, but not __getnewargs_ex__,
as __getnewargs_ex__ isn't available until protocol 4.

This is similar to iterating: first choice for iterating is to call an object's __iter__ method, but if there isn't one
Python will fall back to using __getitem__ using integers from 0 until IndexError is raised.

[Neil Girdhar]

On Sun, Jun 8, 2014 at 3:13 PM, Ethan Furman <et...@stoneleaf.us> wrote:

On 06/08/2014 10:59 AM, Neil Girdhar wrote:

On Sun, Jun 8, 2014 at 6:03 AM, Nick Coghlan wrote:

Those classes shouldn't use reduce for their pickling support.

Of course they should?  What should they use?

They should use the methods that make sense.  Pickling is a protocol.  It will use (in, I believe, this order):

  __getnewargs_ex__

  __getnewargs__

  __reduce_ex__

  __reduce__

In fact, the reduce functions come first.  The order is:

reduce_ex

reduce

getnewargs_ex and getstate (3.4+)

getnewargs_ex (3.4+)

getnewargs and getstate

getnewargs

getstate

default code that copies/pickles the dict

Up until 3.3, a "default" reduce function might be:

def __reduce__(self):

     return type(self), self.__getnewargs__(), self.__getstate__()

It' s no longer possible to write a default reduce function in 3.4 given the addtion __getnewargs_ex__.  What I am suggesting is to add a new protocol so that reduce returns the keyword arguments.

Best,

Neil

 

I'm not sure where __getstate__ and __setstate__ fit in, and happily I don't need to unless I'm subclassing something that makes use of them.

You really should find out where they fit in before you reply to a message about them :)  

Anyway, back to the story.

When you call pickle.dump, that code will look for the most advanced method available on the object you are trying to pickle, and use it.  (Well, the most advanced method for the protocol version you have selected, that's available on the object.)  So, if you selected protocol 2, then the pickle code will look for __getnewargs__, but not __getnewargs_ex__, as __getnewargs_ex__ isn't available until protocol 4.

This is similar to iterating:  first choice for iterating is to call an object's __iter__ method, but if there isn't one Python will fall back to using __getitem__ using integers from 0 until IndexError is raised.

--
~Ethan~
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/