Creating declared class in __declare_first__ causes RuntimeError: deque mutated during iteration

[Tom Kedem]

I'm trying to create a dynamic many_to_many relationship, inspired by:

http://techspot.zzzeek.org/2011/05/17/magic-a-new-orm/

https://bitbucket.org/zzzeek/pycon2014_atmcraft/src/f50cbe745a19/atmcraft/model/meta/?at=master (orm.py and schema.py)

I've got it working with just creating a secondary table in the __declare_first__ hook.

Now I want it to be a "secondary" class and not just a table - and I'm having trouble.

I've created a declared class in the __declare_first__ hook. Creating that class causes (Full stacktrace at the bottom):

RuntimeError: deque mutated during iteration

I suppose new events are being added when I do so. Searching google I came upon these:
https://bitbucket.org/zzzeek/sqlalchemy/issues/3163/use-deque-for-event-lists-so-that-remove

https://bitbucket.org/zzzeek/sqlalchemy/commits/4a4cccfee5a2#Llib/sqlalchemy/event/api.pyT61

But since I'm not creating those events explicitly (I suspect the declared class creates them), I'm not sure about how to proceed.

Full stacktrace if it's helpful:

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\instrumentation.py", line 347, in _new_state_if_none

    state = self._state_constructor(instance, self)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\util\langhelpers.py", line 754, in __get__

    obj.__dict__[self.__name__] = result = self.fget(obj)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\instrumentation.py", line 177, in _state_constructor

    self.dispatch.first_init(self, self.class_)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\event\attr.py", line 256, in __call__

    fn(*args, **kw)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\mapper.py", line 2943, in _event_on_first_init

    configure_mappers()

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\mapper.py", line 2822, in configure_mappers

    Mapper.dispatch._for_class(Mapper).before_configured()

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\event\attr.py", line 217, in __call__

    for fn in self.parent_listeners:

RuntimeError: deque mutated during iteration

[Mike Bayer]

can you attach a quick single-file .py script that I can run, and will
show how you're getting this error? you might be in an "impossible"
block there, in that you are trying to add an event listener within that
same event, but I need to see if what you're doing is really something
that should be made to work. That is, perhaps declarative should use a
different system internally for the __declare_first__ in order to get
around this.

> --
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy+...@googlegroups.com>.
> To post to this group, send email to sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.

[Tom Kedem]

Thanks the for quick reply.

Trying to mush it to a single file I accidentally found the source (not the solution unfortunately) of the problem.

It only happens when I use FlaskSqlalchemy as the base class... so maybe it's something on their part?

I attached a file (python 3). You must use flask-sqlalchemy master to run it, as it uses a custom base class which I think isn't released yet.

I suppose their base class generates those events. I commented out the declarative_base usage that works (see attached file).

> <mailto:sqlalchemy+unsub...@googlegroups.com>.

[Mike Bayer]

is that stacktrace you gave me the *whole* thing? It only includes
lines within SQLAlchemy itself, which is impossible because SQLAlchemy
isn't a standalone application. Can you include the complete trace
with all code lines in it?

Also yes, if flask is doing something weird, I'm more likely to suggest
they fix it on their end.

On 08/15/2016 10:50 AM, Tom Kedem wrote:
> Thanks the for quick reply.
> Trying to mush it to a single file I accidentally found the source (not
> the solution unfortunately) of the problem.
> It only happens when I use FlaskSqlalchemy as the base class... so maybe
> it's something on their part?
> I attached a file (python 3). You must use flask-sqlalchemy master to
> run it, as it uses a custom base class which I think isn't released yet.
> I suppose their base class generates those events. I commented out the
> declarative_base usage that works (see attached file).
>
> On Monday, August 15, 2016 at 4:57:19 PM UTC+3, Mike Bayer wrote:
>
> can you attach a quick single-file .py script that I can run, and will
> show how you're getting this error? you might be in an "impossible"
> block there, in that you are trying to add an event listener within
> that
> same event, but I need to see if what you're doing is really something
> that should be made to work. That is, perhaps declarative should
> use a
> different system internally for the __declare_first__ in order to get
> around this.
>
>
>
> On 08/15/2016 09:38 AM, Tom Kedem wrote:
> > I'm trying to create a dynamic many_to_many relationship, inspired
> by:
> > http://techspot.zzzeek.org/2011/05/17/magic-a-new-orm/

> <http://techspot.zzzeek.org/2011/05/17/magic-a-new-orm/>
> >
> https://bitbucket.org/zzzeek/pycon2014_atmcraft/src/f50cbe745a19/atmcraft/model/meta/?at=master

> <https://bitbucket.org/zzzeek/pycon2014_atmcraft/src/f50cbe745a19/atmcraft/model/meta/?at=master>
>
> > (orm.py and schema.py)
> >
> > I've got it working with just creating a secondary table in the
> > __declare_first__ hook.
> >
> > Now I want it to be a "secondary" class and not just a table - and
> I'm
> > having trouble.
> > I've created a declared class in the __declare_first__ hook. Creating
> > that class causes (Full stacktrace at the bottom):
> > RuntimeError: deque mutated during iteration
> >
> > I suppose new events are being added when I do so. Searching google I
> > came upon these:
> >
> https://bitbucket.org/zzzeek/sqlalchemy/issues/3163/use-deque-for-event-lists-so-that-remove

> <https://bitbucket.org/zzzeek/sqlalchemy/issues/3163/use-deque-for-event-lists-so-that-remove>
>
> >
> https://bitbucket.org/zzzeek/sqlalchemy/commits/4a4cccfee5a2#Llib/sqlalchemy/event/api.pyT61

> > an email to sqlalchemy+...@googlegroups.com <javascript:>
> > <mailto:sqlalchemy+...@googlegroups.com <javascript:>>.

> > To post to this group, send email to sqlal...@googlegroups.com

> <javascript:>
> > <mailto:sqlal...@googlegroups.com <javascript:>>.

> > Visit this group at https://groups.google.com/group/sqlalchemy

> <https://groups.google.com/group/sqlalchemy>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com

> <mailto:sqlalchemy+...@googlegroups.com>.

[Tom Kedem]

I trimmed the business-logic part. Here's a really full stacktrace for the sandbox.py file I attached:

Traceback (most recent call last):

  File "C:/Users/vToMy/PycharmProjects/sandbox/sandbox.py", line 197, in <module>

    user = User()

  File "<string>", line 2, in __init__

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\instrumentation.py", line 347, in _new_state_if_none

    state = self._state_constructor(instance, self)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\util\langhelpers.py", line 754, in __get__

    obj.__dict__[self.__name__] = result = self.fget(obj)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\instrumentation.py", line 177, in _state_constructor

    self.dispatch.first_init(self, self.class_)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\event\attr.py", line 256, in __call__

    fn(*args, **kw)

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\mapper.py", line 2941, in _event_on_first_init

    configure_mappers()

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\orm\mapper.py", line 2820, in configure_mappers

    Mapper.dispatch._for_class(Mapper).before_configured()

  File "C:\Users\vToMy\PycharmProjects\sqlalchemy\lib\sqlalchemy\event\attr.py", line 217, in __call__

    for fn in self.parent_listeners:

RuntimeError: deque mutated during iteration

I'm not sure it's flask-sqlalchemy's issue, but it's related to it. Looking at their base class creation I don't notice anything "funny", but I'm probably overlooking something.

[Mike Bayer]

the issue would be this:

base_class = get_declarative_base(cls)
association_class = type(association_class_name, (base_class,),
association_class_attributes)


that is, you're making a new declarative class inside the
__declare_first__() hook. That class itself also has __declare_first__
so an event is being added within the handling of the event.


I think the issue would be, does this class really need to have a
__declare_first__(). The Flask base issue is likely because their base
class has a __declare_first__() on it at the abstract level, and they
should not do that. They should not need to propagate a
__declare_first__() to every new subclass.

> <http://techspot.zzzeek.org/2011/05/17/magic-a-new-orm/>
> https://bitbucket.org/zzzeek/pycon2014_atmcraft/src/f50cbe745a19/atmcraft/model/meta/?at=master

> <https://bitbucket.org/zzzeek/pycon2014_atmcraft/src/f50cbe745a19/atmcraft/model/meta/?at=master>
> (orm.py and schema.py)
>
> I've got it working with just creating a secondary table in the
> __declare_first__ hook.
>
> Now I want it to be a "secondary" class and not just a table - and
> I'm having trouble.
> I've created a declared class in the __declare_first__ hook.
> Creating that class causes (Full stacktrace at the bottom):
> RuntimeError: deque mutated during iteration
>
> I suppose new events are being added when I do so. Searching google
> I came upon these:
> https://bitbucket.org/zzzeek/sqlalchemy/issues/3163/use-deque-for-event-lists-so-that-remove

> <https://bitbucket.org/zzzeek/sqlalchemy/issues/3163/use-deque-for-event-lists-so-that-remove>
> https://bitbucket.org/zzzeek/sqlalchemy/commits/4a4cccfee5a2#Llib/sqlalchemy/event/api.pyT61

[Tom Kedem]

Figured it out. It wasn't FlaskSqlalchemy, it was me.

The decalrative base class inherited from References which had the __declarefirst__ in it. Trying to instantiate a declarative class (which inherited from References) inside __declarefirst__ fired that event, which caused the error. I removed the References class from the declarative base up the hierarchy (created a base class inheriting from declarative base, which is marked with __abstract__=true).

Solved the problem.

Thanks for the support :)

I hope to release my results, once I finish, as part of sqlalchemy utils.

On Monday, August 15, 2016 at 4:38:34 PM UTC+3, Tom Kedem wrote: