SQLAlchemy 0.8 issue: with appending items to relationship

[Marc Van Olmen]

hi,

Just wanted to quickly report this issue, didn't had time to write unit test for corning the case:

Was upgrading an 0.4.8 legacy project to 0.8.2

Got an SAWarning: This collection has been invalidated in the following case:

Original 0.4.8 code:

            self._serials.append(MetaSerial(value=u'))

And failed to add it to the collection.

The fix needed to get rid of the above warning:

            a_serial = MetaSerial(value=u'')

            self._serials.append(a_serial)

Tables:

metaserial_table = Table(

    'metaserial', metadata,

    Column('id', Integer, primary_key=True),

    Column('id_request', Integer, ForeignKey("request.id"), nullable=True, index=True),

    Column('id_employee', Integer, ForeignKey("item.id"),

           nullable=False, default=lambda: Connection().employee().id, index=True),

    Column('value', UnicodeText(), nullable=True),

    Column('date', LocalDateTime(timezone=True), default=func.now(), nullable=False))

request_table = Table(

    'request', metadata,

    Column('id', Integer, primary_key=True),

        ...

)

Mappers:

requestMapper = mapper(

    Request,

    request_table,

    # save_on_init=False,

    polymorphic_on=request_table.c.type,

    properties={

        ...

        '_serials': relationship(

            MetaSerial,

            primaryjoin=metaserial_table.c.id_request == request_table.c.id, lazy=False),

     ....

}

mapper_cached(

    MetaSerial,

    metaserial_table,

    # save_on_init=False,

    properties={

        'employee': relationship(

            Item, primaryjoin=item_table.c.id == metaserial_table.c.id_employee

        ),

        '_value': metaserial_table.c.value

    }

)

sorry in case already a known issue.

best,

marc

[Michael Bayer]

On Oct 12, 2013, at 11:54 AM, Marc Van Olmen <marcva...@gmail.com> wrote:

> hi,
>
> Just wanted to quickly report this issue, didn't had time to write unit test for corning the case:
> Was upgrading an 0.4.8 legacy project to 0.8.2
>
> Got an SAWarning: This collection has been invalidated in the following case:
>
> Original 0.4.8 code:
>
> self._serials.append(MetaSerial(value=u'))
>
> And failed to add it to the collection.
>
> The fix needed to get rid of the above warning:
>
> a_serial = MetaSerial(value=u'')
> self._serials.append(a_serial)

there's nothing in the mapping or sample code illustrated here that by itself would emit this warning. The warning itself is not a bug, but instead refers to a bug in the calling code, where it is erroneously appending to a collection that is no longer valid, due to expiration. Expiration on session commit was introduced as a default behavior in 0.5 and the collection mechanics also changed significantly from 0.5 on forward.

The code example you have suggests a premature garbage collection of some resource due to the way MetaSerial is not assigned to a variable in one case vs. the other, but that's not really possible with a straight append to a mapped collection as above, and that also doesn't have any direct path to causing this very specific collection-oriented warning from being emitted. So either this code isn't the primary cause of the issue, or there's something more complex going on in the actual classes and/or mappings that produces this.

To produce the warning, you have to do something equivalent to the sample code below:

from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class A(Base):
__tablename__ = 'a'

id = Column(Integer, primary_key=True)
bs = relationship("B")

class B(Base):
__tablename__ = 'b'

id = Column(Integer, primary_key=True)
a_id = Column(Integer, ForeignKey('a.id'))

e = create_engine("sqlite://", echo=True)

Base.metadata.create_all(e)

sess = Session(e)
a1 = A()
a1.bs = [B()]

# hold onto the collection
collection = a1.bs

sess.add(a1)

# "collection" is now no longer associated with a1
sess.commit()

# emits the warning
collection.append(B())

[Marc Van Olmen]

Michael,

Thanks always for your time! amazing those details answers.

I looked at the code and see they did something like this:

        beforeSerials = self._serials[:]

Then a few lines later followed by:  

       self._serials.append(MetaSerial(value=u'))

Then followed by

        afterSerials = self._serials

Then followed by:

       Session.flush()

So It is similar to what you described,

Thanks again.

marc

[Marc Van Olmen]

I did some more digging today to see If I can pin down the exact reason:

I tested with a unit test, and reduced the code above to bare minimum and not doing references to collections anymore.

    def serials(self):

        requiredAmountOfSerials = self.quantity

        if requiredAmountOfSerials > len(self._serials):

            a_len = len(self._serials)

            for a_ctr in range(abs(requiredAmountOfSerials) - a_len):

                self._serials.append( MetaSerial(value=u''))

            Session.add(self)  # MVO? is this needed?

            Session.flush()

        return self._serials

This above code gives a warning when you try to add for the first time something to self._serials collection. (so self._serials has no items)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/Users/marcvano/dev/Acclivity/checkout4_env/Checkout/tests/models/serial_test.py", line 32, in testMakeRequest

    self.assertEqual(len(op1.serials()), op1.allocated)

  File "/Users/marcvano/dev/Acclivity/checkout4_env/Checkout/app/db/request.py", line 2161, in serials

    self._serials.append(MetaSerial(value=u''))

  File "build/bdist.macosx-10.6-intel/egg/sqlalchemy/orm/collections.py", line 1057, in append

    item = __set(self, item, _sa_initiator)

  File "build/bdist.macosx-10.6-intel/egg/sqlalchemy/orm/collections.py", line 1029, in __set

    item = getattr(executor, 'fire_append_event')(item, _sa_initiator)

  File "build/bdist.macosx-10.6-intel/egg/sqlalchemy/orm/collections.py", line 729, in fire_append_event

    self._warn_invalidated()

  File "build/bdist.macosx-10.6-intel/egg/sqlalchemy/orm/collections.py", line 600, in _warn_invalidated

    util.warn("This collection has been invalidated.")

  File "build/bdist.macosx-10.6-intel/egg/sqlalchemy/util/langhelpers.py", line 1036, in warn

    warnings.warn(msg, exc.SAWarning, stacklevel=stacklevel)

SAWarning: This collection has been invalidated.

Changing above code to:

            for a_ctr in range(abs(requiredAmountOfSerials) - a_len):

                a_serial = MetaSerial(value=u'')

                self._serials.append(a_serial)

produces no warnings

[Michael Bayer]

I'm glad you're still poking because I'm still not seeing anything that would trigger this warning.  Here is that exact code in a test case, I can't get it to produce the warning:

from sqlalchemy import *

from sqlalchemy.orm import *

from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class A(Base):

    __tablename__ = 'a'

    id = Column(Integer, primary_key=True)

    _serials = relationship("MetaSerial")

    def serials(self):

        requiredAmountOfSerials = 10

        if requiredAmountOfSerials > len(self._serials):

            a_len = len(self._serials)

            for a_ctr in range(abs(requiredAmountOfSerials) - a_len):

                self._serials.append(MetaSerial())

            session.add(self)  # MVO? is this needed?

            session.flush()

        return self._serials

class MetaSerial(Base):

    __tablename__ = 'b'

    id = Column(Integer, primary_key=True)

    a_id = Column(Integer, ForeignKey('a.id'))

e = create_engine("sqlite://", echo=True)

Base.metadata.create_all(e)

session = Session(e)

a1 = A()

# comment out or not, no warning

session.add(a1)

# same

session.flush()

a1.serials()

--
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.