transaction: getting hooks and synchronizers to work

[Siddhartha Kasivajhula]

Hi there,

I'm not able to get before/after commit hooks or synchronizers to work, hope someone here can explain what I'm doing wrong (or please point me at the relevant list). I'm following the tutorial here:

http://www.zodb.org/zodbbook/transactions.html

.. and got the PickleDataManager working. But when I add the before- and after-commit hooks verbatim from the tutorial, I get the following error:

Traceback (most recent call last):

  File "pickledm.py", line 145, in <module>

    transaction.commit()

  File "/Users/siddhartha/.virtualenvs/pyramid/lib/python2.7/site-packages/transaction/_manager.py", line 107, in commit

    return self.get().commit()

  File "/Users/siddhartha/.virtualenvs/pyramid/lib/python2.7/site-packages/transaction/_transaction.py", line 339, in commit

    self._callBeforeCommitHooks()

  File "/Users/siddhartha/.virtualenvs/pyramid/lib/python2.7/site-packages/transaction/_transaction.py", line 413, in _callBeforeCommitHooks

    hook(*args, **kws)

TypeError: before_commit() got an unexpected keyword argument 'a'

When I pass in a blank dict instead of the {'a':1} from the tutorial, I get this error:

  File "pickledm.py", line 17, in before_commit

    for arg in args:

TypeError: 'int' object is not iterable

... and I find that, in fact, printing 'args' inside the hook function shows that it is equal to 1, and not the tuple (1,2) that was passed.

I also tried adding a synchronizer, and initially got this error:

TypeError: unbound method beforeCompletion() must be called with synch instance as first argument (got Transaction instance instead)

It looked like this may have been due to a typo in the tutorial, I changed:

transaction.manager.registerSynch(sync)

to:

transaction.manager.registerSynch(sync())

...to pass an instance instead of a class and the error goes away. Unfortunately the synchronizer methods don't seem to be called when the transaction is executed, and I don't see the output from those functions.

Could someone point me in the right direction here on how to get these to work? Here is my full code if anyone wants to take a look: https://gist.github.com/countvajhula/4966286

Thanks,

-Sid

[Carlos de la Guardia]

Hi,

there were some problems with the code snippets in the book. I updated
them here:

http://zodb.readthedocs.org/en/latest/transactions.html#before-commit-hooks

For some reason I could only get synch to work using classmethod.

Carlos de la Guardia

> --
> You received this message because you are subscribed to the Google Groups
> "pylons-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-devel...@googlegroups.com.
> To post to this group, send email to pylons...@googlegroups.com.
> Visit this group at http://groups.google.com/group/pylons-devel?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Siddhartha Kasivajhula]

Carlos, great, thanks for the corrections. They appear to be working now :)

[Laurence Rowe]

On Sunday, 17 February 2013 22:54:30 UTC-8, cguardia wrote:

Hi,

there were some problems with the code snippets in the book. I updated
them here:

http://zodb.readthedocs.org/en/latest/transactions.html#before-commit-hooks

For some reason I could only get synch to work using classmethod.

Pass an instance of the MySynch class rather than the class itself, e.g:

class MySynch(object):
    def beforeCompletion(self, transaction):
        print "Commit started"

    def afterCompletion(self, transaction):
        print "Commit finished"

import transaction
synch = MySynch()
transaction.manager.registerSynch(synch)

Laurence

[Carlos de la Guardia]

Laurence, did you get it to work like that? I tried before using
classmethod and wasn't able to make it work.

Carlos de la Guardia

> Pass an instance of the MySynch class rather than the class itself, e.g:
>
> class MySynch(object):
> def beforeCompletion(self, transaction):
> print "Commit started"
>
> def afterCompletion(self, transaction):
> print "Commit finished"
>
> import transaction
> synch = MySynch()
> transaction.manager.registerSynch(synch)
>
> Laurence
>

[Laurence Rowe]

On 3 March 2013 08:37, Carlos de la Guardia

<carlos.de...@gmail.com> wrote:
> Laurence, did you get it to work like that? I tried before using
> classmethod and wasn't able to make it work.

Ah, you need to implement all methods on ISynchronizer:

>>> class MySynch(object):
... def newTransaction(self, transaction):
... pass
...
KeyboardInterrupt
>>> class MySynch(object):
... def newTransaction(self, transaction):
... print "New transaction"
... def beforeCompletion(self, transaction):
... print "Commit started"
... def afterCompletion(self, transaction):
... print "Commit finished"
...

>>> import transaction
>>> synch = MySynch()
>>> transaction.manager.registerSynch(synch)

>>> tx = transaction.begin()
New transaction
>>> tx.commit()
Commit started
Commit finished

If you want a singleton synchronizer then you can use classmethods
everywhere, but you'll still need to implement them all.

Laurence

[Siddhartha Kasivajhula]

Ok, it looks like the synchronizer instance needs to remain in the calling scope for the synchronizer to work. This code (Laurence) works:

synch = MySynch()

transaction.manager.registerSynch(synch)

while this code (what we were doing earlier) doesn't:

transaction.manager.registerSynch(MySynch())

Is this a bug or is it supposed to work this way?

Now, regarding the newTransaction() interface method -- this actually looks very handy and may be just what I need for a mongodb data manager that I'm writing. But it looks like it's only called when there is an explicit call to transaction.begin(), which apparently isn't necessary to use the transaction machinery as it happens implicitly by just using transaction.get() and then commit()ing. These implicitly begun transactions don't appear trigger the newTransaction() method in the synchronizer. Is there any way to make it trigger without an explicit begin()? (this should probably be the default behavior?)

btw I'm not able to find any of these interface methods by using help() or dir() on the interfaces in transaction.interfaces. All of the interfaces appear to show exactly the same set of properties and methods... though I do see them if I look directly in the transaction/interfaces.py source. is there another way to introspect these?

Thanks, -Sid

Laurence

[Laurence Rowe]

On Sunday, March 3, 2013 4:59:58 PM UTC-8, Sid K wrote:

Ok, it looks like the synchronizer instance needs to remain in the calling scope for the synchronizer to work. This code (Laurence) works:

synch = MySynch()

transaction.manager.registerSynch(synch)

while this code (what we were doing earlier) doesn't:

transaction.manager.registerSynch(MySynch())

Is this a bug or is it supposed to work this way?

The TransactionManager keeps the reference to the synchronizer in a WeakSet, so you will need to keep a reference to the synchronizer around some other way in order to avoid it being garbage collected.

 

Now, regarding the newTransaction() interface method -- this actually looks very handy and may be just what I need for a mongodb data manager that I'm writing. But it looks like it's only called when there is an explicit call to transaction.begin(), which apparently isn't necessary to use the transaction machinery as it happens implicitly by just using transaction.get() and then commit()ing. These implicitly begun transactions don't appear trigger the newTransaction() method in the synchronizer. Is there any way to make it trigger without an explicit begin()? (this should probably be the default behavior?)

I think this might be worth bringing up on zodb-dev, I'm not sure why the distinction exists. In Pyramid and Zope the transaction will always be begun explicitly at the start of a request. 

btw I'm not able to find any of these interface methods by using help() or dir() on the interfaces in transaction.interfaces. All of the interfaces appear to show exactly the same set of properties and methods... though I do see them if I look directly in the transaction/interfaces.py source. is there another way to introspect these?

An interface object is not a class, to access the methods and attributes defined in an Interface you can use iface.names() and iface[name'].

I've not used MongoDB myself, but what are you hoping to gain from the Synchronizer? A DataManager sounds more appropriate. You'll only be able to handle a single document update atomically with MongoDB, but that case should be able to integrate safely with the two phase commit protocol by committing during tpc_vote while ensuring the data manager sort key sorts last (take a look at one phase variant of zope.sqlalchemy's DataManager.) For multi-document updates you probably want to treat them more like zope.sendmail does and add them to a queue which then processes them asynchronously.

Laurence

[Siddhartha Kasivajhula]

Hi Laurence,

The TransactionManager keeps the reference to the synchronizer in a WeakSet, so you will need to keep a reference to the synchronizer around some other way in order to avoid it being garbage collected.

I see.

Now, regarding the newTransaction() interface method -- this actually looks very handy and may be just what I need for a mongodb data manager that I'm writing. But it looks like it's only called when there is an explicit call to transaction.begin(), which apparently isn't necessary to use the transaction machinery as it happens implicitly by just using transaction.get() and then commit()ing. These implicitly begun transactions don't appear trigger the newTransaction() method in the synchronizer. Is there any way to make it trigger without an explicit begin()? (this should probably be the default behavior?)

I think this might be worth bringing up on zodb-dev, I'm not sure why the distinction exists. In Pyramid and Zope the transaction will always be begun explicitly at the start of a request. 

I've submitted a request to join that list and will bring it up there once that's approved. Though in the meantime, I did try using pyramid_tm with the data manager that I wrote, and it looks like newTransaction() is not being called. My implementation is currently dependent on that being called so it's failing at the moment. But this seems to suggest that pyramid_tm does not call beginTransaction()... Maybe it's doing a "with transaction.manager" or something else?

An interface object is not a class, to access the methods and attributes defined in an Interface you can use iface.names() and iface[name'].

yup, this works.

I've not used MongoDB myself, but what are you hoping to gain from the Synchronizer? A DataManager sounds more appropriate. You'll only be able to handle a single document update atomically with MongoDB, but that case should be able to integrate safely with the two phase commit protocol by committing during tpc_vote while ensuring the data manager sort key sorts last (take a look at one phase variant of zope.sqlalchemy's DataManager.) For multi-document updates you probably want to treat them more like zope.sendmail does and add them to a queue which then processes them asynchronously.

Yes, I was using the synchronizer as part of the data manager, to avoid making some repeated initialization calls at the start of each transaction. I've put up the data manager here:

https://github.com/countvajhula/mongomorphism

loosely based on this: http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/

Currently each mongo document is managed by a separate data manager instance, and I just have each of them join the current transaction.

Laurence

--