Atomic block termination question

[Mike Dewhirst]

When saving a model I'm getting a TransactionManagementError - You can't
execute queries until the end of the 'atomic' block

Ticket #21540 seems fairly explicit at least where Postgres is
concerned. TransactionManagementError prevents what I want to do and I'm
not a nuclear expert.

How do I terminate the save() method code in that atomic block and then
immediately execute my queries?

I need the save() to complete so I can get_or_create some 1:1 records
belonging to the model being saved.

Thanks

Mike

[Mike Dewhirst]

On 24/06/2015 4:43 PM, Mike Dewhirst wrote:
> When saving a model I'm getting a TransactionManagementError - You can't
> execute queries until the end of the 'atomic' block
>
> Ticket #21540 seems fairly explicit at least where Postgres is
> concerned. TransactionManagementError prevents what I want to do and I'm
> not a nuclear expert.
>
> How do I terminate the save() method code in that atomic block and then
> immediately execute my queries?

I have implemented a workaround but not sure if it is the best way. Any
comment appreciated ...

In the model's clean() method I test for self.pk and if true, execute
the queries which previously caused the problem. This seems to work but
hasn't had any testing in production.

Thanks again for any ideas

Mike

[Gergely Polonkai]

As for me I have a function registered for the post_save signal, so after saving the 1:1 object gets created. If you don't mind that the 1:1 object gets created outside of the atomic block, it should work for you, too.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/558A676C.5080202%40dewhirst.com.au.
For more options, visit https://groups.google.com/d/optout.

[Carl Meyer]

Hi Mike,

On 06/24/2015 02:16 AM, Mike Dewhirst wrote:
> On 24/06/2015 4:43 PM, Mike Dewhirst wrote:
>> When saving a model I'm getting a TransactionManagementError - You can't
>> execute queries until the end of the 'atomic' block
>>
>> Ticket #21540 seems fairly explicit at least where Postgres is
>> concerned. TransactionManagementError prevents what I want to do and I'm
>> not a nuclear expert.
>>
>> How do I terminate the save() method code in that atomic block and then
>> immediately execute my queries?

I'm afraid this description of what you're trying to do is too vague to
be useful. Maybe some sample code would help?

TransactionManagementError is a symptom, not a cause. It means that a
database error occurred inside a transaction, which leaves the
transaction in an unpredictable state, so Postgres wants you to roll
back the transaction (or roll back to a savepoint prior to the error)
before issuing any more database queries.

Possible solutions include:

a) Figuring out why there was a database error, and fixing it so it
doesn't occur.

b) Wrapping the code that might cause a database error in its own
`transaction.atomic` block, so on error that bit of code is rolled back
and later queries within the same transaction can go forward.

> I have implemented a workaround but not sure if it is the best way. Any
> comment appreciated ...
>
> In the model's clean() method I test for self.pk and if true, execute
> the queries which previously caused the problem. This seems to work but
> hasn't had any testing in production.

Again it's hard to tell without seeing code or traceback, but it sounds
like probably what you've done here is fix the condition that was
causing the error in the first place (that is, solution (a) above). It
sounds like you were probably trying to do some related-model queries
with an unsaved model, and now you've guarded those queries to only
occur if the model is saved. If so, that's not a workaround, it's the
best solution.

>> I need the save() to complete so I can get_or_create some 1:1 records
>> belonging to the model being saved.

Again, sample code would really illuminate what you're trying to do.

Carl

[Mike Dewhirst]

Carl

On 25/06/2015 2:34 AM, Carl Meyer wrote:
> Hi Mike,
>
> On 06/24/2015 02:16 AM, Mike Dewhirst wrote:
>> On 24/06/2015 4:43 PM, Mike Dewhirst wrote:
>>> When saving a model I'm getting a TransactionManagementError - You can't
>>> execute queries until the end of the 'atomic' block
>>>
>>> Ticket #21540 seems fairly explicit at least where Postgres is
>>> concerned. TransactionManagementError prevents what I want to do and I'm
>>> not a nuclear expert.
>>>
>>> How do I terminate the save() method code in that atomic block and then
>>> immediately execute my queries?
>
> I'm afraid this description of what you're trying to do is too vague to
> be useful. Maybe some sample code would help?
>
> TransactionManagementError is a symptom, not a cause. It means that a
> database error occurred inside a transaction, which leaves the
> transaction in an unpredictable state, so Postgres wants you to roll
> back the transaction (or roll back to a savepoint prior to the error)
> before issuing any more database queries.

Ok. I thought from reading the ticket that I was trying to do something
illegal in Postgres - that is issuing a query within a transaction which
needed to be finalised or rolled back. I took it as a symptom or signal
and think I understand that Postgres is somewhat more rigorous in this
regard than MySQL.

>
> Possible solutions include:
>
> a) Figuring out why there was a database error, and fixing it so it
> doesn't occur.

I separated out all the pre and post-save stuff without the offending
queries and put them into ...

def save(self, *args, **kwargs):
self._pre_save() # nothing tricky here
super(Substance, self).save(*args, **kwargs)
self._post_save() # nothing tricky here

... which stopped the TransactionManagementError and everything worked
on existing substances which already had the necessary 1:1 relations AND
it kinda "worked" when I [saved as new] except obviously the 1:1
relations were not created.

... then did a _post_post_save() with the offending queries ...

def _post_post_save(self):
if self.physical_state == SOLID:
Solid.objects.get_or_create(substance=self)
elif self.physical_state == LIQUID:
Liquid.objects.get_or_create(substance=self)
elif self.physical_state == GAS:
Gas.objects.get_or_create(substance=self)
elif self.physical_state == AEROSOL:
Aerosol.objects.get_or_create(substance=self)
Aquatic.objects.get_or_create(substance=self)
Tox.objects.get_or_create(substance=self)
Terrestrial.objects.get_or_create(substance=self)
if self.terrestrial:
# We can't do this in terrestrial.save() and it needs
# to be recomputed on every save
self.terrestrial.set_soil_m_factor()
self.terrestrial.set_vertebrate_m_factor()
self.terrestrial.set_invertebrate_m_factor()

... which as I said is now called from substance.clean(). I realise
clean() is called before save() but that's all I can think of at the
moment. Those m_factors are unlikely to change once the concentration
values (EC50, LD50 etc) upon which they are based are set.

>
> b) Wrapping the code that might cause a database error in its own
> `transaction.atomic` block, so on error that bit of code is rolled back
> and later queries within the same transaction can go forward.

That sounds like nuclear physics to me. I could probably follow a recipe
but might have trouble figuring out when to use it.

Thanks for listening

Mike

[Carl Meyer]

Hi Mike,

Yes, that's right. (Well, the transaction needs to be rolled back. Once
there's been an error, there is no other "finalizing" possible besides a
rollback.) My point was that the transaction gets into this state
because some other query causes an error, so your efforts should first
focus on figuring out what that error was and making it not happen, if
possible.

I don't understand why you want to call this from clean() instead of
from save(), but it should work OK as long as you wrap it in an `if
self.pk:` so it doesn't try to run on an unsaved object. Of course that
means it will never run at all when saving a new object.

>> b) Wrapping the code that might cause a database error in its own
>> `transaction.atomic` block, so on error that bit of code is rolled back
>> and later queries within the same transaction can go forward.
>
> That sounds like nuclear physics to me. I could probably follow a recipe
> but might have trouble figuring out when to use it.

Here's the general recipe. This code is problematic:

@transaction.atomic
def do_something():
do_the_first_thing()
try:
do_a_thing_that_might_cause_a_database_error()
except DatabaseError:
pass
do_another_thing()


If `do_a_thing_that_might_cause_a_database_error()` does in fact cause a
database error, then `do_another_thing()` will fail with
`TransactionManagementError` (assuming it tries to use the database)
because it's trying to continue with a transaction that's in an error state.

Assuming you can't prevent the possibility of the database error, here
is how you would solve the problem with the above code:

@transaction.atomic
def do_something():
do_the_first_thing()
try:
with transaction.atomic():
do_a_thing_that_might_cause_a_database_error()
except DatabaseError:
pass
do_another_thing()

By wrapping `do_a_thing_that_might_cause_a_database_error()` in an inner
atomic block (that is, a savepoint), and catching the DatabaseError
outside that atomic block, you allow the DatabaseError to first cause a
rollback of that inner atomic block, putting the transaction back into a
usable state.

This is covered in the docs. See the warning in this section:
https://docs.djangoproject.com/en/1.8/topics/db/transactions/#controlling-transactions-explicitly


Carl

[Mike Dewhirst]

Carl

Thank you very much. Great overview. Confidence boosting.

Mike