How to send task-message after request.tm has committed?

[jens.t...@gmail.com]

Hi,

Within a view function it becomes necessary to run an async task. Celery, Dramatiq, etc are frameworks to handle that. However, my async task needs to read db objects which have been created—and must be committed and persisted—by the view function that spawns the task. Thus, I need to delay spawning the task (i.e. sending the task’s message to the workers) until after the request’s db transaction has committed.

The db transaction is a standard SQLAlchemy transaction. Using the commit veto hook is too early. This SO answer suggests using the repoze.tm.after_end() callback, but the answer is about five years old now.

How do I handle this scenario correctly, i.e. how do I schedule an async job after a request was successful? I guess I can pack up the task object, or a signature, and its parameters and attach them to the request from within the view function. But where is the right place to actually send the task message?

Thanks!

Jens

[jens.t...@gmail.com]

Poking around the documentation some more, it seems that using an after-commit-hook may work: i.e. when a task needs to be scheduled by the view function, then find the request’s transaction (from `request.tm`?) and install the hook function and and its arguments (i.e. which task needs to be scheduled and the task’s args).

But still, what is the official and recommended way of solving this? 🤔

Cheers!

[Thierry Florac]

Hi,

Maybe you should have a look at the "DataManager" component interface provided by the "transaction" package.

This kind of component allows you to include your hook in a "two-phase commit" transaction safely.

Several implementations are available, for example in the "repoze.sendmail" package...

Best regards,

Thierry

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/25d30da3-5079-46f7-b15a-2ffe3d44c06e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

http://www.imagesdusport.com -- http://www.ztfy.org

[jens.t...@gmail.com]

Hello Thierry,

You are talking about these data managers: http://zodb.readthedocs.io/en/latest/transactions.html#data-managers Correct? Other than the Mailer it seems a little difficult to find more verbose documentation?

I have played a little more with the after-commit hook and this short piece of code seems to work:

@view_config(

    …

    )

def some_view(request):                                                                                      

    # Do some view stuff, modify db objects, etc.

    # Local helper function that issues the message to the broker. This

    # function has access to the closure but beware that db objects are

    # committed and invalid!

    def _send_message(success, *args, **kwargs):

        if success:
            # Send message to the broker, thus spawning a worker task.

            dramatiq_actors.some_task.send(*args)  # Or Celery tasks, whatever.

    # Get the transaction for this request.

    t = request.tm.get()

    # Arguments for the task.

    args = (

        foo,

        3,

        )
    kwargs = {}

    # Add the after-commit hook, i.e. call _send_message() when the

    # request's transaction has committed.

    t.addAfterCommitHook(_send_message, args=args, kws=kwargs)

    # And return from the view function.
    return {
        "some": "results",
        }

It looks to me that this is an acceptable solution if the called hook function can not fail (i.e. return success/failure) and therewith can not contribute to the transaction to fail/succeed. However, a data manager solution makes sense if data must be managed and processing/persisting that data may fail which impacts the overall transaction.

Any recommendations here?

Cheers,

Jens

[Mikko Ohtamaa]

Hi Jens,

You can also look here for inspiration:

https://websauna.org/docs/narrative/misc/task.html#delayed-tasks

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/ef6bf67d-acec-4dae-b318-7749743add31%40googlegroups.com.