processing after responding to client

[Zsolt Ero]

On a fairly classic Pyramid app (sync, gunicorn, SQLAlchemy / Postgresql, like the starter template), I am storing analytics-like events in a table for some of my views.

Like this skeleton:

@view_config(route_name='test', request_method='POST', renderer='json')
def test(request):
    # body of the view
    map_id = request.json_body.get('map_id')
    data = {...}
   
    event_data = {'action': 'viewed'}
    event = MapEvent(user=request.user, action=event_data, map_id=map_id)
    request.dbsession.add(event)

    return data

My problem is that while 99% of the views make read-only requests to the database, and thus are very fast, the analytics event is a writing and can be slow occasionally. 

Would it be possible to somehow send the request to the client and still keep processing the view? Like a send() + end() method or something similar, without returning?

// Adding to a tasks queue is not an option as it'd be an overkill and an overcomplicated solution, which wouldn't work for hundreds of thousands of events per day. 

[Tim Hoffman]

Why not publish to a redis qiueue, and then use a separate workers to do the writes.

redis publish is light weight and fast

T

[Ian Wilson]

Are these calls from your own template/client?  If the records are not critical, ie. for billing, throttling, have you considered just firing a second js request to the background in the client to record the event after the first ajax request is processed.  Or if there are a lot of requests per session you could batch them together occasionally like 10 or 20 events to 1 analytic request.  Without batching that would double the amount of requests you would have to process though, but maybe would be simpler than other solutions if the event accuracy is not critical.  Just throwing that out there.

[andi]

I don’t know if there is such a solution inside of the Pyramid scope. Either way, I wouldn’t do it, because it would

 

• Mix business relevant logic with non- business relevant logic in one responsibility level. The latter must not have the ability to influence the first, which is not possible to establish if they are operating on the same level.
• It would be difficult to manage DB transactions independently if you are operating in one request scope.

 

As you say, you’re not willing to put additional infrastructure complexity on that (which sounds fair to me for the given problem), I would look for a solution outside of your service.

 

Personally I would emit HTTP response headers, which contain your events information. Those can be processed in Nginx, e.g. simply logged in json format, to a separate logfile (which is your events log then).  Once you are there, you are free to decide in which cycle, with which tools to continue.

 

Andi

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thierry Florac]

I have several use cases where a request generates long operations (for example, for doing conversion of an uploaded video file) but I want that the end user receives the response as soon as possible (here, before conversion ends).

In these cases, if the long operation is not impacted by the status of my current transaction, and as I don't want to be dependent of a client-side call, my choice was to:

- start a 0MQ listening process on application startup

- when a given request is received, just start a new thread to send the required arguments to the listening 0MQ process, and send response immediately! If this step is just dependent of current transaction status, just be use to use an after-commit hook or a transaction data-manager...

Thierry

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

http://www.imagesdusport.com -- http://pyams.readthedocs.io

[Mike Orr]

I use Redis to queue mail for Sendmail and to record usage stats. I
made a 'request.mailer' object with a 'send' method that takes an
email message (bytes with headers) and uses 'pyramid.tm''s two-phase
transaction to append it to a local list, and then in commit to append
it to a Redis list. I had to do that because you can't roll back in
Redis. Then a half-hourly cron job sends the messages.

It was all straightforward except for setting up the two-phased
commit. That required reading the source of 'pyramid_transaction' and
its dependencies to figure out how to attach the mailer to the
transaction. That required adding several methods, several of which
are empty because they're irrelevant.

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
> To post to this group, send email to pylons-...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/bd6f6169-92dc-47c4-933f-60eb801a1c3a%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--

Mike Orr <slugg...@gmail.com>

[Mikko Ohtamaa]

My problem is that while 99% of the views make read-only requests to the database, and thus are very fast, the analytics event is a writing and can be slow occasionally. 

You can do post commit actions with SQLAlchemy and zope.sqlaclhemy.

1. Response gets served

2. Database transaction get committed for this request

3. Post transaction actions

Example:

https://stackoverflow.com/questions/15593516/how-to-run-a-script-after-pyramids-transaction-manager-has-returned

Please not that thread local `transaction.manager` might be discouraged nowadays, but not sure what is the best practice to get a transaction manager for your request.

Br,

Mikko

Would it be possible to somehow send the request to the client and still keep processing the view? Like a send() + end() method or something similar, without returning?

// Adding to a tasks queue is not an option as it'd be an overkill and an overcomplicated solution, which wouldn't work for hundreds of thousands of events per day. 

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/c44e1af6-8c30-478e-9baf-d7fd8c93e0b5%40googlegroups.com.

[Michael Merickel]

On Mon, Nov 12, 2018 at 4:29 AM Mikko Ohtamaa <mi...@redinnovation.com> wrote:

https://stackoverflow.com/questions/15593516/how-to-run-a-script-after-pyramids-transaction-manager-has-returned

Please not that thread local `transaction.manager` might be discouraged nowadays, but not sure what is the best practice to get a transaction manager for your request.

request.tm and request.tm.get().addAfterCommitHook(...) 

[Zsolt Ero]

I'm just now implementing this. Are you saying that basically, by using addAfterCommitHook, I can replace put relatively fast (say, <10 seconds), functions into async-like behaviour, without requiring any external queue service?

old:

# send email was an async function, via a decorator (using Huey)
send_email(
    to_email=user.email,
    subject='account activation',
    text=email_text,
)

new: 

request.tm.get().addAfterCommitHook(send_email_hook, kws=dict(
    to_email=user.email,
    subject='account activation',
    text=email_text,
))


+ a helper function:


def send_email_hook(success, **kwargs):
    if success:
        send_email(**kwargs)

I have two questions regarding this:

1. When does success == False?

2. What does the warning on transaction's doc page mean?

Hook functions which do raise or propagate exceptions will leave the application in an undefined state. 

Practically speaking, should I put the send_email function into a try-except block?

[Zsolt Ero]

Actually, it doesn't result in async-like behaviour, the client is
still waiting for the function inside the commit hook. I guess I don't
understand the concept then.

> --
> You received this message because you are subscribed to a topic in the Google Groups "pylons-discuss" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylons-discuss/tf5kKsnZRTc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to pylons-discus...@googlegroups.com.

> To post to this group, send email to pylons-...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/6831e434-6e85-4c41-9797-82c71ad7f2b3%40googlegroups.com.

[Thierry Florac]

Hi,

After-commit hooks don't work in async mode!

They allow to declare callbacks which will be called after transaction commit, but server is waiting for these callbacks to be finished (in synchronous mode) before sending response to client.

If you need async mode, you have (at least) to start another thread which will handle your asynchronous tasks.

Regards,

Thierry

You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKw-smAukCYyjhg7wp2PoP4cUCJ0JuchNqYG0LLHpVGGL%2B84mA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Zsolt Ero]

So I guess the simplest possible solution is Celery + Redis (since I'm
already using Redis for sessions), I guess?

> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWCt_SfPDmYsVSK%2Bwt74j5%2BSFj8vb40LrN9PRBPY2%2Bb7Gg%40mail.gmail.com.

[Guillermo Cruz]

I've found dramatiq to be a better alternative to celery for these types of tasks.

https://dramatiq.io/v1.4.3/index.html

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCPd0j-sCuHzMa0b-sXNdXrRzB%2BWs2JHzsx5Tu6w02c7A%40mail.gmail.com.

[Jonathan Vanasco]

On Thursday, January 10, 2019 at 8:27:35 AM UTC-5, Zsolt Ero wrote:

So I guess the simplest possible solution is Celery + Redis (since I'm already using Redis for sessions), I guess?

Yes. The performance you want requires a secondary process to handle the event logging. Celery is probably the simplest was, though there may be faster implementations (like Thierry's use of 0mq).

Two other options that may work for you (I use both of these, and Celery):

* if you can use statsd for logging, that setup just sends the event data via UDP (not TCP) and an external statsd server will handle the logging of the event args.
* you can use a secondary database connection / server that is running in autocommit mode. that can avoid the normal database, and might be against a server that writes faster.  for example, you could write to a mysql database that only has logging tables and is tuned for high-performance writes.

[looking tube]

For the inspiration, look here:

https://websauna.org/docs/narrative/misc/task.html

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAKw-smCPd0j-sCuHzMa0b-sXNdXrRzB%2BWs2JHzsx5Tu6w02c7A%40mail.gmail.com.