'Flush' Like "I'm tapping out but keep working" in Python?

[Brandon Wirtz]

With buffered read/writes to HTML you can’t output a few lines do something, output a few more, I get that.

 

But can I say “Your process will be completed shortly” close the output, and then Log some stuff, make some updates to the data store, but let the user get on with life?

 

-Brandon

 

 

 

[Jeff Schnitzer]

On Mon, Jan 23, 2012 at 1:49 PM, Brandon Wirtz <dra...@digerat.com> wrote:
>
> But can I say “Your process will be completed shortly” close the output, and
> then Log some stuff, make some updates to the data store, but let the user
> get on with life?

How would this be different than shunting any remaining work off to
the task queue?

Jeff

[Brandon Wirtz]

Apparently nothing. But having only been doing python for a year, I was
trying to do it the PHP way.

Now that I know... This looks great, and it appears that what I should have
done with ASync writes should have been done with task queues.

Do Task Queues Execute on the Front end instances? Or will adding tasks spin
up a second instance?

I'm currently serving 30k visitors a day per instance, and I don't know if
tasks would take some users from 1 instance to 2 instances.

-Brandon

Jeff

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengi...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

Also do you know if Queue's share memcache?

-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of Jeff Schnitzer
Sent: Monday, January 23, 2012 11:15 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] 'Flush' Like "I'm tapping out but keep
working" in Python?

Jeff

--

[Jeff Schnitzer]

Task Queues execute on frontend instances just like normal requests.
They will only spin up a 2nd instance if the additional task traffic
crosses the normal latency/idle threshold.

Tasks are processed just like normal requests; they share the same
memcache, instance memory, etc as your normal application.

Tasks are, however, separate requests so if you have some thread local
data it won't be available in the later task execution thread. Also
sometimes the task queues get backed up so it's hard to absolutely
predict when a task will be run. Usually it's pretty instantaneous
though.

One thing that makes tasks easier is the Deferred feature, available
in both Python and Java.

The task queue rocks. I lean on it pretty heavily. Also, you can
enlist some number of tasks (5, I think) transactionally, which
creates a good way to work around 2pc issues.

Jeff

[Brandon Wirtz]

Sigh...

Now I have to go do a re-write... Save MORE money on my GAE bill. Maybe
Google should pay you and me not to post to the forums. :-)

[Anand Mistry]

Yes. All elements of an app, including multiple app versions and task queues, share a memcache.

[stevep]

Very helpful. Any ideas about the what+why of sending tasks to the
default version when running a numbered version? thx-stevep

[James Broberg]

Your use case is perfect for the task queue. We use a similar model:
get a request then fire off a task to log some stuff to the database
and jam a value in memcache if required. This way control is returned
back to the user as quickly as possible.

[Robert Kluin]

Hey Steve,
They actually added a default version "target" recently. You can
use that to send a task back to the default version of your app, just
as you would to send a task to a numbered version. Just set the
target to taskqueue.DEFAULT_APP_VERSION.

Robert

[stevep]

Thanks Robert,

I knew about the addition of the "default" TQ call, but am not sure
about why it should be implemented. Last I checked, there was only one
sentence in the docs about this new feature.

Since I use TQs heavily, my main concern is ensuring in the manner
Brandon is investigating the issue of avoiding "minutes or hours to
task initiation" is important for me. Some tasks are not time-
sensitive (e.g. stats updates), however others such as what Brandon is
describing I would like to run as immediately as possible.

My initial thought about sending tasks to a the default app from a
different numbered version was that it would enable this "highly
reliable" queue. If all the only resource user in the default app is
my TQ, does this ensure more reliable and faster task processing vs.
sending the task to a TQ in a shared front-end handler environment?

That's my personal take on using the default queue feature -- but
influenced by my bias for wanting a highly reliable queue. To date,
I've seen nothing in the forums or docs about why the default queue
should be used. I know there is sound rationale for it since someone
took the time to develop it.

BTW Brandon: be sure to look at pull queues also. May have great
potential for your never-ending pursuit to run GAE for Jeff for Free.
View the video session title "Putting Task Queues to Work" at the
link below. First 15 minutes shows how to reduce datastore writes by a
factor of 1,000 -- something I really needed recently. Link:
http://code.google.com/appengine/docs/videoresources.html

Thanks,
stevep

"Task queues: the red-headed child in a family of blondes"

[stevep]

Sorry about my terrible lack of proof reading. Here is what I meant to
say in that terribly mangled sentence from the previous post:

Since I use TQs heavily in the manner Brandon is investigating, my
main concern is ensuring the issue of avoiding "minutes or hours to
task initiation".

[Robert Kluin]

Hey Steve,
 I'm a bit confused about what you're asking / commenting about here.  Are you asking about the default queue, why there is a taskqueue.DEFAULT_APP_VERSION variable, or whether it is better to send tasks to a different version of your app?

 I generally prefer to use the same version of my apps for both tasks and user requests. The biggest reason queues slow down is because there aren't enough instances. Splitting user requests and tasks up will probably just make that problem worse.

 My tasks generally run quite fast, but they do occasionally back up when I font have enough instances.  The taskqueue team recently added a new header you can use to research the execution latency of your tasks.

 I know several others have asked for the ability to define a high priority push queue as well. I agree that this would be a very powerful feature. Particularly if you could define the relative priority to user requests and other queues.  I'm not sure if there is an existing feature request for this or not, but if you make one I'll star it.


Robert

--
------
Robert Kluin
Ezox Systems, LLC

[stevep]

Hi Robert,

Sorry my ramblings were hard to parse. Basic question is the first you
mentioned, "Why is there the TQ option for DEFAULT_APP_VERSION, and
when is it best used?"

There was a discussion related to a priority TQ a year or so ago.
Someone (Vlad?) initiated a request for a priority TQ, and I did star
it at the time.

Cheers,
Steve

On Jan 27, 10:16 pm, Robert Kluin <robert.kl...@gmail.com> wrote:
> Hey Steve,

[Robert Kluin]

Hey Steve,
The DEFAULT_APP_VERSION var was added so that you could send a task
to whatever version is currently marked as default. Without it there
was no way to target a task at the current default version of your
app. You'd want to use it if you have, for example, a backend that
inserts tasks that need to run on the current default version. There
are a number of cases where this arises, when you hit one you will
know.

Yeah I'd love to see the ability to give task queues a priority
relative to user requests and relative to each other. That would be
pretty cool. Of course if they implement that then we could probably
get even finer control over the scheduler -- which could be handy as
well!

Robert

[stevep]

Thanks so much Robert. Thinking about what you wrote:
Yeah I'd love to see the ability to give task queues a priority
relative to user requests and relative to each other. That would be
pretty cool. Of course if they implement that then we could probably
get even finer control over the scheduler -- which could be handy as
well!

Fingers crossed. Actually we really need both ends of the spectrum. A
high priority+availability TQ, and a very low priority one that only
fires up on instances that have been initiated but are now not too
busy after a spike. The latter would preclude issues like Brandon
thinking to write a process that would use idle cycles to do deletes.
Could really help with instance optimization I'd guess -- although
that statement is made without any understanding of the interplay
between the Instance and TQ schedulers.

On Jan 29, 9:19 pm, Robert Kluin <robert.kl...@gmail.com> wrote:
> Hey Steve,

[Robert Kluin]

On Mon, Jan 30, 2012 at 10:47, stevep <pros...@gmail.com> wrote:
> Thanks so much Robert. Thinking about what you wrote:
>  Yeah I'd love to see the ability to give task queues a priority
> relative to user requests and relative to each other.  That would be
> pretty cool.  Of course if they implement that then we could probably
> get even finer control over the scheduler -- which could be handy as
> well!
>
> Fingers crossed. Actually we really need both ends of the spectrum. A
> high priority+availability TQ, and a very low priority one that only
> fires up on instances that have been initiated but are now not too
> busy after a spike. The latter would preclude issues like Brandon
> thinking to write a process that would use idle cycles to do deletes.
> Could really help with instance optimization I'd guess -- although
> that statement is made without any understanding of the interplay
> between the Instance and TQ schedulers.

Yeah, I love this idea. A way to designate a queue so that it isn't
factored in for instance spinup (or keep alive), but would just run
tasks when there are instances sitting around waiting to get killed
off. I've got a lot of processes I'd like to run in that type of
mode, or at least the "don't cause new spinup mode".

Something to think about with that, you can specify the maximum
concurrent requests for a queue. So you can stop it from rampant
instance spinup. Just set a low concurrent rate, and that would at
least limit the queue to processing as much as X instances can handle.