Architecture question - asynchronous queue

[Bill Moseley]

I'm looking for ideas or examples of asynchronous (background) queue processing in a web environment.  I apologize for the length of this mail. :) 

Say that I have a web app where users upload large photos.  The photos get processed in two steps by different back-end servers.  One process resizes the photos, and another does face recognition once the image is resized.

One requirement is that the user of the web app must confirm that the the image was processed as expected.  If the upload does not require face recognition then must confirm that the image can be viewed (by previewing a thumbnail).  If face recognition is also requested then the confirming means viewing the image after face recognition.

In other words, there's either one or two steps that must be done after the upload and then confirmed.

To meet the confirmation requirement I could run the process synchronously -- that is the web app waits for the job(s) to complete.  That's not desirable as it tightly couples the back-end with the web server and if the back-end processes get bogged down then the web process is blocked.

Should the web tier submit a background job to Gearman for image resizing, then have the resize process create a new job for face recognition once it is done?

When the web process submits a job for image resizing it will receive an opaque id that it can use to check the status of that job.  But, if the image resize job then creates the face recognition job upon completion how does the web tier check the status of this new job?

Does there need to be a parent job that manages the separate image resize and face recognition tasks?  Or should the resize, upon completion, queue the face recognition job as a background task and then return the opaque job id for the face recognition job as the response to it's resize request?   That is, when the web tier checks the status it either gets "job 50% done", "job completed", or "my job complete, but now go look at this other job id now."

Another requirement is we must prevent jobs from running out of sequence.  For example, a user uploads the wrong, yet very large image for their personal profile.  The back-end process takes a long time processing this image.  The the user realizes their mistake and uploads the correct smaller image as a replacement profile image while the first image is still being processed.  What I want to prevent is the the first processing finishes after the second upload clobbering the the second upload.  Granted, that's more of a version management issue, but I wonder if Gearman has a way to either prevent this or a way to say "cancel that first job!"

Thanks for your input.

--
Bill Moseley
mos...@hank.org

[Adrian Otto]

Your use case fits well into a SEDA (Staged Event Driven Architecture) paradigm. Stages are separated using queues so you first queue the job for stage one, and upon completion it is queued for stage two, so on and so forth. It helps to have a metadata service as a global state machine so you can query the state of any given job, and know what stage its in. You can implement the metadata service using SOA if you want, but make sure it issues a unique job id that follows the job through all of its stages. This is also where you can indicate which jobs should go through which stages. You could create an index that would allow you to look up active jobs for a given account/type so it would then be possible for a subsequent job to superceede a previous one. I don't know if gearman has a way to cancel jobs as they are running. It may not even be required for what you want. Let the big job keep processing, and just set up the workers to have some simple absolute timeout ala SIGALRM
.

Having separate queues for each stage makes it easy to scale resources for the stage that needs it. If your pending queue length is consistently nonzero for a stage, then it needs more workers.

For the asyncronous client notification, a callback can work nicely if you have low enough concurrency. Otherwise waiters can periodically poll a completion queue.

Adrian

[Vick Khera]

On Tue, Oct 26, 2010 at 9:53 AM, Bill Moseley <mos...@hank.org> wrote:
> When the web process submits a job for image resizing it will receive
> an opaque id that it can use to check the status of that job.  But, if the
> image resize job then creates the face recognition job upon completion how
> does the web tier check the status of this new job?

I'd make a single job which does two things: run the sizer job
synchronously, and then the face recognition job synchronously. Your
web server task only ever waits on the master job.

What I'm working on is having the web server page auto-reload every
few seconds to probe the job, and updating some eye candy to make the
user think work is really happening. Once it detects the job is done
(and all sub-jobs implicitly) it can move to the next stop of the UI.
Minimal load on the web server this way (ie, you don't tie up a whole
connection waiting for work to be done).

[John Pettitt]

On Oct 26, 2010, at 10:53 AM, Vick Khera wrote:

>
>
> What I'm working on is having the web server page auto-reload every
> few seconds to probe the job, and updating some eye candy to make the
> user think work is really happening. Once it detects the job is done
> (and all sub-jobs implicitly) it can move to the next stop of the UI.
> Minimal load on the web server this way (ie, you don't tie up a whole
> connection waiting for work to be done).

jQuery/AJAX is your friend here - it's trivially easy to do the polling in background and avoid the whole page reload.

We do something similar on http://Curate.Us - when you first ask us to clip a web page that job is async and assigned to a pool of clipping servers. Then you resize or reposition an existing clip we use gearman to assign it to a pool of render servers and run it synchronously. For example all the non-browser image manipulation on this clip of the gearman home page is done using the sync render servers http://s.tt/11sMG+

We also use gearman for logging - all the log events are sent asynchronously to a memcache backed gearman server with workers doing a bunch of de-normalization on the result so that we don't keep the web servers hanging around to write complex logs and update stats - this also lets us treat the log workers as long running batch jobs which can maintain local state for thinks like persistent DB connection, in memory copies of the browscap db etc.

John
j...@p.tt