Multi-request workers proposal for handling transactions

[Manuel Lemos]

Hello,

Sometimes it is necessary to perform transactions in multiple steps that
require calling workers.

For instance, a client needs to call a worker to get some information
for instance to edit a database record, and depending on that
information, it needs to send more requests all within the same transaction.

The problem is that for instance a database transaction must be enclosed
by the same process that establishes the database connection.

It seems that as it is now, Gearman is not able to assure that multiple
requests are handled by the same worker. So it is not guaranteed that a
transaction started by one worker is resumed and finished by the same
worker.

My proposal is to evolve Gearman in order to handle this situation. It
could be something would return a transaction id to a client when it
sends the first request.

The job server that takes the request would call a worker and make sure
that it will be reserved for requests with the same transaction id until
no more requests are sent with that transaction id and the transaction
finishes.

What do you think?

--

Regards,
Manuel Lemos

Find and post PHP jobs
http://www.phpclasses.org/jobs/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Joseph Stein]

Can't you just chain the worker by having it be a client making it request execution of another task?

This seems like an interesting concept perhaps instead of transaction it could be called a token and be more of a workflow type of solution in regards to state.

By making it a token you do not then require the transaction to complete with the same worker (as that worker may no longer be available).

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

[Manuel Lemos]

Hello,

on 11/26/2009 03:52 AM Joseph Stein said the following:

> Can't you just chain the worker by having it be a client making it
> request execution of another task?
>
> This seems like an interesting concept perhaps instead of transaction it
> could be called a token and be more of a workflow type of solution in
> regards to state.
>
> By making it a token you do not then require the transaction to complete
> with the same worker (as that worker may no longer be available).

The problem is very common. In Web scripts we need to process database
transactions that usually run within the same script.

Gearman is great for distributing load to backend workers, so all
database interaction is pushed to the workers.

The problem is that Web scripts still handle user interface logic, like
forms processing.

So the user interface processing often needs data from the backend that
is processed in a way that depends on what is the state of the data and
what the user submitted.

Then the Web script decides what to do in each situation and may need to
request more data or execute more database queries to update the data in
the backend worker. To be consistent, all requests need to be done in
the same database transaction, just like before the backend processing
was pushed to workers.

This is why it is convinient to have it done with the same worker.

Keep in mind that this all happens during the same HTTP request. I do
not intend to start database transacations that span over multiple HTTP
requests.

[Pedro Melo]

Hi,

I think that it goes against gearman basic ideas.

Can't you move all the logic that needs this back-and-forward to a
gearman function?

This way all that is executed inside a single worker and it becomes
trivial do to what you want.

Bye

[Pedro Melo]

Hi,

Take all HTTP request information, create a job with that. A single
worker can then do whatever it needs to inside the same transaction.

Basically, you would be dispatching URLs into Gearman functions.

Bye,

[Manuel Lemos]

Hello,

on 11/26/2009 06:16 AM Pedro Melo said the following:

> Take all HTTP request information, create a job with that. A single
> worker can then do whatever it needs to inside the same transaction.
>
> Basically, you would be dispatching URLs into Gearman functions.

That would mean replacing the Web server by Gearman, or otherwise
passing all the HTTP request data to Gearman worker, which does not seem
to be a good idea.

I understand that Gearman does not do yet what I am proposing. The idea
is to develop support for multi-request workers as I described. Anybody
would see a problem in developing a patch to implement that feature?

[Pedro Melo]

On 2009/11/26, at 08:33, Manuel Lemos wrote:

> Hello,
>
> on 11/26/2009 06:16 AM Pedro Melo said the following:
>> Take all HTTP request information, create a job with that. A single
>> worker can then do whatever it needs to inside the same transaction.
>>
>> Basically, you would be dispatching URLs into Gearman functions.
>
> That would mean replacing the Web server by Gearman, or otherwise
> passing all the HTTP request data to Gearman worker, which does not
> seem
> to be a good idea.

It seemed to me that it was exactly what you where doing.

It might be a good ideia, it really depends on the situation. I know I
do it for some things.

Bye,

[Vick Khera]

On Thu, Nov 26, 2009 at 3:33 AM, Manuel Lemos <mle...@acm.org> wrote:
> I understand that Gearman does not do yet what I am proposing. The idea
> is to develop support for multi-request workers as I described. Anybody
> would see a problem in developing a patch to implement that feature?
>

The problem with this is that you create potentially long-running
"idle in transaction" connections, which would impede the database
from doing some other work, or locking out other transactions. If
you're worker is holding state like this, then it has to have some
notion of a timeout to give up those locks.

The right thing really is to push the whole job off to one worker and
let it do all of the work, and not try to break up your transaction
into smaller chunks which would just get re-assigned back to this same
worker anyway. It just adds a bunch of round-trip overhead to the
gearman server.

As for "replacing the web server with gearman"... well, that's one of
its use cases.

[Manuel Lemos]

Hello,

on 11/26/2009 05:35 PM Vick Khera said the following:

> On Thu, Nov 26, 2009 at 3:33 AM, Manuel Lemos <mle...@acm.org> wrote:
>> I understand that Gearman does not do yet what I am proposing. The idea
>> is to develop support for multi-request workers as I described. Anybody
>> would see a problem in developing a patch to implement that feature?
>>
>
> The problem with this is that you create potentially long-running
> "idle in transaction" connections, which would impede the database
> from doing some other work, or locking out other transactions. If
> you're worker is holding state like this, then it has to have some
> notion of a timeout to give up those locks.

I don't think it is up to Gearman to manage database transactions.

In the worst case, if the client disconnects without finishing the
transaction, the job server should signal the worker so it rolls back
the transaction.

> The right thing really is to push the whole job off to one worker and
> let it do all of the work, and not try to break up your transaction
> into smaller chunks which would just get re-assigned back to this same
> worker anyway. It just adds a bunch of round-trip overhead to the
> gearman server.

As I mentioned it does not make sense to push the whole HTTP request
data to the worker when it may need only just somethings that can be
processed by the Web scripts.

Also the same Web script may not even call the worker, depending what
the user has submitted. The problem is that to make that decision, the
Web script needs to pull data from the worker.

> As for "replacing the web server with gearman"... well, that's one of
> its use cases.

I think it will take take time before Gearman can prove itself as a good
replacement of Web servers.

[Cesar D. Rodas]

Hello Manuel

2009/11/26 Manuel Lemos <mle...@acm.org>

Hello,

on 11/26/2009 06:16 AM Pedro Melo said the following:

> Take all HTTP request information, create a job with that. A single
> worker can then do whatever it needs to inside the same transaction.
>
> Basically, you would be dispatching URLs into Gearman functions.

That would mean replacing the Web server by Gearman, or otherwise
passing all the HTTP request data to Gearman worker, which does not seem
to be a good idea.

I understand that Gearman does not do yet what I am proposing. The idea
is to develop support for multi-request workers as I described. Anybody
would see a problem in developing a patch to implement that feature?

You're  proposal is this  (as far as I understood), right?

$client = new GermanClient();

$t = $client->begin(); /* We're attached to one client; node-a */
$t->begin('foo');      /* Node-a process foo */
sleep(10);               /* Node-a is locked waiting another request */
$t->begin('bar')      /* node-a process bar */
$t->end();              /* node-a is unlocked */


To me this proposal makes a sense and it could be useful. Develop a patch wouldn't be that hard and I can give a hand with it.

Best regards,
 

--

Regards,
Manuel Lemos

Find and post PHP jobs
http://www.phpclasses.org/jobs/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

--
Cesar D. Rodas
http://cesar.la/
Phone: +595-961-974165

[Clint Byrum]

I'm not sure why this is important or how it would actually work.

in pseudo code:

web script:

workload_object = new StuffTodo(webrequest_data)
client = new GearmanClient
client->do('process_workload',workload_object);


worker:

func handle_workload (job)
workload_object = job->getWorkload()
do_sql_begin
do_sql_operation1
do_sql_operation2
do_sql_commit
job->success

worker = new GearmanWorker
worker->addFunction('process_workload',handle_workload)
worker->work()

At what point do we need to do multiple requests in a single
transaction.. ? If op1 and op2 don't always happen in serial.. how are
they going to work reliably in a transaction?

[David Ashwood]

The granularity of a worker is a single work unit - distributing
database locks outside of a work unit (essentially what you're talking
about) is generally very complex and often a bad idea.

Your application should really be separating presentation from logic -
ala the popular MVC type approach to separating form from function.

Here you have a single activity or action - that does the processing
logic and would be processed by a worker.

In your example you mention "may request more information" - I'm
assuming that this is from the database and not from the user?

Regards,

David

[Manuel Lemos]

Hello,

on 11/26/2009 07:14 AM David Ashwood said the following:

> The granularity of a worker is a single work unit - distributing

That is in the current design, but the proposal is to improve it and
make more useful.

> database locks outside of a work unit (essentially what you're talking
> about) is generally very complex and often a bad idea.
>
> Your application should really be separating presentation from logic -
> ala the popular MVC type approach to separating form from function.

You am afraid you are missing the point. That is precisely because the
model is separated from the controller that I want to push the model to
workers. The controller does not access the database, and the model does
not access the HTTP request data.

The presentation does not access neither the database nor the request
data. At most the controller will ask the model to return data, and the
controller makes it available for the view.

> Here you have a single activity or action - that does the processing
> logic and would be processed by a worker.
>
> In your example you mention "may request more information" - I'm
> assuming that this is from the database and not from the user?

Right, I am talking of a single HTTP request. The controller makes the
decisions on what to do based on what the user submitted and what is the
state of the data in the relevant database records.

[Manuel Lemos]

Hello,

on 11/26/2009 10:13 AM Cesar D. Rodas said the following:

> I understand that Gearman does not do yet what I am proposing. The idea
> is to develop support for multi-request workers as I described. Anybody
> would see a problem in developing a patch to implement that feature?
>
> You're proposal is this (as far as I understood), right?
>
> $client = new GermanClient();
>
> $t = $client->begin(); /* We're attached to one client; node-a */
> $t->begin('foo'); /* Node-a process foo */
> sleep(10); /* Node-a is locked waiting another request */
> $t->begin('bar') /* node-a process bar */
> $t->end(); /* node-a is unlocked */

Yep, it is almost like you suggest. I would just make it more compact to
reduce the API calls:

$t = $client->begintransaction("worker start transaction function",
$parameter)
$t->do('something');
$t->do('maybe do something else');
$t->endtransaction("worker end transaction function", $parameter);

> To me this proposal makes a sense and it could be useful. Develop a
> patch wouldn't be that hard and I can give a hand with it.

Great. It would be great if you would like to help to develop a patch to
support this.

I just don't know what is the patch submission process? What is the
right way to develop an enhancement so it gets integrated in the end?

[Manuel Lemos]

hELLO,

on 11/26/2009 07:46 AM Clint Byrum said the following:

> in pseudo code:
>
> web script:
>
> workload_object = new StuffTodo(webrequest_data)
> client = new GearmanClient
> client->do('process_workload',workload_object);
>
>
> worker:
>
> func handle_workload (job)
> workload_object = job->getWorkload()
> do_sql_begin
> do_sql_operation1
> do_sql_operation2
> do_sql_commit
> job->success
>
> worker = new GearmanWorker
> worker->addFunction('process_workload',handle_workload)
> worker->work()
>
> At what point do we need to do multiple requests in a single
> transaction.. ? If op1 and op2 don't always happen in serial.. how are
> they going to work reliably in a transaction?

The part that you seem to be missing is that the Web script needs to
make a decision on what it needs to do next based on what is the state
of the data retrieved from the database and what the Web user submitted.

[dormando]

> You am afraid you are missing the point. That is precisely because the
> model is separated from the controller that I want to push the model to
> workers. The controller does not access the database, and the model does
> not access the HTTP request data.
>
> The presentation does not access neither the database nor the request
> data. At most the controller will ask the model to return data, and the
> controller makes it available for the view.

This still fits your model fine. Why do you think you need to pass the raw
HTTP request data? Pass the form headers that you need processed into the
model for the form processing part. Do some sanity checking first, even.

Sending individual requests of a transaction down to gearman is certainly
missing the point. You're not distributing any significant work other than
the DB requests themselves. This is a logical distribution from a false
MVC abstraction.

Think about it:

client -> gearman -> database (part of it)
for each section. Why not process all of that directly in the client?
Gearman is serving as a mild mannered middleman and adding network
roundtrips for no reason.

[Manuel Lemos]

Hello,

on 11/26/2009 08:07 PM dormando said the following:

>> You am afraid you are missing the point. That is precisely because the
>> model is separated from the controller that I want to push the model to
>> workers. The controller does not access the database, and the model does
>> not access the HTTP request data.
>>
>> The presentation does not access neither the database nor the request
>> data. At most the controller will ask the model to return data, and the
>> controller makes it available for the view.
>
> This still fits your model fine. Why do you think you need to pass the raw
> HTTP request data? Pass the form headers that you need processed into the
> model for the form processing part. Do some sanity checking first, even.

I do not want to pass the whole HTTP request data to the worker. That is
the point. The client (web script) controller) processes the user
submitted data and only passes the necessary values to the worker.

> Sending individual requests of a transaction down to gearman is certainly
> missing the point. You're not distributing any significant work other than
> the DB requests themselves. This is a logical distribution from a false
> MVC abstraction.

> Think about it:
>
> client -> gearman -> database (part of it)
> for each section. Why not process all of that directly in the client?
> Gearman is serving as a mild mannered middleman and adding network
> roundtrips for no reason.

Database access is the most important bottleneck of Web applications.
When your Web application grows, it starts requiring more than one
server to handle the requests. When that happens I just want to add more
backend servers for workers that handle database access.

In the front end I just need to keep Web server threads that have very
low memory requirements, so I can fit much more Web server requests in
one server.

[Pedro Melo]

Hi,

On 2009/11/26, at 20:44, Manuel Lemos wrote:

> on 11/26/2009 05:35 PM Vick Khera said the following:
>> On Thu, Nov 26, 2009 at 3:33 AM, Manuel Lemos <mle...@acm.org> wrote:
>>> I understand that Gearman does not do yet what I am proposing. The
>>> idea
>>> is to develop support for multi-request workers as I described.
>>> Anybody
>>> would see a problem in developing a patch to implement that feature?
>>>
>>
>> The problem with this is that you create potentially long-running
>> "idle in transaction" connections, which would impede the database
>> from doing some other work, or locking out other transactions. If
>> you're worker is holding state like this, then it has to have some
>> notion of a timeout to give up those locks.
>
> I don't think it is up to Gearman to manage database transactions.

Of course not, but workers do have that job, and in your use case they
would be doing that over longer periods of time.

> In the worst case, if the client disconnects without finishing the
> transaction, the job server should signal the worker so it rolls back
> the transaction.

In your scenario, and due to normal race conditions between webserver
and worker, this can also happen.

Please note that this is not new. I don't remember a single web-
framework that will interrupt the processing of a request if the HTTP
TCP connection is killed by the client, because most of the frameworks
only detect dropped TCP connections when they try to write back the
response, usually at the end of the request.

So this scenario where the work is completed even in the absence of a
active client is pretty common.

Also, note that this scenario is much more resilient to temporary
failures in the communication between clients and servers.

>> The right thing really is to push the whole job off to one worker and
>> let it do all of the work, and not try to break up your transaction
>> into smaller chunks which would just get re-assigned back to this
>> same
>> worker anyway. It just adds a bunch of round-trip overhead to the
>> gearman server.
>
> As I mentioned it does not make sense to push the whole HTTP request
> data to the worker when it may need only just somethings that can be
> processed by the Web scripts.

It does not make sense for you :), but it is one of the most used
cases for Gearman deployment.

On some large processing jobs, we will even start a bg gearman job in
the first request, create a response with the "job id" and then let
the browser reload a results page (or use XHR to pool or long-pool for
the result) of that job.

Think image resizing or anything that takes more than a second.

> Also the same Web script may not even call the worker, depending what
> the user has submitted. The problem is that to make that decision, the
> Web script needs to pull data from the worker.

We move all the decision process to the worker side. After the worker
receives all the request information, he should have all the data, or
ways to get at it, that he needs to complete the request.

>> As for "replacing the web server with gearman"... well, that's one of
>> its use cases.
>
> I think it will take take time before Gearman can prove itself as a
> good
> replacement of Web servers.

I think it did that a couple years ago actually. We don't assume that
gearman will receive HTTP requests, we assume that HTTP requests
received by your favorite web framework that will take some time to
process can be off-loaded to a farm of gearman workers (one that can
scale out trivially). And in that scenario, you can find several
reports of real-work usage where Gearman did make a big difference.

The architecture that Gearman represents is widely used today by all
the major sites: Google, Yahoo, Flickr, YouTube, you named it. Anybody
that has long processing times per requests uses something that is
architectural similar to Gearman.

Bye,

[Pedro Melo]

Hi,

On 2009/11/27, at 03:15, Manuel Lemos wrote:

> on 11/26/2009 08:07 PM dormando said the following:
>>> You am afraid you are missing the point. That is precisely because
>>> the
>>> model is separated from the controller that I want to push the
>>> model to
>>> workers. The controller does not access the database, and the
>>> model does
>>> not access the HTTP request data.
>>>
>>> The presentation does not access neither the database nor the
>>> request
>>> data. At most the controller will ask the model to return data,
>>> and the
>>> controller makes it available for the view.
>>
>> This still fits your model fine. Why do you think you need to pass
>> the raw
>> HTTP request data? Pass the form headers that you need processed
>> into the
>> model for the form processing part. Do some sanity checking first,
>> even.
>
> I do not want to pass the whole HTTP request data to the worker.
> That is
> the point. The client (web script) controller) processes the user
> submitted data and only passes the necessary values to the worker.

So you have a different point from what Gearman is trying to solve. I
personally don't like your solution but it might make sense on your
use case.

But I don't think Gearman is the tool you are looking for then. Maybe
a DB connection pool is more what you need, where you can start
several DB connections to increase parallelism.

>> Sending individual requests of a transaction down to gearman is
>> certainly
>> missing the point. You're not distributing any significant work
>> other than
>> the DB requests themselves. This is a logical distribution from a
>> false
>> MVC abstraction.
>
>> Think about it:
>>
>> client -> gearman -> database (part of it)
>> for each section. Why not process all of that directly in the client?
>> Gearman is serving as a mild mannered middleman and adding network
>> roundtrips for no reason.
>
> Database access is the most important bottleneck of Web applications.
> When your Web application grows, it starts requiring more than one
> server to handle the requests. When that happens I just want to add
> more
> backend servers for workers that handle database access.

If the database is your bottleneck, adding more workers querying the
database is not going to solve you problem. In fact it will only make
it worse.

Best regards,

[Manuel Lemos]

Hello,

on 11/27/2009 06:29 AM Pedro Melo said the following:

>>> On Thu, Nov 26, 2009 at 3:33 AM, Manuel Lemos <mle...@acm.org> wrote:
>>>> I understand that Gearman does not do yet what I am proposing. The idea
>>>> is to develop support for multi-request workers as I described. Anybody
>>>> would see a problem in developing a patch to implement that feature?
>>>>
>>>
>>> The problem with this is that you create potentially long-running
>>> "idle in transaction" connections, which would impede the database
>>> from doing some other work, or locking out other transactions. If
>>> you're worker is holding state like this, then it has to have some
>>> notion of a timeout to give up those locks.
>>
>> I don't think it is up to Gearman to manage database transactions.
>
> Of course not, but workers do have that job, and in your use case they
> would be doing that over longer periods of time.

That is not my point. What I am saying is whether I push database access
to Gearman workers or not, the use of database transactions will remain
the same. So it is not the use of Gearman that make it have to manage my
application database transactions.

>> In the worst case, if the client disconnects without finishing the
>> transaction, the job server should signal the worker so it rolls back
>> the transaction.
>
> In your scenario, and due to normal race conditions between webserver
> and worker, this can also happen.
>
> Please note that this is not new. I don't remember a single

> web-framework that will interrupt the processing of a request if the

> HTTP TCP connection is killed by the client, because most of the
> frameworks only detect dropped TCP connections when they try to write
> back the response, usually at the end of the request.
>
> So this scenario where the work is completed even in the absence of a
> active client is pretty common.
>
> Also, note that this scenario is much more resilient to temporary
> failures in the communication between clients and servers.

Sorry, I was not clear. I do not want Gearman to kill or otherwise
interrupt workers. What I man is that by the end of worker process that
was part of multi-request worker, the job server should tell the worker
somehow that the client disconnected. Then the worker script should do
what needs to do to eventually rollback an ongoing database transaction.

I am not concerned with the use case that the Web browser disconnects
from the Web server. I am more concerned with the situation that the Web
script is abnormally terminated (think, somebody killed the Web script
process).

>>> The right thing really is to push the whole job off to one worker and
>>> let it do all of the work, and not try to break up your transaction
>>> into smaller chunks which would just get re-assigned back to this same
>>> worker anyway. It just adds a bunch of round-trip overhead to the
>>> gearman server.
>>
>> As I mentioned it does not make sense to push the whole HTTP request
>> data to the worker when it may need only just somethings that can be
>> processed by the Web scripts.
>
> It does not make sense for you :), but it is one of the most used cases
> for Gearman deployment.

OK, each developer uses Gearman the way they think is best for his
application.

For me, I just do not think that is a good idea to pass all the HTTP
request variables, HTTP headers, etc.. on all requests like CGI does. It
is needless overhead.

> On some large processing jobs, we will even start a bg gearman job in
> the first request, create a response with the "job id" and then let the
> browser reload a results page (or use XHR to pool or long-pool for the
> result) of that job.
>
> Think image resizing or anything that takes more than a second.

Right, but you do not need to pass the whole HTTP request data to the
worker. If you do image resizing, you just pass the the image data and
probably a few more parameters.

>> Also the same Web script may not even call the worker, depending what
>> the user has submitted. The problem is that to make that decision, the
>> Web script needs to pull data from the worker.
>
> We move all the decision process to the worker side. After the worker
> receives all the request information, he should have all the data, or
> ways to get at it, that he needs to complete the request.

It may make sense for you to do that, but for me that is the same as
using FastCGI for distributing HTTP requests to multiple backend
servers. If I can avoid passing the whole HTTP request data to back-end
workers, I would rather do it.

[Manuel Lemos]

Hello,

on 11/27/2009 06:39 AM Pedro Melo said the following:

>>>> You am afraid you are missing the point. That is precisely because the
>>>> model is separated from the controller that I want to push the model to
>>>> workers. The controller does not access the database, and the model
>>>> does
>>>> not access the HTTP request data.
>>>>
>>>> The presentation does not access neither the database nor the request
>>>> data. At most the controller will ask the model to return data, and the
>>>> controller makes it available for the view.
>>>
>>> This still fits your model fine. Why do you think you need to pass
>>> the raw
>>> HTTP request data? Pass the form headers that you need processed into
>>> the
>>> model for the form processing part. Do some sanity checking first, even.
>>
>> I do not want to pass the whole HTTP request data to the worker. That is
>> the point. The client (web script) controller) processes the user
>> submitted data and only passes the necessary values to the worker.
>
> So you have a different point from what Gearman is trying to solve. I
> personally don't like your solution but it might make sense on your use
> case.
>
> But I don't think Gearman is the tool you are looking for then. Maybe a
> DB connection pool is more what you need, where you can start several DB
> connections to increase parallelism.

It would be if my concern was just about processes that take more time.
I am also concerned with processes that take more memory.

The number of simultaneous requests that a Web server process can take
depend on the available RAM. If the RAM is exhausted with all
simultaneous request you are getting, you need more servers.

A typical PHP Web script process takes 10MB. However, the processes
increase of size over time because some requests use a lot more RAM than
average. Overtime you start seeing processes taking 20MB or more, often
because you are doing database queries that buffer results and take
space in large arrays. That memory is not freed until the Web server
processes are recycled (killed and restarted).

Pushing database access to workers, will make my Web server processes
use less memory in average, and so deal with more simultaneous requests.

Database access worker processes will still keep growing eventually in
memory usage. However, they will use memory of back end servers, which
eventually will not be the same machines as the Web servers.

The point of doing this, is that not all Web server requests need
database access, so the need to add more servers to workers will not
necessary follow the need to deal with more simultaneous user Web
accesses, specially in sites that are mostly content based, as most
content can be cached, thus no database access.

>>> Sending individual requests of a transaction down to gearman is
>>> certainly
>>> missing the point. You're not distributing any significant work other
>>> than
>>> the DB requests themselves. This is a logical distribution from a false
>>> MVC abstraction.
>>
>>> Think about it:
>>>
>>> client -> gearman -> database (part of it)
>>> for each section. Why not process all of that directly in the client?
>>> Gearman is serving as a mild mannered middleman and adding network
>>> roundtrips for no reason.
>>
>> Database access is the most important bottleneck of Web applications.
>> When your Web application grows, it starts requiring more than one
>> server to handle the requests. When that happens I just want to add more
>> backend servers for workers that handle database access.
>
> If the database is your bottleneck, adding more workers querying the
> database is not going to solve you problem. In fact it will only make it
> worse.

Please, read the above.

[Eric Day]

Hi Manuel,

On Fri, Nov 27, 2009 at 01:15:07AM -0200, Manuel Lemos wrote:
> > This still fits your model fine. Why do you think you need to pass the raw
> > HTTP request data? Pass the form headers that you need processed into the
> > model for the form processing part. Do some sanity checking first, even.
>
> I do not want to pass the whole HTTP request data to the worker. That is
> the point. The client (web script) controller) processes the user
> submitted data and only passes the necessary values to the worker.

That seems to be what dormando was suggesting, only pass a subset
of request information to the worker, only what is required to make
any decisions and respond with the result you need. Passing a few
hundred bytes of form data to the worker is cheap, and can easily fit
in the same TCP packets as the other protocol request information,
so no extra overhead there if you are using Gearman.

> > client -> gearman -> database (part of it)
> > for each section. Why not process all of that directly in the client?
> > Gearman is serving as a mild mannered middleman and adding network
> > roundtrips for no reason.
>
> Database access is the most important bottleneck of Web applications.
> When your Web application grows, it starts requiring more than one
> server to handle the requests. When that happens I just want to add more
> backend servers for workers that handle database access.

Yes, most of us quite familiar with basic web scaling topics. :)

What dormando was saying though is rather than using Gearman to manage
your database connections, either use a direct database connections to
other servers or use a database-specific load balancer/proxy. Gearman
was not designed to have transactional context within it. Once a job
is started, the client does not have the ability to send any other
information to the connected worker. While it's not impossible to add,
it starts to introduce new functionality that goes against some of
the basic design principals Gearman promotes.

> In the front end I just need to keep Web server threads that have very
> low memory requirements, so I can fit much more Web server requests in
> one server.

If managing database connections directly in the front-end web
requests, you should be able to keep memory utilization low with proper
result set cleanup. The load balancing is a little trickier if you
are dealing with multiple masters or read slaves, but it can be done.

Like I said above, you may be looking for something else besides
Gearman, but Gearman could be a great solution for you if you just
pass the few pieces of request information needed by the worker to
make a decision.

In another thread you ask about contributing to Gearman. Check out
all the various projects on the gearman.org download page. Adding
some features is easy if it's just client or server behavior. New
protocol changes are not trivial because to be useful within the
community it needs to be generally accepted and implemented by a
handful of project maintainers. Sending out any such proposals to
the mailing list for community feedback is a good way to start.

-Eric

[Clint Byrum]

On Nov 26, 2009, at 7:15 PM, Manuel Lemos wrote:

Database access is the most important bottleneck of Web applications.
When your Web application grows, it starts requiring more than one
server to handle the requests. When that happens I just want to add more
backend servers for workers that handle database access.

Your goal is definitely something suited to gearman... limiting or multiplying backend processing.

However, what isn't suited to gearman, is lots of time spent communicating instead of working.

Basically it sounds like you desire to have only the "model" in the worker, and no controller code. However, other than some code separation ideal, I don't see a good reason for that. Move both pieces into the worker, and you are doing just fine. In the "MVC" world, dos it matter if the controller *and* model are *called* in one binary? You can still have the model code separate, but called from the "controller" worker thread.

In the front end I just need to keep Web server threads that have very
low memory requirements, so I can fit much more Web server requests in
one server.

If you keep only "view" related code in those webserver "threads" then one would expect them to use less memory on account of having less code and data to work with.

[David Ashwood]

On Sat, 2009-11-28 at 23:41 -0800, Clint Byrum wrote:

>
> Basically it sounds like you desire to have only the "model" in the
> worker, and no controller code. However, other than some code
> separation ideal, I don't see a good reason for that. Move both pieces
> into the worker, and you are doing just fine. In the "MVC" world, dos
> it matter if the controller *and* model are *called* in one binary?
> You can still have the model code separate, but called from the
> "controller" worker thread.

Exactly - the worker is just another controller/action.

We have a separate MVC app for workers and map the routing when we
register gearman functions. This gives us the same logical paradigm
throughout all layers and allow us to keep things dry. Symfony (PHP MVC
Framework) apps are logical subsets of functionality and mapping the
routing allows us to easily and quickly extend the gearman services.

Results from gearman actions are generally json objects, occasionally
with typed results that indicate if the actual data should be retrieved
from memcached, mongo, mongoFS or the database.

This does come at a cost - gearman processes have a heavier footprint
due to the framework overheads - but this comes down to how well you
streamline it's configuration, what's loaded, etc - but brings a bag of
benefits - common application logic & processes, logging, debugging,
etc.

[Manuel Lemos]

Hello Eric,

on 11/27/2009 04:16 PM Eric Day said the following:

>>> This still fits your model fine. Why do you think you need to pass the raw
>>> HTTP request data? Pass the form headers that you need processed into the
>>> model for the form processing part. Do some sanity checking first, even.
>> I do not want to pass the whole HTTP request data to the worker. That is
>> the point. The client (web script) controller) processes the user
>> submitted data and only passes the necessary values to the worker.
>
> That seems to be what dormando was suggesting, only pass a subset
> of request information to the worker, only what is required to make
> any decisions and respond with the result you need. Passing a few
> hundred bytes of form data to the worker is cheap, and can easily fit
> in the same TCP packets as the other protocol request information,
> so no extra overhead there if you are using Gearman.

I am afraid the point you are still missing is that often the Web script
only knows what information needs to be passed to the worker only after
it retrieves database records to know the state of the stored data. Only
after that it can decide what to do next.

>>> client -> gearman -> database (part of it)
>>> for each section. Why not process all of that directly in the client?
>>> Gearman is serving as a mild mannered middleman and adding network
>>> roundtrips for no reason.
>> Database access is the most important bottleneck of Web applications.
>> When your Web application grows, it starts requiring more than one
>> server to handle the requests. When that happens I just want to add more
>> backend servers for workers that handle database access.
>
> Yes, most of us quite familiar with basic web scaling topics. :)
>
> What dormando was saying though is rather than using Gearman to manage
> your database connections, either use a direct database connections to
> other servers or use a database-specific load balancer/proxy. Gearman

I already explained that. If you keep doing database access directly
from your Web scripts, the scripts end up tying more memory, thus
reducing the ability of the Web server to handle more simultaneous requests.

> was not designed to have transactional context within it. Once a job

That is what my proposal is aimed to improve.

> is started, the client does not have the ability to send any other
> information to the connected worker. While it's not impossible to add,
> it starts to introduce new functionality that goes against some of
> the basic design principals Gearman promotes.

I never suggested that workers should interrupted or get more
information while a job is still being executed.

What I am saying is that after a worker executes a job that is part of
multi-request process, if the client disconnected for some reason, the
job server just needs to send a new message to the worker telling that,
so the worker can do what it needs to do, eventually rolling back an
ongoing database transaction. This is the same with databases.

>> In the front end I just need to keep Web server threads that have very
>> low memory requirements, so I can fit much more Web server requests in
>> one server.
>
> If managing database connections directly in the front-end web
> requests, you should be able to keep memory utilization low with proper
> result set cleanup. The load balancing is a little trickier if you
> are dealing with multiple masters or read slaves, but it can be done.

I suppose you are aware that freeing results sets and whatever memory
scripts use during executing (at least in PHP) does not result in having
the process memory returned to the OS to be made available to other
processes running on the same machine (at least in Unix-like OS).

So, if you have a script that accesses the database, occasionally
retrieves and processes a reasonably large result set, it will allocate
a lot of memory that will not be returned to the OS until that process dies.

It does not matter if the same process subsequently only handles scripts
that need much less memory. The memory remains tied with the process and
your Web server will not be able to handle so many more simultaneous Web
requests.

The processes tend to grow in memory usage. The exceeding memory usage
could have been better used to fit more Web script processes in memory
to handle more simultaneous request.

My idea is to push the database access logic to workers, so Web scripts
will not waste so much memory. I hope that I am being clearer now and it
makes more sense to everybody that raised concerns.

> Like I said above, you may be looking for something else besides
> Gearman, but Gearman could be a great solution for you if you just
> pass the few pieces of request information needed by the worker to
> make a decision.

What I am proposing, I assume it will not affect the way other people
use Gearman. So, I suppose there is nothing for others to be concerned
with the improvements I am suggesting to make.

> In another thread you ask about contributing to Gearman. Check out
> all the various projects on the gearman.org download page. Adding
> some features is easy if it's just client or server behavior. New
> protocol changes are not trivial because to be useful within the
> community it needs to be generally accepted and implemented by a
> handful of project maintainers. Sending out any such proposals to
> the mailing list for community feedback is a good way to start.

Right, that is what I am doing, explaining the proposal so others
provide feedback. Thank you for the guidance.

I still need to study the protocol, in order to determine if it needs to
be extended. Maybe not. I will post more on this once I figure out.

[Manuel Lemos]

Hello Clint,

on 11/29/2009 05:41 AM Clint Byrum said the following:

>> Database access is the most important bottleneck of Web applications.
>> When your Web application grows, it starts requiring more than one
>> server to handle the requests. When that happens I just want to add more
>> backend servers for workers that handle database access.
>>
>
> Your goal is definitely something suited to gearman... limiting or
> multiplying backend processing.
>
> However, what isn't suited to gearman, is lots of time spent
> communicating instead of working.

I am not sure if that is what you intended to say.

If spending a lot of time communicating with Gearman was a bad idea, the
example in the documentation of sending large images for workers to
resize would also be a bad idea.

http://gearman.org/?id=getting_started

I do not see much problem in communication, as long as clients, workers
and job servers are in the same network. There is a little overhead, but
that is the price you have to pay for scalability.

> Basically it sounds like you desire to have only the "model" in the
> worker, and no controller code. However, other than some code separation
> ideal, I don't see a good reason for that. Move both pieces into the
> worker, and you are doing just fine. In the "MVC" world, dos it matter
> if the controller *and* model are *called* in one binary? You can still
> have the model code separate, but called from the "controller" worker
> thread.

Sorry, that is not what I meant. I suppose you jumped to the middle of
the thread and may not have ready what I said before.

I just want to push database access to workers. Database access is often
what takes more time and memory to execute.

Another person claimed that seemed that I was not employing a good MVC
approach. I just said that precisely because I have already the models
and controllers in separate classes, I can easily push the database
access part of the models to the workers.

Whether models and workers are running on the same process or not, is an
ideal that is not really my concern.

>> In the front end I just need to keep Web server threads that have very
>> low memory requirements, so I can fit much more Web server requests in
>> one server.
>>
>
> If you keep only "view" related code in those webserver "threads" then
> one would expect them to use less memory on account of having less code
> and data to work with.

Exactly, but to push the controller logic together with model to the
worker, I also need to send the whole HTTP request data (think of form
data, file uploads, HTTP headers, etc..) to the worker. That way I would
forcing what you object above, which is the communication of much more
data to job servers and works than it is actually necessary.

[Clint Byrum]

On Nov 29, 2009, at 11:43 AM, Manuel Lemos wrote:

Hello Clint,

on 11/29/2009 05:41 AM Clint Byrum said the following:

Database access is the most important bottleneck of Web applications.

When your Web application grows, it starts requiring more than one

server to handle the requests. When that happens I just want to add more

backend servers for workers that handle database access.

Your goal is definitely something suited to gearman... limiting or

multiplying backend processing.

However, what isn't suited to gearman, is lots of time spent

communicating instead of working.

I am not sure if that is what you intended to say.

If spending a lot of time communicating with Gearman was a bad idea, the
example in the documentation of sending large images for workers to
resize would also be a bad idea.

http://gearman.org/?id=getting_started

I do not see much problem in communication, as long as clients, workers
and job servers are in the same network. There is a little overhead, but
that is the price you have to pay for scalability.

That example actually reinforces my point. All of the data needed to complete the resize operation is pushed into gearman in one client request, and the response is done in one request. If you add the ability to communicate directly with the worker, you have to be talking back and forth between client/worker, adding latency (even on a LAN, things take a few ms. It adds up as your app gets bigger).

Another person claimed that seemed that I was not employing a good MVC
approach. I just said that precisely because I have already the models
and controllers in separate classes, I can easily push the database
access part of the models to the workers.

Whether models and workers are running on the same process or not, is an
ideal that is not really my concern.

Basically what we're trying to understand is why you'd want to keep talking between controller and model.. when you can do this all in one worker.

Say you have one bit of code that accepts a user's web request:

$c = new GearmanClient(); $c->addServer('gearman-server');

$response = $c->do('handle_web_request',serialize($_REQUEST));

render_response($response);

handle_web_request can be a gearman worker that just does

function handle_web_request($job)

{

  $con = Model::getConnection();

  $con->begin();

  $result = $con->doSomeInsert();

  $data = file_get_contents($result['url']); // just an example

  $result = $con->doSomeUpdate($data);

  $con->commit();

}

at what point have we saved anything by doing those things separated by gearman? Since the update is dependent on the url fetch, which is dependent on the database, this is not a good use case for async operation, and moving just the database parts in another worker just means wasting time talking to gearmand.

Exactly, but to push the controller logic together with model to the
worker, I also need to send the whole HTTP request data (think of form
data, file uploads, HTTP headers, etc..) to the worker. That way I would
forcing what you object above, which is the communication of much more
data to job servers and works than it is actually necessary.

One other thing that is bothering me about your design is that you can pretty much solve the "occasional heavy PHP request" problem with this (in apache):

MaxRequestsPerChild 10

In this case, each child process just does 10 PHP requests and then dies, prompting apache to fork another one. This would solve your "memory not returning to the OS" problem, as when the process dies, the memory would indeed be returned to the OS, and forking a new process for every 10 (or maybe 100..) PHP requests is really not very painful.

I would hope Apache is better at process management than anything one would write in PHP. So unless your worker code is in something other than PHP ... you're probably better off *not* using gearman for database backend work.

[Eric Day]

Hi Manuel,

On Sun, Nov 29, 2009 at 05:26:15PM -0200, Manuel Lemos wrote:
> > That seems to be what dormando was suggesting, only pass a subset
> > of request information to the worker, only what is required to make
> > any decisions and respond with the result you need. Passing a few
> > hundred bytes of form data to the worker is cheap, and can easily fit
> > in the same TCP packets as the other protocol request information,
> > so no extra overhead there if you are using Gearman.
>
> I am afraid the point you are still missing is that often the Web script
> only knows what information needs to be passed to the worker only after
> it retrieves database records to know the state of the stored data. Only
> after that it can decide what to do next.

Actually, I'm fully aware of what is going on here. :) My point is
that this is not a black box, you have some idea of what is happen,
some context. We're also dealing with a finite amount of data, probably
just a handful of fields from a web form. I'm not suggesting shipping
the entire HTTP request over (although that would probably be fine
too), but just the fields that you know you need. I understand the
desire to keep things really generic, but like many things they all
have trade-offs. I would focus more on reducing RTTs than total data
sent with a single RTT.

> > Yes, most of us quite familiar with basic web scaling topics. :)
> >
> > What dormando was saying though is rather than using Gearman to manage
> > your database connections, either use a direct database connections to
> > other servers or use a database-specific load balancer/proxy. Gearman
>
> I already explained that. If you keep doing database access directly
> from your Web scripts, the scripts end up tying more memory, thus
> reducing the ability of the Web server to handle more simultaneous requests.

The same issue will come up in the workers, and like Apache, you just
need to set a max # of requests per process to ensure the memory
is given back into the kernel. I think Clint also mentined this in
another email in this thread.

> > was not designed to have transactional context within it. Once a job
>
> That is what my proposal is aimed to improve.

Understood, and I think there is some room for improvement in Gearman
for further client->worker communication. I think the main thing
people are questioning is this particular use case, since many of us
have done something similar with Gearman, just without needing the
extra communication.

> > is started, the client does not have the ability to send any other
> > information to the connected worker. While it's not impossible to add,
> > it starts to introduce new functionality that goes against some of
> > the basic design principals Gearman promotes.
>
> I never suggested that workers should interrupted or get more
> information while a job is still being executed.

To accomplish the thing you want to do in the next paragraph, you
need to do just this. :)

> What I am saying is that after a worker executes a job that is part of
> multi-request process, if the client disconnected for some reason, the
> job server just needs to send a new message to the worker telling that,
> so the worker can do what it needs to do, eventually rolling back an
> ongoing database transaction. This is the same with databases.

So, right now the job server does not have any means in the protocol
to notify workers of client activity (ie, the attached client on the
other end is gone). This is because many Gearman use cases expect
clients to be transient. They can attach/detach and have the jobs
coalesced by unique key without the worker even knowing what's going
on. If the functionality you propose were to be added, what happens
if two clients are connected and just one disconnects? Do you only
notify workers when all disconnect? Do you notify when more than one
connect? This also assumes the worker pauses to check for new status
from the job server during execution of the jobs. Unfortunately it's
not just a matter of adding in a new protocol command, it changes
the semantics of the job server in a way that may not be able to be
backwards compatible with how Gearman works today.

> > Like I said above, you may be looking for something else besides
> > Gearman, but Gearman could be a great solution for you if you just
> > pass the few pieces of request information needed by the worker to
> > make a decision.
>
> What I am proposing, I assume it will not affect the way other people
> use Gearman. So, I suppose there is nothing for others to be concerned
> with the improvements I am suggesting to make.

As I mentioned above, I don't think this is the case.

> > In another thread you ask about contributing to Gearman. Check out
> > all the various projects on the gearman.org download page. Adding
> > some features is easy if it's just client or server behavior. New
> > protocol changes are not trivial because to be useful within the
> > community it needs to be generally accepted and implemented by a
> > handful of project maintainers. Sending out any such proposals to
> > the mailing list for community feedback is a good way to start.
>

> I still need to study the protocol, in order to determine if it needs to
> be extended. Maybe not. I will post more on this once I figure out.

The protocol is fairly simple, but there are some behaviors that
come from it that will make it hard to extend Gearman in the way you
want. I could possibly see adding a command that allows a client to
send more data to a connected worker, but the trick here is having
workers receive this data while avoiding blocking conditions. It also
ends up being more data to track and manage in the job server should
the worker die and it needs to be executed again on another worker.

Personally, I would take another stab at making things work with how
Gearman behaves today before trying to extend it.

Best regards,
-Eric

[Manuel Lemos]

Hello Clint,

on 11/30/2009 05:48 AM Clint Byrum said the following:

>>> However, what isn't suited to gearman, is lots of time spent
>>> communicating instead of working.
>>
>> I am not sure if that is what you intended to say.
>>
>> If spending a lot of time communicating with Gearman was a bad idea, the
>> example in the documentation of sending large images for workers to
>> resize would also be a bad idea.
>>
>> http://gearman.org/?id=getting_started
>>
>> I do not see much problem in communication, as long as clients, workers
>> and job servers are in the same network. There is a little overhead, but
>> that is the price you have to pay for scalability.
>>
>
> That example actually reinforces my point. All of the data needed to
> complete the resize operation is pushed into gearman in one client
> request, and the response is done in one request. If you add the ability

Here you seem to be agreeing with me that communication is not the worst
of the problems in terms of overhead. That is particularly true when the
actual job that the worker will do, will take much more time to finish.

> to communicate directly with the worker, you have to be talking back and
> forth between client/worker, adding latency (even on a LAN, things take
> a few ms. It adds up as your app gets bigger).

It seems to me if sending large pictures via local network to a worker
is not a big problem, sending small 2 or 3 requests to the same worker,
instead of just one, shouldn't be the problem either.

Also keep in mind, if it was not clear, if a client is going to send
multiple requests to the same worker, it is intended that it does not
disconnect from the job server to avoid eventual reconnection overhead.

>> Another person claimed that seemed that I was not employing a good MVC
>> approach. I just said that precisely because I have already the models
>> and controllers in separate classes, I can easily push the database
>> access part of the models to the workers.
>>
>> Whether models and workers are running on the same process or not, is an
>> ideal that is not really my concern.
>>
>
> Basically what we're trying to understand is why you'd want to keep
> talking between controller and model.. when you can do this all in one
> worker.

I already explained that several times. The controller does not have the
information that is in the database to decide what to do next, also
depending on what came in the HTTP request from the user browser.

Maybe all the client needs is returned in one request to the worker. It
may need one, two, or more requests depending on the state that it is
processing.

> at what point have we saved anything by doing those things separated by
> gearman? Since the update is dependent on the url fetch, which is
> dependent on the database, this is not a good use case for async
> operation, and moving just the database parts in another worker just
> means wasting time talking to gearmand.

I am not sure what you are asking. But I mainly want to avoid doing
database access on the client because that is often what takes more memory.

>> Exactly, but to push the controller logic together with model to the
>> worker, I also need to send the whole HTTP request data (think of form
>> data, file uploads, HTTP headers, etc..) to the worker. That way I would
>> forcing what you object above, which is the communication of much more
>> data to job servers and works than it is actually necessary.
>>
>
> One other thing that is bothering me about your design is that you can
> pretty much solve the "occasional heavy PHP request" problem with this
> (in apache):
>
> MaxRequestsPerChild 10

I already do this with Apache since a long time, except that I set it to
500 to avoid the overhead of frequent kill and re-forking of processes.

The problem is that, as you said, it just minimizes the problem of
occasional PHP request that uses a lot more memory.

What I am trying to explain is that once you do database requests on
Apache processes, the processes immediately start increasing the memory
usage of PHP scripts.

Simple PHP scripts only tend to use 8 to 10MB of RAM if you avoid
running database queries. Once you start doing regular queries on the
Apache processes, they tend to grow to 18/20MB, thus halving the
capacity of your Web server to deal with simultaneous requests.

You may kill the processes after 500 requests (or less if your prefer),
but the memory will end up used way before the processes are killed.

[Manuel Lemos]

Hello Eric,

on 12/01/2009 05:27 PM Eric Day said the following:

>>> That seems to be what dormando was suggesting, only pass a subset
>>> of request information to the worker, only what is required to make
>>> any decisions and respond with the result you need. Passing a few
>>> hundred bytes of form data to the worker is cheap, and can easily fit
>>> in the same TCP packets as the other protocol request information,
>>> so no extra overhead there if you are using Gearman.
>> I am afraid the point you are still missing is that often the Web script
>> only knows what information needs to be passed to the worker only after
>> it retrieves database records to know the state of the stored data. Only
>> after that it can decide what to do next.
>
> Actually, I'm fully aware of what is going on here. :) My point is
> that this is not a black box, you have some idea of what is happen,
> some context. We're also dealing with a finite amount of data, probably
> just a handful of fields from a web form. I'm not suggesting shipping
> the entire HTTP request over (although that would probably be fine
> too), but just the fields that you know you need. I understand the
> desire to keep things really generic, but like many things they all
> have trade-offs. I would focus more on reducing RTTs than total data
> sent with a single RTT.

Right, my point is that each Web script deals with several different use
cases. It needs to use different sets of information from the Web
request to determine what to do next. So, it is not a trivial job to
guess all the data that needs to be passed without checking what is in
the database.

Furthermore, there is another reason for wanting to just move the
database access layer to the workers. It is that we have already
separated that layer to individual model classes in our current
implementation.

Moving the model data access to the worker, is just a matter of
replacing in the client the current model classes by proxy classes that
provide the same API but make calls to workers. This is the easiest path
when upgrading from a single server approach (not based in Gearman) to a
distributed approach based in Gearman.

Meaning, it will dramatically cut the development cost of implementing
Gearman based distribution on existing code.

>>> Yes, most of us quite familiar with basic web scaling topics. :)
>>>
>>> What dormando was saying though is rather than using Gearman to manage
>>> your database connections, either use a direct database connections to
>>> other servers or use a database-specific load balancer/proxy. Gearman
>> I already explained that. If you keep doing database access directly
>> from your Web scripts, the scripts end up tying more memory, thus
>> reducing the ability of the Web server to handle more simultaneous requests.
>
> The same issue will come up in the workers, and like Apache, you just
> need to set a max # of requests per process to ensure the memory
> is given back into the kernel. I think Clint also mentined this in
> another email in this thread.

Right, but since workers will be distributed, if I need to handle more
requests, I just need to add more servers for the workers.

What happens is that if your sites are mostly for content serving, and
most of us develop such sites, 90% of the requests serve cached data. So
no worker access is needed. Single cache access, eventually to
memcached, will do.

However, if I keep doing database access on the Apache processes to
handle those 10% or less of the requests that needed, they will
increment significantly the usage of RAM on the Web server machines.
Thus reducing the ability to handle more Web server requests per server.

>>> was not designed to have transactional context within it. Once a job
>> That is what my proposal is aimed to improve.
>
> Understood, and I think there is some room for improvement in Gearman
> for further client->worker communication. I think the main thing
> people are questioning is this particular use case, since many of us
> have done something similar with Gearman, just without needing the
> extra communication.

Well I do not thing is such a particular case. I am sure a lot of us
have done plenty of content sites, on which most data comes from a
database and is cached somewhere to avoid further database access. Other
than that, if you need to change content, you need to do it
consistently, usually encapsulating it in a database transaction.

All those types of requests that require database access, even though
they represent 10% or less of the Web requests, those are the ones that
make the RAM usage of Web server processes, eventually double, depending
on what you are doing.

>> What I am saying is that after a worker executes a job that is part of
>> multi-request process, if the client disconnected for some reason, the
>> job server just needs to send a new message to the worker telling that,
>> so the worker can do what it needs to do, eventually rolling back an
>> ongoing database transaction. This is the same with databases.
>
> So, right now the job server does not have any means in the protocol
> to notify workers of client activity (ie, the attached client on the
> other end is gone). This is because many Gearman use cases expect
> clients to be transient. They can attach/detach and have the jobs
> coalesced by unique key without the worker even knowing what's going
> on. If the functionality you propose were to be added, what happens
> if two clients are connected and just one disconnects? Do you only
> notify workers when all disconnect? Do you notify when more than one
> connect? This also assumes the worker pauses to check for new status
> from the job server during execution of the jobs. Unfortunately it's
> not just a matter of adding in a new protocol command, it changes
> the semantics of the job server in a way that may not be able to be
> backwards compatible with how Gearman works today.

No, that is not my proposal. Here is a draft of some pseudo-code:

1. Client connects and sends the first job request of a multi-request
access. It also tells what (rollback) job the worker should execute, in
case the transaction fails because the client disconnected before the
final request.

2. Job server assigns an ID to that multi-request access, assigns and
reserves a worker to that multi-request access, send the first job
request to the worker.

3. Worker gets the request, executes the job, sends response to the job
server, waits for the next request.

4. job server gets response from worker passing it back to the client
along with the multi-request ID assigned in step 2.

5. client gets the multi-request id and the first job response, does
whatever he wants.

6. client sends a new multi-request job request along with the ID
returned by the server, and some information telling the server whether
it is the final request.

7. server gets the client request, looks up for the worker that was
reserved for the the given multi-request ID, forwards the request to
that worker.

8. Repeat from 3 if it was not a final request. Otherwise, after getting
the response from the work and passing it to the client, just free the
ID assigned for this multi-request access.


Exception: if after getting the response from the worker, the client
disconnected, just send a new job to the worker defined in step one
precisely for the case when the client disconnects.


I am not sure if I covered all the cases. Anyway, the protocol to
communicate with the worker does not change because it is never aware
that is a part of multi-request access.

As for the protocol to communicate with the client, it seems to need to
support new types of message to accomodate the additional information
related with multi-request accesses (ID, and additional rollback job
defined in step 1);

So, to the question whether the protocol needs to be changed, the answer
seems to be yes, but I do not think it will change the existing types of
messages exchanged, only new ones to add.

>>> In another thread you ask about contributing to Gearman. Check out
>>> all the various projects on the gearman.org download page. Adding
>>> some features is easy if it's just client or server behavior. New
>>> protocol changes are not trivial because to be useful within the
>>> community it needs to be generally accepted and implemented by a
>>> handful of project maintainers. Sending out any such proposals to
>>> the mailing list for community feedback is a good way to start.
>> I still need to study the protocol, in order to determine if it needs to
>> be extended. Maybe not. I will post more on this once I figure out.
>
> The protocol is fairly simple, but there are some behaviors that
> come from it that will make it hard to extend Gearman in the way you
> want. I could possibly see adding a command that allows a client to
> send more data to a connected worker, but the trick here is having
> workers receive this data while avoiding blocking conditions. It also
> ends up being more data to track and manage in the job server should
> the worker die and it needs to be executed again on another worker.

I don't know if the pseudo-code above was clear enough, but when the
client disconnects, the server does not need to communicate with the
worker until the worker ends what it is doing. Only when the worker is
done, the server sends the (rollback) job request defined in step 1.

As for when the worker dies, you just need to return to the client
whatever error condition that is returned for simple request accesses.

[Kiall Mac Innes]

Now - I've been partially following this thread.. and have a few comments that cover a small part of your proposal...

On Fri, Dec 4, 2009 at 4:46 AM, Manuel Lemos <mle...@acm.org> wrote:

However, if I keep doing database access on the Apache processes to
handle those 10% or less of the requests that needed, they will
increment significantly the usage of RAM on the Web server machines.
Thus reducing the ability to handle more Web server requests per server.

 

...

All those types of requests that require database access, even though
they represent 10% or less of the Web requests, those are the ones that
make the RAM usage of Web server processes, eventually double, depending
on what you are doing.

I really don't understand your reasoning around memory usage. It sounds to me like your worried about the amount of RAM available to web servers, but are not worried about RAM on workers.

From the sounds of it - All you are doing is shifting memory usage from web servers, to workers. How does this provide any benefit?

Why not allocate more memory to the web servers, and less memory to the workers? That way, you avoid the gearman request for the 10% of requests requiring DB access and can handle the same number of rps as with gearman doing the DB access!

At the end of the day - in regards to memory - you are not reducing the overall amount of RAM required to run the site in any way and you are actually adding unnecessary gearman requests resulting in .... a less scalable application!

Feel free to correct me, or back me up on this anyone!

[Manuel Lemos]

Hello,

on 12/04/2009 03:09 AM Kiall Mac Innes said the following:

> I really don't understand your reasoning around memory usage. It sounds
> to me like your worried about the amount of RAM available to web
> servers, but are not worried about RAM on workers.
>
> From the sounds of it - All you are doing is shifting memory usage from
> web servers, to workers. How does this provide any benefit?

The point you seem to be missing is that once a Web server process that
performs database access doubles the allocated RAM, from then on all
requests sent to the same process will keep that memory allocated, even
if they do not need it, because memory is only freed when the process dies.

If I move all database accesses to separate machines that only run
workers, the processes may still double in memory usage, but since only
10% or less of the requests will result in calls to the work, I obtain 5
times more efficiency in RAM usage, thus fiting more work is less servers.

> Why not allocate more memory to the web servers, and less memory to the

Even if that solved the problem, you cannot put infinite memory in each
server. No matter how much RAM per server you have, if your processes
take half of the memory, you can fit more simultaneous request per Web
server, thus reducing in half the need to add more Web servers.

Ideally PHP would be able to run with 100% stability in multithreaded
Web servers natively (without FastCGI), so different threads would share
and reuse the same memory pool.

However, PHP core developers avoided trying to make PHP 100%
multi-thread safe, because it is complicated, due to the nature of the
libraries PHP need to run.

So, we need to deal with the consequences of a more robust model which
is the multi-process system used in most PHP installations.

[Clint Byrum]

On Dec 3, 2009, at 8:46 PM, Manuel Lemos wrote:

Well I do not thing is such a particular case. I am sure a lot of us
have done plenty of content sites, on which most data comes from a
database and is cached somewhere to avoid further database access. Other
than that, if you need to change content, you need to do it
consistently, usually encapsulating it in a database transaction.

This is something I'm not so sure is true.

Transactions are nice, but in reality relying on them will hamper scalability quite a bit.

Systems that can deal with "eventual consistency" are going to be far cheaper and simpler to scale than systems requiring transactional accuracy 100% of the time.

BTW you can totally do what you're talking about using background jobs for two way communication.

frontend:

function handle_response($job)

{

  $response_data = $job->geWorkload();

  // do stuff with it

  $this->part2Handle = $this->client->doBackground('trans_part2');

}

$me = uniqid();

$worker->addFunction('handle_response_'.$me);

$handle = $client->doBackground('start_transaction', serialize(array('respond_to'=>$me, 'body'=>$body)));

$worker->work();

backend:

class my_worker {

  __construct() {

    $this->worker = new GearmanWorker();

    $this->worker->addFunction('start_transaction', array($this,'start_transaction'));

    $this->worker->addFunction('trans_part2', array($this, 'trans_part2'));

    $this->worker->work();

  }

  function start_transaction ($job) {

    $wl = $job->getWorkload();

    $this->respond_to = $wl['respond_to'];

    $body = $wl['body'];

   $this->db->begin();

   $response = $db->query();

   $this->client->doBackground('respond_to_'.$this->respond_to, $response);

  }

  function trans_part2 ($job) {

     //...

  }

}

Get it?

[Eric Day]

On Fri, Dec 04, 2009 at 02:07:32AM -0200, Manuel Lemos wrote:
> > One other thing that is bothering me about your design is that you can
> > pretty much solve the "occasional heavy PHP request" problem with this
> > (in apache):
> >
> > MaxRequestsPerChild 10
>
> I already do this with Apache since a long time, except that I set it to
> 500 to avoid the overhead of frequent kill and re-forking of processes.
>
> The problem is that, as you said, it just minimizes the problem of
> occasional PHP request that uses a lot more memory.

But if you can minize the problem to the point where it is
insignificant, that might be good enough. Sure, it's not ideal,
but again it's a trade. Using Gearman for all DB access is not free,
especially if you want to add multiple RTTs on your network.

Forking is relatively cheap, and if that's not your bottleneck
and memory is, set the max requests low (to like 10 as Client
suggests). I'm not going to work out the exact probabilities (since the
source numbers are fuzzy to begin with), but on average you should only
have half your apache processses eating wasted memory from DB requests,
as opposed to nearly all of them with a high max requests setting.

I'm not saying this change will remove the need to use Gearman for
wrapping DB requests, but it will at least buy you some more capacity.
I agree it's not ideal, but it's better. It's all about tradeoffs. :)

-Eric

[Eric Day]

Hi Manuel,

On Fri, Dec 04, 2009 at 02:46:59AM -0200, Manuel Lemos wrote:
> > So, right now the job server does not have any means in the protocol
> > to notify workers of client activity (ie, the attached client on the
> > other end is gone). This is because many Gearman use cases expect
> > clients to be transient. They can attach/detach and have the jobs
> > coalesced by unique key without the worker even knowing what's going
> > on. If the functionality you propose were to be added, what happens
> > if two clients are connected and just one disconnects? Do you only
> > notify workers when all disconnect? Do you notify when more than one
> > connect? This also assumes the worker pauses to check for new status
> > from the job server during execution of the jobs. Unfortunately it's
> > not just a matter of adding in a new protocol command, it changes
> > the semantics of the job server in a way that may not be able to be
> > backwards compatible with how Gearman works today.
>
> No, that is not my proposal. Here is a draft of some pseudo-code:

That was not intended to state your proposal, but rather explaining
some of the behaviors of the job server today and how you would
consider handling those cases.

> 1. Client connects and sends the first job request of a multi-request
> access. It also tells what (rollback) job the worker should execute, in
> case the transaction fails because the client disconnected before the
> final request.

If you want to go down this path, I would avoid storing a "rollback
job" and instead just let the worker figure out what to do with a
generic 'client disconnected' notification. Any specific instructions
on what to do on client disconnect could be sent in the initial job
to the worker.

> 2. Job server assigns an ID to that multi-request access, assigns and
> reserves a worker to that multi-request access, send the first job
> request to the worker.

It may be easier to do multi-packet jobs (ie, client->worker data
packets) rather than multi-job. I've considered the former before
and it would not be too hard to do. Like the multi-job idea, you
just need to handle cases with multiple connected clients when one
or all disconnect.

> As for the protocol to communicate with the client, it seems to need to
> support new types of message to accomodate the additional information
> related with multi-request accesses (ID, and additional rollback job
> defined in step 1);
>
> So, to the question whether the protocol needs to be changed, the answer
> seems to be yes, but I do not think it will change the existing types of
> messages exchanged, only new ones to add.

I wasn't suggesting it would change existing messages, but rather
it would change the way workers are handled in the job server. It
essentially creates a different worker manager to handle the new
types of jobs you propose. Would you expect workers to be available
for other jobs between multi-request jobs? How will you handle client
coalescing to the same job (by the unique ID)?

If you are thinking about implementing this in the C server, please
keep us updated. We'll want to review any protocol additions you
propose and I can help give feedback on the server internals.

-Eric

[Yehuda Tuvia]

Hey,

Do you submit gearman scatter-gather tasks from a UI thread?