Task queue only does POST?

[hawkett]

Hi,

I've just about managed to get my application running on TyphoonAE
OS X, after a few false starts, but have finally run into a problem
I'm not sure how to fix. My app makes extensive use of the task queue
for POST, PUT and DELETE operations. The task queue seems to be
working fine, except that it is not honouring the HTTP method set when
raising the task, and always executing POST operations. For example a
task created with

task = taskqueue.Task(url=url, method="PUT", payload=...)

will actually arrive at my app as a POST. Just wondering if I have
missed something, or if there is something I can configure to fix
this? Since my app operates on REST principles, it is a core
requirement to handle and use the basic HTTP methods.

Out of interest, the other issues I had were that using mongodb I was
getting false errors about transactionally updating entities in
different entity groups, and bdb was throwing
'java.util.ConcurrentModificationException' almost immediately. MySQL
seemed to do the trick though.

Also, the default build expects mysql to be installed (crashing if it
can't find mysql_config), which at the time was a hassle, but turned
out to be necessary :)

Anyway - feels so close to getting everything working - it's a great
project - hope there's an easy fix for the HTTP method thing. Cheers,


Colin

[Joaquin Cuenca Abela]

(I first replied to Colin in private, but my answer was slightly
wrong. I've corrected a couple of things.)

In /src/typhoonae/taskqueue/worker.py, near line 90 change:

req = urllib2.Request(
url='http://%(host)s:%(port)s%(url)s' % task,
data=base64.b64decode(task['payload']),
headers=headers
)

by:

req = RequestWithMethod(
method=task['method'],
url='http://%(host)s:%(port)s%(url)s' % task,
data=base64.b64decode(task['payload']),
headers=headers
)

and at line 77 add:

class RequestWithMethod(urllib2.Request):
def __init__(self, method, *args, **kwargs):
self._method = method
urllib2.Request.__init__(self, *args, **kwargs)

def get_method(self):
return self._method

And change line 157 from:

method=add_request.method()

to:

method=add_request.RequestMethod_Name(add_request.method())

Cheers,

> --
> You received this message because you are subscribed to the Google
> Groups "typhoonae" group.
> To post to this group, send email to typh...@googlegroups.com
> To unsubscribe from this group, send email to
> typhoonae+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/typhoonae?hl=en

--
Joaquin Cuenca Abela

[Tobias]

Great, Joaquin!

That seems to do the trick. I'm going to commit your changes later
today, since I'm at the Mongo Berlin conference right now.

Thanks!

- Tobias

On Oct 4, 11:53 am, Joaquin Cuenca Abela <joaq...@cuencaabela.com>
wrote:

[Colin H]

Thanks Joaquin - can also confirm those changes work for me - and my app is running locally on OS X 10.6 using mySQL! Cool. Thanks for the speedy turn around - Colin

[Tobias]

Just added the changes (slightly modified) to the trunk.

http://code.google.com/p/typhoonae/source/detail?r=0659d944bf4d388660ff6ff792dfc68a5d797b28

Thanks again!

- Tobias

> > > typhoonae+...@googlegroups.com<typhoonae%2Bunsubscribe@googlegroups .com>

[Joaquin Cuenca Abela]

Hi Tobias,

I have merged the support of celery tasks here:

https://code.google.com/r/e98cuenc-typhoonae/source/detail?r=30727e09a1bad5596eeeb434f1e77365f986505c

If you like the code please merge it in your tree. I based these
changes on typhoonae before the change you just made, so maybe it will
be easier to rollback the last change you made and pull my changes.

What this patch adds is:

- workers managed by celery. So you can have several workers on
several servers (ie you can scale the workers).
- deferred tasks managed by the same workers.
- support for rate_limit.

Note that bucket_size is still not supported.

it's the first time I work with buildout, so probably I made something
stupid. Please Tobias let me know if I screwed it up.

Cheers,

[Tobias]

Hi Joaquin,

Awesome! I think you did a great job and these are real improvements.
So, integrating your changes is definitely next on my list. I have
minor objections, though. Celery support might better be optional,
because it seems to pull in a number of dependencies like sqlalchemy.

I'd like to get back to you on this with further questions and a
clearer opinion after having had a deeper look.

Thanks!

- Tobias

On Oct 4, 4:43 pm, Joaquin Cuenca Abela <joaq...@cuencaabela.com>
wrote:

> Hi Tobias,
>
> I have merged the support of celery tasks here:
>

> https://code.google.com/r/e98cuenc-typhoonae/source/detail?r=30727e09...

>
> If you like the code please merge it in your tree. I based these
> changes on typhoonae before the change you just made, so maybe it will
> be easier to rollback the last change you made and pull my changes.
>
> What this patch adds is:
>
> - workers managed by celery. So you can have several workers on
> several servers (ie you can scale the workers).
> - deferred tasks managed by the same workers.
> - support for rate_limit.
>
> Note that bucket_size is still not supported.
>
> it's the first time I work with buildout, so probably I made something
> stupid. Please Tobias let me know if I screwed it up.
>
> Cheers,
>
>
>
>
>
> On Mon, Oct 4, 2010 at 3:10 PM, Tobias <tobias.rodae...@googlemail.com> wrote:
> > Just added the changes (slightly modified) to the trunk.
>

> >http://code.google.com/p/typhoonae/source/detail?r=0659d944bf4d388660...