Python 2.7 multithreaded: ereporter & mapreduce compatibility

262 views
Skip to first unread message

Pol

unread,
Nov 10, 2011, 10:44:17 AM11/10/11
to Google App Engine
Hi,

What are the plans to get these 2 extensions compatible with Python
2.7 multithreaded?

$PYTHON_LIB/google/appengine/ext/mapreduce/main.py
$PYTHON_LIB/google/appengine/ext/ereporter/report_generator.py

Thanks,

- Pol

Brian Quinlan

unread,
Nov 10, 2011, 9:10:02 PM11/10/11
to google-a...@googlegroups.com

I assume that you are seeing an error during the parsing of your appcfg.py file?

The error message should say that you can't use a CGI with threadsafe.
The solution is to use WSGI in your handlers instead i.e.

google.appengine.ext.mapreduce.application
google.appengine.ext.ereporter/report_generator.application

Cheers,
Brian


> Thanks,
>
> - Pol
>
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

Pol

unread,
Nov 11, 2011, 3:46:46 PM11/11/11
to Google App Engine
Indeed the problem is the parsing of app.yaml:
- url: /mapreduce(/.*)?  script: $PYTHON_LIB/google/appengine/ext/
mapreduce/main.py  login: admin- url: /ereporter.*  script:
$PYTHON_LIB/google/appengine/ext/ereporter/report_generator.py  login:
admin
So what do I use instead for script paths? Looking into the SDK
source, these extensions are not 2.7 multithreaded ready.

Brian Quinlan

unread,
Nov 11, 2011, 4:26:09 PM11/11/11
to google-a...@googlegroups.com
Hi Pol,

On Sat, Nov 12, 2011 at 7:46 AM, Pol <p...@everpix.net> wrote:
> Indeed the problem is the parsing of app.yaml:
> - url: /mapreduce(/.*)?  script: $PYTHON_LIB/google/appengine/ext/
> mapreduce/main.py  login: admin- url: /ereporter.*  script:
> $PYTHON_LIB/google/appengine/ext/ereporter/report_generator.py  login:
> admin
> So what do I use instead for script paths?

script: google.appengine.ext.mapreduce.application
script: google.appengine.ext.ereporter.report_generator.main.APP

> Looking into the SDK
> source, these extensions are not 2.7 multithreaded ready.

How so?

Cheers,
Brian

Pol

unread,
Nov 11, 2011, 9:09:30 PM11/11/11
to Google App Engine
Ah OK it works, thanks. Didn't know had to replace path entirely...
Was wondering about compatibility since there were still some CGI
leftovers in these files.

working

unread,
Nov 12, 2011, 4:05:52 PM11/12/11
to Google App Engine
Hi Brian,

As for
google.appengine.ext.mapreduce.application

Does it mean the SDK/runtime already has its mapreduce? Right now, I
still supply an additional copy in my app folder. If SDK/runtime
already has it, I will delete my own copy.

How about the pipeline?

Thanks,
coronin


On Nov 10, 6:10 pm, Brian Quinlan <bquin...@google.com> wrote:

Brian Quinlan

unread,
Nov 12, 2011, 4:10:12 PM11/12/11
to google-a...@googlegroups.com
Hi coronin,

On Sun, Nov 13, 2011 at 8:05 AM, working <cor...@gmail.com> wrote:
> Hi Brian,
>
> As for
> google.appengine.ext.mapreduce.application
>
> Does it mean the SDK/runtime already has its mapreduce? Right now, I
> still supply an additional copy in my app folder. If SDK/runtime
> already has it, I will delete my own copy.
>
> How about the pipeline?

The SDK and runtime does include a copy of mapreduce but it is
recommended that you include your own copy because mapreduce is
experimental and incompatible changes may still be made. Including
your own copy will protect your from this.

Cheers,
Brian

Pol

unread,
Nov 13, 2011, 1:10:12 AM11/13/11
to Google App Engine
Hi Brian,

Although we can deploy to production using 2.7, accessing /mapreduce
fails:

Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/runtime/wsgi.py", line 168, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/runtime/wsgi.py", line 217, in _LoadHandler
__import__(cumulative_path)
ImportError: No module named application

The app.yaml is set like this:

- url: /mapreduce(/.*)?
script: google.appengine.ext.mapreduce.application
login: admin

Thanks for your help!

On Nov 11, 1:26 pm, Brian Quinlan <bquin...@google.com> wrote:
> Hi Pol,
>
> On Sat, Nov 12, 2011 at 7:46 AM, Pol <p...@everpix.net> wrote:
> > Indeed the problem is the parsing of app.yaml:
> > - url: /mapreduce(/.*)?  script: $PYTHON_LIB/google/appengine/ext/
> >mapreduce/main.py  login: admin- url: /ereporter.*  script:
> > $PYTHON_LIB/google/appengine/ext/ereporter/report_generator.py  login:
> > admin
> > So what do I use instead for script paths?
>
> script: google.appengine.ext.mapreduce.application
> script: google.appengine.ext.ereporter.report_generator.main.APP
>
> > Looking into the SDK
> > source, these extensions are not2.7multithreaded ready.

Brian Quinlan

unread,
Nov 13, 2011, 1:20:10 AM11/13/11
to google-a...@googlegroups.com
On Sat, Nov 12, 2011 at 8:26 AM, Brian Quinlan <bqui...@google.com> wrote:
> Hi Pol,
>
> On Sat, Nov 12, 2011 at 7:46 AM, Pol <p...@everpix.net> wrote:
>> Indeed the problem is the parsing of app.yaml:
>> - url: /mapreduce(/.*)?  script: $PYTHON_LIB/google/appengine/ext/
>> mapreduce/main.py  login: admin- url: /ereporter.*  script:
>> $PYTHON_LIB/google/appengine/ext/ereporter/report_generator.py  login:
>> admin
>> So what do I use instead for script paths?
>
> script: google.appengine.ext.mapreduce.application
> script: google.appengine.ext.ereporter.report_generator

Oops, I meant:

script: google.appengine.ext.mapreduce.main.APP
script: google.appengine.ext.ereporter.report_generator.application

Cheers,
Brian

Reply all
Reply to author
Forward
0 new messages