Python27 and threadsafe = true

[Sylvain]

Hi,

I've tried ProtoRCP (GAE 1.5.5 version) with python27 and threadsafe=true but it doesn't work because ProtoRPC uses CGI (and not WSGI).

Is there a solution to make it works ?

Thank you.

Sylvain

[Kyle Finley]

Sylvain,

I haven't tried this myself, but protorpc.experimental seems to support WSGI.

http://code.google.com/p/google-protorpc/source/browse/demos/experimental/wsgi/services.py

Here's the instruction for including your own version of protorpc:

https://groups.google.com/forum/#!topic/google-protorpc-discuss/1-N0lCAWils

- Kyle

[Rafe Kaplan]

The original service handlers were written using webapp which is
WSGI compatible. However, I imagine that webapp would have some
threading issues.

Could you provide more details about what didn't work for you?

[Rafe Kaplan]

Kyle,

Mind you, if you are using protorpc from the source code repository,
there is now a wsgi module. It contains the proper implementation of
a single URL WSGI handler (it cannot yet handle multiple mappings).

On Thu, Oct 13, 2011 at 7:16 PM, Kyle Finley <kylef...@gmail.com> wrote:

[Sylvain]

When I start the dev_server, I've a CGI error :  "threadsafe cannot be enabled with CGI handler: xxxxxx.py"

I've read the GAE forum and it seems that the dev_server is not python27 compatible.
It must be the issue.

Did you try to run a the Proto Example with GAE 1.5.5 on the dev_server ?

Sylvain

[Rafe Kaplan]

I'm not sure how much luck you will be able to have running python
2.7 using the dev_appserver.

Based on what I am seeing, there might be another issue. When you
enable 2.7 and want to use it in threadsafe mode, you will need to use
"wsgi" handlers vs. "cgi" handlers. This is not a protorpc issue,
because protorpc provides wsgi implementations, including webapp.

The difference between "wsgi" and "cgi" in the app engine sense has
to do with how you define handlers. Handlers that end in ".py" are
cgi handlers, meaning the script is executed as a cgi call and your
script can handle it however it wants. One way to handle cgi requests
is using wsgi, which webapp does.

"wsgi" handlers in the app engine sense require you to define a wsgi
application. This wsgi application must then be placed in a global
variable of a module and is referred to directly in the handler. For
example:

handlers:
- url: /my_service.*
script: my_service.app

In this case, my_service refers to a module (a .py file) that
contains a cgi application object 'app'. A wsgi application can be
defined as simply as this:

my_service.py:
===========

def app(environ. start_response):
start_response('200', [('content-type', 'application/json')])
return ['{"message": "hello"}']

If you used webapp, you would call webapp.WSGIApplication to create
the application with the protorpc mappings. I would also consider
using webapp2 instead of webapp, although it has not been tested with
protorpc.

Make sense?

[Ubaldo Huerta]

I wonder the status of ProtoRPC and webapp2. The webapp2 source has an extra that supports protorpc. See unittest http://code.google.com/p/webapp-improved/source/browse/tests/extras_protorpc_test.py

and documentation http://webapp-improved.appspot.com/api/webapp2_extras/protorpc.html

I haven't dare to use protorpc yet because with webapp2 because of the webapp2's claim that it's compatible with protorpc source shipped in 1.5.1 SDK. I wonder if if anyone is making sure that current or future versions of protorpc are compatible with webapp2, which has superseded webapp and it's even required for the upcoming python27 runtime.

In other words, is anyone in charge of keeping protorpc and webapp2 working together? 

Thanks

PS: I'm, by the way, currently using webapp2 (version 2.3, including its source code with my app's code) for regular handlers. 

[Rafe Kaplan]

Using ProtoRPC ServiceHandler with webapp2 will have some problems,
particularly if you use the service_mapping functions. The reason is
that webapp and webapp2 consider functions differently. Webapp2
treats functions as if they are themselves handlers. webapp (original
version) treats functions as if they are factories for service handler
objects. It's possible to write your own service handler mappings or
write a wrapper, but it's largely inconvenient.

ProtoRPC is moving to a raw WSGI handler mode of configuration so in
the long run this incompatibility will not be an issue. For now, it's
problematic. Either use webapp 1 for ProtoRPC main scripts or use the
new wsgi module (which is not in the SDK yet, but will be soon).

[Sylvain]

Yes,

So, currently, it's not possible to make it works.

I've just tried the Echo services example and I've got this error :

Fatal error when loading application configuration:
Invalid object:
threadsafe cannot be enabled with CGI handler: main.py
  in "C:\Outils\Eclipse\workspace\ProtoRPCEchoDEMO\src\app.yaml", line 32, column 1

I've tried to follow Nick's blog : http://blog.notdot.net/2011/10/Migrating-to-Python-2-7-part-1-Threadsafe
But I didn't success. Too much changes to make it work.

For the next GAE release (1.6 ?) will you plan to make ProtoRPC thresafe=true compatible ?

Regards

Sylvain

[Rafe Kaplan]

main.py will not work in threadsafe mode because it does not refer
to a CGI handler, just a python script. The main.py file would have
to be written to expose a WSGI application and then the app.yaml
rewritten to refer to the handler itself. Even then it would not
likely work because by default python 2.7 app engine uses webapp2.
You would need to configure Python 2.7 to use webapp, and I don't feel
like webapp is threadsafe.

I am planning to get better support as soon as I can. Mainly, I've
been writing WSGI based ProtoRPC handlers that are not based on webapp
at all. Those will eventually be the officially supported way to
write ProtoRPC servers. You can use your own local version from the
source repository using package protorpc.wsgi.service. Unfortunately,
it does not have the ability to automatically set up a registry yet,
nor does it handle multiple urls. You would need to write your own
multi url handler or use a different wsgi handler for each service.
Should be easy to do under python 2.7 app engine.

[Uri]

Hi

Any updates about this issue?

I'm having trouble defining the handler:

My code:

# Register mapping with application.

application = webapp.service_handlers.service_mapping(['/guestRPC'], GuestService)

YAML file:

- url: /guestRPC.*

  script: guestRPC.application

Getting the following error:

AttributeError: 'module' object has no attribute 'service_handlers'

[Uri]

changed the import into

from protorpc.webapp import service_handlers

now getting a new error;

AttributeError: 'str' object has no attribute 'definition_name'

[Uri]

ok now I changed the import again to the wsgi

from protorpc.wsgi import service

now it seems to work ok

I think the documentation here is a bit lacking...

[Chris Kuzak]

I was able to use protorpc wsgi interface in python27 runtime / webapp2 by adding some custom functions in place of the default webapp2 router dispatcher and handler adapter. I'm not sure yet if its fully correct functionality (especially in error conditions)... here's essentially the strawman

send webapp2 wsgi app constructer the list of rpc services all pointing to the same webapp2 handler. This handler should just route the the call to protorpc like "return self.request.get_response(protorpc_wsgi_app_returned_by_service_mappings())". By doing this, you are essentially using the webob.Request class to call the wsgi application generated by protorpc's service_mappings. I also created my own dispatcher for webapp2 to pass down the protorpc_wsgi_app into the webapp2 handler. 

Basically this lets you use protorpc wsgi module as-is + webapp2 to process protorpc endpoints. Happy to provide all code if anyone wants it.

[Ubaldo Huerta]

Yes, that would be great. I could run my tests, report de results, etc. Perhaps the author of proto-rpc could take a look as well. 

[Chris Kuzak]

Attached an example module to demonstrate launching an app that supports protorpc. Its possible lower level wsgi app stuff is being called twice (like start_response) but I've stuck this example code into an existing app that was on webapp/protorpc and I so far so good. 

[Rafe Kaplan]

Sorry for the long delay. There is now a module
protorpc.wsgi.service.* which has simpler WSGI handlers that you can
use outside of webapp. This is better than trying to integrate with
webapp2 and support for the webapp version will be deprecated and if
possible removed.

Working to get the documentation updated for this.

--
- Rafe Kaplan

[Ubaldo Huerta]

Hi, can you provide a "hello world" type of example? I'll appreciate it

[Rafe Kaplan]

The example here is a bit more up to date:

http://code.google.com/p/google-protorpc/

--
- Rafe Kaplan

[Ubaldo Huerta]

Ok, good, it does work well in python 2.5 runtime, it's all clear.

Now, has it been tested in python 2.7 (with concurrency)?

I imagine that the example simply needs to be modified according to the migration guide https://developers.google.com/appengine/docs/python/python27/migrate27

That is, this last block 

# Map the RPC service and path (/hello)
hello_service = service.service_mapping(HelloService, '/hello.*')

def main():
  util.run_wsgi_app(hello_service)

if __name__ == '__main__':
  main()

Should be changed for

# Map the RPC service and path (/hello)
app = service.service_mapping(HelloService, '/hello.*')

[Rafe Kaplan]

It was written with the understanding that servers could run on
non-App Engine servers that may well be highly concurrent, even before
we launched 2.7. I use it internally for mission critical projects
running on Python 2.7 concurrently. Any issues arising from
concurrency will become a high priority fix.

On Thu, Jul 12, 2012 at 3:43 AM, Ubaldo Huerta <uba...@gmail.com> wrote:
> Ok, good, it does work well in python 2.5 runtime, it's all clear.
>
> Now, has it been tested in python 2.7 (with concurrency)?
>
> I imagine that the example simply needs to be modified according to the
> migration guide
> https://developers.google.com/appengine/docs/python/python27/migrate27
>
> That is, this last block
>
> # Map the RPC service and path (/hello)
> hello_service = service.service_mapping(HelloService, '/hello.*')
>
> def main():
> util.run_wsgi_app(hello_service)
>
> if __name__ == '__main__':
> main()
>
>
> Should be changed for
>
> # Map the RPC service and path (/hello)
> app = service.service_mapping(HelloService, '/hello.*')
>

Yes, that code is there just so that there is one example for 2.5
developers. There is no reason why a Python 2.7 user should not route
directly to hello_service via app.yaml.

Also, while it has not been documented yet, there is also a
service_mappings function which helps if mapping multiple services and
configures the registry service, if desired.

# Automatically defines /protorpc which implements RegistryService.
services = service.service_mappings([('/hello', HelloService)])

--
- Rafe Kaplan