how to use protobuf with Google App Engine?

[Dan]

Hello-
I'm working with GAE and ran into this error when trying to use
protocol buffers:

from google.protobuf import descriptor
ImportError: No module named protobuf

I saw some mention of people using hacks to do this in the past, but
also saw that the release notes for protocol buffers v 2.10 said this:
* Changes to input_stream.py should make protobuf compatible with
appengine.

I'm not sure what is meant by "compatible" in that statement, but I'd
like to know if GAE still requires the hackery described in these
threads referenced below. Is there an up-to-date howto or something
like that somewhere that I missed?

http://groups.google.com/group/protobuf/browse_thread/thread/6430dc5d7ceb145a/b8634dddbe4425f3?lnk=gst&q=gae#b8634dddbe4425f3

http://groups.google.com/group/google-appengine/browse_thread/thread/db910ed40ae495fc/f41873a476db744e

Dan

[Kenton Varda]

You'd have to ask the AppEngine people.  Someone from that team submitted the input_stream.py change.

The problem is that both protocol buffers and AppEngine place themselves in the "google" package.  Protocol Buffers has, from day one, declared the package as a "namespace package" using setuptools, which theoretically allows sharing of the package.  For things to work, though, AppEngine has to cooperate by doing the same thing in their own code.

[Dan]

OK, I'll ask over there. thanks

On Jul 2, 6:55 pm, Kenton Varda <ken...@google.com> wrote:
> You'd have to ask the AppEngine people.  Someone from that team submitted
> the input_stream.py change.
> The problem is that both protocol buffers and AppEngine place themselves in
> the "google" package.  Protocol Buffers has, from day one, declared the
> package as a "namespace package" using setuptools, which theoretically
> allows sharing of the package.  For things to work, though, AppEngine has to
> cooperate by doing the same thing in their own code.
>

> On Thu, Jul 2, 2009 at 6:46 PM, Dan <danbr...@gmail.com> wrote:
>
> > Hello-
> > I'm working with GAE and ran into this error when trying to use
> > protocol buffers:
>
> > from google.protobuf import descriptor
> > ImportError: No module named protobuf
>
> > I saw some mention of people using hacks to do this in the past, but
> > also saw that the release notes for protocol buffers v 2.10 said this:
> > * Changes to input_stream.py should make protobuf compatible with
> > appengine.
>
> > I'm not sure what is meant by "compatible" in that statement, but I'd
> > like to know if GAE still requires the hackery described in these
> > threads referenced below. Is there an up-to-date howto or something
> > like that somewhere that I missed?
>

> >http://groups.google.com/group/protobuf/browse_thread/thread/6430dc5d...
>
> >http://groups.google.com/group/google-appengine/browse_thread/thread/...
>
> > Dan

[Yegor]

Hi, Dan,

I believe by "compatible" they mean compatible with the runtime. You
see, C extensions are prohibited in GAE. The patch changed
input_stream.py to fall-back to using arrays if buffers, which are
implemented in C, are not available. That said, it does not fix the
issue of restricted access to google.* packages. Because GAE does not
provide protobuf library by default (at least I haven't heard anything
yet) you have to bundle it with your application. At the same time,
the libs you bundle are not allowed to use google.* namespace, thus
the ImportError. The hack (http://code.google.com/p/protobuf-gae-hack)
seems still valid.

Cheers,

Yegor

On Jul 2, 7:46 pm, Dan <danbr...@gmail.com> wrote:
> Hello-
> I'm working with GAE and ran into this error when trying to use
> protocol buffers:
>
> from google.protobuf import descriptor
> ImportError: No module named protobuf
>
> I saw some mention of people using hacks to do this in the past, but
> also saw that the release notes for protocol buffers v 2.10 said this:
> * Changes to input_stream.py should make protobuf compatible with
> appengine.
>
> I'm not sure what is meant by "compatible" in that statement, but I'd
> like to know if GAE still requires the hackery described in these
> threads referenced below. Is there an up-to-date howto or something
> like that somewhere that I missed?
>

> http://groups.google.com/group/protobuf/browse_thread/thread/6430dc5d...
>
> http://groups.google.com/group/google-appengine/browse_thread/thread/...
>
> Dan