download_data: How do we deal with blobs?

[Jonathan Feinberg]

How should we deal with blobs on the way out? Should we build
(potentially large) Base64 strings?

[Nick Johnson (Google)]

Hi Jonathan,

It's not clear what you're asking. On the way out of what?

-Nick Johnson

On Mon, Jun 22, 2009 at 7:02 PM, Jonathan Feinberg <e.e....@gmail.com> wrote:

How should we deal with blobs on the way out? Should we build
(potentially large) Base64 strings?

--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[Jonathan Feinberg]

> It's not clear what you're asking. On the way out of what?

On the way out of App Engine. :)

Is there a preferred "correct" way to encode blobs in an exporter?

[Nick Johnson (Google)]

No, just whatever suits you best. You can write them in binary, or to separate files, if you want.

-Nick Johnson

[GenghisOne]

Hi Jonathan

Here's a sample that might provide some value to you...

http://code.google.com/p/google-app-engine-samples/source/browse/trunk/image_sharing

Cheers

[Jonathan Feinberg]

On Jun 23, 8:13 am, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> No, just whatever suits you best. You can write them in binary, or to
> separate files, if you want.

In reading the documentation for

appcfg.py download_data

, which is what I'm talking about here, I don't see any sample code or
documentation that describes how to do what you're suggesting. Could
you give me a pointer? Here's the documentation I've seen:
http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_Data_from_App_Engine

Thanks.

[Jonathan Feinberg]

On Jun 23, 8:57 am, GenghisOne <mdkach...@gmail.com> wrote:

> Here's a sample that might provide some value to you...
>

> http://code.google.com/p/google-app-engine-samples/source/browse/trun...

I couldn't find any reference to download_data there; which file
defines the exporter?

[GenghisOne]

The logic for pulling an image out of the datastore is stored in the
image_sharing.py file....


class ImageSharingServeImage(webapp.RequestHandler):
"""Handler for dynamically serving an image from the datastore.

Very simple - it just pulls the appropriate data out of the
datastore
and serves it.
"""

def get(self, display_type, pic_key):
"""Dynamically serves a PNG image from the datastore.

Args:
type: a string describing the type of image to serve (image or
thumbnail)
pic_key: the key for a Picture model that holds the image
"""
image = db.get(pic_key)

if display_type == 'image':
self.response.headers['Content-Type'] = 'image/png'
self.response.out.write(image.data)
elif display_type == 'thumbnail':
self.response.headers['Content-Type'] = 'image/png'
self.response.out.write(image.thumbnail_data)
else:
self.error(500)
self.response.out.write(
'Couldn\'t determine what type of image to serve.')

On Jun 23, 7:02 am, Jonathan Feinberg <e.e.c...@gmail.com> wrote:

[Jonathan Feinberg]

On Jun 23, 9:13 am, GenghisOne <mdkach...@gmail.com> wrote:

>   def get(self, display_type, pic_key):

I'm afraid I still don't see how this addresses my question about
download_data.

http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_Data_from_App_Engine

[Nick Johnson (Google)]

Check out the source code for the Exporter in the bulkloader:  http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/bulkloader.py#3308 . The docstrings describe how to define your own exporter class that stores data however you want.

-Nick Johnson

Thanks.

[Jonathan Feinberg]

On Tue, Jun 23, 2009 at 9:26 AM, Nick Johnson (Google) <
nick.j...@google.com> wrote:

> Check out the source code for the Exporter in the bulkloader:

> http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/bulkloader.py#3308. The docstrings describe how to define your own exporter class that stores
> data however you want.
>

The docstring on the Exporter class is incorrect; it suggests
overriding a
method on the Loader class.

The Exporter class's __init__ method expects a bunch of tuples that
connect
entity properties to functions that return strings. There doesn't seem
to be
a way to decouple this assumption (that properties are serialized as
strings) from the other initialization machinery (registering the
exporter
for a given entity class). Am I missing something?

[Nick Johnson (Google)]

Raw strings (that is, ones that aren't unicode) can contain any data, including blobs, and all the conversion functions should return raw strings. You can then override output_entities to output the data to the format of your choice.

-Nick Johnson

[Jonathan Feinberg]

On Jun 23, 9:57 am, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> Raw strings (that is, ones that aren't unicode) can contain any data,
> including blobs, and all the conversion functions should return raw strings.
> You can then override output_entities to output the data to the format of
> your choice.

Huzzah! Thank you.

Am I write in my understanding that by the time these Exporter methods
are called, the entities have already been transferred in some opaque
(and presumably efficient) form from GAE to the client?

[Nick Johnson (Google)]

Correct - the bulkloader uses remote_api. If you want, of course, you can write your own importer, exporter, or anything else, using remote_api.

-Nick Johnson
 

[Jonathan Feinberg]

On Jun 23, 10:04 am, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> > Am I write in my understanding that by the time these Exporter methods
> > are called, the entities have already been transferred in some opaque
> > (and presumably efficient) form from GAE to the client?
>
> Correct - the bulkloader uses remote_api. If you want, of course, you can
> write your own importer, exporter, or anything else, using remote_api.

Again, thanks. One final question (I think): is my undertsnading of
the code correct in that the result_db is not deleted at the
conclusion of a successful transfer, and that, therefore, if i run the
export a couple of weeks later, only the newly added entities will be
transferred?

[Nick Johnson (Google)]

I'm not certain, actually. I would expect that the database is deleted or marked as completed once everything's downloaded.  Let me know if you find out! ;)

-Nick Johnson