What's up with Blobstore?

129 views
Skip to first unread message

Joshua Fox

unread,
May 17, 2017, 3:23:23 AM5/17/17
to google-a...@googlegroups.com
Blobstore is is not officially deprecated, and I hear that there are no plans to deprecate it.

Yet it is de facto deprecated -- Google recommends against using it.

The problem with Blobstore is
- No backup tool.
-  The data is locked up -- very difficult to extract.   The API  is oriented towards downloading to a  browser, using a key passed from an HTML form.  It should be possible to write code that behaves like  a browser, but I'd rather not have to figure that out.
- There is no API usable on Flex Env, or anywhere else except Standard Env 

Google should provide a super-easy way to extract and migrate data. 

Most simply, note that Cloud Storage can be a backing store for Blobstore. 

Google should provide a one-button migration that takes a Blobstore and alters it so that it is backed by Cloud Storage. Then existing Blobstore clients would continue to work, but it would become possible to access the data by a better-supported API.

George (Cloud Platform Support)

unread,
May 17, 2017, 2:37:49 PM5/17/17
to google-a...@googlegroups.com
Hello Joshua, 

There is work in progress on this issue, as one can gather from the "Add full blobstore support to bulkloader" issue

Meanwhile, similar functionality can be set up on user side. As an example, the "Backup GAE Data/Files from Blob Store to Google Cloud Storage / other space" page provides useful information. 

Nicholas (Google Cloud Support)

unread,
May 17, 2017, 2:51:37 PM5/17/17
to Google App Engine
Here's a link to the above mentioned public issue: https://issuetracker.google.com/35888043

Jason Collins

unread,
May 17, 2017, 5:15:08 PM5/17/17
to Google App Engine
This (unsupported, open source) tool might help too: https://github.com/GoogleCloudPlatform/appengine-blobstoremigrator-python

Joshua Fox

unread,
May 18, 2017, 7:44:52 AM5/18/17
to google-a...@googlegroups.com
Thank you Jason, Nicholas,  and George. I hope that  Google comes through with a transparent migration tool before deprecating Blobstore.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f246b4b2-6f27-4ac7-8ee8-9f9e925e697a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
JOSHUA FOX
Principal Software Architect | Freightos


 (Israel): +972-545691165 |  (US):  +1-3123400953 | Skypejoshuafox.freightos
Smoother shipping with the world's online freight marketplace.

Joshua Fox

unread,
May 18, 2017, 7:45:46 AM5/18/17
to google-a...@googlegroups.com
Thank you, that is a useful illustration of  how to access Blobstore from code that is not a browser.

On Thu, May 18, 2017 at 12:15 AM, Jason Collins <jason.a...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

Jeff Schnitzer

unread,
May 18, 2017, 3:01:11 PM5/18/17
to Google App Engine
One programmer’s opinion:

I did a migration from the Blobstore to GCS some years ago. The docs for GCS were a little bit obtuse and took a little work to figure out, but the end result was satisfying - GCS is a vastly better blobstore than the old Blobstore. And the GCS API is much more powerful than the BlobstoreService API.

It’s worth putting effort into the transition. Copying the data is relatively straighforward; the hard part is transitioning your code away from the Blobstore API, and Google can’t really help with that.

Jeff

Joshua Fox

unread,
May 21, 2017, 9:15:34 AM5/21/17
to google-a...@googlegroups.com
On Thu, May 18, 2017 at 10:00 PM, Jeff Schnitzer <je...@infohazard.org> wrote:
One programmer’s opinion:

I did a migration from the Blobstore to GCS some years ago. The docs for GCS were a little bit obtuse and took a little work to figure out, but the end result was satisfying - GCS is a vastly better blobstore than the old Blobstore. And the GCS API is much more powerful than the BlobstoreService API.

It’s worth putting effort into the transition. Copying the data is relatively straighforward; the hard part is transitioning your code away from the Blobstore API, and Google can’t really help with that.

Google could help by transparently migrating the backing store of the Blobstore data to GCS (which is a supported backing store for Blobstore). 

Then, existing Blobstore-API code would  continue to work and it would be easier to transition to a GCS-API without downtime.


For more options, visit https://groups.google.com/d/optout.

Attila-Mihaly Balazs

unread,
May 29, 2017, 7:52:15 AM5/29/17
to Google App Engine
I'm confused a little bit. Isn't Blobstore backed by GCS? Why would one need to "migrate"? I only use Blobstore for the "create_upload_url" functionality and all the post-processing of the data is done trough GCS.

Attila

Joshua Fox

unread,
May 29, 2017, 8:46:18 AM5/29/17
to google-a...@googlegroups.com
Blobstore can either use GCS as a backing store or use its own internal backing-store. My problem is what to do with a Blobstore that is not backed by GCS. 

On Mon, May 29, 2017 at 2:52 PM, Attila-Mihaly Balazs <dify...@gmail.com> wrote:
I'm confused a little bit. Isn't Blobstore backed by GCS? Why would one need to "migrate"? I only use Blobstore for the "create_upload_url" functionality and all the post-processing of the data is done trough GCS.

Attila

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

For more options, visit https://groups.google.com/d/optout.

Jason Collins

unread,
May 29, 2017, 1:01:40 PM5/29/17
to Google App Engine
It depends on when you created your project. Projects created since late 2016 use GCS by default. 

Projects created before that date used an internal storage by default where the blobs do not appear in GCS, though you can choose to use GCS instead by explicitly specifying a bucket.

On Mon, 29 May 2017 at 04:52 Attila-Mihaly Balazs <dify...@gmail.com> wrote:
I'm confused a little bit. Isn't Blobstore backed by GCS? Why would one need to "migrate"? I only use Blobstore for the "create_upload_url" functionality and all the post-processing of the data is done trough GCS.

Attila

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/2puYf1l7vVs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages