Google App Engine SDK version 1.9.21 now available

[Sarah Murphy]

Joy! the latest version of the SDK is available for download.

Please take a look at these new additions:

All

• The Files API, which has been deprecated since June 2013, will soon be restricted to existing users only in preparation for its future removal. Further information about the removal of this API is available here: https://cloud.google.com/appengine/docs/deprecations/
  

PHP

• Add support for basic authorization when using HTTP streams (https://code.google.com/p/googleappengine/issues/detail?id=11901)

• Upgraded the version of libcurl to 7.42.0.

[Emanuele Ziglioli]

• The Files API, which has been deprecated since June 2013, will soon be restricted to existing users only in preparation for its future removal. Further information about the removal of this API is available here: https://cloud.google.com/appengine/docs/deprecations/

That's a shame because Cloud Storage is an inferior solution from the GAE point of view: with Blobstore, we could serve large files without using CPU instance time. With Cloud Storage that's not possible, unless the file is public. We have to load each byte into the instance first therefore adding latency and wasting RAM.

Finally, using the Blobstore API to serve Cloud Storage files is not an option either. First the generated key is bound to the file name, so it's not a key.

I feel like Google is discouraging us from using GAE, see also the latest price reduction that only applies to Compute Engine VMs.

I'll start looking into alternatives, GAE increasily looks like an orphan.

Emanuele

- on GAE since 2011

[Alex Martelli]

On Wed, May 20, 2015 at 2:16 PM, Emanuele Ziglioli <the...@emanueleziglioli.it> wrote:

• The Files API, which has been deprecated since June 2013, will soon be restricted to existing users only in preparation for its future removal. Further information about the removal of this API is available here: https://cloud.google.com/appengine/docs/deprecations/

That's a shame because Cloud Storage is an inferior solution from the GAE point of view: with Blobstore, we could serve large files without using CPU instance time. With Cloud Storage that's not possible, unless the file is public.

Non proprio!-)  "Signed URLs" are the Cloud Storage way to "serve large files without using CPU instance time" AND without making them public -- see https://cloud.google.com/storage/docs/access-control#Signed-URLs .

Alex

 

We have to load each byte into the instance first therefore adding latency and wasting RAM.

Finally, using the Blobstore API to serve Cloud Storage files is not an option either. First the generated key is bound to the file name, so it's not a key.

I feel like Google is discouraging us from using GAE, see also the latest price reduction that only applies to Compute Engine VMs.

I'll start looking into alternatives, GAE increasily looks like an orphan.

With 16 SDK releases (1.9.5 to 1.9.20 included) in exactly one year (May 20 to May 20) this "looks like" seems peculiar.  How many releases per month would suffice to convince you otherwise, if 1.33 are not enough?-)

Ciao,

Alex

Emanuele

- on GAE since 2011

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/96169256-0e0d-49e6-b8e1-70565cdb2567%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Emanuele Ziglioli]

Non proprio!-)  "Signed URLs" are the Cloud Storage way to "serve large files without using CPU instance time" AND without making them public -- see https://cloud.google.com/storage/docs/access-control#Signed-URLs .

Ciao Alex, 

I've been looking for something like that, I just didn't know it was called "signed URLs"!

Although I suspect that once my app creates a file and generates a signed URL, that URL should still be exposed to the user, there's no way to "internally forward" a GET request to that external address, is there?

 

With 16 SDK releases (1.9.5 to 1.9.20 included) in exactly one year (May 20 to May 20) this "looks like" seems peculiar.  How many releases per month would suffice to convince you otherwise, if 1.33 are not enough?-)

Point taken, also I've been following the correspondence between various Googlers and developers on this forum.

There's been an increased pace in addressing long standing bugs, that should be a good sign.

I hope to see some features added to GAE in the future too, easier integration between Datastore and BigQuery/Dataflow for example.

Grazie mille,

Emanuele

 

[Alex Martelli]

On Wed, May 20, 2015 at 2:36 PM, Emanuele Ziglioli <the...@emanueleziglioli.it> wrote:

Non proprio!-)  "Signed URLs" are the Cloud Storage way to "serve large files without using CPU instance time" AND without making them public -- see https://cloud.google.com/storage/docs/access-control#Signed-URLs .

Ciao Alex, 

I've been looking for something like that, I just didn't know it was called "signed URLs"!

Although I suspect that once my app creates a file and generates a signed URL, that URL should still be exposed to the user, there's no way to "internally forward" a GET request to that external address, is there?

Right -- redirects or the like are done client-side, that's how HTTP works.  But you set the timeout as tight or lax as you want, so that for example it can be acted upon "right then and there or not at all" -- or endure for the duration of e.g a subscription to your service the user's paid up for -- or whatever else you want, it's really a very flexible arrangement (esp. once you consider how many other things you can, if you so choose, bind into a signed URL).

 

 

With 16 SDK releases (1.9.5 to 1.9.20 included) in exactly one year (May 20 to May 20) this "looks like" seems peculiar.  How many releases per month would suffice to convince you otherwise, if 1.33 are not enough?-)

Point taken, also I've been following the correspondence between various Googlers and developers on this forum.

There's been an increased pace in addressing long standing bugs, that should be a good sign.

Definitely not the dynamics you'd typically observe about a product that's orphaned.

 

I hope to see some features added to GAE in the future too, easier integration between Datastore and BigQuery/Dataflow for example.

Sorry, no comments about the roadmap.  But, e.g, is https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/io/DatastoreIO.Source (a Dataflow Source that reads the result rows of a Datastore query as Entity objects) the kind of things you're asking for?  Anyway, feature requests are always welcome, e.g at https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues for the Dataflow Java SDK!

 

Grazie mille,

Emanuele

Ma prego, dovere!

Alex

 

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/23a57b26-8ecf-4462-b953-558982ae7c46%40googlegroups.com.

[hiroshi saito]

Hi Sarah,

Are there any permanent links of SDK to download?
I need them for CI use.
I can have a copy or a cache as a workaround, but why not Google provides a way?

Please see http://stackoverflow.com/q/29113596/338986 for details.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/google-appengine/3b2e3b6e-634c-458d-87bd-ef404980b30e%40googlegroups.com.

[Emanuele Ziglioli]

Update: I've tried again by serving GCS files using the Blobstore API and, after fixing some problems, it's been working well all day. The problems were:

1. lack of BlobInfo (as documented)

2. file names containing some characters such as blank spaces

Because I used Blobinfo in order to get the file size and to see if the file was there, I now use GcsService.list().

A method that checks for an individual file would be better than querying for the whole list.

Also, some headers I've set don't appear in the Http Response, so I've had to provide that information another way.

Regardless, blobstoreService.serve() now works for me off GCS and I recommend it to fellow GAE users.
It's definitely simpler than creating "signed URLs" on demand.

But I'll keep this information for future use, thank you for the tip, Alex

Emanuele

[Jeff Schnitzer]

I also recently started using GCS through the Blobstore API. I found the docs to be super confusing but I'm happier with my new GCS-based solution than my Old-Blobstore solution.

The documentation doesn't make it clear when you should use BlobKey and when you should use the gcsobjectname. It appears that the BlobKey should be ignored now; use BlobstoreService.getFileInfos() instead of getBlobs() or getBlobInfos(). The images service getServingUrl() has an option that takes the objectname too. The one piece missing is delete() with an objectname; but it appears you can make a BlobKey from the objectname and delete that.

I'm pretty happy that there are no system-maintained __BlobInfo__s anymore. I find it much more useful to maintain my own BlobInfo entity, creating it in the callback from the FileInfo (and storing the servingurl along with it). I take the opportunity to de-duplicate uploaded blobs here, discarding repeat uploads.

The one bit of advice I have for google is to make the documentation really obvious about the "old way" and "new way" of doing things. Right now it's really not clear that BlobKey is on its way out.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/689faeec-dbe6-4889-8be3-b612b1916c7c%40googlegroups.com.

[husayt]

Hi Jeff,

you mentioned:

>appears you can make a BlobKey from the objectname and delete that.

I thought it would be useful to mention how:

final BlobKey blobKey = ImagesServiceFactory.makeImageFromFilename(cloudStorageUrl).getBlobKey();

   if (blobKey != null) {

            ImagesService imagesService = ImagesServiceFactory.getImagesService();

            imagesService.deleteServingUrl(blobKey);

    }

Thanks.

[Wilson MacGyver]

My usage case is simple, I just need to serve data.

I use createGsBlobKey("Google Storage filename")

https://cloud.google.com/appengine/docs/java/javadoc/com/google/appengine/api/blobstore/BlobstoreService#createGsBlobKey(java.lang.String)

then just call .serve on it

https://cloud.google.com/appengine/docs/java/javadoc/com/google/appengine/api/blobstore/BlobstoreService#serve(com.google.appengine.api.blobstore.BlobKey,%20com.google.appengine.api.blobstore.ByteRange,%20HttpServletResponse)

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f846416d-c3b0-40ef-85f1-8aeeef13a875%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Omnem crede diem tibi diluxisse supremum.

[Josh Moore]

@Hiroshi,

While it doesn't solve the entire issue, incorporating the gcloud SDK into the automation process would allow the use of a service account to download the App Engine SDKs via gsutil.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAM0dQnnwuRFAED7gSao_u6yyBpLGJ1Wa_Uw-c1Pto5TWe-_Pfg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Josh Moore |

Cloud Solutions Engineer |

josh...@google.com |

[hiroshi saito]

Thanks, Josh.

> While it doesn't solve the entire issue, incorporating the gcloud SDK into the
automation process would allow the use of a service account to download the
App Engine SDKs via gsutil.

Yeah, it is a way.

> To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAOvpJmVPp5yxMGM8xBSB4R%2Btt%3DBdtjmV_-zpXWOTWzkP%2BKFPrA%40mail.gmail.com.