GAE/J - Completely lost on migrating from Files API to GCS Client Library

[Gilberto Torrezan Filho]

Hi everyone,

I'm very confused about what do I have to do to migrate from the now deprecated Files API to GCS Client Library. Before anything I'd like to express my complete dissatisfaction with the fact that I have to manage a second service from now on just to write files on the server. Anyway, let's stick to the problem:

I have this piece of code which is working very well for years:

FileService fileService = FileServiceFactory.getFileService();
AppEngineFile file = fileService.createNewBlobFile("text/plain");
FileWriteChannel writeChannel = fileService.openWriteChannel(file, true);
Writer writer = Channels.newWriter(writeChannel, "UTF8");
//write my file
writeChannel.closeFinally();
BlobKey blobKey = fileService.getBlobKey(file);
//I got my BlobKey and I'm happy

But FileService is now deprecated. When I try to use the GCS client library (after discovering that the library isn't part of the GAE SDK just to make me even more "happy"), I got this situation:

GcsService gcsService = GcsServiceFactory.createGcsService(); //no problems
GcsFilename gcsFileName = new GcsFilename(bucketName, objectName); //I have to create this object to create the file below
gcsService.createOrReplace(gcsFileName, gcsFileOptions); //I really miss the day when file.createNewFile(); was sufficient

The problem is the second line. As it seems now I have to provide a file name. Ok, I can gerente a random one, just to keep the API satisfied. Now the problem is the "bucket name". I don't even know what it means. I just want to save a file.

After searching a bit I found that it is something related to the Google Cloud Storage service. Ok, let's signup another service, accept another terms, and so on (... and I just want to save a file). When logging on the Cloud Console, I can see my App Engine project, with the following options: App Engine (which links to the App Engine Dashboard), BigQuery (not using) and Cloud Datastore (which links to a view of my stored entities). No Cloud Storage. Following this activation guide, I have to create another project (!!!).

So no Cloud Storage dashboard or console anywhere, nowhere to configure or get that "bucket name" and no, I'm not going to drop my 2-years project to create another one just to enable file writing.

What do I have to do to just-save-a-file on the server? Why is that so hard?

Thank you.

(Sorry for the text but I'm really upset with this situation)

[Kaan Soral]

+1

I think it also contradicts with AppEngine in the sense that it complicates things, whereas AppEngine is for simplifying stuff (scalability, maintenance, development etc.)

I've also took a quick look at the state of GCS integration and I've decided to wait until the integration matures (I'm not saying it isn't mature now, I will just wait until all problems/issues are solved / improved)

[Jason Collins]

Ya, it seems like all the bits and pieces are in place, but it's not quite all stitched together yet.

Anxiously awaiting the future I can clearly see, especially the bits to do with automatic authentication via Service Accounts,

j

[Vinny P]

Hello Gilberto,

On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho <gilberto...@gmail.com> wrote:

The problem is the second line. As it seems now I have to provide a file name. Ok, I can gerente a random one, just to keep the API satisfied. Now the problem is the "bucket name". I don't even know what it means. I just want to save a file.

 

 

A bucket is basically a container for all your files. Bucket names are unique across Google Cloud Storage - once you reserve your bucket, it's yours forever.

 

 

 On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho <gilberto...@gmail.com> wrote:

After searching a bit I found that it is something related to the Google Cloud Storage service. Ok, let's signup another service, accept another terms, and so on (... and I just want to save a file). When logging on the Cloud Console, I can see my App Engine project, with the following options: App Engine (which links to the App Engine Dashboard), BigQuery (not using) and Cloud Datastore (which links to a view of my stored entities). No Cloud Storage. Following this activation guide, I have to create another project (!!!).

 

You don't need to create a new project. Go into your Application Settings page and press the button marked "Add Project" under the Cloud Integration section. See here for a picture: http://imgur.com/9jMWHMj 

 

I saw in an earlier email that the button wasn't working for you. Try it again once or twice, sometimes it fails but it'll eventually integrate your project.

 

 

On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho <gilberto...@gmail.com> wrote:

So no Cloud Storage dashboard or console anywhere, nowhere to configure or get that "bucket name" and no, I'm not going to drop my 2-years project to create another one just to enable file writing.

 

You don't have to drop your current project. If your current project isn't integrating with Cloud Integration, what you can do is keep your current project (holding your application) and create a new project that contains only the Google Cloud Storage bucket.

 

Go to https://cloud.google.com/console - click the button marked Create Project - fill in all the forms to create a new project (you may have to activate billing). Click on the new project you've created (the screen will look like this: http://imgur.com/iszb7lw ). Click on the Google Cloud Storage link, and press the button marked "New Bucket." Name your new bucket whatever you want.

 

You can then configure your application to access the bucket (it's irrelevant that they're technically within different projects).

 

On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho <gilberto...@gmail.com> wrote:

(Sorry for the text but I'm really upset with this situation)

 

 Don't apologize, it's much easier to help when people give detailed comments on what they need :-). 

 

I sympathize with you - I miss having the Files API as well.

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

 

[Kaan Soral]

What do you do for the SDK?

I better create a sample app and check the situation, with the blobstore, it has a nice directory structure, which you can move around, change the directory with an argument etc.
Also wondering what can be done to move already existing blobstore sdk stuff to blobstore gcs structure, if that's possible

Can't imagine the frustration to do this for a live app

On Wednesday, July 24, 2013 10:38:01 PM UTC+3, Vinny P wrote:

Hello Gilberto,

On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho

The problem is the second line. As it seems now I have to provide a file name. Ok, I can gerente a random one, just to keep the API satisfied. Now the problem is the "bucket name". I don't even know what it means. I just want to save a file.

 

 

A bucket is basically a container for all your files. Bucket names are unique across Google Cloud Storage - once you reserve your bucket, it's yours forever.

 

 

 On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho

After searching a bit I found that it is something related to the Google Cloud Storage service. Ok, let's signup another service, accept another terms, and so on (... and I just want to save a file). When logging on the Cloud Console, I can see my App Engine project, with the following options: App Engine (which links to the App Engine Dashboard), BigQuery (not using) and Cloud Datastore (which links to a view of my stored entities). No Cloud Storage. Following this activation guide, I have to create another project (!!!).

 

You don't need to create a new project. Go into your Application Settings page and press the button marked "Add Project" under the Cloud Integration section. See here for a picture: http://imgur.com/9jMWHMj 

 

I saw in an earlier email that the button wasn't working for you. Try it again once or twice, sometimes it fails but it'll eventually integrate your project.

 

 

On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho 

So no Cloud Storage dashboard or console anywhere, nowhere to configure or get that "bucket name" and no, I'm not going to drop my 2-years project to create another one just to enable file writing.

 

You don't have to drop your current project. If your current project isn't integrating with Cloud Integration, what you can do is keep your current project (holding your application) and create a new project that contains only the Google Cloud Storage bucket.

 

Go to https://cloud.google.com/console - click the button marked Create Project - fill in all the forms to create a new project (you may have to activate billing). Click on the new project you've created (the screen will look like this: http://imgur.com/iszb7lw ). Click on the Google Cloud Storage link, and press the button marked "New Bucket." Name your new bucket whatever you want.

 

You can then configure your application to access the bucket (it's irrelevant that they're technically within different projects).

 

On Sat, Jul 20, 2013 at 9:07 PM, Gilberto Torrezan Filho 

(Sorry for the text but I'm really upset with this situation)

[Gilberto Torrezan Filho]

Hi Vinny P,

Thank you for your reply. Indeed I'm having problems to activate the cloud integration for my project, due to this issue. I'll wait for a position from Google about it (a patch?) before creating more projects or trying workarounds.

Kaan Soral, I'm worried about the migration of existing blobstore as well. I think (and I want to believe) Google will make some "migration magic button" similar to when they migrated to HRD. I simply can't rely on a structure which makes me migrate data and rework code at every release, though.

[alex]

I think one could create their own "magic button" with just a couple piplines/mapreduces.

Here's a sample I just did:

https://github.com/crhym3/bs2gcs

(though it's for python27)

[Kaan Soral]

Nice project

but I think the hardest part would be to refactor url references, an url might be used in thousands of PickleProperties etc.

I don't know whether image urls keep serving after deleting a blob, however If I were to refactor any data, I would keep the old blobs for simplicity

If the storage costs are extreme, so would be the refactoring costs, in that case it might be logical to keep the old blobstore logic for actions that need editing, and after a blob/image is edited just save it to the gcs instead

depending on the use case, switching to gcs without refactoring also seems logical, assuming old blobs/urls will indefinitely keep serving

I will not switch to GCS for my old/relatively big app

[Celio Marcos]

In Python I get the bucket using this:

        bucket_name = os.environ.get('BUCKET_NAME', app_identity.get_default_gcs_bucket_name())
        bucket = '/' + bucket_name
        filename = bucket + '/' + "picture.jpg"