How to access Google Cloud Storage from Firebase functions on the server

[Chris Curnow]

I started this topic on StackOverflow: https://stackoverflow.com/questions/59098666/how-to-access-google-cloud-storage-from-firebase-functions-on-the-server?noredirect=1#comment104475241_59098666

I probably should have started it here. Anyway, I'm bringing it over here now.

I want to upload some data to Google Cloud Storage and then run a Firebase Function on the server to process the data.

My reason for this is that the data (representing information I want to import into Firestore) could be quite large – maybe 50MB. I could easily have data that represents 30,000 documents. I don't want to process the data on my local machine because it would possibly take several hours.

My problem is I can't find a way to access Cloud Storage from the server itself. I can easily download and upload files between local and server. I've spent the last several hours reading through the docs but everything seems to be based on accessing Cloud Storage from a local instance.

Here are the comments already received over at SO:

There are lots of SDKs for various languages that you can use to work with Cloud Storage in backend code. Are you just having problems finding those? – Doug Stevenson Nov 29 at 4:17

Yes @DougStevenson, I guess there are just too many SDKs and it's quite confusing. I've been following your videos on "Getting started with Cloud Functions" and I have no problems accessing Cloud Firestore. I just can't see what I would have to do to access Cloud Storage. Right at the moment I just want to post to a HTTP end point and then within that function I want to grab hold of a file from Cloud Storage. I just can't see how to do that. – Chris Curnow Nov 29 at 6:47  

Which Admin SDK are you using? – Renaud Tarnec Nov 29 at 11:12

To be clear, I am using Cloud Functions and when I say “server” I mean Google Cloud. In this scenario I thought I had to use the Node SDK. I found this tutorial codelabs.developers.google.com/codelabs/… but it only shows how to respond to Cloud Storage events. – Chris Curnow Nov 29 at 21:38

There is a node SDK for Cloud Functions. You can also use the Firebase Admin SDK, which wraps the Cloud node SDK. These are easy to find. – Doug Stevenson 2 days ago 

OK, I have been using the node SDK for Cloud Functions. I still can't see anyway from there where I can go and grab a document from Storage. I found this in the docs: firebase.google.com/docs/reference/functions/providers_storage_. I hope you can understand that page because I can't. And I found this: firebase.google.com/docs/reference/admin/node/admin.storage which makes no more sense to me than the first one. I have been using Cloud Functions for a while now and have written some pretty extensive stuff. But Storage has got me stumped – Chris Curnow yesterday

 

 

 

 

 

 

[Kato Richardson]

Hi Chris,

This is very broad, which is why you're having trouble getting an answer. How to trigger functions and how to use Cloud Storage are things best learned by checking out the getting started guides or by walking through a codelab.

I provided a quick overview of the getting started contents as an answer to your question on Stack Overflow and some notes on execution time limits that will be important to you.

☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/adc34acd-dc93-4d05-85d1-28883092dfb1%40googlegroups.com.

--

Kato Richardson |

Developer Programs Eng |

kato...@google.com |

775-235-8398

[Chris Curnow]

Thanks for your answer Kato.

The thing is I don't think my question is very broad.

I want to do ONE thing and ONE thing only.

I have a file I have uploaded to Cloud Storage.

I want to get the contents of this file on the server.

That's it. Nothing more. I don't want to create a trigger that runs when I upload a file. I actually don't even want to write the file back to Cloud Storage when I've finished. I just want its contents.

I can't find ANYTHING in the docs that mentions how to do this. I can get and modify the metadata of an object but not its actual data.

I have spent hours and hours reading the docs on this. I didn't just jump on SO and expect someone to give me a getting started tutorial.

Some of the documentation is confusing and I find myself going around in circles. I keep thinking I've missed something somewhere but if I have, I can't find it.

The more I read in the docs, including the links you sent me, the more I am coming to believe this simply cannot be done.

(As I said in one of my comments I have been using Cloud Functions for quite a while now and feel confident I can do pretty much anything I want to do. So it's not like I can't read the documentation.)

Anyway, thanks again for jumping in. I do appreciate it.

Chris

[Hiranya Jayathilaka]

Hi Chris,

You will use the Admin SDK to access Cloud Storage from a Function. See https://firebase.google.com/docs/storage/admin/start to get started. And here's an example Function that does some file upload/download work: https://github.com/firebase/functions-samples/blob/master/generate-thumbnail/

Hope this helps.

Thanks,

Hiranya

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/9e756c0e-ec52-4882-afd9-a0d6eeae4dc9%40googlegroups.com.

--

Hiranya Jayathilaka | Software Engineer | h...@google.com | 650-203-0128

[Chris Curnow]

Hi Hiranya,

Thanks for chiming in.

Unfortunately this doesn't help.

I know how to use a trigger on an event in Cloud Storage and I know how to upload and download to Cloud Storage from Cloud Functions.

What I CAN'T do, is get the contents of a file already on Cloud Storage from a Cloud Function.

From what I've read, I don't think it is possible.

Chris

[Hiranya Jayathilaka]

The above example calls file.download(). If you want to access file contents as raw bytes, then you should probably use file.createReadStream(). See https://googleapis.dev/nodejs/storage/latest/File.html#createReadStream

--

You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/229e8721-3332-4510-8534-20e99ec1645f%40googlegroups.com.

[Chris Curnow]

Sorry, I haven't replied for a while. I got caught up in some other stuff and haven't really had a chance to get back to this.

I think I finally worked out where I was going wrong. I was thinking I was running functions on 'the server' therefore didn't need to download from Cloud Storage. Like I was thinking they are the same server. Of course this is a result of being stuck in the old server paradigm and not really thinking about what 'serverless' really means in practice.

Thanks to everyone for taking the time to respond. I really appreciate that.

Chris