Unable to upload file to cloud storage

1,784 views
Skip to first unread message

thribhu kumar

unread,
Aug 14, 2019, 10:32:17 AM8/14/19
to Firebase Google Group
Hello, I am new to firebase. I have been trying to work with file upload from server-side. I have followed the documentation and created a busboy middleware.
The middleware return all the files in req.files, and all fields in req.body.


But I am unable to upload the file to storage. I am using @google-cloud/storage package.
I am confused about uploading file to storage.

MY MIDDLEWARE CODE
exports.filesUpload = function(req, res, next) {
const busboy = new Busboy({
headers: req.headers,
limits: {
// Cloud functions impose this restriction anyway
fileSize: 10 * 1024 * 1024
}
});

const fields = {};
const files = [];
const fileWrites = [];
// Note: os.tmpdir() points to an in-memory file system on GCF
// Thus, any files in it must fit in the instance's memory.
const tmpdir = os.tmpdir();

busboy.on("field", (key, value) => {
// You could do additional deserialization logic here, values will just be
// strings
fields[key] = value;
});

busboy.on("file", (fieldname, file, filename, encoding, mimetype) => {
const filepath = path.join(tmpdir, filename);
console.log(
`Handling file upload field ${fieldname}: ${filename} (${filepath})`
);
const writeStream = fs.createWriteStream(filepath);
file.pipe(writeStream);

fileWrites.push(
new Promise((resolve, reject) => {
file.on("end", () => writeStream.end());
writeStream.on("finish", () => {
fs.readFile(filepath, (err, buffer) => {
const size = Buffer.byteLength(buffer);
firebaseConfig.storage.upload(filepath);
console.log(`${filename} is ${size} bytes`);
if (err) {
return reject(err);
}

files.push({
fieldname,
originalname: filename,
encoding,
mimetype,
buffer,
size
});

try {
fs.unlinkSync(filepath);
} catch (error) {
return reject(error);
}

resolve();
});
});
writeStream.on("error", reject);
})
);
});

busboy.on("finish", () => {
Promise.all(fileWrites)
.then(() => {
req.body = fields;
req.files = files;
next();
})
.catch(next);
});

busboy.end(req.rawBody);
};


CLOUD FUNCTION:
Enter code here...
app.post("/uploadProduct", filesUpload, (req, res) => {
res.json(req.files);
});

I am able to see buffer, type and all other stuff I need. But i am stuck on how to upload this req.files object to cloud storage. thanks in advance.

Kato Richardson

unread,
Aug 14, 2019, 12:15:16 PM8/14/19
to Firebase Google Group
Hello Thribhu,

A few thoughts here: 
  1. You are finishing after busboy is done, but you aren't waiting on storage.upload() to complete before resolving. Your app could be terminating before the upload is done.
  2. You aren't catching or logging any errors from storage.upload(). Try adding .catch(e => console.error(e));
  3. I think it should be storage.bucket().upload(...) although I'm not positive on that.
☼, Kato

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/cc9abfcb-3592-4db2-9db2-ad62f3bb4876%40googlegroups.com.


--

Kato Richardson | Developer Programs Eng | kato...@google.com | 775-235-8398

thribhu kumar

unread,
Aug 14, 2019, 5:24:03 PM8/14/19
to Firebase Google Group
Hello Kato,  
Thanks for your reply. I have followed your steps and modified the code with some changes. I have added a then and catch to the upload function.
I have referenced the storage bucket in my parent, so that would not be the problem if I am not wrong. 
When I try to perform the file upload after updating the code I am getting the following error.

Enter code here... storage error { [Error: ENOENT: no such file or directory, open '/tmp/call-icon-png-3.jpg']
>    errno: -2,
>    code: 'ENOENT',
>    syscall: 'open',
>    path: '/tmp/call-icon-png-3.jpg' }
firebaseConfig.storage.upload(filepath).then(() => {console.log("upload done to storage")}).catch(e => console.error(e));
To unsubscribe from this group and stop receiving emails from it, send an email to fireba...@googlegroups.com.

Kato Richardson

unread,
Aug 14, 2019, 7:26:39 PM8/14/19
to Firebase Google Group
try {
  fs.unlinkSync(filepath);
} catch (error) {
  return reject(error);
}

Did you move this code inside storage.bucket().upload(...).finally()? If not, it's likely deleting the file before Storage upload is taking place.



To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/f668f41a-c9f3-4750-bd01-5ab81c82ae1c%40googlegroups.com.

thribhu kumar

unread,
Aug 15, 2019, 10:24:01 AM8/15/19
to Firebase Google Group
Hello Kato, 
Upon reading your comment I have added a few changes with code.
fileWrites.push(
new Promise((resolve, reject) => {
file.on("end", () => writeStream.end());
writeStream.on("finish", () => {
fs.readFile(filepath, (err, buffer) => {
const size = Buffer.byteLength(buffer);
console.log(`${filename} is ${size} bytes`);
firebaseConfig.storage
.upload(filepath)
.then(() => {
console.log("uploaded image");
})
.catch(e => console.error(e))
.finally(() => {
try {
fs.unlinkSync(filepath);
} catch (error) {
return reject(error);
}
});
if (err) {
return reject(err);
}
files.push({
fieldname,
originalname: filename,
encoding,
mimetype,
buffer,
size
});
resolve();
});
});
writeStream.on("error", reject);
})
); I have moved the fs.unlinkSync to finally() method. But now the storage function is not at all triggering. Meanwhile I am running the functions locally.

thribhu kumar

unread,
Aug 15, 2019, 6:00:43 PM8/15/19
to Firebase Google Group
I see things are working around fine when I am trying to use functions after deploying it to console. But I can't upload to the storage bucket when running locally.

Kato Richardson

unread,
Aug 16, 2019, 12:51:43 PM8/16/19
to Firebase Google Group
That seems correct. I don't think the local emulator works with Storage uploads yet.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/366b18f0-ac0d-4074-9a65-939db50ddbeb%40googlegroups.com.

thribhu kumar

unread,
Aug 16, 2019, 7:38:31 PM8/16/19
to Firebase Google Group
Thanks a lot for your support. I am really happy to be part of this community. 


On Wednesday, August 14, 2019 at 8:02:17 PM UTC+5:30, thribhu kumar wrote:
Reply all
Reply to author
Forward
0 new messages