Parallel bulk file(object) operations on cloud storage from PHP flexible environment app engine.
37 views
Skip to first unread message
Armen Haroyan
Dec 14, 2017, 12:17:53 PM12/14/17
to Google App Engine
Hello
We have a php app running on app engine, flexible environment, and we need to do a bulk file operation on on a google cloud storage, like copy a 200 files from one place to another in a bucket.
We were using php libraries out there, but as they all are doing the copy operations file by file with api, it was taking too long. No multi-thread is available for them, so we went more and found the gsutil.
As gsutil can do bulk operations in multi-thread, we used shell_exec() in php to call gsutil with -m option, and it was running really fast on local development.
But we are getting "command not found" error on app engine, we tried to manually put gsutil there, but after that we start getting "permission denied".
Our task is to do bulk operations of google cloud storage in parallel manner, anything suggestions?
We have a php app running on app engine, flexible environment, and we need to do a bulk file operation on on a google cloud storage, like copy a 200 files from one place to another in a bucket.
We were using php libraries out there, but as they all are doing the copy operations file by file with api, it was taking too long. No multi-thread is available for them, so we went more and found the gsutil.
As gsutil can do bulk operations in multi-thread, we used shell_exec() in php to call gsutil with -m option, and it was running really fast on local development.
But we are getting "command not found" error on app engine, we tried to manually put gsutil there, but after that we start getting "permission denied".
Our task is to do bulk operations of google cloud storage in parallel manner, anything suggestions?
Kenworth (Google Cloud Platform)
Dec 14, 2017, 3:07:46 PM12/14/17
to google-a...@googlegroups.com
Since you are using PHP on App Engine Flex, I would recommend using copy/insert method of the Cloud Storage API. This way, the communication will all be RESTful services.
Please note that Google Groups is for general product discussions and not for technical support. If you require further technical support for copying files from one place to another, it is recommended to post your full detailed question to Stack Overflow using the supported Cloud tags.
Reply all
Reply to author
Forward
0 new messages