Deploying to App Engine Standard Env + Laravel Project + Read-only file system error

[Hadil Charafeddine]

Hello,

Our team has been using App Engine Standard for this project (PHP and Laravel 5.8.38) for over a year now without any issues with deployment. However, a couple of weeks ago we started getting a 500 HTTP error every time we deploy (with and without --no-promote option).

When we check our logs, we find that the error is caused by:
2020/08/18 06:15:15 [error] 25#25: *2 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught ErrorException: file_put_contents(/workspace/storage/framework/views/c1b305dfa33e081e1fc836f46439e09f829e8770.php): failed to open stream: Read-only file system

It's very weird that we're receiving this message now even though the project has been working fine for more than a year and we've made countless deployments to date without this issue. Furthermore, we did not update the configuration of the project nor did we update the Laravel version.

What's even stranger is that over the past couple of weeks, 95% of the deployments fail with 500 error and 5% succeed without issues. What's also strange is that our team is comprised of 4 developers and 3 of us have 95% unsuccessful deployments while one of has has 100% successful deployments. There's absolutely no difference between the code the 3 of us are trying to deploy and the code the 4th member is successfully deploying.

This inconsistency is the error is baffling us. We've tried all the solutions online we can find but nothing is working.

Can anyone offer any insight on why this could be happening? Please keep in mind that this issue started only 2 weeks ago.

[Hadil Charafeddine]

Also, something else worth noting is that we're not sure why Laravel is trying to the write to the folder "/workspace/storage/framework/views/" instead of "/tmp/storage/framework/views/". Also, in case it's helpful to someone, below is a copy of our app.yaml (it's been configured the same way for over a year):

APP_KEY: XXX
APP_URL: 'XXXXX'
APP_NAME: XXX
APP_STORAGE: /tmp
APP_ENV: production
APP_DEBUG: true
VIEW_COMPILED_PATH: /tmp

[Mary (Google Cloud Support)]

Hello Hadil,

Yes that is correct. Since App Engine Standard the disk is read-only[1] with the exception of the /tmp directory, the app.yaml file will be need modified for the APP_STORAGE to reflect /tmp  as you have mentioned. If these are static files that Laravel needs to retrieve you can use a Cloud Storage bucket to store them. There is an open source Laravel Google Cloud Storage Package[2] which can help you with this. More information can be found on how to configure Laravel with the above package[3]. Please be aware since this package is open source, it is not supported or endorsed by GCP.

[1] https://cloud.google.com/appengine/docs/the-appengine-environments#comparing_high-level_features

[2] https://github.com/Superbalist/laravel-google-cloud-storage

[3] https://stackoverflow.com/questions/57177524/laravel-how-to-access-stored-image-from-google-cloud-storage-using-php-laravel

[Hadil Charafeddine]

Hello Mary,

Thank you for your reply.

The app.yaml file already includes "/tmp" for both "VIEW_COMPILED_PATH" and "APP_STORAGE". For more permanent files, we're already using Cloud Buckets. What Laravel is trying to do is store compiled views (which are temporary files) onto "/workspace/storage/framework/views/". However, from app.yaml we already set "VIEW_COMPILED_PATH" as tmp, why is the code still trying to save onto the "workspace" directory? I need to emphasize again that this project has been working fine for more than a year on app engine and it started breaking only 2 weeks ago. I appreciate the links you shared but if you read the problem carefully you'll see that the links do not address our issue.

With the above in mind, can you please direct us towards a solution?

[Alexis (Google Cloud Platform Support)]

Hi Hadil,

The error says: "Failed to open stream: Read-only file system". I think this means that it's finding a file there already (under "/workspace/storage/framework/views/") where it shouldn't be. This could be a caching issue with Laravel where the files are upload by certain machines only. However, if it isn't a caching issue, then something is different about those local machines and what it uploads since one dev doesn't have this problem 100% of the time. It could also be a config issue in Laravel.

Scenario 1: If all the devs are pulling from the same repo:

- What if you try one of these[1] commands to clear view cache on non-working machines?

- What if you try clearing the general[2] cache on non-working machines?

- Check the gitignore file if it's the same across stations that work and don't work

Scenario 2: If the devs are not pulling from same repo:

- Where is the part of the code that should have VIEW_COMPILED_PATH? Is that code different across repos?

- Check the gitignore file if it's the same across stations that work and don't work

There's a lot of possibilities, but I suspect it might be caching being uploaded into the wrong path somehow. Probably due to local testing.

[1] https://laravel.com/docs/7.x/views#optimizing-views

[2] https://laravel.com/docs/7.x/configuration#configuration-caching

[Hadil Charafeddine]

Hello Alexis,

Thank you for the above. We believe the root of the problem is that Laravel is trying to same files inside the "workspace" directory and not the "tmp" directory as instructed to by our app.yaml file. In the app.yaml file, we're telling Laravel to save everything in "tmp":

VIEW_COMPILED_PATH: /tmp

APP_STORAGE: /tmp

Why is it still trying to save to "workspace"? Did the app engine team do an update these past few weeks that introduced the workspace directory perhaps?

After clearing the cache, views, sessions and routes locally, I did "gcloud components update". After the update was done, I tried to deploy a --no-promote version of the project but this time, instead of failing online, it failed during the build. The error was again related to the "workspace" directory:

In PackageManifest.php line 170:

                                                                          

  The /workspace/bootstrap/cache directory must be present and writable.

Do you have any idea why Laravel would ignore app.yaml's instructions to save files onto "/tmp" and save files in "/workspace" instead?

[Mohammad I (Cloud Platform Support)]

Hello Hadil, 

It is difficult to determine the root cause and provide work around to resolve the issue without inspecting your Google Cloud Platform project, Stackdriver Logging Logs and internal monitoring tools. 

I would recommend you to create an App Engine Public Issue under the “Compute” section by going here as the issue may be required to refer to the App Engine Engineering team to resolve it. Please select the private component as probably you will have to share project specific information, so that it is not visible to the public. One of the Google Cloud Platform Support Agents will assist you to resolve your issue.

You can share the link of this Google Group thread at the Public Issue as reference. 

Please note that Google Groups is only meant for general discussion related to Google Cloud Platform products which are not technical in nature, service status updates and release notes, and ranging from book recommendations to creative shortcuts.