gcloud beta app deploy : This deployment has too many files

[Lourdu rajan]

I got the below error, when I tried to deploy my Google Cloud Flex Environment Endpoint project (NodeJs) through gcloud command (gcloud beta app deploy).

ERROR: (gcloud.beta.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.- '@type': type.googleapis.com/google.rpc.BadRequest fieldViolations:  - description: This deployment has too many files. 

[Ani Hatzis]

Does your app deployment indeed include more than 10,000 files? You cannot deploy an app with more than 10,000 files. The deployment is supposed to contain mainly code and configuration files. You probably want to also host a bunch of other files (images, PDFs etc.) and have added them to the deployment, I guess. You would not include them in the deployment. Instead you can host these files in Cloud Storage or Firebase Hosting.

On Tue, May 15, 2018 at 7:01 PM Lourdu rajan <lourd...@gmail.com> wrote:

I got the below error, when I tried to deploy my Google Cloud Flex Environment Endpoint project (NodeJs) through gcloud command (gcloud beta app deploy).

ERROR: (gcloud.beta.app.deploy) INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app.- '@type': type.googleapis.com/google.rpc.BadRequest fieldViolations:  - description: This deployment has too many files. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/83d04d8f-8642-4f28-a55d-4c5ae89b1f08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lourdu rajan]

Hi Ani Hatzis,

Thanks for your reply.

I don't have any process like file storage or file processing. Just trying to deploy my code into the Google app engine flexible environment, which has basic functionality and it's reference. Like, app.yaml, openapi.yaml, app.js, db.js, package.json and node modules (reference). I checked the file size of each files, its all just below 250 kB.

Any other solutions/suggestions please.

Regards,

Lourdu Rajan G

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CALdDuQ5KaTRA7f8RS%3D5yo%3DekvqgtZ9dSSrA9rSsKCSpisHZ6CA%40mail.gmail.com.

[Ani Hatzis]

Hi,

well, that doesn't sound like 10,000+ files. If it's a bug you could file a report in the Public Issue Tracker. Is there a particular reason that you use gcloud beta app deploy instead of gcloud app deploy? The first one gives you the commands that are still in beta, the latter one the commands that are GA. And my last question: Do you have used gcloud beta app deploy with nothing else? I believe that if you don't specify the location, e.g. gcloud app deploy src/app.yaml, the command will try to automatically detect where the app is, and maybe it gets confused and tries to upload some other path with 10,000+ files.

Good luck,

Ani

[Lourdu rajan]

Hi Ani Hatzis,

Thanks for your reply.

I am trying to publish my code as Endpoint Services in Google Flex Environment. Currently Endpoint backend deploy methodlogy is in beta version. So "gcloud beta app deploy" command used for deploying my code into the Flex environment. 

I also tried to publish the code with specifying the app.yaml file in the gcloud command (i.e. gcloud beta app deploy app.yaml), but still got the same issue.

If you would like, then I may be share the code.

Also implemented below items inthis code,

- Swagger Docs

- OpenAPI file for Endpoint

- OAuth Security in OpenAPI file as Security Definition

- Encode/Decode for Database credentials in app.yaml

- Swagger Project structure

Do you think any of these item may cause for this issue?, because your sample code from Git works fine for me. 

Regards

Lourdu Rajan G

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/add00054-e4c6-47fc-a7e7-129d78323192%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,
Lourdu Rajan.G

[Ani Hatzis]

Ok, beta makes sense then and maybe it's really just a bug. If your code is public I could try to deploy myself into a temporary project and see if the deployment gives me the same error. Have you ever used Cloud Shell? It makes reproducing such steps very easy and it comes with pre-installed tools. Just to be sure the problem is not related to the local environment, e.g. outdated or broken Cloud SDK.

Regards,

Ani

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/add00054-e4c6-47fc-a7e7-129d78323192%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,
Lourdu Rajan.G

[Lourdu rajan]

Hi Ani Hatzis,

Could you provide some steps to make my code as public, so that I can upload my code for your exploration.

Yeah, my gCloud SDK is up-to date.

Regards

Lourdu Rajan G

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/3a9cb943-c5b1-4fb4-8ff0-4578326303f8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,
Lourdu Rajan.G

[Kenworth (Google Cloud Platform)]

@Lourdu

You can try GitHub, Bitbucket, or similar repository tools. Preferably, post publicly only a skeleton app that would reproduce the issue.

You mentioned files are < 250 kb but how many files do you have on your application? For example in Linux you can issue the following command to count the files on a directory:

ls directory | wc -l

[Ani Hatzis]

Hey Lourdu Rajan,

gcloud endpoints services deploy openapi.yaml

Everything fine as expected.

gcloud beta app deploy

at some point confirmed the upload of 24 files:

╔════════════════════════════════════════════════════════════╗╠═ Uploading 24 files to Google Cloud Storage ═╣╚════════════════════════════════════════════════════════════╝File upload done.Updating service [default] (this may take several minutes)...done.

My understanding is that the error occurs before you get this message right?

However, the deploy fails later where it reads the Cloud SQL connection (I don't have one yet, I'll fix this later).

ERROR: (gcloud.beta.app.deploy) Error Response: [13] Invalid Cloud SQL name: <The connection name of your instance>

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f71cfa13-4b96-4cea-b2b5-9fa2b92352a7%40googlegroups.com.

[Esther Escobodo]

yes

On Sat, May 19, 2018, 12:46 PM Esther Escobodo <escobod...@gmail.com> wrote:

yes

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CALdDuQ5iDjwHVNfGTu54iwWp64An2smvA%3Dqy9v76FF2m1bbvdA%40mail.gmail.com.

[Ani Hatzis]

Although I'm not sure whether this is the right place to discuss the trouble-shooting, here some things to try. My apologies if I repeat things you already have tried.

1. cd into the directory, where the app.yaml file of your application is located
   
2. With the "." and the debug level (see gcloud Reference):
   gcloud beta app deploy . --verbosity=debug
3. Next the command will prompt you to confirm the deployment and shows a few values. In the first lines you should see values for "descriptor" and "source". "descriptor" should be the absolute path to the app.yaml file, and "source" to the parent directory of the app.yaml file.
4. The --verbosity=debug flag will also print more information (that otherwise is only written to the gcloud log file). It will show every file that is skipped and every file that is uploaded during deployment (with full paths). After the skipping part, the "Uploading ..." part should only mention 24 files or so:
   ╔════════════════════════════════════════════════════════════╗╠═ Uploading 1 file to Google Cloud Storage ═╣╚INFO: Uploading [/home/REDACTED/shipping-backend/app.yaml] to [8a185114176aa5cdd7b0e1233915bb2bf03aeb33]
   
   (Since it isn't my first deployment of your app it uploads just the one file that I have changed)
   
   The supposed 10,000 files would hopefully will show up here or right before that, and then you should get a better idea what is causing the error. This answer suggests to check the directory for hidden content (e.g. .git, .venv, anything from your IDE). This other answer suggests to check if you have (circular) symlinks in that directory.

If you are 100% sure the directory contains only the 42 items that you have shared with me:

1. You can evaluate whether you have the same error if you don't use the beta command. For avoiding the Endpoints beta feature of a managed rollout strategy, we would need to change the app.yaml file as described in this Troubleshooting App Engine Flexible Deployment.
2. Search for the current config_id of your deployed Endpoints API service
   gcloud endpoints services describe [PROJECT_ID].appspot.com
   
3. Very likely there is one serviceConfig. Copy the value of "id:", something like 2018-05-19r0
4. Add the value as config_id to your app.yaml and change the value of rollout_strategy from "managed" to "fixed", so it looks something like this:
   endpoints_api_service:
     name: "[PROJECT_ID].appspot.com"
     config_id: "2018-05-19r0"
     rollout_strategy: fixed
   #  rollout_strategy: managed
5. Run the normal command:
   gcloud app deploy . --verbosity=debug

There is still the probability that your gcloud installation or setup is "kaput", which you could verify with a new installation and setup as explained in Before you begin in Cloud Endpoints > OpenAPI > Getting Started with Endpoints on App Engine Flexible Environment. If this doesn't help, then it might be a bug, and I would file a bug-report in the Public Issue Tracker.

For any other trouble-shooting, I would recommend to create your own question on Stack Overflow. GCP support monitors certain tags, such as google-app-engine. Also include the output of gcloud version, and --maybe redacted for secrets and project IDs-- the output of gcloud info - Don't forget to a post the link to your question here, too.

Good luck

Ani

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CALegnbHUnA4gTZ%2B9sV-khnDxsq0J3zpHNbWJ22LvGFiw8Mb_Mw%40mail.gmail.com.