Max 10,000 file limit is not suitable for modern full featured wordpress site or any PHP app

[Shaharia Azam]

Hello Everyone,

I know this is a common questions but actually this is now becoming more and more strange to me day by day. Why Google still feels about the hard limit related to max 10,000 file? If I want to install Wordpress latest version with any premium themes, few e-commerce related mandatory plugins then easily from the beginning of my development this would cross 10,000 files. 

We are a web solution provider company and we are facing few very hard questions that is forbidding us to suggest GCP to our client's for their needs.

1. Advance & Full featured E-Commerce site powered by Wordpress due to 10,000 file limit issue.

2. SMTP outgoing issue due to block of Port 25. Even our own cPanel server is hosted on AWS for that. Rest of our servers that doesn't need email communication are on GCP.

Does Google think that it is irrelevant? Even from the quota request page, this file limit quota can't be increased. In this case, not only us but also lots of people are going away from using GCP and specially Google App Engine (our use case depends on it). So I want to hear something from any internal team members about this. Is there any plan in future to remove this limit or increase this limit or you are suggesting us to go away from App engine to somewhere else?

- @shaharia

[Nicolas (Google Cloud Platform Support)]

Hi Shaharia,

Thank you for bringing those inconveniences that you are experiencing , I’ll make sure to relay those to the App Engine engineering team.

In the meantime for the default 10,000 file upload issue, if you have a Silver, Gold, or Platinum support package you can contact a support representative [2] to file a proper quota increase request.

Also concerning the SMTP port 25 being block you are right we block it because of the risk of abuse [3] you could also be interested in other alternative solution like SendGrid, MAilgun or Mailjet [4].

We value our customers and take into consideration every feedback about our products.

[1] https://cloud.google.com/appengine/quotas#When_a_Resource_is_Depleted

[2] https://cloud.google.com/support/docs/#contacting_technical_support

[3] https://cloud.google.com/compute/docs/tutorials/sending-mail/

[4]https://cloud.google.com/compute/docs/tutorials/sending-mail/#choosing_an_email_service_to_use

[Shaharia Azam]

From the GCP Console quota request, I couldn't find any option to increase this specific file limit quota on App Engine section. This quota is changeable? I didn't know that. So what would happen if I don't have any support package, to send a quota increase request, I have to upgrade my support package first. This is really inconvenience and should not be welcomed by public GCP user.

So in this situation, what can I do?

- Shaharia

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/uxLbmkWZPqM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/4f5421d9-0dc9-4218-860d-4dbb76146c0a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

Hello Shaharia, 

You should have a look at the "Getting Support for Google Cloud Platform" documentation page, "Contacting Technical support" sub-chapter, and proceed in accordance with the instructions provided there. 

[G. M. Shaharia Azam]

George,

Thanks for your reply with instructions.

But that's not the case. In my previous reply i wrote that to get file limit increase i have to upgrade my support plan with money and then tell someone from GCP support to increase. That's not convenient. That's what i wanted to express.

------
Shaharia Azam
Managing Director/ Preview Holdings Limited
CEO/ Preview Technologies Limited
Sr. Executive VP/ BluSheet LLC

Phone: +8801717530114
Tel: +880 9611 738 439 (Office)

On Sun, Apr 21, 2019, 10:01 PM 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hello Shaharia, 

You should have a look at the "Getting Support for Google Cloud Platform" documentation page, "Contacting Technical support" sub-chapter, and proceed in the accordance with the instructions provided there. 

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/uxLbmkWZPqM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d7810f2b-55cb-41f4-8c50-5e1738a9d042%40googlegroups.com.

[diogoa...@google.com]

Hello Shaharia,

I understand the inconvenience, however, please note that the Google Cloud Platform free tier coverage does not provide quota increase requests. Please check here more details about the free tier.

On Sunday, April 21, 2019 at 12:25:11 PM UTC-4, G. M. Shaharia Azam wrote:

George,

Thanks for your reply with instructions.

But that's not the case. In my previous reply i wrote that to get file limit increase i have to upgrade my support plan with money and then tell someone from GCP support to increase. That's not convenient. That's what i wanted to express.

------
Shaharia Azam
Managing Director/ Preview Holdings Limited
CEO/ Preview Technologies Limited
Sr. Executive VP/ BluSheet LLC

Phone: +8801717530114
Tel: +880 9611 738 439 (Office)

On Sun, Apr 21, 2019, 10:01 PM 'George (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:

Hello Shaharia, 

You should have a look at the "Getting Support for Google Cloud Platform" documentation page, "Contacting Technical support" sub-chapter, and proceed in the accordance with the instructions provided there. 

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/uxLbmkWZPqM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

[Dmitry Ivanov]

It's an idiotic restriction. The tensorflow package alone contains over 5000 files. I have a tiny app and I exceeded the limit. What were they thinking..

[Henry Saldyga]

Our Java GAPE application ( and it's not a free tier with billing enabled ) since 2014 was using a Google BLOB storage . 

Since that our clients  performed about 180'000 file uploads . 

Than on February 15th Google BLOB API stopped to upload any files just returning 500 Internal server error. 

A private issue  was created and Google support  informed that it was a restriction of 10'000 uploads set. 

And on February 28th they simple stopped to respond in the private issue board. 

I have to create another project and redirect all clients traffic there. But we and our clients experienced a financial losses due to service interruption.

Right now on a new project we are getting to 10'000 uploads limit. 

Looks Google support does not care about their clients at all.

[Jad El Houssami]

Hello and thank you all for your feedback.

@Dmitry I can certainly understand how the 10,000 file limit could be insufficient to your use case, however you must keep in mind that a free trial is meant for customers to learn the platform and experiment new solutions. We don’t normally expect those uses to require more than 10,000 files however we do acknowledge that it is possible in some cases. If you would like, I invite you to submit a feature request to increase the max file limit for App Engine’s Free Tier. Don’t forget to include details regarding your use case(s) and any other relevant information that might help us when evaluating the request as everything will be considered.

@Henry Thank you for reporting this. We take all reported issues seriously; an unanswered private thread in our issue tracker is unacceptable and I sincerely apologize for that. I have looked into our Issue Tracker and was able to locate your thread. I will be directly in contact with the person handling it to ensure that you can get an update as soon as possible.

[Henry Saldyga]

Thanks Jad.

When Google set a 10'000 upload limit to BLOB  to a commercial project where hundreds of external applications   upload  about 10'000 files in a month  after they already uploaded 180'000 it  ruins the  business.  

You are our last hope, Jad

Henry.

[Henry Saldyga]

Hello Jad,

That;s what I got from the person you are directly in contact and  the person handling it ( The issue I experienced ).

Status: Won't Fix (Infeasible)

02:14AM

Hi Henry,

I there has been some confusion here. We are not Google Cloud Platform support. Google Issue Tracker is intended for tracking reproducible issues and feature requests during product development 1.

Since you issue is a technical one, I believe you should raise this on the right channel, which would be Google Cloud Platform support 2.

Which mean after 45 days they are figured out that they  cannot reproduce it  and do not want to fix.

Henry

On Wednesday, 25 March 2020 08:49:12 UTC-7, Jad El Houssami wrote:

[Michael Targino]

Hi Henry,

I also could locate the private thread you made reference to. 

Indeed the App Engine team couldn't reproduce this issue you're experiencing. But I can see a lot of communication issues on that thread by the person who was talking to you. More specifically he should've explained that it seemed that these requests didn’t make their way into App Engine stack. The App Engine team wasn’t able to reproduce this issue, then more information is required. 

As you can easily reproduce it. I suggest you to do so, gather additional information such as reproduction url, screenshot or any other information on where/how do you observe the 500s? Can you tell where you are coming from?

Then you can refer to one of the support channels [1] for further technical assistance. 

[1] console.cloud.google.com/support

[Henry Saldyga]

Hi  Michael Targino,

You stated that  App Engine team couldn't reproduce the issue. That means they managed to upload a file to the BLOB storage using BLOB API  on my project. 

Could you please provide the blob-key for this uploaded  file ?

Henry.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a0485fb6-01bc-4d7a-9588-16dbf1a5890a%40googlegroups.com.

[Michael T]

Thanks for your prompt answer, Henry!

The team did try to reproduce the issue on their test environment, not on your project (which we're not allowed to make any changes/updates). Ty

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.

[Henry Saldyga]

Hi Michael,

Thank you for the fast respond too. 

This is actually the key.

The issue occurs in  the one ( old ) project only that already have about 180'000 uploads. 

I  created another project and  uploaded there the same source  code and everything works as expected and how it used to be in the old project for 6 years up to 2020-02-15. 

All those details are provided in the private tracker issue.

Henry

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Henry Saldyga]

Hi Michael,

Also this what I got in private issue from your colleague: 

ba...@google.com added comment #10:
Hi Henry,

There is a hidden quota of 10000 files uploaded per app ( i.e per project). That would explain why you are not facing this error in other projects with the exact same code. Could you check that you are not hitting this quota?

So I'm afraid that when my new project reached 10'000 uploads it will stop working as well. It will be disaster  for us.

Henry

[Manpreet Sidhu (Google Cloud Support)]

Hi Henry,

The Quota of 10,000 files is an App Engine deployment quota. This information is publicly available on our App Engine documentation regarding Quotas[1]. This quota enforces a maximum of 10,000 files per day.

As an alternative to uploading files when deploying App Engine Applications, I suggest that you consider using Cloud Storage[2] with your application[3]. This will allow you to theoretically have millions of files and you will not need to worry about the deployment quotas.

If you wish to keep using this method (not make any changes to your codebase), when your application does face this deployment issue, you can attempt to request a higher quota as mentioned in the documentation[4]. Keep in mind that all quota requests are subject to the approval of engineering, which means that it can be denied.

Regarding your other projects, I cannot tell you why you are not faced with the limitation of 10,000 files. Google Groups are for general questions, product discussions and not in-depth technical issues. As such, I am unable to tell you why it works in the other projects. My hypothesis is that a Quota increase request was made and it was approved, but this is my theory and only that.

TL;DR: Use Cloud Storage. Less stress in regards to quotas.

[1]: https://cloud.google.com/appengine/quotas#Deployments

[2]: https://cloud.google.com/storage

[3]: https://cloud.google.com/appengine/docs/standard/java/using-cloud-storage

[4]: https://cloud.google.com/appengine/quotas#When_a_Resource_is_Depleted

[Henry Saldyga]

Hi Manpreet Sidhu,

Thank you for the clarification. 

We upload about 10'000 files to the BLOB in a MONTH not in a DAY. In a day it's no more that 500 files now days and have a plenty of room while our business is growing. 

It's hard to switch to google Cloud storage  as we have hundreds of application copies that our clients using all around the world. And it's Android application  and Cloud java client library does not have Android support. 

The 500 Internal server started on 2020-02-15  when we  made more than 180'000 uploads  in about 6 years.

If the quota is 10'000 per day  and not in total  as you said  than we do not have what to worry about in term of quota. 

Henry

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/98c5598a-c501-4012-b660-e04283086119%40googlegroups.com.

[Katayoon (Cloud Platform Support)]

Hello Henry,

Looking into your issue report, it seems your issue is not related to the App Engine deployment quota and is related to the Blobstore. So, you may consider the maximum number of files in a single form POST which the limit is 500. I have reopened your issue tracker ticket for further investigation and you can continue with it.

[Henry Saldyga]

Thanks  Katayoon

we are uploading just 1 file per POST request.  About 10'000 files per month. 

Henry

On Tue, 31 Mar 2020 at 10:45, 'Katayoon (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hello Henry,

Looking into your issue report, it seems your issue is not related to the App Engine deployment quota and is related to the Blobstore. So, you may consider the maximum number of files in a single form POST which the limit is 500. I have reopened your issue tracker ticket for further investigation and you can continue with it.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/cb50ade3-c13b-4de0-a0ab-3ac9311157ef%40googlegroups.com.

[Sandeep Koduri]

I was looking at what files/folders has most files. Realized its plugins, I am not sure if I can use plugins from Cloud Storage. 

I dont understand the solution here, if appengine is infrastructure as service and I am paying for resources I use., why do I need special support plan. 

Addressing responses from Google here, Cloud Storage Cant be answer for problem here. And its not App Engine Free Tire issue(we have a billing account).

[yananc]

Hi Sandeep,

According to the documentation [1], Apps can serve static files from a Google Cloud option such as Cloud Storage, serve them directly, or use a third-party CDN. And also, the Google Cloud Platform free tier coverage does not provide quota increase requests, you may have to upgrade to a paid billing account if you would like to use features that are not included in the Free trial, such as GPUs, Windows servers, and quota increases [2].

[1]: https://cloud.google.com/appengine/docs/standard/go111/serving-static-files

[2]: https://cloud.google.com/free/docs/gcp-free-tier#how-to-upgrade

[Sandeep Koduri]

These are plugins which are in PHP not static files. 

We are using a paid account. 

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/uxLbmkWZPqM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/731d2dc7-490d-4c3b-80e5-fa1da8314bbcn%40googlegroups.com.

[George (Cloud Platform Support)]

Hello Sandeep, 

What form do your plugins take? Are these plugins not files? If the plugins are included in your PHP application, you may need to deploy your app again with the new plugins. 

You may use the advantages offered by a paid account to request a quota increase, if this solves the issue. 

Access to resources is controlled by quotas, Cloud Platform resources not being limitless. The purpose of this quota-imposed limitation is to avoid a situation whereby an app uses resources on a large scale, preventing other apps and projects to access the same resources. Everyone pays for the received services, but resources in Cloud Platform, even if plentiful, are not unlimited, hence imposed quotas.