no response headers from Google Frontend (or not paying for them), denial of wallet attacks

[Waltraud Siller]

Hi,

in LogsExplorer I see by a request (protoPayload):

responseSize: "95"

cost: 1.0617e-8

I guess it is byte and dollar, from: outgoing bandwidth cost of 0.12$/gb.

This request has 0 content length and I tried to remove each and every response headers via Jetty. Still Google Frontend adds response headers and I guess these are counted towards my bill too, even if I do not see how I could stop App Engine to add these response headers. In Chrome developer console I see following response headers:

1. content-length:0
2. content-type:text/html
3. date:Sun, 13 Jun 2021 12:27:59 GMT
4. server:Google Frontend
5. x-cloud-trace-context:01c5ef694*******

My question(s): Do browsers need these response headers? Can I ask Google Frontend to skip all headers? (app.yaml or so). If not, why is it counted towards my bill? Please do not say it is not much, because if it is sooo little, you might as well not count this 100 bytes...

It seems odd to me that the system adds data to a response (if it is not needed for the browser) and I have to pay for it.

How is this relevant? I try to minimise my costs associated with a future denial of wallet like attack. GET requests to static resources may cause a huge bill through outgoing bandwidth / gb, which is a big vulnerability I believe so.

In the process of minimizing data sent for first GET requests to /, I realized I pay for these unwanted response headers which might not be much in case of an attack relative to data I must send the first time, still it is somehow disturbing :)

If you could, I would also be happy to get some suggestions how I can defend against denial of wallet attacks. I do not find anything useful.

Actually I have a system now and to be honest if I send 500 bytes of data the extra 100 bytes will not make a lot of difference, still now I am curious why I have to pay for the data if it is just garbage (I do not know at this point, it seems to be garbage to me cause I do not need it) and I do not want to send it to the client.

Thanks!

[George (Cloud Platform Support)]

Hello, 

You are right, some headers are added, as a service to the app. This is documented in the App Engine-specific headers sub-chapter. However, the headers you mention are not added, but removed from request responses. You are right, outgoing traffic is counted towards your bill. 

This discussion group is oriented more towards general opinions, trends, and issues of general nature touching App Engine and Cloud SQL. For coding and programming architecture, as well as adding or removing headers, you may be better served in dedicated forums such as stackoverflow, where experienced programmers are within reach and ready to help. 

[Waltraud Siller]

Hi!

Thanks for the reply.

The incoming request headers are somewhat manipulated and you are right, for instance "X-Forwarded-For" is added and as a service to my app, it tells me the source request ip address. Then my app and my jetty web server handles the request. I remove all response headers and send the reply. It goes through Google Frontend again. They would remove "server" and such if I set them and instead they add these 5 response headers to the reply. Even if I do not want it. Moreover, I do not see how I could tell Google Frontend from my application that it should not add any response headers.

It makes sense that they set these response headers, normally. The problem is, I cannot tell from my App Engine application not to set them, in special cases. And still, these 100 bytes are counted towards my bill. My app is public and if I get a DDoS like attack, even if I served zero content, App Engine adds these 100 byte response headers to hundreds of request per second(!) and I am going to be billed for it.

It is not a question for stackoverflow because I am 95% sure now I cannot change this behavior of App Engine... my question is

1. is it true or there exists an option to make App Engine (Google Frontend) behave exactly as I want it (in special cases, skip adding these aforementioned response headers... I tested it, I send absolutely zero content and headers and these 5 are added before the response leaves Google Frontend... it is client -> GoogleFrontend -> my app with my embedded jetty server -> Google Frontend (here happens the unwanted byte-adding) -> client)

2. then I guess I would have a feature request to be able to tell App Engine that I want a completely header free response for security/financial reasons

It would be also ok for me if I werent billed for these bytes that are always added.

Security and request handling is a common responsibility in App Engine. These added response headers are a (financial) security threat in serverless architectures like the pay as you go App Engine.

https://arxiv.org/pdf/2104.08031.pdf

I myself cannot really close this attack surface with session management because the first requests to "/" are inherently public. I think it is somewhat theoretical because 100 bytes are not that much, still it makes my wallet bleed in case of an http flood attack. For no good reasons, apparently. Or for none that I am aware of.

Could you please point me somewhere if this is not the right place where could I get an explanation why I am forced to play extra dollars/euros in case of an attack? Or where can I make a feature request?

My suggested solution:

1. I should be able to tell Google Frontend to skip adding anything to the requests or at least to the inherently public request to "/"

2. or if they add these response headers (100 bytes) for no good reason they should not bill me for them

Thanks!

[Shawn Wu]

 as you mentioned it may  DDos attack, please go cloud console to file support case, it will allow us to look into the issue specific based on your logs and service behavior, to assist you fix the issue.