Google CDN log parameters

[Stevo Novkovski]

We export Google CDN logs into our cloud storage for future parsing and calculations. However, we cant really understand this parameters and i think some parameters are missing.

Typical export log is like: 

{"httpRequest":{"cacheHit":true,"cacheLookup":true,"referer":"http://referer.com/","remoteIp":"1.1.1.1","requestMethod":"GET","requestSize":"1285","requestUrl":"https://cdn.site.com/image.jpg","responseSize":"76","status":304,"userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0"},"insertId":"111111111","log":"requests","metadata":{"labels":{"network.googleapis.com/forwarding_rule_name":"forwarding_rule_name","network.googleapis.com/target_proxy_name":"target_proxy_name","network.googleapis.com/url_map_name":"url_map_name"},"projectId":"project-id","serviceName":"network.googleapis.com","severity":"INFO","timestamp":"2016-12-23T02:59:58.676436503Z","zone":"global"},"structPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry","statusDetails":"response_from_cache"}}

Questions:

1. Why sometimes we have log with:

• Only "cacheLookup":true
• Only "cacheHit":true
• "cacheHit":true,"cacheLookup":true
  
• "cacheFillBytes":"85435","cacheHit":true,"cacheLookup":true
  

Can anyone explain meaning of this parameters.

2. Google CDN have cache fill from origin and cache-to-cache fill. How you can know if the cache fill is from origin or from other cache. Also how you can identify cache-to-cache location just as Google bills us:

Compute Engine Network CDN Cache Fill from Europe to Other Destinations: 1 Gibibytes

Compute Engine Network CDN Cache Fill from Europe to North America: 1 Gibibytes

Cloud CDN North America Intra-Region Cache Fill: 1 Gibibytes

[Faizan (Google Cloud Support)]

Hello Stevo,

On Monday, January 2, 2017 at 5:48:00 PM UTC-5, Stevo Novkovski wrote:

We export Google CDN logs into our cloud storage for future parsing and calculations. However, we cant really understand this parameters and i think some parameters are missing.

Typical export log is like: 

{"httpRequest":{"cacheHit":true,"cacheLookup":true,"referer":"http://referer.com/","remoteIp":"1.1.1.1","requestMethod":"GET","requestSize":"1285","requestUrl":"https://cdn.site.com/image.jpg","responseSize":"76","status":304,"userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0"},"insertId":"111111111","log":"requests","metadata":{"labels":{"network.googleapis.com/forwarding_rule_name":"forwarding_rule_name","network.googleapis.com/target_proxy_name":"target_proxy_name","network.googleapis.com/url_map_name":"url_map_name"},"projectId":"project-id","serviceName":"network.googleapis.com","severity":"INFO","timestamp":"2016-12-23T02:59:58.676436503Z","zone":"global"},"structPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry","statusDetails":"response_from_cache"}}

Questions:

1. Why sometimes we have log with:

• Only "cacheLookup":true
• Only "cacheHit":true
• "cacheHit":true,"cacheLookup":true
  
• "cacheFillBytes":"85435","cacheHit":true,"cacheLookup":true
  

Can anyone explain meaning of this parameters.

The details on cacheLookup, cacheHit and cacheFillBytes entries can be found on this link.

 

2. Google CDN have cache fill from origin and cache-to-cache fill. How you can know if the cache fill is from origin or from other cache. Also how you can identify cache-to-cache location just as Google bills us:

Compute Engine Network CDN Cache Fill from Europe to Other Destinations: 1 Gibibytes

Compute Engine Network CDN Cache Fill from Europe to North America: 1 Gibibytes

Cloud CDN North America Intra-Region Cache Fill: 1 Gibibytes

If you are serving using cacheable content the network egress charges for your GCE instance will be replaced by cache fill and cache egress charges, as stated on this link:

Cache fill and cache egress charges apply to cacheable content. On cache hits, you pay for cache egress bandwidth. On cache misses, you additionally pay for cache fill bandwidth. These charges replace the network egress charges that apply when serving directly from Compute Engine.

I hope that helps.

Faizan

[Stevo Novkovski]

My questions were other.

Let me ask on other way.

1. cacheHit : Whether or not an entity was served from cache (with or without validation). 

• What is with or without validation?
  
• How can request with cacheFillBytes: true ( request wasnt cached that`s why cacheFill is performed) also can have cacheHit:true ?

2. cacheLookup: When this is true? From what i understand that is a request when load balancer try to get data from origin and if request is cacheable, cache the entry for next requests? If that is how is working, how can request have both cacheHit and cacheLookup true?

3. How you can know what cache was filled from cache-to-cache not from origin?

4. responseSize is always int64 but i have logs with null on 200 response codes. http://prntscr.com/drpbo2  

[Faizan (Google Cloud Support)]

Hello Stevo,

Sorry for the misunderstanding, I'll find answer to your questions and will get back to you by next week.

Faizan

[Faizan Hussain]

Hello Stevo, 

Thank you for your patience. I don't have any useful information to share with you at this point of time. I'm working with the backend team to get answer to your questions. I'll get in touch with you by next week. 

Faizan 

[Faizan (Google Cloud Support)]

Hello Stevo,

You can find the answer to your questions below:

1. cacheHit : Whether or not an entity was served from cache (with or without validation). 

Validation in regards to CDN is the determination of whether or not the cached content is still valid to be served, or if it needs to be downloaded again. You can refer to this link [1] which describes validation in an easy to understand way. 

2. cacheLookup: When this is true? From what i understand that is a request when load balancer try to get data from origin and if request is cacheable, cache the entry for next requests? If that is how is working, how can request have both cacheHit and cacheLookup true? 

Both are true when the client made a request that could be cacheable, and there was a cached object that was returned. You can refer to this [2] link for more information. 

3. How you can know what cache was filled from cache-to-cache not from origin?

For this you can look at google::logging::type::HttpRequest which is available through GCLB/CDN Stackdriver logs. If field HttpRequest.cache_fill_bytes > 0 then it's a cache fill. Now to identify whether this was cache-to-cache fill or not there is no direct way but you can look at server_ip field. if HttpRequest.server_ip IS NOT populated and HttpRequest.cache_fill_bytes > 0 then it's cache-to-cache fill otherwise it's cache fill from origin. 

4. responseSize is always int64 but i have logs with null on 200 response codes. http://prntscr.com/drpbo2 

A responseSize of 0b with 'cacheHit:True' means that the client was able to fetch the data from cache, so the server did not need to respond with the data. 

I hope that answer your questions. Let me know if you have further questions or concerns.

Faizan

[1] - https://symfony.com/doc/current/http_cache/validation.html

[2] - https://lyte.id.au/2014/08/28/x-cache-and-x-cache-lookupheaders/

On Friday, January 13, 2017 at 12:52:36 PM UTC-5, Faizan (Google Cloud Support) wrote:

Hello Stevo, 

Thank you for your patience. I don't have any useful information to share with you at this point of time. I'm working with the backend team to get answer to your questions. I'll get in touch with you by next week. 

Faizan 

On Fri, Jan 6, 2017 at 1:28 PM, Faizan (Google Cloud Support)

Hello Stevo,