Parsing logs from GKE Ingress through StackDriver to BigQuery

258 views
Skip to first unread message

Tim Child

unread,
Oct 23, 2017, 9:02:21 AM10/23/17
to Google Cloud Developers
Hi!

Currently, I have GKE cluster with a fairly standard ingress running nginx, and this is logging to StackDriver. I wish to then take the logs and perform some analysis on them using BigQuery.

The logs are coming into StackDriver looking like:

{
insertId: "dbu6vtg2qwb8qg"  
labels: {
 compute.googleapis.com/resource_name: "gke-test-service-default-pool-3ffbb053-jvdd"  
  container.googleapis.com/namespace_name: "default"  
  container.googleapis.com/pod_name: "ingress-nginx-3490004333-123456"  
  container.googleapis.com/stream: "stdout"  
 }
logName: "projects/service-test/logs/ingress-nginx"  
receiveTimestamp: "2017-10-23T12:01:06.567436153Z"  
resource: {
 labels: {
  cluster_name: "test-service"    
  container_name: "ingress-nginx"    
  instance_id: "8940435729859120887"    
  namespace_id: "default"    
  pod_id: "ingress-nginx-3490004333-3493l"    
  project_id: "service-test"    
  zone: "europe-west3-a"    
 }
 type: "container"  
 }
severity: "INFO"  
textPayload: "1.2.3.4 - [1.2.3.4] - - [23/Oct/2017:12:01:02 +0000] "POST /API/search/v1/search/?per_page=50 HTTP/2.0" 200 62528 "https://test.service.cloud/search" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38" 693 10.773 [default-gateway-80] 10.48.8.76:8000 402456 10.771 200
"  
timestamp: "2017-10-23T12:01:02Z"  
}

The information that I want to run BigQuery against is really in the textPayload, is this correct from StackDriver and I should somehow parse the information I want from the textPayload in BigQuery? Or should I be using another service to do that? 

Thanks,

Tim.




Carlos (Cloud Platform Support)

unread,
Oct 23, 2017, 4:37:04 PM10/23/17
to Google Cloud Developers

Hi Tim,


If you are already getting the information on Stackdriver logging, you can create a sink service to get those logs exported to BQ.


Reply all
Reply to author
Forward
0 new messages