Logs-bases metrics - no data for special json

[Henner Graubitz]

Hi there,

I am using gcloud to write logs (to make it more clear: I am written the records written by big query) using the command:

gcloud logging write --payload-type=json bq-written-record-log "$OUTPUT"

The variable $OUTPUT contains a json file of the form (only the important fields are mentioned)

insertId:  "current-insert-id"
 jsonPayload: {
  . . .
  recordsWritten:  "146225280"
  tableId:  "my-table"
  . . .
  }
  type:  "global"
 }
 timestamp:  "2018-10-01T07:43:17.791103336Z"
}

Successfully I am using Logs-bases metrics to filter all json logs using the filter:

resource.type="global"
logName="projects/my-project/logs/bq-written-record-log"

jsonPayload.tableId="my-table"

All necessary json logs will be displayed when I press "submit filter"


What I want is to get a graphic from my records written daily.

What I did: I filled out the following fields in the Metric Editor:

Name: MyImports
Type: Distribution
Field name: jsonPayload.recordsWritten

Even in the Regexp Editor I can see the values.


But when I save this stuff and Click on "View in Metric Explorer" no data will be displayed. Message is
"No data is available for the selected time frame" - but it should. (I switched to 1w and the log is about 1 day old)


I there a possibility to see the records written in a Logs-bases metrics. What am I do wrong.

Any help, links, whatever  are really appreciated.

Thanks from Germany,


Henner

[Summit Tuladhar]

Hi Henner,

A few things to keep in mind: 

1. logs-based metrics do not backfill - it only starts capturing values after the metric is created
   
2. there is some delay (about a minute or two) when you initially create a metric to when the data is seen in Metrics Explorer
   
3. A distribution metric only captures the statistical distribution of captured values - not the individual values (not directly related to your question, but just making sure it matches to your requirements)
   

I think the issue may be either 1. or 2. If that's not the case, we will need information about your project. It would be easier to debug this further if you could file a support ticket or a feedback using various ways that are documented in https://cloud.google.com/stackdriver/docs/getting-support.

Regards,

Summit

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/34de4c1e-5874-47fd-8924-04f6b136a96f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henner Graubitz]

Hi Summit,

thanks for your information.

Acutally step one and step two are not problems.

I played around with counter, as I think this would be the solution. With counter as type I can see datapoints, but only between 1 or nothing.

Idea is to "track" the number of records in the Metric Explorer, e.g.

01.01.2000 - Point on 200.453M (for 200.435.000 entries written in BigQuey)

02.01.2000 - Point on 199.632M (for 200.435.000 entries written in BigQuey)

03.01.2000 - Point on 201.410M (for 200.435.000 entries written in BigQuey)

and so on

At the moment I am only able to write

01.01.2000 - Point on 1

02.01.2000 - Point on 1

03.01.2000 - Point on 1

The unit in counter is set to 1, but only 0 or 1 is allowed. I would more prefer something like "set the unit in counter to the record written"

My problem is similar to https://cloud.google.com/blog/products/gcp/extracting-value-from-your-logs-with-stackdriver-logs-based-metrics: they use "Distribution" to track the number of burger they sold (not zero or one to say: yes, I sold a burger,

But maybe there is a trick behind. Maybe somebody else has an idea. Thanks again,

Greetings from Germany,

Henner

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[Summit Tuladhar]

On Tue, Oct 9, 2018 at 9:50 AM Henner Graubitz <henne...@gmail.com> wrote:

Hi Summit,

thanks for your information.

Acutally step one and step two are not problems.

I played around with counter, as I think this would be the solution. With counter as type I can see datapoints, but only between 1 or nothing.

- How many matching log entries were there? 

- What Aligner are you using in Metrics Explorer? 

If you want the SUM of counts over an alignment period, you need to use SUM aligner. You may be using COUNT, which just counts the number of timeseries. 

 

Idea is to "track" the number of records in the Metric Explorer, e.g.

01.01.2000 - Point on 200.453M (for 200.435.000 entries written in BigQuey)

02.01.2000 - Point on 199.632M (for 200.435.000 entries written in BigQuey)

03.01.2000 - Point on 201.410M (for 200.435.000 entries written in BigQuey)

and so on

At the moment I am only able to write

01.01.2000 - Point on 1

02.01.2000 - Point on 1

03.01.2000 - Point on 1

The unit in counter is set to 1, but only 0 or 1 is allowed. I would more prefer something like "set the unit in counter to the record written"

For a counter logs-based metric, the unit should always be 1.

 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/34de4c1e-5874-47fd-8924-04f6b136a96f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/a48d52da-149a-4cc1-997d-8b6d30de82da%40googlegroups.com.

[Henner Graubitz]

Hi Summit,

I solved it. Compared to my information before here is the solution (also for other user):

1. Create the filter for the JSON-Logs (keep in mind I want to have the records written by big query): e.g.

resource.type="global"
logName="projects/my-project/logs/bq-written-record-log"
jsonPayload.tableId="my-table"


2. Type in a Name on the right site, e.g. "Records Written"

3. Units (REALLY IMPORTANT): "number"

4. Type: Distribution

5. Field Name: jsonPayload.recordsWritten

6. RegExp: recordsWritten: (\d+)

If you save the metric, write a new entry, wait a minute und go ot your logs-based metrics, call "View in metric explorer"

Switch "Line" to "Stacked Bar" and you should see a nice bar with the height of the records written.

Perfect. Thanks once again for your help.

Greetings from Germany,

Henner

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/34de4c1e-5874-47fd-8924-04f6b136a96f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.