MQL Query to pull data between top 500-1000 rows

[Chavya Khanna]

Hello folks,

Curious is there a way in MQL if we can pull just specific number of rows?

like between 500-1000, etc?

[Haskell Garon]

Hi,

Both MQL and PromQL support Top and Bottom operators and should work in our metrics explorer, dashboard builder, and other interfaces:

https://cloud.google.com/monitoring/mql/examples#qlx-top

https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators (see bottomk, topk)

Best,

Haskell Garon
Product Manager
Google Cloud 

[Chavya Khanna]

Hi @Haskell

Apologies for delayed response, but just a top or bottom won't help in this situation.

As we won't know number of rows returned by the query in real time, it can be 100, 500 , 900, 1000 or 1500.

Now since Grafana - Stackdriver Plugin has a limitation of 1000 rows per API Call.  We were thinking of processing Query Data in bits - 1000 records per request so we will have two queries one query will return 1000 and next query to return the remaining number of rows - which will be from row number 1001 to 1500 - That way we can avoid overlapping of data. In Regular SQL we'd use SELECT * FROM your_table

ORDER BY some_column LIMIT 500 OFFSET 500;

LIMIT and OFFSET to capture number of records we are interested in, is something we can do in MQL?

[Lee Yanco]

Hey Chayva,

This isn't really a common use case for time series databases and therefore we don't have any specific function for this. Generally speaking you're not pulling out raw rows from a TSDB. The time component - and the fact that individual time series often don't persist for long periods over time - makes doing something like this messy.

If you're trying to export data out of our system and into another, you can either set up a service to poll the ListTimeSeries API regularly and parse the results, or you can use the open-source stackdriver_exporter to do something similar, but in Prometheus format. You'd pull the raw data every minute or so and then store the results in whatever DB you want.

If you can speak a little more as to what you want to do with the raw data I might be able to provide some more pointers.

--
© 2021 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/c12eb039-6097-442b-b410-c666061a8288n%40googlegroups.com.

--

Lee Yanco Senior Product Manager lya...@google.com

[Igor Peshansky]

Chavya,

If you're using the Monitoring API to query, the results are paged (customizable by the supplied pageSize parameter), and you can issue follow-up calls with the returned pageToken until you've retrieved all of the rows: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/query .

This may be something that the Grafana/Stackdriver plugin already knows how to do, or a feature that could be implemented.

Hope this helps.

        Igor

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CA%2BjoHoPOKSncCFTT8E88PWTkuuQnkN7SMXuwYVO8box50BM-DA%40mail.gmail.com.