Logging API: /v2/entries:list is crazy slow, is that expected?

24 views
Skip to first unread message

Ryan B

unread,
Jun 4, 2020, 3:30:17 PM6/4/20
to Google Stackdriver Discussion Forum
Hi all! I migrated from the App Engine Python 2 runtime to Stackdriver Logging a bit ago, and it's mostly great, but I've hit a few quirks. One big one is that reading from the logging API seems crazy slow. I'm using the google-cloud-logging pip package, and when i instantiate a LoggingServiceV2Client and call list_log_entries() - which I'm guessing uses the /v2/entries:list API call - it usually takes a minute or more. For example, a recent call that returned 21 entries took ~330 seconds. Is this expected?!

My code looks like this:

logging_client = LoggingServiceV2Client()
...
for log in logging_client.list_log_entries(('projects/brid-gy',), filter_=query):
  ...

where query is of the form:

logName="projects/brid-gy/logs/stdout" AND trace="projects/brid-gy/traces/72f7ab04fb705cb891c1e96654660bc6"

Thanks in advance!

Igor Peshansky

unread,
Jun 4, 2020, 3:32:19 PM6/4/20
to Ryan B, Mary Koes, Google Stackdriver Discussion Forum

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/7b5c1f82-1033-4522-8bb2-0cfe76ddbe7fo%40googlegroups.com.

Ryan B

unread,
Jun 4, 2020, 4:22:20 PM6/4/20
to Google Stackdriver Discussion Forum
Thanks Igor! Just FYI, you cc'ed Mary the same way on my last question about the logging API, and she never replied.


On Thursday, June 4, 2020 at 12:32:19 PM UTC-7, Igor Peshansky wrote:
On Thu, Jun 4, 2020 at 3:30 PM Ryan B <goo...@ryanb.org> wrote:
Hi all! I migrated from the App Engine Python 2 runtime to Stackdriver Logging a bit ago, and it's mostly great, but I've hit a few quirks. One big one is that reading from the logging API seems crazy slow. I'm using the google-cloud-logging pip package, and when i instantiate a LoggingServiceV2Client and call list_log_entries() - which I'm guessing uses the /v2/entries:list API call - it usually takes a minute or more. For example, a recent call that returned 21 entries took ~330 seconds. Is this expected?!

My code looks like this:

logging_client = LoggingServiceV2Client()
...
for log in logging_client.list_log_entries(('projects/brid-gy',), filter_=query):
  ...

where query is of the form:

logName="projects/brid-gy/logs/stdout" AND trace="projects/brid-gy/traces/72f7ab04fb705cb891c1e96654660bc6"

Thanks in advance!

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

Mary Koes

unread,
Jun 4, 2020, 5:05:04 PM6/4/20
to Ryan B, Google Stackdriver Discussion Forum
Hi Ryan,

Apologies for the lack of response. Our support here is best effort and I just missed your last message. :-) 

To your first message, 
Hi all! I have an app with bursty read usage of the Logging API, and recently it's been hitting the 1qps entries.list quota pretty regularly. I went to request an increase on the IAM Quotas page, but the checkboxes for the logging read APIs are all disabled, so I can't select them to edit and request an increase.

I expect this is intentional on Google's end. Has anyone else hit this? Any ideas for workarounds? Or other ways to request an increase? 

Do you have any more information on your use case? 
Yes, this is intentional on our side - logs can be quite voluminous so we recommend exporting logs via the logs router if you need to access large volumes of logs. Two possibilities:
  • make sure you have the number of logs being returned set to the max (1000 I believe)
  • you can request an increase by filing a ticket with support.
To your second message:
I agree that 330 seconds is slow but it depends on volume. If we're search a PB of logs, that might be reasonable, for example. Looking at your query, 
logName="projects/brid-gy/logs/stdout" AND trace="projects/brid-gy/traces/72f7ab04fb705cb891c1e96654660bc6"
it's good practice to include the logName. You can also speed up the search by adding the resource type and a time range. For anything more specific, we'd likely need to look at your project which would require a support ticket. 

Hope that helps,

Mary


On Thu, Jun 4, 2020 at 4:22 PM Ryan B <goo...@ryanb.org> wrote:
Thanks Igor! Just FYI, you cc'ed Mary the same way on my last question about the logging API, and she never replied.

On Thursday, June 4, 2020 at 12:32:19 PM UTC-7, Igor Peshansky wrote:
On Thu, Jun 4, 2020 at 3:30 PM Ryan B <goo...@ryanb.org> wrote:
Hi all! I migrated from the App Engine Python 2 runtime to Stackdriver Logging a bit ago, and it's mostly great, but I've hit a few quirks. One big one is that reading from the logging API seems crazy slow. I'm using the google-cloud-logging pip package, and when i instantiate a LoggingServiceV2Client and call list_log_entries() - which I'm guessing uses the /v2/entries:list API call - it usually takes a minute or more. For example, a recent call that returned 21 entries took ~330 seconds. Is this expected?!

My code looks like this:

logging_client = LoggingServiceV2Client()
...
for log in logging_client.list_log_entries(('projects/brid-gy',), filter_=query):
  ...

where query is of the form:

logName="projects/brid-gy/logs/stdout" AND trace="projects/brid-gy/traces/72f7ab04fb705cb891c1e96654660bc6"

Thanks in advance!

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/a6012132-bc79-49df-97b9-a749324b455ao%40googlegroups.com.

Ryan B

unread,
Jun 4, 2020, 6:18:34 PM6/4/20
to Google Stackdriver Discussion Forum
Hi Mary! No worries, I figured, totally ok. Thanks for the info!

I'm using the logging API for an interactive user-facing feature that only reads small amounts of logs at a time, so exporting wouldn't work. And I'm a small customer, on the bronze support package, so I can't actually file support tickets, only billing tickets. Please do let me know if there's any other way to request an increase. Not a big deal if not.

I only have 5-20GB or so of logs at any given time, so the total volume these calls search seems manageable? Definitely nowhere near PB. The trace filter already limits them to only return ~10-200 or so entries on average, but I'll try adding the type and time filters you mentioned. Thanks for the suggestion!
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

Ryan B

unread,
Jun 7, 2020, 12:44:31 PM6/7/20
to Google Stackdriver Discussion Forum
Adding timestamp filters fixed this! Filtering on resource type didn't have an effect, but filtering timestamp to just the few minutes around the request sped up the fetch from multiple minutes to effectively instantaneous. You can see the code here. Thank you Mary!

Mary Koes

unread,
Jun 7, 2020, 9:02:15 PM6/7/20
to Ryan B, Google Stackdriver Discussion Forum
Awesome--thank you for sharing your code as well!

Best,

Mary

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/abc00c5c-7722-4b4d-ac83-4ea8337e2104o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages