Is there a limit to the number of log entries per request?

[Stephan]

Hi there

Well, subject says it all: I have a request (on the backend), which potentially produces a high number of log entries (multiple of hundreds of lines).

Besides the question whether it is a good style to generate so many log entries: Is there a limit that I can hit? 

Cheers

Stephan

[Robert Kluin]

This is exactly the type of thing I'd love to see included in the big list of limits. Please star issue 5677.
http://code.google.com/p/googleappengine/issues/detail?id=5677

I can tell you I've logged a lot of lines in backends tasks without issue. I'm not sure exactly how many, but several hundred would probably work.  Keep in mind, if you're doing this a lot you're going to fill your log buffers more quickly.

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/IDekOw0k_G4J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

--
------
Robert Kluin
Ezox Systems, LLC

[Brandon Wirtz]

If you are logging lots of things don’t use logs. Especially if you want to look at them later. Logs are not permanent, Not guaranteed, and not something you should make a dependency.

 

PS

Disabling logging speeds our app up quite a bit (30-50 ms per request)

[Robert Kluin]

On Wednesday, January 25, 2012, Brandon Wirtz <dra...@digerat.com> wrote:
> If you are logging lots of things don’t use logs.

This is great, until you need to debug a complex process.  Better advice is probably to use logging intelligently, and be sure you're using appropriate logging levels. I think too much logging is generally obvious: if it is noise that is hard to filter through and doesn't aid in debugging, it is of no value.

> Especially if you want to look at them later. Logs are not permanent, Not guaranteed, and not something you should make a dependency.
>

This is true. If you want a permanent record you need to periodically download your app's logs.

>  
>
> PS
>
> Disabling logging speeds our app up quite a bit (30-50 ms per request)
>

While it is true that logging adds some overhead, he's talking about a backend task. So it is presumably a longer-running more involved process. Logging is great in these cases because it allows you to watch the process running live. I find this immensely valuable.


Robert

[Brandon Wirtz]

We use logs some, but we find that for the really critical stuff or really detailed stuff we are happier with writes to the datastore.

 

I didn’t really mean don’t log for debugging, I meant consider Logging to datastore.

 

Logging also changes results sometimes…

 

Don’t ask me why.

 

But for example we used to do performance numbers like

 

 

Log time and what we are about to do.

Do something

Log time and what we just did.

 

Do some things.

 

Log time and what we are about to do.

Do something

Log time and what we just did.

 

 

Then we switched to

Variable = time and what we are going to do

Do something

Variable .= time and what we just did.

 

 

Then the last line before Return is either log it, or write it to the data store

[Stephan]

Hi all

On Wednesday, January 25, 2012 8:23:11 AM UTC+1, Robert Kluin wrote:

[...] 

While it is true that logging adds some overhead, he's talking about a backend task. So it is presumably a longer-running more involved process. Logging is great in these cases because it allows you to watch the process running live. I find this immensely valuable.


Robert

This is correct, I have a long running task on a backend, which processes hundreds of items. Each item generates only a few lines in the log, but with hundreds of items, I apparently hit an unknown limit. I could think about spliiting the task in small subtasks, but I don't see the benefit yet other than getting below the unknown log limit.

Thanks

Stephan

PS: starred the issue- thanks 

[Robert Kluin]

Hey Stephan,
Mike Wesner and I were actually just discussing this earlier today.
He also mentioned there being some limit on log entries, we couldn't
recall exactly what it was though. I just ran a simple test; it seems
that only the last 1000 lines are kept. At least that is what it
looks like to me with my test script:

import logging
for i in xrange(1500):
logging.info(i)

Results: first line logged was 500, last line logged was 1499.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/_SlFWr3v6PgJ.