How to download logs from a backend?

[Dave Loomer]

The docs for downloading logs make no specific mention of backends,
and from my attempts it seems that you can only download logs for your
frontend. That would be strange though, and to make matters worse a
web search for "google app engine download backend logs" (no quotes
obvs.) reveals nothing relevant. Either I'm a true pioneer here or no
one else has had any troubles with this -- or no one else cares. Here
is what I've tried.

First, the documented syntax (which gets you the frontend logs - no
issues here):
appcfg.py request_logs myapp/ mylogs.txt

So to get the backend logs I have tried:

--------
appcfg.py backends request_logs myapp/ mylogs.txt <backend-name>
(Returns the error: appcfg.py: error: Expected a <directory> and
<action> argument.)

--------
appcfg.py backends request_logs myapp/ <backend-name> mylogs.txt
(As expected, returns the same error: appcfg.py: error: Expected a
<directory> and <action> argument.)


appcfg.py seems confused by "backends request_logs" so I then tried,
without the "backends" argument:
--------
appcfg.py request_logs myapp/ <backend-name> mylogs.txt
(Returns the error: appcfg.py: error: Expected a <directory> argument
and an <output_file> argument.)

--------
appcfg.py request_logs myapp/ mylogs.txt <backend-name>
(Same thing: returns the error: appcfg.py: error: Expected a
<directory> argument and an <output_file> argument.)


And there is also the documented -vhost to restrict by domain:
--------
appcfg.py --vhost=http://<backend-name>.<app-name>.appspot.com
request_logs myapp/ mylogs.txt
(console just shows that is downloading a frontend version number, and
the result is that I get nothing downloaded, probably because there
are no requests going to my frontend using my backend host name.)


So ... did Google really leave this out, or what am I missing?

[Amy Unruh]

Dave, 

The same logs include both backend and frontend instance logging- they are not separated.  You could be experiencing a delay before long-running backend log writes get flushed.

You might be interested in the LogService API,

 http://code.google.com/appengine/docs/python/backends/logserviceapi.html

which lets you periodically flush logs during long-running requests, and to examine an application's request logs and application logs.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Dave Loomer]

Amy, I don't think it has anything to do with flushing or delays, as
even when I run with the --num_days=2 option, I still only get
requests for frontend objects (HTML, images, etc.) from the past two
days. In reality, since my app is not yet "launched" I probably have
100x as many log entries for backends as for frontends in the admin
console. I use logging.info quite liberally, and can find plenty of
Info-level entries in the admin console from my backends, so this
seems unrelated to the --severity option as well (and I have tried
playing around with severity levels to no avail).

What additional information would you like for me? The exact command I
issue from the shell is:

appcfg.py --no_cookies --email=<mye...@domain.com> --passin --
num_days=2 request_logs "/<path>/mn-live" "/log.txt"

[Dave Loomer]

I should add that I don't have any truly long-running backend
processes -- typically, they all complete in a few minutes, and I run
thousands per day.

On Jan 18, 12:19 pm, Dave Loomer <dloo...@gmail.com> wrote:
> Amy, I don't think it has anything to do with flushing or delays, as
> even when I run with the --num_days=2 option, I still only get
> requests for frontend objects (HTML, images, etc.) from the past two
> days. In reality, since my app is not yet "launched" I probably have
> 100x as many log entries for backends as for frontends in the admin
> console. I use logging.info quite liberally, and can find plenty of
> Info-level entries in the admin console from my backends, so this
> seems unrelated to the --severity option as well (and I have tried
> playing around with severity levels to no avail).
>
> What additional information would you like for me? The exact command I
> issue from the shell is:
>

> appcfg.py --no_cookies --email=<myem...@domain.com> --passin --

> num_days=2 request_logs "/<path>/mn-live" "/log.txt"
>
> On Jan 15, 7:00 pm, Amy Unruh <amyu+gro...@google.com> wrote:
>
>
>
>
>
>
>
> > Dave,
>

> > The samelogsinclude both backend and frontend instance logging- they are

> > not separated.  You could be experiencing a delay before long-running
> > backend log writes get flushed.
> > You might be interested in the LogService API,
> >  http://code.google.com/appengine/docs/python/backends/logserviceapi.html

> > which lets you periodically flushlogsduring long-running requests, and to

> > examine an application's requestlogsand applicationlogs.
>
> > On Sun, Jan 15, 2012 at 3:27 AM, Dave Loomer <dloo...@gmail.com> wrote:

> > > The docs for downloadinglogsmake no specific mention of backends,
> > > and from my attempts it seems that you can only downloadlogsfor your

> > > frontend.  That would be strange though, and to make matters worse a

> > > web search for "google app engine download backendlogs" (no quotes

> > > obvs.) reveals nothing relevant. Either I'm a true pioneer here or no
> > > one else has had any troubles with this -- or no one else cares.  Here
> > > is what I've tried.
>

> > > First, the documented syntax (which gets you the frontendlogs- no

> > > issues here):
> > > appcfg.py request_logs myapp/ mylogs.txt
>

> > > So to get the backendlogsI have tried:

[Amy Unruh]

My earlier post was incorrect-- sorry!  You do need to specifically request the logs for a backend.  You can do that by passing the backend name as the version name, e.g.:

 appcfg.py request_logs --version=<backend_name> <project_dir> <outfile>

You can add the --include_all flag to see details of which instance handled the request.

To see the logs for a particular backend instance only, you can filter by its vhost. E.g., to see the logs for the 2nd instance of a backend named 'worker', this should work:

 appcfg.py request_logs  --version="worker" --vhost="2.worker.<your_appid>.appspot.com" <project_dir> <outfile>

[Dave Loomer]

This is great - thanks Amy!

On Jan 18, 7:43 pm, Amy Unruh <amyu+gro...@google.com> wrote:
> My earlier post was incorrect-- sorry!  You do need to specifically request
> the logs for a backend.  You can do that by passing the backend name as the
> version name, e.g.:
>
>  appcfg.py request_logs --version=<backend_name> <project_dir> <outfile>
>
> You can add the --include_all flag to see details of which instance handled
> the request.
> To see the logs for a particular backend instance only, you can filter by
> its vhost. E.g., to see the logs for the 2nd instance of a backend named
> 'worker', this should work:
>  appcfg.py request_logs  --version="worker" --vhost="2.worker.<your_appid>.
> appspot.com" <project_dir> <outfile>
>
>
>
>
>
>
>
> On Thu, Jan 19, 2012 at 5:19 AM, Dave Loomer <dloo...@gmail.com> wrote:
> > Amy, I don't think it has anything to do with flushing or delays, as
> > even when I run with the --num_days=2 option, I still only get
> > requests for frontend objects (HTML, images, etc.) from the past two
> > days. In reality, since my app is not yet "launched" I probably have
> > 100x as many log entries for backends as for frontends in the admin
> > console. I use logging.info quite liberally, and can find plenty of
> > Info-level entries in the admin console from my backends, so this
> > seems unrelated to the --severity option as well (and I have tried
> > playing around with severity levels to no avail).
>
> > What additional information would you like for me? The exact command I
> > issue from the shell is:
>

> > appcfg.py --no_cookies --email=<myem...@domain.com> --passin --