Log Parser for downloaded request_logs

[Matthew Blain]

Happy Monday!

I recently posted a tool which lets you import the request logs
downloaded from the App Engine Admin Console into a sqlite3 database.
This is useful for ad-hoc analysis and aggregation. It also
understands the logs which you as a developer wrote with logging.error
or similar methods--I am not sure if any of the existing Apache
Combined Log Format parsers understand that extension, it would be
great if they did. You can also describe a custom column on those
logs.

You can view it at
http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py
You can download using a subversion client, or directly at
http://google-app-engine-samples.googlecode.com/svn/trunk/logparser/logparser.py

Here are some examples from the docstring of the tool:

Example:

# Download request logs into requests.txt
appcfg.py request_logs --severity 0 --include_all <appdirectory>
requests.txt
# Run the logparser to insert them into requests.db.
# (You can specify multiple input files.)
logparser.py --db requests.db requests.txt
# Query them using the sqlite3 interactive command line interface.
sqlite3 requests.db
sqlite> -- what are the most common 404s?
sqlite> select distinct request_line, count(*) from requests
...> where status == 404 group by request_line order by
request_line desc;
sqlite> -- what requests see 'Deadline Exceeded'?
sqlite> select distinct request_line from requests
...> where applog like '%DeadlineExceeded%';
sqlite> -- How many loading requests were seen?
sqlite> select count(*) from requests where loading_request=1;
sqlite> -- What was the average cpm across all pages?
sqlite> select sum(cpm_usd)/count(cpm_usd) from requests;

The flag
--custom_column "widgets:^1:[0-9.]+ Found ([0-9]+) widgets"
will locate all Info logs (severity 1) which look like this:
1:1286325423.286856 Found 12 widgets
and pull out the "12" part. Now you can run a query like these:

sqlite> -- How many widgets were seen?
sqlite> select select sum(widgets) from requests;
sqlite> -- What was the approximate the processing time per widget?
sqlite> select sum(cpu_ms)/cast(sum(widgets) as float)
...> from requests where widgets > '';

Let me know if you find this tool useful.

--Matthew

[vivpuri]

Matthew

Thank you for building this tool. While this tool helps, in my case, i
am looking to download logs per request url, since downloading logs
for complete app is not really required to fix specific issues.


-V

[Matthew Blain]

I'm glad you find it useful--once the logs are parsed you can filter
them using sqlite3.
At the moment, the request_logs tool only downloads logs by version
and date--It look like you've filed http://code.google.com/p/googleappengine/issues/detail?id=2909
for this, if other people would find it helpful they may want to star
it.

--Matthew