logger branch

[Jeff Lindsay]

I started a logger branch to tackle how Ginkgo will handle basic logging. It's focus is on a logger module:

https://github.com/progrium/ginkgo/blob/logger/ginkgo/logger.py

However, it also adds two actions to ginkgoctl: log and logtail. Yes, given a service name or config file you can use ginkgoctl to cat or tail its log file. Just a nice convenience inspired by the heroku CLI tool. 

Anyway, as you can see, the logger module provides a Logger object that does basic logfile configuration and management. Nothing fancy at all. But any log related logic can go here. You'll notice we don't tell Python logging about the logfile, we just let it log to stdout by default and we redirect stdout and stderr to this logfile. Maybe this is a dumb way to do it -- I didn't think about it much. 

Later on, when we get into threads and subprocesses, perhaps Logger will become a service that provides a network interface for services in other threads / processes to publish logs to it so there is only one writer to the logfile. 

--
Jeff Lindsay
http://progrium.com

[Jeff Lindsay]

A few of us (Sean, Alan) know better the actual requirements we have for logging. Care to share? Anybody else have requirements?

[Sean McQuillan]

There's three logical log files that need to get generated.  The naming and details aren't really important.

1. access.log (possibly separated by webservers?)

2. info.log (everything)

3. errors.log (error/warn+)

access.log is useful mostly due to the overwhelming number of tools written that can parse it.  Info.log is the main debugging log, and the log that most people want to read (crashes etc should end up here).  errors.log should be the errors only, and is commonly separated from info.log so that nagios plugins can grep it quickly without loading hundreds of megs of info level logs into ram.

Cheers,

Sean

[Jeff Lindsay]

On Mon, Apr 9, 2012 at 2:39 PM, Sean McQuillan <se...@twilio.com> wrote:

There's three logical log files that need to get generated.  The naming and details aren't really important.

1. access.log (possibly separated by webservers?)

2. info.log (everything)

3. errors.log (error/warn+)

FYI to everyone else, these are Twilio requirements. 

Access.log is specific to web servers, which is a common case, but not the only case. Supporting the use of python logging to have multiple log files for this case is obviously a requirement. Same goes for info and errors. 

So as expected, we'll have to support python logging to write files, and allow you to set full python logging configuration. I see 3 different logging use cases / scenarios:

1. No configuration. Good defaults. Single log file.

2. Basic configuration. Using logging.basicConfig. Single file.

3. Advanced configuration. Using logging configuration files. Whatever you want.

Does that sound reasonable?

-jeff

 

access.log is useful mostly due to the overwhelming number of tools written that can parse it.  Info.log is the main debugging log, and the log that most people want to read (crashes etc should end up here).  errors.log should be the errors only, and is commonly separated from info.log so that nagios plugins can grep it quickly without loading hundreds of megs of info level logs into ram.

Cheers,

Sean

On Fri, Apr 6, 2012 at 2:30 PM, Jeff Lindsay <prog...@gmail.com> wrote:

A few of us (Sean, Alan) know better the actual requirements we have for logging. Care to share? Anybody else have requirements?

On Fri, Apr 6, 2012 at 4:29 AM, Jeff Lindsay <prog...@gmail.com> wrote:

I started a logger branch to tackle how Ginkgo will handle basic logging. It's focus is on a logger module:

https://github.com/progrium/ginkgo/blob/logger/ginkgo/logger.py

However, it also adds two actions to ginkgoctl: log and logtail. Yes, given a service name or config file you can use ginkgoctl to cat or tail its log file. Just a nice convenience inspired by the heroku CLI tool. 

Anyway, as you can see, the logger module provides a Logger object that does basic logfile configuration and management. Nothing fancy at all. But any log related logic can go here. You'll notice we don't tell Python logging about the logfile, we just let it log to stdout by default and we redirect stdout and stderr to this logfile. Maybe this is a dumb way to do it -- I didn't think about it much. 

Later on, when we get into threads and subprocesses, perhaps Logger will become a service that provides a network interface for services in other threads / processes to publish logs to it so there is only one writer to the logfile. 

--
Jeff Lindsay
http://progrium.com

--
Jeff Lindsay
http://progrium.com