insanely huge log files

[Neptronix]

Sooooo....

After 2 months of having a more recent ( Oct. 2012 ) Canvas installed, i noticed that my production.log has grown to about 75mb.

My delayed_job.log has grown to about 819mb.

This is only during testing Canvas out, with maybe 1 user poking around the system every other day, at most.

Is there some kind of limit that can be set for these files?

[Trương Hoàng Dũng]

I think you can make a cron job to delete those log files periodically.

[Neptronix]

I suppose i can do that, it's just that Canvas produces an amazing amount of logs.

In just a few seconds, it wrote a dozen of entries that look like this:

[- -]   ^[[4;36;1mDelayed::Backend::ActiveRecord::Job Load (0.8ms)^[[0m   ^[[0;1mSELECT * FROM "delayed_jobs" WHERE ((queue = 'canvas_que

[Neptronix]

After 1 minute, the file is already up to 30kb..

[Neptronix]

Config file looks like this:

  workers:

  - queue: canvas_queue

    workers: 2

    max_priority: 10

  - queue: canvas_queue

    workers: 4

  max_run_time: 28800

That's strange. Are there duplicate entries here? two queues working simultaneously? i have not edited this file, by the way..

[Neptronix]

Well, i guess i fixed my own problem, lol. Maybe this tip will be helpful...

I edited the config file to say: 

production:

  workers:

  - queue: canvas_queue

    workers: 2

    max_priority: 10

  max_run_time: 28800

In the same time period, the log size has only grown 5kb versus 30kb. 

We are a small school; so i'm sure this is OK.

I bet this cuts down CPU load a little. Definitely cuts down constant disk writes.

[Neptronix]

anyway to turn off production logging / delayed_job logging? i can't find anything just yet..

On Wednesday, November 21, 2012 8:32:47 PM UTC-8, Neptronix wrote:

[Steve Hillman]

Consider switching to syslog logging and then configure syslog to do whatever you want with the log entries (keep some, keep none, split them based on their content (which is what we do))

[Zach Wily]

The two separate canvas_queue worker blocks are so that one set of workers can be reserved to only process high priority jobs. You removed the set of workers that will do lower-priority jobs, so your job count is probably growing. Check /jobs on your instance to see.

Logs are very verbose because the default log level is debug. You can change that to info to greatly reduce log volume.

--

Zach Wily | VP Engineering

instructure.com

--
 
 
 

[Neptronix]

Thanks; i have changed back to the old configuration, but set it to 1 worker & 1 worker respectively.

Now my current jobs list shows nothing.

Where can i change the default log level? i hunted around for some kind of parameter to change this, and couldn't find one.

[Neptronix]

OK, i checked out /var/rails/canvas/config/environment.rb and it says that production uses :info by default.

I changed it to :info anyways.

This didn't change anything. Log files still exploding in size.

Then i tried this in /var/rails/canvas/config

cp logging.yml.example logging.yml

added this line into the production section:

log_level: info

Rebooted;

And now delayed_jobs.log is virtually stagnant compared to it's previous growth rate - now it only reports each finished job.

production.log now grows at a fraction of the pace - before, it was growing 150kb from just 3 page loads.. after, it has grown to maybe 3kb per 3 page loads.

Zach; maybe the open source Canvas LMS should be set this way by default? or maybe mention something about this in the setup wiki ? the debug mode produces an excessive amount of data to be a default setting. 

On Wednesday, November 21, 2012 8:32:47 PM UTC-8, Neptronix wrote: