Excessive logging

[Joel]

This isn't really a question, I guess it could be an issue, but I don't know where it belongs. Due to an unfinished job (I think), I had this error repeated several times a second resulting in 68 GB of logs:

2016-11-09 03:28:13,332 - [ERROR] - from application in application-akka.actor.default-dispatcher-8

[WS clientlib] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5 failed:

 

java.lang.Throwable: Job not found

        at org.daisy.pipeline.webservice.impl.GenericResource.getErrorRepresentation(GenericResource.java:26)

        at org.daisy.pipeline.webservice.impl.JobResource.getResource(JobResource.java:77)

        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:506)

        at org.restlet.resource.ServerResource.get(ServerResource.java:707)

        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:589)

        at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:649)

        at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:348)

        at org.restlet.resource.ServerResource.handle(ServerResource.java:951)

        at org.restlet.resource.Finder.handle(Finder.java:246)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.routing.Router.doHandle(Router.java:431)

        at org.restlet.routing.Router.handle(Router.java:648)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:155)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:211)

        at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:84)

        at org.restlet.Application.handle(Application.java:381)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.routing.Router.doHandle(Router.java:431)

        at org.restlet.routing.Router.handle(Router.java:648)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.routing.Filter.handle(Filter.java:206)

        at org.restlet.routing.Router.doHandle(Router.java:431)

        at org.restlet.routing.Router.handle(Router.java:648)

        at org.restlet.routing.Filter.doHandle(Filter.java:159)

        at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:155)

[Jostein Austvik Jacobsen]

Hi Joel.

Logging for the engine is configured in etc/config-logging.xml. It looks like the default is 10MB rolling log files with a maximum of 4 log files before the first file is overwritten. This seems to work fine on the Pipeline 2 v1.9 server we've had running here at NLB for a while:

daisy-pipeline/data/log$ du -sh *

520K daisy-pipeline.1.log.zip

472K daisy-pipeline.2.log.zip

472K daisy-pipeline.3.log.zip

472K daisy-pipeline.4.log.zip

1.1M daisy-pipeline.log

4.0K derby.log

And the logs you mention are the Web UI logs. I see that the logger.xml for the Web UI are set to a time based rolling policy, with a max filesize of 50MB and a max history of 30 days. I suppose you got 1000+ log files at 50MB? It should probably be changed to the same as the engine to avoid the enormous logs you are getting. If you fix the logging configuration it would be great if you create a PR in the webui repo so that we can include the fix in future releases.

As for the exception itself, I get that error as well when using the Web UI when the Web UI tries to get a job that for some reason doesn't exist. I don't know why the engine gives a stack trace, but it does give an appropriate 404 header, so it's not a bug. The response is logged to the webui log file. I think either the exception should not be logged to the webui log files, or the exception should not be included in the engine response in the first place.

I created a couple of issues:

- https://github.com/daisy/pipeline/issues/491

- https://github.com/daisy/pipeline/issues/492

Jostein

--
You received this message because you are subscribed to the Google Groups "DAISY Pipeline Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to daisy-pipeline-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joel]

Thanks Jostein!

When I compared with your calculations, it turned out that it was only 23GB in /var/opt/daisy-pipeline2-webui/logs/ so I saved 45 GB just by counting correctly! :)

/Joel

Jostein

To unsubscribe from this group and stop receiving emails from it, send an email to daisy-pipeline-dev+unsub...@googlegroups.com.