Re: [go-cd] trying to configure logback and access logs

[Ketan Padegaonkar]

Assuming you're on version 17.11+ — checkout these docs.

For versions older than 17.10, you'd likely have a log4j.properties file and you'd be expected to uncomment this line.

On Tue, Dec 19, 2017 at 9:26 PM 'Barry Greenwood' via go-cd <go...@googlegroups.com> wrote:

• Hi I am trying to configure logback and access logs but this doesnt appear to work. following the documentation i've come up with the below jetty.xml. However the logback-access.xml doesn't ever seem to be read. 

• <Get name="handler"> 
  <Call name="addHandler"> 
  <Arg> 
  <New class="org.eclipse.jetty.server.handler.RequestLogHandler"> 
  <Set name="requestLog"> 
  <Set name="requestLogHandler"> 
  <New id="requestLogImpl" class="ch.qos.logback.access.jetty.RequestLogImpl"></New> 
  <Set name="fileName">/etc/go/logback-access.xml</Set> 
  </Set> 
  </Set> 
  </New> 
  </Arg> 
  </Call> 
  </Get>

the below is the logback.xml (which appears to be needed but is missing from the documentation.

• <configuration> 
  <!-- To Be Managed by puppet --> 
  <include file="/etc/go/logback-include.xml"/>

  <appender name="SOCKET" class="ch.qos.logback.classic.net.SocketAppender"> 
  <remoteHost>SD-LOGSTASH.sysdev.zen.co.uk</remoteHost> 
  <port>3319</port> 
  <reconnectionDelay>10000</reconnectionDelay> 
  <includeCallerData>true</includeCallerData> 
  <encoder> 
  <pattern>%h %l %u [%t] "%r" %s %b "%i{Referer}" "%i{User-Agent}"</pattern> 
  </encoder> 
  </appender>

  <appender name="FILE" class="ch.qos.logback.core.FileAppender"> 
  <file>/var/log/go-server/lbfile.log</file> 
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> 
  <fileNamePattern>lbfile.%d{yyyy-MM-dd}.log.zip</fileNamePattern> 
  </rollingPolicy> 
  <maxHistory>7</maxHistory> 
  <totalSizeCap>3GB</totalSizeCap> 
  <append>true</append> 
  <!-- set immediateFlush to false for much higher logging throughput --> 
  <immediateFlush>true</immediateFlush> 
  <!-- encoders are assigned the type 
  ch.qos.logback.classic.encoder.PatternLayoutEncoder by default --> 
  <encoder> 
  <pattern>%-4relative [%thread] %-5level %logger{35} - %date{ISO8601} - %msg%n</pattern> 
  </encoder> 
  </appender>

  <appender name="FILEac" class="ch.qos.logback.core.rolling.RollingFileAppender"> 
  <file>/var/log/go-server/lbaccess.log</file> 
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> 
  <fileNamePattern>lbaccess.%d{yyyy-MM-dd}.log.zip</fileNamePattern> 
  </rollingPolicy> 
  <maxHistory>7</maxHistory> 
  <totalSizeCap>1GB</totalSizeCap> 
  <append>true</append> 
  <encoder> 
  <pattern>combined</pattern> 
  </encoder> 
  </appender>

  <root level="WARN"> 
  <appender-ref ref="FILE" />

• finally the below is the logback-access.xml which doenst seem to be imported (the base log file isnt created)

  <configuration> 
  <!-- always a good activate OnConsoleStatusListener --> 
  <statusListener class="ch.qos.logback.core.status.OnConsoleStatusListener" />

  <appender name="ROLFILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> 
  <file>/var/log/go-server/lb-access.log</file> 
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> 
  <fileNamePattern>lbaccess.%d{yyyy-MM-dd}.log.zip</fileNamePattern> 
  </rollingPolicy>

  <encoder> 
  <pattern>combined</pattern> 
  </encoder> 
  </appender>

  <appender-ref ref="ROLFILE" /> 
  </configuration>

• Many Thanks for any help, Barry

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Barry Greenwood]

I am on 17.12, I have got logback working for application logs however the access logs state i need to import an logback-access.xml, however it doesnt specify how to force this import, I have followed the docs as far as they go and am still unable to get this working (https://github.com/logstash/logstash-logback-encoder#accessevent-fields) 

Thanks for helping

[Ketan Padegaonkar]

On Tue, Dec 19, 2017 at 10:27 PM 'Barry Greenwood' via go-cd <go...@googlegroups.com> wrote:

I am on 17.12, I have got logback working for application logs however the access logs state i need to import an logback-access.xml

I don't believe there is any GoCD document that says so.

[Varsha Varadarajan]

@Barry - Are you looking for request logs? This can be setup using org.eclipse.jetty.server.RequestLog. Checkout the docs that Ketan pointed to - https://docs.gocd.org/current/advanced_usage/logging.html

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.

[Barry Greenwood]

Hi Varsha 

I've tried in the  logback.xml

<appender name="FILEac" class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>/var/log/go-server/lbaccess.log</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

      <fileNamePattern>lbaccess.%d{yyyy-MM-dd}.log.zip</fileNamePattern>

    </rollingPolicy>

    <maxHistory>7</maxHistory>

    <totalSizeCap>1GB</totalSizeCap>

    <append>true</append>

    <encoder>

      <pattern>combined</pattern>

    </encoder>

  </appender>

  <logger name="org.eclipse.jetty.server.RequestLog" level="INFO">

    <appender-ref ref="FILEac" />

  </logger>

and that generates the file but no logs appear.

The docs imply i need to use a logback-access.xml but i cant get that to import.

Thanks

Barry

[Ketan Padegaonkar]

The logs do not mention any logback-access.xml anywhere. To get access logs —

* create a file called `/etc/default/logback-include.xml`, with the following contents

<?xml version="1.0" encoding="UTF-8"?>

<included>

  <logger name="org.eclipse.jetty.server.RequestLog" level="INFO" />

</included>

--
You received this message because you are subscribed to the Google Groups "go-cd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

[Barry Greenwood]

Hi

Thanks for the help however i am still having issues.

/etc/default/logback-include.xml as described but with a with a file appender in it does not create the file, that implies this isn't being read

/etc/go/logback.xml 

does create the file but nothing is put into the file.

"

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

  <appender name="FILEic" class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>/var/log/go-server/lbinclude.log</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

      <fileNamePattern>lbinclude.%d{yyyy-MM-dd}.log.zip</fileNamePattern>

    </rollingPolicy>

    <maxHistory>7</maxHistory>

    <totalSizeCap>1GB</totalSizeCap>

    <append>true</append>

  </appender>

  <logger name="org.eclipse.jetty.server.RequestLog" level="INFO">

    <appender-ref ref="FILEic" />

  </logger>

</configuration>

[Aravind SV]

Hello Barry,

On Tue, Jan 2, 2018 at 9:49 AM, 'Barry Greenwood' via go-cd <go...@googlegroups.com> wrote:

Thanks for the help however i am still having issues.

/etc/default/logback-include.xml as described but with a with a file appender in it does not create the file, that implies this isn't being read

You're right. It's a bug. It'll be fixed in the next release (probably through this PR). Meanwhile, temporarily, can you see if moving the file logback-include.xml into /var/lib/go-server/config/ works?

Here's what I tried, to ensure that this works:

$ ls -l /var/lib/go-server/config/
total 4
-rw-r--r-- 1 go go 759 Jan  2 18:58 logback-include.xml
$ cat /var/lib/go-server/config/logback-include.xml

<?xml version="1.0" encoding="UTF-8"?>
<included>

  <appender name="test-appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/var/log/go-server/test.log</file>
    <encoder>
      <pattern>%date{ISO8601} %-5level [%thread] %logger{0}:%line - %msg%n</pattern>
    </encoder>

    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>/var/log/go-server/test.log.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
      <maxFileSize>10 MB</maxFileSize>
      <maxHistory>10</maxHistory>
      <totalSizeCap>512 MB</totalSizeCap>
    </rollingPolicy>

  </appender>

  <logger name="org.eclipse.jetty.server.RequestLog" level="INFO">

    <appender-ref ref="test-appender" />
  </logger>
</included>

Once 18.1 is released, this will stop working and you can move it back to its correct place, /etc/go/logback-include.xml. Or, you can see if a symbolic link from /var/lib/go-server/config/logback-include.xml to /etc/go/logback-include.xml works.

Cheers,
Aravind

[Aravind SV]

Just FYI. Yes, it looks like doing this will work:

mkdir config
chown go:go config
ln -s /etc/go/logback-include.xml /var/lib/go-server/config/

[Barry Greenwood]

Hi

 I have tied these its a little better in the file is now being created, 

but i'm still not getting any access logs in the file

i've tried both

cat /var/lib/go-server/config/logback-include.xml

<?xml version="1.0" encoding="UTF-8"?>

<included>

  <appender name="FILEic" class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>/var/log/go-server/lbinclude.log</file>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

      <fileNamePattern>lbinclude.%d{yyyy-MM-dd}.log.gz</fileNamePattern>

    </rollingPolicy>

    <maxHistory>7</maxHistory>

    <totalSizeCap>1GB</totalSizeCap>

    <append>true</append>

    <encoder>

      <pattern>%date{ISO8601} - %-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>

    </encoder>

  </appender>

  <logger name="org.eclipse.jetty.server.RequestLog" level="INFO" />

  <root level="ERROR">

    <appender-ref ref="FILEic" />

  </root>

</included>

and 

  <logger name="org.eclipse.jetty.server.RequestLog" level="INFO" >

    <appender-ref ref="FILEic" />

  </logger >

the first generates some data in the file the second leaves the file empty.

logback access comes from the documentation on https://logback.qos.ch/access.html which is linked from https://github.com/logstash/logstash-logback-encoder

Many thanks

Barry

[Ketan Padegaonkar]

I ran these sequence of commands and it worked for me.

--

[Aravind SV]

Maybe check permissions of that directory and file? I tried it too and it worked. The second snippet (with the appender inside the logger).

--
You received this message because you are subscribed to the Google Groups "go-cd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.

[Barry Greenwood]

Deeper digging has allowed me to confirm nothing was appearing in the file due to it being not a new server, the issue with failing to log was related to an old version of the jetty.xml config file.

I can now log to files successfully, next step will be to send the data to a logstash server as well as log locally

Many thanks for help with this.

 Barry