CAS logging analysis

[Trenton D. Adams]

Hi Guys,

When it comes to system administration and diagnostics, it is quite
common to use standard Linux utilities for debugging. Things like grep,
awk, sed, cut, etc. The CAS logs make this quite difficult. Could we
maybe start composing the logs into single line "records" of a sort?

for example, the following is not easily processed using the cli...

=============================================================
WHO: someone
WHAT: ST-54765-7eEtYFJT1VBZ2Ssexczzf7FE5ow-tst-cas-01
ACTION: SERVICE_TICKET_VALIDATED
APPLICATION: CAS
WHEN: Mon Aug 26 15:27:53 MDT 2019
CLIENT IP ADDRESS: 123.123.123.123
SERVER IP ADDRESS: server.example.com
=============================================================

I've been working around it in a somewhat clunky way by doing the
following, but this really is just a hack.

tail -f /var/log/cas/cas.log | egrep 'CLIENT IP ADDRESS|ACTION|WHAT'

Also, for service ticket validations it would be nice if they included
the service url.

Thanks.

--
Trenton D. Adams
Senior Systems Analyst/Web Software Developer
Applications Unit - ITS
Athabasca University
(780) 675-6195

It is only when you are surrounded by a supportive team, that you can achieve
your best. Instead of tearing people down, try building them up!

--
This communication is intended for the use of the recipient to whom it is addressed, and may contain confidential, personal, and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communications received in error, or subsequent reply, should be deleted or destroyed.
---

[Matthew Uribe]

Trenton,

What version of CAS are you on? When we were on 5.2, we had a line in our cas.properties which made the logging all on one line:  cas.audit.useSingleLine=true

However, since having gone to CAS 5.3, that property seems to have gone away. I've yet to dig into log4j2.xml to get the same result: one line log entries. It does make audits much easier. If you're still on 5.2 that property should take care of you.

[Daniel Ellentuck]

Hi Trenton, Matthew,

In CAS 5.3:

cas.audit.slf4j.useSingleLine=true

...

Dan Ellentuck

Columbia University I.T.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/6d250ff7-c261-4e16-8055-a64dc13f4495%40apereo.org.

[Patrick Proniewski]

Hi

> When it comes to system administration and diagnostics, it is quite
> common to use standard Linux utilities for debugging. Things like grep,
> awk, sed, cut, etc. The CAS logs make this quite difficult. Could we
> maybe start composing the logs into single line "records" of a sort?
>
> for example, the following is not easily processed using the cli...
>
> =============================================================
> WHO: someone
> WHAT: ST-54765-7eEtYFJT1VBZ2Ssexczzf7FE5ow-tst-cas-01
> ACTION: SERVICE_TICKET_VALIDATED
> APPLICATION: CAS
> WHEN: Mon Aug 26 15:27:53 MDT 2019
> CLIENT IP ADDRESS: 123.123.123.123
> SERVER IP ADDRESS: server.example.com
> =============================================================

The only way I know to process multiline log files properly is to use a programming language like AWK, Perl, Python… You won't make it with grep & friends.
Here is a sample AWK program you can adapt to your needs:

------
BEGIN {
eot = ":"; eor = "\r";
eol = "\r";
}

FNR == 1 {
if (_filename_ != "")
endfile(_filename_)
_filename_ = FILENAME
beginfile(FILENAME)
}

END { endfile("finished"); }

/^WHO: / {
sub(/^WHO: /,"");
WHO = $0; next
}

/^WHAT: / {
sub(/^WHAT: /,""); # WHAT
WHAT = $0; next
}

/^ACTION: / {
sub(/^ACTION: /,""); # ACTION
ACTION = $0; next
}

/^APPLICATION: / {
sub(/^APPLICATION: /,""); # APPLICATION
APPLICATION = $0; next
}

/^WHEN: / {
sub(/^WHEN: /,""); # WHEN
APPLICATION = $0; next
}

/^CLIENT IP ADDRESS: / {
sub(/^CLIENT IP ADDRESS: /,""); # CLIENT IP ADDRESS
CLIENT_IP_ADDRESS = " "$0; next
}

/^SERVER IP ADDRESS: / {
sub(/^SERVER IP ADDRESS: /,""); # SERVER IP ADDRESS
SERVER_IP_ADDRESS = " "$0; next
}

$0 !~/.+/ {
prfields();
WHO=""; WHAT =""; APPLICATION = ""; ACTION=""; CLIENT_IP_ADDRESS=""; SERVER_IP_ADDRESS="";
}

#============================ fonctions =====================

function beginfile(fichier) {
# init
WHO = ""; WHAT =""; APPLICATION = ""; ACTION=""; CLIENT_IP_ADDRESS=""; SERVER_IP_ADDRESS="";
}

function endfile(fichier) { # end of file
printf("%s",eor); nfich++;
printf(".") > "/dev/stderr";
}

function prfields() { # output to stdout
printf("%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s", WHO, eot, WHAT, eot, APPLICATION, eot, APPLICATION, eot, ACTION,",",SERVER_IP_ADDRESS,",",APPLICATION,",",CLIENT_IP_ADDRESS":", "\n");
}

------


You save it as foo.awk on your server, and use it like this:

awk -f /path/to/foo.awk /path/to/cas.log

To change the output, just tweak function prfields and eot. It needs testing, it's a quick & dirty script.

If you don't have more than 500 MB of cas.log daily, I would suggest you give Splunk a try. Under 500 MB per day you wont need a paid licence and can live with a free licence. It's incredibly powerful and will allow you to parse your log with great efficiency: no more headaches and 100% of your time on valuable task ;)


Patrick PRONIEWSKI
--
Chef du Service Opérations - DSI - Université Lumière Lyon 2
Responsable Sécurité des Systèmes d'Information

[Matthew Uribe]

Thank you Dan!

Matt Uribe ERP Architect/Administrator Information Technology Aims Community College 970.339.6375 matthe...@aims.edu 5401 W. 20th Street Greeley, CO, 80634 www.aims.edu

IT staff will never ask you for your username and password.

Always decline to provide the information and report such 
attempts to the Help Desk (x6380).

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAFqYg5LDdEssa9UuwuNxiKhrWTqBiDgb%3DC1f_ivkWA3%3DV%2B%2B4uA%40mail.gmail.com.

[Fabrice Bacchella]

Did you try to set 'cas.audit.slf4j.auditFormat' to 'JSON' ?

The service URL/id is missing indeed.

[Trenton D. Adams]

Interesting, thanks for the note. I will see if we can use that.

--

- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/6d250ff7-c261-4e16-8055-a64dc13f4495%40apereo.org.

[Trenton D. Adams]

No, I was thinking along those lines as well, but didn't know it had that feature.  I know log4j2 has that feature, but then you'd get a bunch of newlines wrapped as '\n'.  I'll look into it, thanks!

[Fabrice Bacchella]

The  JSON is generated directly by CAS and generated in a single log4j event. So perhaps if you play with the log4j and use a separator that is not a new line, a \0 for example, it might do the trick. I'm using mpsgack to remotely send them and so I have no separator problems.

[Ray Bon]

Auditing should be converted to a single line with the property given by Daniel.

If you have other multi line log output (I am looking at you exception stack traces), one of our logging experts came up with this (you will have to look up the specifics of %replace but I think in this case it just removes them):

        <Socket name="syslogAppender" host="localhost" port="1514" protocol="TCP">

            <PatternLayout>

                <pattern>&lt;%level{WARN=28, DEBUG=31, ERROR=27, TRACE=31, INFO=30, FATAL=25}&gt;%d{MMM dd HH:mm:ss} ${hostName} CAS: %c %replace{%m}{\n+}{&lt;31&gt;CAS: TRACE: }%n</pattern>

            </PatternLayout>

        </Socket>

Ray

-- 

Ray Bon

Programmer Analyst

Development Services, University Systems

2507218831 | CLE 019 | rb...@uvic.ca

I respectfully acknowledge that my place of work is located within the ancestral, traditional and unceded territory of the Songhees, Esquimalt and WSÁNEĆ Nations.

[Trenton D. Adams]

Interesting, thanks Ray!

I was thinking more along the lines of audit info only, but outputting exceptions in one line might be useful too.  I wonder if my IDE would auto-link to the source code lines if it was all jumbled into one line, I'll have to take a look. :D