anyone having issues with syslog after going to 7.2.1.1?

641 views
Skip to first unread message

Matthew Kauffmann

unread,
Feb 11, 2016, 9:33:44 AM2/11/16
to Isilon Technical User Group
I've opened an SR with EMC so we'll see. We went to 7.2.1.1 on Feb 1 with our test cluster and now the only output we get to our syslog server is the isi audit syslog output, no regular syslog output.  We're using this to output to feed our Splunk instance which has proven to be really helpful.  If I try to 'fix' the issue by redoing the setup with isi_log_server commands all I get is this:

# isi_log_server list
isi_log_server: need more than 1 value to unpack
Use 'isi_log_server help' for more information

I get this for all the options (list, clear, add, delete.) I also tried to just do the roll your own OneFS: How to configure remote logging from a cluster to a remote server?

I'd meant to get more in depth by going down this path anyway but that also does not work.  

Just wondering if anyone else is seeing this.

Peter Serocka

unread,
Feb 11, 2016, 10:00:20 AM2/11/16
to isilon-u...@googlegroups.com
Check the file /etc/mcp/override/syslog.conf

The lines must have the format:

server filter

with some whitespace in between.

For example, a line with a single
#
cannot be parsed, and results in the error you’ve shown.

— Peter
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


Peter Serocka

unread,
Feb 11, 2016, 10:05:14 AM2/11/16
to isilon-u...@googlegroups.com
Minor correction, the format is actually

filter server

but the message still is: parsing requires
exactly two strings separated by whitespace

Matthew Kauffmann

unread,
Feb 11, 2016, 11:15:14 AM2/11/16
to isilon-u...@googlegroups.com
Hey Peter,
o.k. but I was getting this error before I even touched the /etc/mcp/templates/syslog.conf 

You received this message because you are subscribed to a topic in the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/isilon-user-group/zo5DgSxphLs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Matt Kauffmann
Network Storage Engineer
Office of Information Technology
University of Minnesota
(six-one-two) 626 0893
(six-one-two)) 419-1491 (cell)
m...@umn.edu
   __O
 _-\<,_
(_)/ (_)

The wolves thin the RAID arrays,
removing the slowest and weakest disks
to keep the average seek speed high.

Peter Serocka

unread,
Feb 12, 2016, 5:07:21 AM2/12/16
to isilon-u...@googlegroups.com
Matt

the file is o.k. but the issue persists?
or has the issue been solved?

Matthew Kauffmann

unread,
Feb 12, 2016, 4:21:43 PM2/12/16
to isilon-u...@googlegroups.com

EMC is going through the gather.  I turned off the audit syslog traffic and there is what appears to be the normal traffic for regular syslog events BUT the isi_log_server error still persists so . . . we'll see

Matthew Kauffmann

unread,
Mar 9, 2016, 2:14:12 PM3/9/16
to Isilon Technical User Group
I'm being told it's a bug in the python code that underlies the isi_log_server. They've opened a bug fix SR.  We had NOT manipulated /etc/mcp/override/syslog.conf until AFTER this issue came up. I backed up the rsyslog.conf to and rolled it back after it didn't change the behaviour I still had the issue it seems once you touch this file you may have this issue.  So updating to 7.2.1.1 may mess with your syslog configuration

From the SR:

#isi_log_server command cannot process this kind of entries:

*.warn;*.notice;kern.*;ifs.info;istat.none              @fqdn
!lsass
cifs.*                                                  @xxx.xxx.xxx.xxx
!lwio
cifs.*                                                  @xxx.xxx.xxx.xxx

even though KB 304052 says it is ok. 

February 05, 2016 - 304052 - OneFS: How to configure remote logging from a cluster to a remote server? 

Dan Pritts

unread,
Jun 30, 2016, 11:33:38 AM6/30/16
to isilon-u...@googlegroups.com
Hi Matthew -

I just came across this email again; coincidentally I was just about to start mucking with the logging configuration on my cluster, running 7.2.1.2.    I want to forward protocol logging events to a remote syslog server. 
 
I don't see anything that looks related to your bug in the release notes for 7.2.1.2 or 7.2.1.3 - did you get a resolution?

March 9, 2016 at 2:14 PM
I'm being told it's a bug in the python code that underlies the isi_log_server. They've opened a bug fix SR.  We had NOT manipulated /etc/mcp/override/syslog.conf until AFTER this issue came up. I backed up the rsyslog.conf to and rolled it back after it didn't change the behaviour I still had the issue it seems once you touch this file you may have this issue.  So updating to 7.2.1.1 may mess with your syslog configuration

From the SR:

#isi_log_server command cannot process this kind of entries:

*.warn;*.notice;kern.*;ifs.info;istat.none              @fqdn
!lsass
cifs.*                                                  @xxx.xxx.xxx.xxx
!lwio
cifs.*                                                  @xxx.xxx.xxx.xxx

even though KB 304052 says it is ok. 

February 05, 2016 - 304052 - OneFS: How to configure remote logging from a cluster to a remote server? 
On Thursday, February 11, 2016 at 8:33:44 AM UTC-6, Matthew Kauffmann wrote:
--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan 

Matthew Kauffmann

unread,
Jun 30, 2016, 2:02:33 PM6/30/16
to isilon-u...@googlegroups.com
Dan

so there's still some kind of bug in the python code on the isi_log_server and that as far as I know is not fixed yet.  The work around they gave me as far is the software for some reason expects an ipv6 address first so I had to do this

isi_log_server add ::ffff:xxx.xxx.xxx.xxx (that's the normal ipv4 address of your rsyslog server.)

--
You received this message because you are subscribed to a topic in the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/isilon-user-group/zo5DgSxphLs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages