syslog file access logs have wrong datestamp on one node
Dan Pritts
Hi folks -
Really weird one here.
We upgraded our cluster from 8.something to 9.2.1.7 over the holiday.
We do filesystem access logging via syslog. We have a 4-node cluster of H400's.
Since the upgrade, syslog filesystem access logging has been sending bad datestamps from node 4 only.
The datestamp it's sending is about 45 days and 22.5 hours in the past.
e.g., these two came in together:
Jan 5 11:41:29 (from node 3)
Nov 20 05:51:09 (from node 4)
The system time on all nodes is correct; NTP is running. I've restarted (killed and let mcp restart) syslogd. Similarly killed /usr/libexec/isilon/isi_audit_syslog.
I have a ticket open which is being escalated to level 2 support.
Anyone run into this before?
I am, um, flummoxed.
thanks
danno
Dan Pritts
Devops Lead
ICPSR Computing & Network Services
Peter Serocka
--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/73F47E9B-9CFA-4BAF-9B14-E04FC1F85602%40umich.edu.
Dan Pritts
Turns out that this one node was not properly forwarding to syslog since a reboot in November. The good news is that it's now replaying the logs to syslog, so our syslog server will get them all eventually, maybe. The bad news is that it seems to be doing this in realtime, and doesn't seem to be catching up.
Support told us how to reset it so it just emits back logs from here on out. Which is nice and all but 5then I'm missing 45 days of logs on my syslog server.
The good news is that the cluster stores raw logs indefinitely in /ifs/.ifsvar/audit/logs. You can access them directly in a binary format, or dump them with isi_audit_viewer.
isi_audit_viewer has this nice output format that is excellent to work with. It starts with an arbitrary event ID, which always starts at 0, and a timestamp, surrounded in []. Then it gives a json formatted log entry (which includes a timestamp).
[0: Sun Jan 9 17:49:35 2022] {"id":"6ace3e01-719e-11ec-bf37-90e2baf723c0","timestamp":1641768575231542,"payloadType":"c411a642-c139-4c7a-be58-93680bc20b41","payload":{"protocol":"NFS","zoneID":13,"zoneName": [...] }
The fields are documented here: https://www.dell.com/support/kbdoc/en-uk/000019850/isilon-audit-payload-values
That doc goes on about the timestamp value being a 64-bit number with the top 32 bits being the unix timestamp and the bottom 32 bits being microseconds. Luckily, it's simpler than that, just a unix timestamp with 9 additional digits for µs.
Nice how it chooses 4 GUIDs for the 4 possible values of payloadType. No doubt from some larger standard, but yuck.
The binary format is the json that it outputs, separated by a small binary blob. The blob is not obviously the 64-bit binary timestamp, but maybe. Regardless, isi_audit_viewer handles that for you.
The paths reported in the JSON are written as so:
\\ifs\\foo\\bar\\ack\\baz
...even if they were served via NFS.
Conveniently, it also gives you \ if your filename has a \ in the name. The syslog output simply replaces the \ in the filename with a / - what else is it gonna do, I guess.
A newline was maybe better - \u000a in the json, or #015 in the syslog output.
thanks
danno
Dan Pritts
Devops Lead
ICPSR Computing & Network Services
To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/7BFD3AC3-A441-4E2D-95F6-A2E588BD6953%40gmail.com.