lnav . not sorting all files by timestamp
311 views
Skip to first unread message
hagfelsh
Nov 16, 2015, 4:12:04 PM11/16/15
to lnav
I just discovered something that explained why my sequence of events for this system I'm troubleshooting made no sense--the various logs in var/run/log/* aren't sorted by timestamp when they're all loaded at once! In partcular, the yellow lines are what's out of order, from vmkwarning.log.
I'm running lnav 0.7.3-145-g97699e9 compiled from source on Centos 7.
Here is a screenshot:
Here's a debug dump from the run:
2015-11-16T15:03:06.010 I lnav.cc:2362 lnav started
2015-11-16T15:03:06.012 D log_format_loader.cc:554 Loading default formats
2015-11-16T15:03:06.012 D log_format_loader.cc:65 Loading format -- access_log
2015-11-16T15:03:06.012 D log_format_loader.cc:83 format regex: access_log/ts-first-noquotes = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?) (?<c_ip>[^ ]+) (?<cs_username>[^ ]+) (?<cs_method>[A-Z]+) (?<cs_uri_stem>[^ \?]+)(?:\?(?<cs_uri_query>[^ ]*))? (?:-1|\d+) (?<sc_status>\d+) \d+\s*(?<body>.*)
2015-11-16T15:03:06.012 D log_format_loader.cc:83 format regex: access_log/ts-first = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?) (?<c_ip>[^ ]+) (?<cs_username>[^ ]+) (?<cs_method>[A-Z]+) "(?<cs_uri_stem>[^ \?]+)(?:\?(?<cs_uri_query>[^ ]*))?" (?:-1|\d+) (?<sc_status>\d+) \d+\s*(?<body>.*)
2015-11-16T15:03:06.012 D log_format_loader.cc:83 format regex: access_log/std = ^(?<c_ip>[\w\.:\-]+)\s+[\w\.\-]+\s+(?<cs_username>\S+)\s+\[(?<timestamp>[^\]]+)\] "(?:\-|(?<cs_method>\w+) (?<cs_uri_stem>[^ \?]+)(?:\?(?<cs_uri_query>[^ ]*))? (?<cs_version>[\w/\.]+))" (?<sc_status>\d+) (?<sc_bytes>\d+|-)(?: "(?<cs_referer>[^"]+)" "(?<cs_user_agent>[^"]+)")?\s*(?<body>.*)
2015-11-16T15:03:06.012 D log_format_loader.cc:83 format regex: access_log/mod-std = ^(?<c_ip>[\w\.:\-]+)\s+[\w\.\-]+\s+(?<cs_username>\S+)\s+"(?:\-|(?<cs_method>\w+) (?<cs_uri_stem>[^ \?]+)(?:\?(?<cs_uri_query>[^ ]*))? (?<cs_version>[\w/\.]+))" (?<sc_status>\d+) (?<sc_bytes>\d+|-)(?: "(?<cs_referer>[^"]+)" "(?<cs_user_agent>[^"]+)")?\s*(?<body>.*)
2015-11-16T15:03:06.013 D log_format_loader.cc:65 Loading format -- autodeploy_log
2015-11-16T15:03:06.013 D log_format_loader.cc:83 format regex: autodeploy_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?) \[(?<pid>\d+)\](?<level>\w+):(?<module>[\w-]+):(?<body>.*$)
2015-11-16T15:03:06.013 D log_format_loader.cc:65 Loading format -- block_log
2015-11-16T15:03:06.013 D log_format_loader.cc:83 format regex: block_log/std = ^(?<timestamp>\w{3} \w{3}\s+\d{1,2} \d{2}:\d{2}:\d{2} \w+ \d{4})(?<body>(?:.|\n)*)$
2015-11-16T15:03:06.013 D log_format_loader.cc:65 Loading format -- candlepin_log
2015-11-16T15:03:06.013 D log_format_loader.cc:83 format regex: candlepin_log/reqorg = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(req=(?<req>[0-9a-f-]+)|=), org=(?<org>\w*)\] (?<alert_level>\w+) (?<module>[\w.]+) - (?<body>.*)$
2015-11-16T15:03:06.013 D log_format_loader.cc:83 format regex: candlepin_log/other = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}[+-]\d{4}) (?<body>.*)$
2015-11-16T15:03:06.014 D log_format_loader.cc:65 Loading format -- choose_repo_log
2015-11-16T15:03:06.014 D log_format_loader.cc:83 format regex: choose_repo_log/std = ^\[(?<level>\w+):[^\]]+] [^:]+:\d+ (?<timestamp>\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:[\.,]\d{3})?):(?<body>.*)
2015-11-16T15:03:06.014 D log_format_loader.cc:65 Loading format -- cups_log
2015-11-16T15:03:06.014 D log_format_loader.cc:83 format regex: cups_log/system = ^(?<level>[IEW]) \[(?<timestamp>\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{2,4})\] (?<section>\w+): (?<body>.*)$
2015-11-16T15:03:06.014 D log_format_loader.cc:83 format regex: cups_log/default = ^(?<level>[IEW]) \[(?<timestamp>\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{2,4})\] (?<body>.*)$
2015-11-16T15:03:06.014 D log_format_loader.cc:65 Loading format -- dpkg_log
2015-11-16T15:03:06.014 D log_format_loader.cc:83 format regex: dpkg_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d{3})?) (?:(?:(?<action>startup|status|configure|install|upgrade|trigproc|remove|purge)(?: (?<status>config-files|failed-config|half-configured|half-installed|installed|not-installed|post-inst-failed|removal-failed|triggers-awaited|triggers-pending|unpacked))? (?<package>[^ ]+) (?<installed_version>[^ ]+)(?: (?<available_version>[^ ]+))?)|update-alternatives: (?<body>.*))$
2015-11-16T15:03:06.014 D log_format_loader.cc:65 Loading format -- engine_log
2015-11-16T15:03:06.014 D log_format_loader.cc:83 format regex: engine_log/std = ^(?<timestamp>.+) (?<level>.+) \[(?<logger>.+)\] \((?<tid>.+)\) (?<body>(?:-|\n)*)
2015-11-16T15:03:06.014 D log_format_loader.cc:65 Loading format -- error_log
2015-11-16T15:03:06.014 D log_format_loader.cc:83 format regex: error_log/cups = ^(?<level>\w) \[(?<timestamp>[^\]]+)\] (?<body>.*)
2015-11-16T15:03:06.015 D log_format_loader.cc:65 Loading format -- fsck_hfs_log
2015-11-16T15:03:06.015 D log_format_loader.cc:83 format regex: fsck_hfs_log/std = ^(?<device>[^:]+): fsck_hfs (?:run|started) at (?<timestamp>\w{3} \w{3}\s+\d{1,2} \d{2}:\d{2}:\d{2} \d{4})(?<body>(?:.|\n)*)
2015-11-16T15:03:06.015 D log_format_loader.cc:65 Loading format -- glog_log
2015-11-16T15:03:06.015 D log_format_loader.cc:83 format regex: glog_log/std = ^(?<level>[IWECF])(?<timestamp>\d{4} \d{2}:\d{2}:\d{2}\.\d{6}) (?<thread>\d+) (?<src_file>[^:]+):(?<src_line>\d+)\] (?<body>(?:.|\n)*)
2015-11-16T15:03:06.015 D log_format_loader.cc:65 Loading format -- java_log
2015-11-16T15:03:06.015 D log_format_loader.cc:83 format regex: java_log/jvm = ^(?<alert_level>\w+)\s+\|\s+jvm (?<jvm_no>\d+)\s+\|\s(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\s+\| (?<timestamp_f>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<function>\w+-\d+)\]\s+(?<debug_level>\w+)\s+(?<class>[\w.]+)\s+-\s+(?<body>\S.*)
2015-11-16T15:03:06.015 D log_format_loader.cc:83 format regex: java_log/dump = ^(?<alert_level>\w+)\s+\|\s+jvm (?<jvm_no>\d+)\s+\|\s(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\s+\| JVMDUMP\w+\s(?<body>\S.*)$
2015-11-16T15:03:06.015 D log_format_loader.cc:83 format regex: java_log/tasko = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<function>\w+-\d+)\]\s+(?<alert_level>\w+)\s+(?<class>[\w.]+)\s+-\s+(?<body>\S.*)$
2015-11-16T15:03:06.015 D log_format_loader.cc:65 Loading format -- katello_log
2015-11-16T15:03:06.015 D log_format_loader.cc:83 format regex: katello_log/log = ^\[\s?(?<alert_level>\w+)\s(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s(?<module>\w+)\]\s+(?<message>\S.*)$
2015-11-16T15:03:06.016 D log_format_loader.cc:65 Loading format -- openam_log
2015-11-16T15:03:06.016 D log_format_loader.cc:83 format regex: openam_log/std = ^"(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})"\s+(?<data>[^ "]+|"(?:[^"]*|"")*")\s+(?<loginid>[^ "]+|"(?:[^"]*|"")*")\s+(?<contextid>[^ "]+|"(?:[^"]|"")*")\s+(?<ipaddr>[^ "]+|"(?:[^"]|"")*")\s+(?<level>[^ "]+|"(?:[^"]|"")*")\s+(?<domain>[^ "]+|"(?:[^"]|"")*")\s+(?<loggedby>[^ "]+|"(?:[^"]|"")*")\s+(?<messageid>[^ "]+|"(?:[^"]|"")*")\s+(?<modulename>[^ "]+|"(?:[^"]|"")*")\s+(?<nameid>[^ "]+|"(?:[^"]|"")*")\s+(?<hostname>[^ "]+|"(?:[^"]|"")*")(?<body>.*)$
2015-11-16T15:03:06.016 D log_format_loader.cc:65 Loading format -- openamdb_log
2015-11-16T15:03:06.016 D log_format_loader.cc:83 format regex: openamdb_log/std = ^(?<module>[\w]+):(?<timestamp>\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}:\d{3} [AP]M \w+): Thread\[(?<thread>[^,]+,\d+,[^,]+)\]\n?(?:\*+|(?<body>.*))$
2015-11-16T15:03:06.016 D log_format_loader.cc:65 Loading format -- openstack_log
2015-11-16T15:03:06.016 D log_format_loader.cc:83 format regex: openstack_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}) (?<pid>\d+) (?<level>\w+) (?<logger>.+) \[(?<tid>.+)\] (?<body>.*)
2015-11-16T15:03:06.016 D log_format_loader.cc:65 Loading format -- page_log
2015-11-16T15:03:06.016 D log_format_loader.cc:83 format regex: page_log/pre-1.7 = ^(?<printer>[\w_\-\.]+) (?<username>[\w\.\-]+) (?<job_id>\d+) \[(?<timestamp>[^\]]+)\] (?<page_number>total|\d+) (?<num_copies>\d+) (?<job_billing>[^ ]+) (?<job_originating_hostname>[\w\.:\-]+)(?<body>.*)$
2015-11-16T15:03:06.016 D log_format_loader.cc:83 format regex: page_log/1.7 = ^(?<printer>[\w_\-\.]+) (?<username>[\w\.\-]+) (?<job_id>\d+) \[(?<timestamp>[^\]]+)\] (?<page_number>total|\d+) (?<num_copies>\d+) (?<job_billing>[^ ]+) (?<job_originating_hostname>[\w\.:\-]+) (?<job_name>.+) (?<media>[^ ]+) (?<sides>.+)(?<body>.*)$
2015-11-16T15:03:06.017 D log_format_loader.cc:65 Loading format -- papertrail_log
2015-11-16T15:03:06.017 D log_format_loader.cc:65 Loading format -- snaplogic_log
2015-11-16T15:03:06.017 D log_format_loader.cc:83 format regex: snaplogic_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?) (?:(?:(?<level>\w{4,}) (?<logger>[^ ]+) (?<facility>[^ ]+) (?<msgid>[^ ]+) (?<pipe_rid>-|\d+)(?:\.(?<comp_rid>[^ ]+))? (?<resource_name>[^ ]+) (?<invoker>[^ ]+))|(?:(?:stdout|stderr): ))(?<body>.*)
2015-11-16T15:03:06.017 D log_format_loader.cc:65 Loading format -- sssd_log
2015-11-16T15:03:06.018 D log_format_loader.cc:83 format regex: sssd_log/core = ^\((?<timestamp>\w{3} \w{3} ( \d|\d{2}) \d{2}:\d{2}:\d{2} \d{4})\) \[sssd\] \[(?<function>\w+)\] \((?<debug_level>0x[0-9a-fA-F]{4})\): (?<body>.*)$
2015-11-16T15:03:06.018 D log_format_loader.cc:83 format regex: sssd_log/module = ^\((?<timestamp>\w{3} \w{3} ( \d|\d{2}) \d{2}:\d{2}:\d{2} \d{4})\) \[sssd(?<module>\[.*?\])\] \[(?<function>\w+)\] \((?<debug_level>0x[0-9a-fA-F]{4})\): (?<body>.*)$
2015-11-16T15:03:06.018 D log_format_loader.cc:65 Loading format -- strace_log
2015-11-16T15:03:06.018 D log_format_loader.cc:83 format regex: strace_log/std = ^(?<timestamp>\d{2}:\d{2}:\d{2}\.\d{6}) (?<syscall>\w+)\((?<body>.*)\)\s+=\s+(?<rc>[-\w]+)(?: (?<errno>\w+) \([^\)]+\))?(?: <(?<duration>\d+\.\d+)>)?$
2015-11-16T15:03:06.018 D log_format_loader.cc:65 Loading format -- sudo_log
2015-11-16T15:03:06.018 D log_format_loader.cc:83 format regex: sudo_log/std = ^(?<login>\S+)\s*: (?:(?<error_msg>[^;]+);)?\s*TTY=(?<tty>[^;]+)\s+;\s*PWD=(?<pwd>[^;]+)\s+;\s*USER=(?<user>[^;]+)\s+;\s*COMMAND=(?<command>(\n|.)*)$
2015-11-16T15:03:06.018 D log_format_loader.cc:65 Loading format -- syslog_log
2015-11-16T15:03:06.018 D log_format_loader.cc:83 format regex: syslog_log/std = ^(?<timestamp>\w{3}\s+\d{1,2} \d{2}:\d{2}:\d{2})(?: (?<log_hostname>[a-zA-Z0-9:][^ ]+[a-zA-Z0-9]))?(?:(?: (?<log_procname>(?:[^\[:]+|[^:]+))(?:\[(?<log_pid>\d+)\])?:(?<body>(?:.|\n)*))$|:?(?:(?: ---)? last message repeated \d+ times?(?: ---)?))
2015-11-16T15:03:06.019 D log_format_loader.cc:65 Loading format -- tcf_log
2015-11-16T15:03:06.019 D log_format_loader.cc:83 format regex: tcf_log/std = ^TCF (?<timestamp>\d{2}:\d{2}.\d{3}): (?:Server-Properties: (?:.*)|channel server|\w+: (?<dir>--->|<---) (?<type>\w)(?: (?<token>\w+))?(?: (?<service>\w+))?(?: (?<name>\w+))?(?: (?<msg>.*))?(?: <eom>))(?<body>.*)$
2015-11-16T15:03:06.019 D log_format_loader.cc:65 Loading format -- tcsh_history
2015-11-16T15:03:06.019 D log_format_loader.cc:83 format regex: tcsh_history/std = ^#(?<timestamp>\+\d+)\n?(?<body>.*)?$
2015-11-16T15:03:06.019 D log_format_loader.cc:65 Loading format -- uwsgi_log
2015-11-16T15:03:06.019 D log_format_loader.cc:83 format regex: uwsgi_log/std = ^\[pid: (?<s_pid>\d+)\|app: (?<s_app>[\-\d]+)\|req: (?<s_req>[\-\d]+)/(?<s_worker_reqs>\d+)\] (?<c_ip>[^ ]+) \((?<cs_username>[^\)]*)\) \{(?<cs_vars>\d+) vars in (?<cs_bytes>\d+) bytes\} \[(?<timestamp>[^\]]+)\] (?<cs_method>[A-Z]+) (?<cs_uri_stem>[^ \?]+)(?:\?(?<cs_uri_query>[^ ]*))? => generated (?<sc_bytes>\d+) bytes in (?<s_runtime>\d+) (?<rt_unit>\w+) \((?<cs_version>[^ ]+) (?<sc_status>\d+)\) (?<sc_headers>\d+) headers in (?<sc_header_bytes>\d+) bytes \((?<s_switches>\d+) switches on core (?<s_core>\d+)\)(?<body>.*)
2015-11-16T15:03:06.020 D log_format_loader.cc:65 Loading format -- vdsm_log
2015-11-16T15:03:06.020 D log_format_loader.cc:83 format regex: vdsm_log/std = ^(?<tid>.+)::(?<level>.+)::(?<timestamp>.+)::(?<module>.+)::(?<src_line>.+)::(?<logger>.+)::\((?<func>[^\)]+)\)(?<body>(?:.|\n)*)
2015-11-16T15:03:06.020 D log_format_loader.cc:65 Loading format -- vmk_log
2015-11-16T15:03:06.020 D log_format_loader.cc:83 format regex: vmk_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z) cpu(?<cpu>\d+):(?<world_id>\d+)\)((?:(?<level>WARNING|ALERT)|(?<subsystem>[^:]+)): )?(?<body>.*)
2015-11-16T15:03:06.020 D log_format_loader.cc:65 Loading format -- vmw_log
2015-11-16T15:03:06.020 D log_format_loader.cc:83 format regex: vmw_log/6.0+ = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}(?:Z|[-+]\d{2}:\d{2})) (?:Section for VMware VirtualCenter,.*|(?<level>\w+) (?<prc>\w+)\[(?<tid>\w+)\] \[(?<src>[^ \]]+)\s*(?: sub=(?<sub>[^ \]]+))?(?: opID=(?<opid>[^ \]]+))?(?: user=(?<user>[\w\-]+))?\](?<body>.*)(?:\n(?:.|\n)*)?)$
2015-11-16T15:03:06.020 D log_format_loader.cc:83 format regex: vmw_log/5.0+ = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}(?:Z|[-+]\d{2}:\d{2})) \[(?<tid>\w+) (?<level>\w+) '(?<comp>[^']+)'(?: opID=(?<opid>[^ \]]+))?(?: user=(?<user>[\w\-]+))?\](?<body>.*)(?:\n(?:.|\n)*)?$
2015-11-16T15:03:06.020 D log_format_loader.cc:83 format regex: vmw_log/pre-5.0 = ^\[(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) (?<tid>\w+) (?<level>\w+) '(?<comp>[^']+)'(?: opID=(?<opid>[^ \]]+))?(?: user=(?<user>[\w\-]+))?\](?<body>.*)(?:\n(?:.|\n)*)?$
2015-11-16T15:03:06.020 D log_format_loader.cc:65 Loading format -- xmlrpc_log
2015-11-16T15:03:06.021 D log_format_loader.cc:83 format regex: xmlrpc_log/main = ^(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} [+-]?\d{2}:\d{2}) (?<pid>\d+) (?<client_ip>\S+): (?<module>\w+)/(?<function>.*)(?<arguments>\(.*?\))?(?<body>.*)$
2015-11-16T15:03:06.021 I log_format_loader.cc:523 loading formats from path: /etc/lnav/formats/*/*.json
2015-11-16T15:03:06.021 I log_format_loader.cc:523 loading formats from path: /usr/etc/lnav/formats/*/*.json
2015-11-16T15:03:06.021 I log_format_loader.cc:523 loading formats from path: /home/username/.lnav//formats/*/*.json
2015-11-16T15:03:06.021 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/candlepin.json
2015-11-16T15:03:06.021 D log_format_loader.cc:83 format regex: candlepin_log/reqorg = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(req=(?<req>[0-9a-f-]+)|=), org=(?<org>\w*)\] (?<alert_level>\w+) (?<module>[\w.]+) - (?<body>.*)$
2015-11-16T15:03:06.021 D log_format_loader.cc:83 format regex: candlepin_log/other = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}[+-]\d{4}) (?<body>.*)$
2015-11-16T15:03:06.021 I log_format_loader.cc:537 found format: candlepin_log
2015-11-16T15:03:06.021 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/ehcmd-log.json.json
2015-11-16T15:03:06.021 D log_format_loader.cc:65 Loading format -- eqlogic_ehcmd_log
2015-11-16T15:03:06.021 D log_format_loader.cc:83 format regex: eqlogic_ehcmd_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})Z (?<level>\w{1}) EHCMD (?<method>\w+):(?<method_id>\d+) (?<body>.*)
2015-11-16T15:03:06.022 D log_format_loader.cc:83 format regex: eqlogic_ehcmd_log/mem1.3 = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})Z (?<level>\w{1}) EHCMD \w+\|(?<method>\w+)\|(?<method_id>\d+)\|(?<body>.*)
2015-11-16T15:03:06.022 I log_format_loader.cc:537 found format: eqlogic_ehcmd_log
2015-11-16T15:03:06.022 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/esxupdate-log.json
2015-11-16T15:03:06.022 D log_format_loader.cc:65 Loading format -- esxupdate_log
2015-11-16T15:03:06.022 D log_format_loader.cc:83 format regex: esxupdate_log/non-level = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?<system>\w+): (?<subsystem>[a-zA-Z0-9\.]+): (?<level>[A-Z]+): (?<body>.*)$
2015-11-16T15:03:06.022 I log_format_loader.cc:537 found format: esxupdate_log
2015-11-16T15:03:06.022 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/goofysyslog.json
2015-11-16T15:03:06.022 D log_format_loader.cc:65 Loading format -- vmkernel_log_sub
2015-11-16T15:03:06.022 D log_format_loader.cc:83 format regex: vmkernel_log_sub/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z) (?<cpu>cpu\d+):(?<world_id>\d+)\)((?:(?<level>WARNING|ALERT)|(?<subsystem>[a-zA-Z0-9_\<\>]+)): )?(?<body>.*)
2015-11-16T15:03:06.022 I log_format_loader.cc:537 found format: vmkernel_log_sub
2015-11-16T15:03:06.022 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/katello.json
2015-11-16T15:03:06.022 D log_format_loader.cc:83 format regex: katello_log/log = ^\[\s?(?<alert_level>\w+)\s(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s(?<module>\w+)\]\s+(?<message>\S.*)$
2015-11-16T15:03:06.023 I log_format_loader.cc:537 found format: katello_log
2015-11-16T15:03:06.023 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/log4j.json
2015-11-16T15:03:06.023 D log_format_loader.cc:83 format regex: java_log/jvm = ^(?<alert_level>\w+)\s+\|\s+jvm (?<jvm_no>\d+)\s+\|\s(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\s+\| (?<timestamp_f>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<function>\w+-\d+)\]\s+(?<debug_level>\w+)\s+(?<class>[\w.]+)\s+-\s+(?<body>\S.*)
2015-11-16T15:03:06.023 D log_format_loader.cc:83 format regex: java_log/dump = ^(?<alert_level>\w+)\s+\|\s+jvm (?<jvm_no>\d+)\s+\|\s(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\s+\| JVMDUMP\w+\s(?<body>\S.*)$
2015-11-16T15:03:06.023 D log_format_loader.cc:83 format regex: java_log/tasko = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<function>\w+-\d+)\]\s+(?<alert_level>\w+)\s+(?<class>[\w.]+)\s+-\s+(?<body>\S.*)$
2015-11-16T15:03:06.023 I log_format_loader.cc:537 found format: java_log
2015-11-16T15:03:06.023 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/phlog.json
2015-11-16T15:03:06.023 D log_format_loader.cc:65 Loading format -- Compellent_phlog
2015-11-16T15:03:06.023 D log_format_loader.cc:83 format regex: Compellent_phlog/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})\s+(?<hsn>\d+)\s+(?<level>\w{1}) (?<subsystem>\w+)\s+(?<body>.*)
2015-11-16T15:03:06.023 I log_format_loader.cc:537 found format: Compellent_phlog
2015-11-16T15:03:06.023 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/rayford_strace.json
2015-11-16T15:03:06.023 D log_format_loader.cc:65 Loading format -- strace_timefork_log
2015-11-16T15:03:06.023 D log_format_loader.cc:83 format regex: strace_timefork_log/std = ^(?<pid>\d+)\s+(?<timestamp>\d{2}:\d{2}:\d{2}\.\d{6}) (?<syscall>\w+)\((?<body>.*)\)\s+=\s+(?<rc>[-\w]+)(?: (?<errno>\w+) \([^\)]+\))?(?: <(?<duration>\d+\.\d+)>)?$
2015-11-16T15:03:06.024 I log_format_loader.cc:537 found format: strace_timefork_log
2015-11-16T15:03:06.024 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/rhn_xmlrpc.json
2015-11-16T15:03:06.024 D log_format_loader.cc:83 format regex: xmlrpc_log/main = ^(?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} [+-]?\d{2}:\d{2}) (?<pid>\d+) (?<client_ip>\S+): (?<module>\w+)/(?<function>.*)(?<arguments>\(.*?\))?$
2015-11-16T15:03:06.024 I log_format_loader.cc:537 found format: xmlrpc_log
2015-11-16T15:03:06.024 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/sssd_log.json
2015-11-16T15:03:06.024 D log_format_loader.cc:83 format regex: sssd_log/core = ^\((?<timestamp>\w{3} \w{3} ( \d|\d{2}) \d{2}:\d{2}:\d{2} \d{4})\) \[sssd\] \[(?<function>\w+)\] \((?<debug_level>0x[0-9a-fA-F]{4})\): (?<body>.*)$
2015-11-16T15:03:06.024 D log_format_loader.cc:83 format regex: sssd_log/module = ^\((?<timestamp>\w{3} \w{3} ( \d|\d{2}) \d{2}:\d{2}:\d{2} \d{4})\) \[sssd(?<module>\[.*?\])\] \[(?<function>\w+)\] \((?<debug_level>0x[0-9a-fA-F]{4})\): (?<body>.*)$
2015-11-16T15:03:06.024 I log_format_loader.cc:537 found format: sssd_log
2015-11-16T15:03:06.024 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/syslog_offset.json
2015-11-16T15:03:06.024 D log_format_loader.cc:65 Loading format -- syslog_offset
2015-11-16T15:03:06.024 D log_format_loader.cc:83 format regex: syslog_offset/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})-\d{2}:\d{2}\s+(?:(?<log_hostname>[a-zA-Z0-9:][^ ]+[a-zA-Z0-9]))?(?:(?: (?<log_procname>(?:[^\[:]+|[^:]+))(?:\[(?<log_pid>\d+)])?:(?<body>(?:.|\n)*))$|:?(?:(?: ---)? last message repeated \d+ times?(?: ---)?))
2015-11-16T15:03:06.025 I log_format_loader.cc:537 found format: syslog_offset
2015-11-16T15:03:06.025 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmauthd.json
2015-11-16T15:03:06.025 D log_format_loader.cc:65 Loading format -- vmauthd_log
2015-11-16T15:03:06.025 D log_format_loader.cc:83 format regex: vmauthd_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?<system>\w+)\[(?<psid>\d+)\]: (?<subsystem>\S+)\| (?<level>\w)(?<errornumber>\d+): (?<body>.*)$
2015-11-16T15:03:06.025 I log_format_loader.cc:537 found format: vmauthd_log
2015-11-16T15:03:06.025 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmkdevmgr.json
2015-11-16T15:03:06.025 D log_format_loader.cc:65 Loading format -- vmkdevmgr_log
2015-11-16T15:03:06.025 D log_format_loader.cc:83 format regex: vmkdevmgr_log/non-level = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?<system>vmkdevmgr): (?:(?<main>\w+) \[(?<level>\w+)\]:|(?<subsystem>\w+):) (?<body>.*)$
2015-11-16T15:03:06.025 I log_format_loader.cc:537 found format: vmkdevmgr_log
2015-11-16T15:03:06.025 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmkernel.json
2015-11-16T15:03:06.025 D log_format_loader.cc:83 format regex: vmkernel_log_sub/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z) (?<cpu>cpu\d+):(?<world_id>\d+)\)((?:(?<level>WARNING|ALERT)|(?<subsystem>[a-zA-Z0-9_\<\>]+)): )?(?<body>.*)
2015-11-16T15:03:06.025 I log_format_loader.cc:537 found format: vmkernel_log_sub
2015-11-16T15:03:06.026 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmkernel_combined.json
2015-11-16T15:03:06.026 D log_format_loader.cc:65 Loading format -- vmk_comb_logs
2015-11-16T15:03:06.026 D log_format_loader.cc:83 format regex: vmk_comb_logs/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z) (?<hostname>\S+) (?<cpu>cpu\d+):(?<world_id>\d+)(?:( opID=\d+)\)|\))((?:(?<level>WARNING|ALERT)|(?<subsystem>[a-zA-Z0-9_\<\>]+)): )?(?<body>.*)
2015-11-16T15:03:06.026 I log_format_loader.cc:537 found format: vmk_comb_logs
2015-11-16T15:03:06.026 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmkeventd.json
2015-11-16T15:03:06.026 D log_format_loader.cc:65 Loading format -- vmkeventd_log
2015-11-16T15:03:06.026 D log_format_loader.cc:83 format regex: vmkeventd_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?<processname>\w+)\[(?<psid>\d+)\]: (?<level>\w{1})(?<errornumber>\d+): (?:(?<messagetype>\w+):|(?<body>.*))$
2015-11-16T15:03:06.026 I log_format_loader.cc:537 found format: vmkeventd_log
2015-11-16T15:03:06.026 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmksummary.json
2015-11-16T15:03:06.026 D log_format_loader.cc:65 Loading format -- vmksummary_log
2015-11-16T15:03:06.026 D log_format_loader.cc:83 format regex: vmksummary_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?<eventtype>\w+): \w+ (?:(?<duration>\d+d\d+h\d+m\d+s), (?<vmcount>\d+) VMs\; |(?<body>.*))$
2015-11-16T15:03:06.026 I log_format_loader.cc:537 found format: vmksummary_log
2015-11-16T15:03:06.026 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmware_log.json
2015-11-16T15:03:06.027 D log_format_loader.cc:65 Loading format -- vmware_guest_log
2015-11-16T15:03:06.027 D log_format_loader.cc:83 format regex: vmware_guest_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\|\s+(?<processname>[0-9a-zA-Z\-\#]+)\|\s+(?<level>\w{1})(?<errornumber>\d+)[+:]\s+(?:(?<object>\S+):\s+|(?<body>.*))$
2015-11-16T15:03:06.027 D log_format_loader.cc:83 format regex: vmware_guest_log/alt = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\|\s+(?<processname>[0-9a-zA-Z\-\#]+)\|\s+(?:(?<object>\S+):\s+|(?<body>.*))$
2015-11-16T15:03:06.027 I log_format_loader.cc:537 found format: vmware_guest_log
2015-11-16T15:03:06.027 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vmware_syslog_log.json
2015-11-16T15:03:06.027 D log_format_loader.cc:65 Loading format -- vm_syslog_log
2015-11-16T15:03:06.027 D log_format_loader.cc:83 format regex: vm_syslog_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?<system>[a-zA-Z0-9-_\[\]]+): (?<subsystem>[a-zA-Z0-9 _-]+):? (?<method>[a-zA-Z0-9 -_]+):? (?<body>.*)$
2015-11-16T15:03:06.027 I log_format_loader.cc:537 found format: vm_syslog_log
2015-11-16T15:03:06.027 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/vobd_log.json
2015-11-16T15:03:06.027 D log_format_loader.cc:65 Loading format -- vobd_log
2015-11-16T15:03:06.027 D log_format_loader.cc:83 format regex: vobd_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z): \[(?<subsystem>[0-9a-zA-Z]+)\] (?<ussinceboot>\d+)us: \[(?<state>[0-9a-zA-Z.]+)\] (?<body>.*)$
2015-11-16T15:03:06.027 I log_format_loader.cc:537 found format: vobd_log
2015-11-16T15:03:06.027 I log_format_loader.cc:463 loading formats from file: /home/username/.lnav//formats/installed/xencenter.json
2015-11-16T15:03:06.028 D log_format_loader.cc:65 Loading format -- XenCenter_log
2015-11-16T15:03:06.028 D log_format_loader.cc:83 format regex: XenCenter_log/std = ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<level>\w+)\s+(?<method>[a-zA-Z0-9\.]+) \[(?<targetobject>[a-zA-Z0-9\s]+)\] \- (?<body>.*)
2015-11-16T15:03:06.028 I log_format_loader.cc:537 found format: XenCenter_log
2015-11-16T15:03:06.031 W log_format_loader.cc:602 Format collision, format 'cups_log' matches sample from 'error_log'
2015-11-16T15:03:06.032 E pcrepp.hh:458 pcre err -8
2015-11-16T15:03:06.033 W log_format_loader.cc:602 Format collision, format 'error_log' matches sample from 'cups_log'
2015-11-16T15:03:06.034 W log_format.cc:1255 katello_log/regex/log/pattern:body field 'body' not found in pattern
2015-11-16T15:03:06.038 W log_format.cc:1255 sudo_log/regex/std/pattern:body field 'body' not found in pattern
2015-11-16T15:03:06.041 W log_format_loader.cc:602 Format collision, format 'vm_syslog_log' matches sample from 'esxupdate_log'
2015-11-16T15:03:06.041 W log_format_loader.cc:602 Format collision, format 'vm_syslog_log' matches sample from 'vmkdevmgr_log'
2015-11-16T15:03:06.041 W log_format_loader.cc:602 Format collision, format 'vm_syslog_log' matches sample from 'vmkeventd_log'
2015-11-16T15:03:06.041 W log_format_loader.cc:602 Format collision, format 'vm_syslog_log' matches sample from 'vmksummary_log'
2015-11-16T15:03:06.041 W log_format_loader.cc:602 Format collision, format 'vmk_log' matches sample from 'vmkernel_log_sub'
2015-11-16T15:03:06.042 W log_format_loader.cc:602 Format collision, format 'vmkernel_log_sub' matches sample from 'vmk_log'
2015-11-16T15:03:06.042 W log_format_loader.cc:602 Format collision, format 'vmksummary_log' matches sample from 'vmkdevmgr_log'
2015-11-16T15:03:06.043 W log_format.cc:1245 vmware_guest_log/regex/alt/pattern:level field 'level' not found in pattern
2015-11-16T15:03:06.044 W log_format.cc:1255 xmlrpc_log/regex/main/pattern:body field 'body' not found in pattern
2015-11-16T15:03:06.044 W log_format_loader.cc:634 Detected a cycle, breaking by picking cups_log
2015-11-16T15:03:06.044 W log_format_loader.cc:634 Detected a cycle, breaking by picking vmk_log
2015-11-16T15:03:06.044 I log_format_loader.cc:650 Format order:
2015-11-16T15:03:06.044 I log_format_loader.cc:654 Compellent_phlog
2015-11-16T15:03:06.044 I log_format_loader.cc:654 XenCenter_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 access_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 autodeploy_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 block_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 candlepin_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 choose_repo_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 dpkg_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 engine_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 eqlogic_ehcmd_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 esxupdate_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 fsck_hfs_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 glog_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 java_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 katello_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 openam_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 openamdb_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 openstack_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 page_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 papertrail_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 snaplogic_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 sssd_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 strace_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 strace_timefork_log
2015-11-16T15:03:06.044 I log_format_loader.cc:654 sudo_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 syslog_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 syslog_offset
2015-11-16T15:03:06.045 I log_format_loader.cc:654 tcf_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 tcsh_history
2015-11-16T15:03:06.045 I log_format_loader.cc:654 uwsgi_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vdsm_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmauthd_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmk_comb_logs
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmkdevmgr_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmkeventd_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmw_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmware_guest_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vobd_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 xmlrpc_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmksummary_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vm_syslog_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 cups_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 error_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmk_log
2015-11-16T15:03:06.045 I log_format_loader.cc:654 vmkernel_log_sub
2015-11-16T15:03:06.048 D log_vtab_impl.cc:147 creating log format table: all_logs = 0xc208b8
2015-11-16T15:03:06.048 D log_vtab_impl.cc:147 creating log format table: generic_log = 0xc25818
2015-11-16T15:03:06.049 D log_vtab_impl.cc:147 creating log format table: Compellent_phlog = 0xc1ebe8
2015-11-16T15:03:06.049 D log_vtab_impl.cc:147 creating log format table: XenCenter_log = 0xc25458
2015-11-16T15:03:06.049 D log_vtab_impl.cc:147 creating log format table: access_log = 0xc29458
2015-11-16T15:03:06.050 D log_vtab_impl.cc:147 creating log format table: autodeploy_log = 0xc25248
2015-11-16T15:03:06.050 D log_vtab_impl.cc:147 creating log format table: block_log = 0xc28068
2015-11-16T15:03:06.050 D log_vtab_impl.cc:147 creating log format table: candlepin_log = 0xc27ba8
2015-11-16T15:03:06.050 D log_vtab_impl.cc:147 creating log format table: choose_repo_log = 0xc2b018
2015-11-16T15:03:06.051 D log_vtab_impl.cc:147 creating log format table: dpkg_log = 0xc2fb68
2015-11-16T15:03:06.051 D log_vtab_impl.cc:147 creating log format table: engine_log = 0xc1eb48
2015-11-16T15:03:06.051 D log_vtab_impl.cc:147 creating log format table: eqlogic_ehcmd_log = 0xc32088
2015-11-16T15:03:06.052 D log_vtab_impl.cc:147 creating log format table: esxupdate_log = 0xc206f8
2015-11-16T15:03:06.052 D log_vtab_impl.cc:147 creating log format table: fsck_hfs_log = 0xc31df8
2015-11-16T15:03:06.052 D log_vtab_impl.cc:147 creating log format table: glog_log = 0xc37928
2015-11-16T15:03:06.053 D log_vtab_impl.cc:147 creating log format table: java_log = 0xc39798
2015-11-16T15:03:06.053 D log_vtab_impl.cc:147 creating log format table: katello_log = 0xc3b0b8
2015-11-16T15:03:06.053 D log_vtab_impl.cc:147 creating log format table: openam_log = 0xc3b198
2015-11-16T15:03:06.054 D log_vtab_impl.cc:147 creating log format table: openamdb_log = 0xc2e6b8
2015-11-16T15:03:06.054 D log_vtab_impl.cc:147 creating log format table: openstack_log = 0xc3d998
2015-11-16T15:03:06.054 D log_vtab_impl.cc:147 creating log format table: page_log = 0xc3c3d8
2015-11-16T15:03:06.055 D log_vtab_impl.cc:147 creating log format table: papertrail_log = 0xc40248
2015-11-16T15:03:06.055 D log_vtab_impl.cc:147 creating log format table: snaplogic_log = 0xc40028
2015-11-16T15:03:06.055 D log_vtab_impl.cc:147 creating log format table: sssd_log = 0xc3fea8
2015-11-16T15:03:06.056 D log_vtab_impl.cc:147 creating log format table: strace_log = 0xc2dbc8
2015-11-16T15:03:06.056 D log_vtab_impl.cc:147 creating log format table: strace_timefork_log = 0xc21008
2015-11-16T15:03:06.056 D log_vtab_impl.cc:147 creating log format table: sudo_log = 0xc45828
2015-11-16T15:03:06.057 D log_vtab_impl.cc:147 creating log format table: syslog_log = 0xc45968
2015-11-16T15:03:06.057 D log_vtab_impl.cc:147 creating log format table: syslog_offset = 0xc478a8
2015-11-16T15:03:06.057 D log_vtab_impl.cc:147 creating log format table: tcf_log = 0xc4e848
2015-11-16T15:03:06.058 D log_vtab_impl.cc:147 creating log format table: tcsh_history = 0xc4ce08
2015-11-16T15:03:06.058 D log_vtab_impl.cc:147 creating log format table: uwsgi_log = 0xc4cce8
2015-11-16T15:03:06.058 D log_vtab_impl.cc:147 creating log format table: vdsm_log = 0xc4d018
2015-11-16T15:03:06.059 D log_vtab_impl.cc:147 creating log format table: vmauthd_log = 0xc4e998
2015-11-16T15:03:06.059 D log_vtab_impl.cc:147 creating log format table: vmk_comb_logs = 0xc20d58
2015-11-16T15:03:06.059 D log_vtab_impl.cc:147 creating log format table: vmkdevmgr_log = 0xc54388
2015-11-16T15:03:06.060 D log_vtab_impl.cc:147 creating log format table: vmkeventd_log = 0xc545f8
2015-11-16T15:03:06.060 D log_vtab_impl.cc:147 creating log format table: vmw_log = 0xc59108
2015-11-16T15:03:06.060 D log_vtab_impl.cc:147 creating log format table: vmware_guest_log = 0xc51308
2015-11-16T15:03:06.061 D log_vtab_impl.cc:147 creating log format table: vobd_log = 0xc5a078
2015-11-16T15:03:06.061 D log_vtab_impl.cc:147 creating log format table: xmlrpc_log = 0xc5a0c8
2015-11-16T15:03:06.061 D log_vtab_impl.cc:147 creating log format table: vmksummary_log = 0xc5b628
2015-11-16T15:03:06.062 D log_vtab_impl.cc:147 creating log format table: vm_syslog_log = 0xc58d88
2015-11-16T15:03:06.062 D log_vtab_impl.cc:147 creating log format table: cups_log = 0xc5c848
2015-11-16T15:03:06.062 D log_vtab_impl.cc:147 creating log format table: error_log = 0xc37258
2015-11-16T15:03:06.062 D log_vtab_impl.cc:147 creating log format table: vmk_log = 0xc60ca8
2015-11-16T15:03:06.063 D log_vtab_impl.cc:147 creating log format table: vmkernel_log_sub = 0xc372e8
2015-11-16T15:03:06.065 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/debug_log
2015-11-16T15:03:06.065 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/Xorg.log
2015-11-16T15:03:06.065 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/auth.log
2015-11-16T15:03:06.065 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/clomd.log
2015-11-16T15:03:06.065 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/dhclient.log
2015-11-16T15:03:06.065 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/epd.log
2015-11-16T15:03:06.066 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/esxupdate.log
2015-11-16T15:03:06.066 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/fdm.log
2015-11-16T15:03:06.066 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/hostd-probe.log
2015-11-16T15:03:06.066 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/hostd.log
2015-11-16T15:03:06.066 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/hostprofiletrace.log
2015-11-16T15:03:06.066 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/iofiltervpd.log
2015-11-16T15:03:06.067 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/lacp.log
2015-11-16T15:03:06.067 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/nfcd.log
2015-11-16T15:03:06.067 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/osfsd.log
2015-11-16T15:03:06.067 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/rabbitmqproxy.log
2015-11-16T15:03:06.067 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/rhttpproxy.log
2015-11-16T15:03:06.067 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/sdrsinjector.log
2015-11-16T15:03:06.068 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/shell.log
2015-11-16T15:03:06.068 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/storagerm.log
2015-11-16T15:03:06.068 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/swapobjd.log
2015-11-16T15:03:06.068 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/syslog.log
2015-11-16T15:03:06.068 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/usb.log
2015-11-16T15:03:06.068 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmauthd.log
2015-11-16T15:03:06.069 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmkdevmgr.log
2015-11-16T15:03:06.069 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmkernel.log
2015-11-16T15:03:06.069 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmkeventd.log
2015-11-16T15:03:06.069 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmksummary.log
2015-11-16T15:03:06.069 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmkwarning.log
2015-11-16T15:03:06.069 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmsyslogd-dropped.log
2015-11-16T15:03:06.070 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vobd.log
2015-11-16T15:03:06.070 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vprobe.log
2015-11-16T15:03:06.070 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vprobed.log
2015-11-16T15:03:06.070 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vpxa.log
2015-11-16T15:03:06.070 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vsanvpd.log
2015-11-16T15:03:06.070 I lnav.cc:1259 loading new file: /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vvold.log
2015-11-16T15:03:06.071 I lnav.cc:2767 startup: lnav 0.7.3-145-g97699e9
2015-11-16T15:03:06.071 I lnav_log.cc:163 uname:
2015-11-16T15:03:06.071 I lnav_log.cc:164 sysname=Linux
2015-11-16T15:03:06.071 I lnav_log.cc:165 nodename=hostname.domain.tld
2015-11-16T15:03:06.071 I lnav_log.cc:166 machine=x86_64
2015-11-16T15:03:06.071 I lnav_log.cc:167 release=3.10.0-123.13.2.el7.x86_64
2015-11-16T15:03:06.071 I lnav_log.cc:168 version=#1 SMP Thu Dec 18 14:09:13 UTC 2014
2015-11-16T15:03:06.071 I lnav_log.cc:169 Environment:
2015-11-16T15:03:06.071 I lnav_log.cc:170 HOME=/home/username
2015-11-16T15:03:06.071 I lnav_log.cc:171 LANG=en_US.UTF-8
2015-11-16T15:03:06.071 I lnav_log.cc:172 PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/username/.local/bin:/home/username/bin:/home/username/scripts:/mnt/workspace_lvm:/home/username/.local/bin:/home/username/bin:/home/username/scripts:/mnt/workspace_lvm
2015-11-16T15:03:06.071 I lnav_log.cc:173 TERM=screen-256color
2015-11-16T15:03:06.071 I lnav_log.cc:174 TZ=(null)
2015-11-16T15:03:06.071 I lnav_log.cc:175 Process:
2015-11-16T15:03:06.071 I lnav_log.cc:176 pid=22830
2015-11-16T15:03:06.071 I lnav_log.cc:177 ppid=8289
2015-11-16T15:03:06.071 I lnav_log.cc:178 pgrp=22830
2015-11-16T15:03:06.071 I lnav_log.cc:179 uid=1000
2015-11-16T15:03:06.071 I lnav_log.cc:180 gid=1001
2015-11-16T15:03:06.071 I lnav_log.cc:181 euid=1000
2015-11-16T15:03:06.071 I lnav_log.cc:182 egid=1001
2015-11-16T15:03:06.071 I lnav_log.cc:184 cwd=/mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log
2015-11-16T15:03:06.071 I lnav_log.cc:185 Executable:
2015-11-16T15:03:06.071 I lnav_log.cc:186 version=lnav 0.7.3-145-g97699e9
2015-11-16T15:03:06.071 I lnav.cc:2769 Libraries:
2015-11-16T15:03:06.071 I lnav.cc:2776 ncurses=5.9
2015-11-16T15:03:06.071 I lnav.cc:2777 pcre=8.32 2012-11-30
2015-11-16T15:03:06.071 I lnav.cc:2778 readline=6.2
2015-11-16T15:03:06.071 I lnav.cc:2779 sqlite=3.7.17
2015-11-16T15:03:06.071 I lnav.cc:2780 zlib=1.2.7
2015-11-16T15:03:06.071 I lnav.cc:2781 lnav_data:
2015-11-16T15:03:06.071 I lnav.cc:2782 flags=0
2015-11-16T15:03:06.071 I lnav.cc:2783 commands:
2015-11-16T15:03:06.071 I lnav.cc:2790 files:
2015-11-16T15:03:06.071 I lnav.cc:2795 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/*
2015-11-16T15:03:06.071 I session_data.cc:269 init_session: time=1447707786; id=b57bcbeeb9fa411dc06633445f96cdc0
2015-11-16T15:03:06.071 I lnav.cc:2891 session_id=b57bcbeeb9fa411dc06633445f96cdc0
2015-11-16T15:03:06.417 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/auth.log:1:log format found -- vm_syslog_log
2015-11-16T15:03:06.418 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/dhclient.log:1:log format found -- vm_syslog_log
2015-11-16T15:03:06.418 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/esxupdate.log:1:log format found -- esxupdate_log
2015-11-16T15:03:06.418 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/hostd-probe.log:1:log format found -- generic_log
2015-11-16T15:03:06.419 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/hostd.log:1:log format found -- generic_log
2015-11-16T15:03:06.419 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/hostprofiletrace.log:1:log format found -- generic_log
2015-11-16T15:03:06.420 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/lacp.log:1:log format found -- generic_log
2015-11-16T15:03:06.420 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/nfcd.log:1:log format found -- vm_syslog_log
2015-11-16T15:03:06.420 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/rabbitmqproxy.log:1:log format found -- generic_log
2015-11-16T15:03:06.421 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/rhttpproxy.log:1:log format found -- generic_log
2015-11-16T15:03:06.421 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/sdrsinjector.log:1:log format found -- generic_log
2015-11-16T15:03:06.422 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/shell.log:1:log format found -- vmksummary_log
2015-11-16T15:03:06.422 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/storagerm.log:1:log format found -- generic_log
2015-11-16T15:03:06.422 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/swapobjd.log:1:log format found -- generic_log
2015-11-16T15:03:06.423 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/syslog.log:1:log format found -- generic_log
2015-11-16T15:03:06.423 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/usb.log:1:log format found -- vm_syslog_log
2015-11-16T15:03:06.506 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmkeventd.log:1:log format found -- vm_syslog_log
2015-11-16T15:03:06.507 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmksummary.log:1:log format found -- vmksummary_log
2015-11-16T15:03:06.507 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vmkwarning.log:3:log format found -- vmk_log
2015-11-16T15:03:06.508 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vobd.log:1:log format found -- vobd_log
2015-11-16T15:03:06.509 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vpxa.log:1:log format found -- generic_log
2015-11-16T15:03:06.509 I logfile.cc:173 /mnt/vmware/bundle01/esx-localhost-2015-11-16--17.41/var/run/log/vvold.log:3:log format found -- generic_log
2015-11-16T15:03:06.539 I session_data.cc:356 loading bookmark db: /home/username/.lnav/log_metadata.db
2015-11-16T15:03:06.545 I session_data.cc:812 loading session file: /home/username/.lnav/view-info-b57bcbeeb9fa411dc06633445f96cdc0.ts1447707129.ppid8289.json
2015-11-16T15:03:06.547 I command_executor.cc:80 Executing: filter-out firewall
2015-11-16T15:03:06.547 D lnav_commands.cc:965 filter-out [0] firewall
2015-11-16T15:03:07.628 D session_data.cc:1294 Saved session: /home/username/.lnav/view-info-b57bcbeeb9fa411dc06633445f96cdc0.ts1447707786.ppid8289.json
Timothy Stack
Nov 16, 2015, 5:14:14 PM11/16/15
to hagfelsh, lnav
Can you move to those lines and press 'P' to show what lnav thinks the time is
for those lines. That might help explain what is going on. If nothing shows up
when you press 'P' on one of those lines, then lnav didn't recognize it as a log
message and that's why it's not sorted correctly.
If you're able to send me the files, I can take a closer look.
tim
--
You received this message because you are subscribed to the Google Groups "lnav" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lnav+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
hagfelsh
Nov 17, 2015, 4:16:14 PM11/17/15
to lnav, hagf...@gmail.com
Tim,
I can't send you those files in particular, but I'll see about sanitizing a sample.
In the meantime, I just encountered this again, this time from stdin!
I was looking for a particular string out of vobd.* and found that the results, having come from different files, were being sent to stdin out of order. Lnav didn't seem to reorder them, despite recognizing every line as vobd_log.
Here's a Ctrl + L of the data as recieved from stdin:
2015-10-30T04:39:03.010Z: [UserLevelCorrelator] 609238511223us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore Datastore3, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-31T21:50:04.153Z: [UserLevelCorrelator] 757499653842us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-11-01T06:39:12.802Z: [UserLevelCorrelator] 789248303192us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-11-01T14:29:42.320Z: [UserLevelCorrelator] 817477820590us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-25T01:53:01.882Z: [UserLevelCorrelator] 167277382867us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-25T17:32:36.264Z: [UserLevelCorrelator] 223651764723us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-26T01:13:07.075Z: [UserLevelCorrelator] 251282576351us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-26T06:32:55.816Z: [UserLevelCorrelator] 270471317086us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-27T04:08:19.711Z: [UserLevelCorrelator] 348195211891us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-20T22:43:41.477Z: [UserLevelCorrelator] 270093480823us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-21T03:19:20.617Z: [UserLevelCorrelator] 286632619996us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 1, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-21T07:38:42.293Z: [UserLevelCorrelator] 11810669788us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-24T06:23:14.431Z: [UserLevelCorrelator] 97089931782us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-24T17:19:39.732Z: [UserLevelCorrelator] 136475232849us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-18T06:58:31.898Z: [UserLevelCorrelator] 40583901673us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-18T10:20:06.781Z: [UserLevelCorrelator] 52678784797us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-18T12:52:05.671Z: [UserLevelCorrelator] 61797673967us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-18T15:20:22.085Z: [UserLevelCorrelator] 70694088179us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-18T23:21:16.725Z: [UserLevelCorrelator] 99548728181us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T01:17:18.989Z: [UserLevelCorrelator] 106510992751us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 1, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T03:05:14.453Z: [UserLevelCorrelator] 112986455972us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T05:17:16.811Z: [UserLevelCorrelator] 120908814161us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T07:21:15.459Z: [UserLevelCorrelator] 128347462210us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T10:21:17.943Z: [UserLevelCorrelator] 139149946781us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T13:12:53.810Z: [UserLevelCorrelator] 149445813116us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 1, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T13:13:01.830Z: [UserLevelCorrelator] 149453833617us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T15:49:29.714Z: [UserLevelCorrelator] 158841717580us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-19T23:51:31.425Z: [UserLevelCorrelator] 187763427925us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-20T02:47:20.804Z: [UserLevelCorrelator] 198312807253us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-20T05:04:30.274Z: [UserLevelCorrelator] 206542276982us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-10-20T10:29:32.708Z: [UserLevelCorrelator] 226044711662us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
2015-11-16T18:08:51.696Z: [UserLevelCorrelator] 2126627196926us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
Here is a p of the top line:
Pattern: vobd_log/regex/std/pattern = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z): \[(?<subsystem>[0-9a-zA-Z]+)\] (?<ussinceboot>\d+)us: \[(?<state>[0-9a-zA-Z.]+)\] (?<body>.*)$
Current Time: 2015-11-11T11:02:57.874 Original Time: 2015-11-11T11:02:57.874 Offset: +0.000
Known message fields: (SQL table -- vobd_log)
├ subsystem = UserLevelCorrelator
├ ussinceboot = 1669073374860
├ state = esx.problem.iorm.nonviworkload
Discovered message fields: (SQL table -- logline)
├ col_0 = IO
├ col_1 = Datastore3
├ col_2 = .
├ col_3 = IO
└ col_4 = VMware
Here is a p of the 2nd line:
Pattern: vobd_log/regex/std/pattern = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z): \[(?<subsystem>[0-9a-zA-Z]+)\] (?<ussinceboot>\d+)us: \[(?<state>[0-9a-zA-Z.]+)\] (?<body>.*)$
Current Time: 2015-11-12T18:29:37.789 Original Time: 2015-11-12T18:29:37.789 Offset: +0.000
Known message fields: (SQL table -- vobd_log)
├ subsystem = UserLevelCorrelator
├ ussinceboot = 1782273290088
├ state = esx.problem.iorm.nonviworkload
Discovered message fields: (SQL table -- logline)
├ col_0 = IO
├ col_1 = EQL-DataStore
├ col_2 = 2
├ col_3 = .
├ col_4 = IO
└ col_5 = VMware
Here is a p of the 3rd line:
Pattern: vobd_log/regex/std/pattern = ^(?<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z): \[(?<subsystem>[0-9a-zA-Z]+)\] (?<ussinceboot>\d+)us: \[(?<state>[0-9a-zA-Z.]+)\] (?<body>.*)$
Current Time: 2015-11-13T02:36:23.969 Original Time: 2015-11-13T02:36:23.969 Offset: +0.000
Known message fields: (SQL table -- vobd_log)
├ subsystem = UserLevelCorrelator
├ ussinceboot = 1811479470113
├ state = esx.problem.iorm.nonviworkload
Discovered message fields: (SQL table -- logline)
├ col_0 = IO
├ col_1 = Datastore3
├ col_2 = .
├ col_3 = IO
└ col_4 = VMware
Timothy Stack
Nov 17, 2015, 7:39:43 PM11/17/15
to hagfelsh, lnav
On Tue, Nov 17, 2015 at 1:16 PM, hagfelsh <hagf...@gmail.com> wrote:
Tim,I can't send you those files in particular, but I'll see about sanitizing a sample.In the meantime, I just encountered this again, this time from stdin!I was looking for a particular string out of vobd.* and found that the results, having come from different files, were being sent to stdin out of order. Lnav didn't seem to reorder them, despite recognizing every line as vobd_log.Here's a Ctrl + L of the data as recieved from stdin:2015-10-30T04:39:03.010Z: [UserLevelCorrelator] 609238511223us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore Datastore3, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.2015-10-31T21:50:04.153Z: [UserLevelCorrelator] 757499653842us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.2015-11-01T06:39:12.802Z: [UserLevelCorrelator] 789248303192us: [esx.problem.iorm.nonviworkload] An external IO activity is detected on datastore EQL-DataStore 2, this is an unsupported configuration. Please consult IO Resource Management Documentation or follow the Ask VMware link for more information.
The output below does not seem to match up with the above lines (the dates don't match
and the ussinceboot values don't match either). Are you sure those are the right lines?
tim
Message has been deleted
Message has been deleted
hagfelsh
Nov 20, 2015, 10:51:14 AM11/20/15
to lnav, hagf...@gmail.com
Oh dear I see that. Here's some raw data instead. To simulate the stdin issue, just cat the *nonviworkload.txt files into lnav. If you open them with lnav *_nonviworkload.txt, lnav sorts them as expected. Interesting!
Timothy Stack
Nov 25, 2015, 12:05:52 AM11/25/15
to hagfelsh, lnav
On Fri, Nov 20, 2015 at 7:48 AM, hagfelsh <hagf...@gmail.com> wrote:
Oh dear I see that. Here's some raw data instead. To simulate the stdin issue, just cat the *nonviworkload.txt files into lnav. If you open them with lnav *_nonviworkload.txt, lnav sorts them as expected. Interesting!
lnav won't reorder log messages within a file, it maintains the natural order of the messages
since it's a bit confusing to have the file contents being changed around. So, if you're
piping everything into stdin, that is considered one file and those lines won't be rearranged.
tim
hagfelsh
Dec 1, 2015, 1:30:52 PM12/1/15
to lnav, hagf...@gmail.com
Ah, well there we go! Good to know :)
Reply all
Reply to author
Forward
0 new messages