Clarify syslog-ng vs. logstash
don m.
My Palo sends data in from A.A.A.6. syslog-ng has a line, unmodified, that says s_network and accepts data from tcp, udp, with no IP address specified. I assume this means it listens to any ipv4 address.
To route my incoming data I added the lines below about line 68, after the various source _bro* lines (these look like they read a bro written file). My objective is to have palo FW data going to both a rotated log file as well as being routed through logstash to elastic.
filter f_palo { host ( "A.A.A.6" ); };
destination d_palo { file "/var/log/palo.log" ); };
log { source (s_network); filter (f_palo ); destination (d_palo ); };
My general understanding is that these stmts will "take any data that comes in from the network source and if the host is A.A.A.6, send it do a specific log file", and then line 113 will also send it to logstash.
I created blocks like this for the dns name, and the host name itself.
I do see Palo Alto traffic logs inside of Kibana. And I did check the times to make sure they were AFTER I attempted to edit the syslog-ng conf file.
Wes Lambert
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
don m.
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/eZhjC1kRmD8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
don m.
Yep, still not experiencing success. What I did over the weekend - built out a generic ELK platform using the instructions provided here:
https://github.com/sm-biz/paloalto-elasticstack-viz/wiki
End result - I had authenticated access to Kibana via nginx, was receiving Palo Alto data, didn't have to change the configuration of the PAN-OS logstash file. I did have to get lots of other things right - fair amount of type-o's, but in the end, the instructions got me to a running ELK instance w/ Palo alto data, 2 of the 4 necessary indexes, searches, and dashboards.
My assumption is that I can lift out the PAN-OS conf file, update the listening port to 5614 (something on SO is listening on 5514), update my Palo to point there, verify w/ tcpdump I am getting data, and it ?should? work. OH - and yes, I did upload the two templates (traffic, and threat) to the elastic search site.
Well, it does not appear to work, meaning:
In the "discover" area, I can't search for "palo" and get firewall data, just certificate matches.
I don't see the other two indexes show up in the kibana/management area either.
Wes Lambert
Don,If you update FORWARD in iptables and edit the script used to start logstash( so-logstash-start ), you could certainly specify an arbitrary port for input.However, the best way to do this would be send logs to the SO syslog-ng on port 514, then syslog-ng will forward it to logstash, which you can configure to properly grok the input. I did something similar with my firewall, and the only thing I needed to modify was the Logstash pipeline.Hope that helps.
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
don m.
First – thanks for the time in helping out.
I was able to build up a parallel environment using vanilla ELK, following these instructions: https://github.com/sm-biz/paloalto-elasticstack-viz/wiki
In answer – I am attempting to “port” that config into SO.
Yes, I have allowed inbound access to the syslog service, and other ports, to try and get my Palo Alto data to come into SO. For example, when I send the data in using 514/UDP (default syslog), and don’t have my own attempt w/ a logstash file, I can search in the discovery tab for the term “paloalto” and find my log records. They are marked with a “syslogng” tag. (that’s good). I am making the assumption I can’t build out searches and visualizations in the sm-biz stuff until all of the fields are properly parsed through. I do want to use MUTEX (I think) to remap all of the PA’s field sets to be consistent w/ your field sets. For example, I want the palo alto provided data to use the same term for source ip address. Only makes sense.
Log stash config – Yep, stood up a standalone ELK, got something working based on getting the config from sm-biz on git hub, actually got most of his dashboards working.
I am at the place where my own logstash config, w/ very few mod’s from sm-biz’s tutorial, is now generating errors in the logstash.log file => at the point where it looks like logstash is trying to push info into elastic. So from a plumbing perspective, I consider that a very good sign. I can post back the error later today.
Indexes – the ELK environment I setup had the ability to set up indexes, and late last night I saw the two panos indexes show up in the SO Kibana interface (so this is a super positive step in the right direction).
don m.
I've finally gotten most of my Palo Alto traffic and threat logs coming into SO.
Followed much of the info here: https://github.com/sm-biz/paloalto-elasticstack-viz
I reworked the single panos.conf file from the github site, made one for filter-traffic and filter-threat, and two output files, added the two templates after I changed the names from traffic/threat to 'palotraffic/palothreat', and now I have two new indexes. Wondering if they will have dates applied to them... can't wait to pull in the searches, visualizations, and dashboards. So - it looks like I've made some progress.
There are still errors in the logstatsh file though. These errors must be coming in from a slightly different palo alto log record type that doesn't match the parsing in the file. Messages come in a few every few seconds, but there is still plenty of information in the system itself. SO - my guess is that these are some records that don't match. I enabled the "dead letter" function, thinking that would show me plain text records that failed, but not quite...helpful.
SO the question is: can I have multiple output clauses in either the syslog-ng file (which is what I wanted originlly) or in the logstash output file so maybe I could find my mismatched records?
[2018-09-19T22:44:44,394][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :a
ction=>["index", {:_id=>nil, :_index=>"panos-traffic", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x555d4df2>], :re
sponse=>{"index"=>{"_index"=>"panos-traffic", "_type"=>"doc", "_id"=>"smgB9GUBTW6gdOmV0L9f", "status"=>400, "error"=>{"typ
e"=>"mapper_parsing_exception", "reason"=>"failed to parse [LogForwardingProfile]", "caused_by"=>{"type"=>"illegal_argumen
t_exception", "reason"=>"Invalid format: \"siem\""}}}}}
[2018-09-19T22:44:44,395][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :a
ction=>["index", {:_id=>nil, :_index=>"panos-traffic", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x32e6bed7>], :re
sponse=>{"index"=>{"_index"=>"panos-traffic", "_type"=>"doc", "_id"=>"uWgB9GUBTW6gdOmV0L9f", "status"=>400, "error"=>{"typ
e"=>"mapper_parsing_exception", "reason"=>"failed to parse [LogForwardingProfile]", "caused_by"=>{"type"=>"illegal_argumen
t_exception", "reason"=>"Invalid format: \"siem\""}}}}}
Wes Lambert
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
don m.
I looked at that. In the two template files, the field is defined as "text", and in the system itself, there are numerous events that record "siem" for the logforwardprofile field. This puts me back to the original question - can I send the incoming data to a flat file via syslog-ng so that I can find the records that I suspect are miss-configured (she's one of the harpies of IT... you may have heard of her, she is often seen miss-communication...).
Thanks
Wes Lambert
--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.