Incorporating Moloch-like PCAP analysis
1,110 views
Skip to first unread message
Jay Hawk
Feb 17, 2018, 7:27:50 PM2/17/18
to security-onion
Hey Guys,
My team recently stood up an instance of Moloch to analyze large repos of PCAP. Does SecurityOnion have any plans to incorporate something like that for searching through PCAP data? CAPme's tcpflow is nice but being able to index and search all of your PCAP from a web interface could be pretty sweet for certain usecases.
My team recently stood up an instance of Moloch to analyze large repos of PCAP. Does SecurityOnion have any plans to incorporate something like that for searching through PCAP data? CAPme's tcpflow is nice but being able to index and search all of your PCAP from a web interface could be pretty sweet for certain usecases.
Doug Burks
Feb 18, 2018, 6:54:37 AM2/18/18
to securit...@googlegroups.com
Hi Jay,
Could you provide more detail on the use cases you're referring to?
Perhaps there is some way you can accomplish the same using Security
Onion today.
Also see this previous discussion:
https://groups.google.com/d/topic/security-onion/fAn5eoJYfFY/discussion
On Sat, Feb 17, 2018 at 7:27 PM, Jay Hawk <id1010...@gmail.com> wrote:
> Hey Guys,
> My team recently stood up an instance of Moloch to analyze large repos of PCAP. Does SecurityOnion have any plans to incorporate something like that for searching through PCAP data? CAPme's tcpflow is nice but being able to index and search all of your PCAP from a web interface could be pretty sweet for certain usecases.
>
> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
--
Doug Burks
Could you provide more detail on the use cases you're referring to?
Perhaps there is some way you can accomplish the same using Security
Onion today.
Also see this previous discussion:
https://groups.google.com/d/topic/security-onion/fAn5eoJYfFY/discussion
On Sat, Feb 17, 2018 at 7:27 PM, Jay Hawk <id1010...@gmail.com> wrote:
> Hey Guys,
> My team recently stood up an instance of Moloch to analyze large repos of PCAP. Does SecurityOnion have any plans to incorporate something like that for searching through PCAP data? CAPme's tcpflow is nice but being able to index and search all of your PCAP from a web interface could be pretty sweet for certain usecases.
>
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
--
Doug Burks
Jay Hawk
Feb 18, 2018, 8:02:15 PM2/18/18
to security-onion
Hey Doug,
The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
Another big thing is correlating data points back to PCAP easily, there are several instances where quickly transitioning to the correct PCAP files related to an event don’t seem possible via kibana and require manual command-line searches of that PCAP data. This happens for example with:
IPV6
Port not specified
Not TCP or UDP
Additionally, while I definitely prefer the UX and UI of SecurityOnion at this point, there are one of two features in Moloch that I haven’t seen SecurityOnion Elastic take advantage of (bear in mind I haven’t implemented your latest release yet, so I could just be missing some things):
Graph View - https://www.elastic.co/products/x-pack/graph
This is implemented in Moloch as seen in the demo here: https://molo.ch/#demo
It reminds me of the maltego link analysis charts that could help us better visualize relationships in traffic.
I also feel the help page inside of SecurityOnion could be fleshed out, I think Molochs use of the Markdown/Help page is on point. As it acts as a mobile wiki/reference all by itself. Building this out could make the transition from ELSA to Elastic a lot easier. I plan on building out this type of help page for my team as a reference
Their implementation: https://demo.molo.ch/help?date=-1#sessions
There may be something to their implimentation of cyberchef when performing network analysis. https://gchq.github.io/CyberChef/
While it clearly has more applications in malware analysis the Entropy detection, deobfuscation, regex searching, exif extraction, hashing, etc still have applications in network analysis and could make life easier in some instances.
That said, you guys are doing amazing work, and I look forward to setting up the latest release.
Thanks,
Jay
The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
Another big thing is correlating data points back to PCAP easily, there are several instances where quickly transitioning to the correct PCAP files related to an event don’t seem possible via kibana and require manual command-line searches of that PCAP data. This happens for example with:
IPV6
Port not specified
Not TCP or UDP
Additionally, while I definitely prefer the UX and UI of SecurityOnion at this point, there are one of two features in Moloch that I haven’t seen SecurityOnion Elastic take advantage of (bear in mind I haven’t implemented your latest release yet, so I could just be missing some things):
Graph View - https://www.elastic.co/products/x-pack/graph
This is implemented in Moloch as seen in the demo here: https://molo.ch/#demo
It reminds me of the maltego link analysis charts that could help us better visualize relationships in traffic.
I also feel the help page inside of SecurityOnion could be fleshed out, I think Molochs use of the Markdown/Help page is on point. As it acts as a mobile wiki/reference all by itself. Building this out could make the transition from ELSA to Elastic a lot easier. I plan on building out this type of help page for my team as a reference
Their implementation: https://demo.molo.ch/help?date=-1#sessions
There may be something to their implimentation of cyberchef when performing network analysis. https://gchq.github.io/CyberChef/
While it clearly has more applications in malware analysis the Entropy detection, deobfuscation, regex searching, exif extraction, hashing, etc still have applications in network analysis and could make life easier in some instances.
That said, you guys are doing amazing work, and I look forward to setting up the latest release.
Thanks,
Jay
Doug Burks
Feb 19, 2018, 6:53:54 AM2/19/18
to securit...@googlegroups.com
Hi Jay,
Replies inline.
On Sun, Feb 18, 2018 at 8:02 PM, Jay Hawk <id1010...@gmail.com> wrote:
> Hey Doug,
>
> The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
>
> But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
I have an idea for how we could implement a replay function that would
retain timestamps, I just haven't had the time to implement it.
> Another big thing is correlating data points back to PCAP easily, there are several instances where quickly transitioning to the correct PCAP files related to an event don’t seem possible via kibana and require manual command-line searches of that PCAP data. This happens for example with:
> IPV6
> Port not specified
> Not TCP or UDP
How often does this happen?
> Additionally, while I definitely prefer the UX and UI of SecurityOnion at this point, there are one of two features in Moloch that I haven’t seen SecurityOnion Elastic take advantage of (bear in mind I haven’t implemented your latest release yet, so I could just be missing some things):
>
> Graph View - https://www.elastic.co/products/x-pack/graph
X-Pack is not open source and so we can't include it.
> This is implemented in Moloch as seen in the demo here: https://molo.ch/#demo
> It reminds me of the maltego link analysis charts that could help us better visualize relationships in traffic.
There is an open source plugin called kbn_network
(https://github.com/dlumbrer/kbn_network) that you can install as
follows:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Kibana
Additionally, Kibana 6.2 includes Vega for additional visualization
capabilities:
https://github.com/nyurik/kibana-vega-vis
We hope to include Kibana 6.2 in our Release Candidate 3.
> I also feel the help page inside of SecurityOnion could be fleshed out, I think Molochs use of the Markdown/Help page is on point. As it acts as a mobile wiki/reference all by itself. Building this out could make the transition from ELSA to Elastic a lot easier. I plan on building out this type of help page for my team as a reference
> Their implementation: https://demo.molo.ch/help?date=-1#sessions
We recently rewrote our Help page and have plans on extending it as well.
>
> There may be something to their implimentation of cyberchef when performing network analysis. https://gchq.github.io/CyberChef/
>
> While it clearly has more applications in malware analysis the Entropy detection, deobfuscation, regex searching, exif extraction, hashing, etc still have applications in network analysis and could make life easier in some instances.
Yes, I've seen cyberchef and we may add it at some point as well.
> That said, you guys are doing amazing work, and I look forward to setting up the latest release.
RC2 is scheduled for release tomorrow, but you can test it today if you like!
https://groups.google.com/d/topic/security-onion-testing/_NuzYTnN38c/discussion
Thanks!
--
Doug Burks
Replies inline.
On Sun, Feb 18, 2018 at 8:02 PM, Jay Hawk <id1010...@gmail.com> wrote:
> Hey Doug,
>
> The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
>
> But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
retain timestamps, I just haven't had the time to implement it.
> Another big thing is correlating data points back to PCAP easily, there are several instances where quickly transitioning to the correct PCAP files related to an event don’t seem possible via kibana and require manual command-line searches of that PCAP data. This happens for example with:
> IPV6
> Port not specified
> Not TCP or UDP
> Additionally, while I definitely prefer the UX and UI of SecurityOnion at this point, there are one of two features in Moloch that I haven’t seen SecurityOnion Elastic take advantage of (bear in mind I haven’t implemented your latest release yet, so I could just be missing some things):
>
> Graph View - https://www.elastic.co/products/x-pack/graph
> This is implemented in Moloch as seen in the demo here: https://molo.ch/#demo
> It reminds me of the maltego link analysis charts that could help us better visualize relationships in traffic.
(https://github.com/dlumbrer/kbn_network) that you can install as
follows:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Kibana
Additionally, Kibana 6.2 includes Vega for additional visualization
capabilities:
https://github.com/nyurik/kibana-vega-vis
We hope to include Kibana 6.2 in our Release Candidate 3.
> I also feel the help page inside of SecurityOnion could be fleshed out, I think Molochs use of the Markdown/Help page is on point. As it acts as a mobile wiki/reference all by itself. Building this out could make the transition from ELSA to Elastic a lot easier. I plan on building out this type of help page for my team as a reference
> Their implementation: https://demo.molo.ch/help?date=-1#sessions
>
> There may be something to their implimentation of cyberchef when performing network analysis. https://gchq.github.io/CyberChef/
>
> While it clearly has more applications in malware analysis the Entropy detection, deobfuscation, regex searching, exif extraction, hashing, etc still have applications in network analysis and could make life easier in some instances.
> That said, you guys are doing amazing work, and I look forward to setting up the latest release.
https://groups.google.com/d/topic/security-onion-testing/_NuzYTnN38c/discussion
Thanks!
--
Doug Burks
Doug Burks
Feb 19, 2018, 4:59:19 PM2/19/18
to securit...@googlegroups.com
On Mon, Feb 19, 2018 at 6:53 AM, Doug Burks <doug....@gmail.com> wrote:
> Hi Jay,
>
> Replies inline.
>
> On Sun, Feb 18, 2018 at 8:02 PM, Jay Hawk <id1010...@gmail.com> wrote:
>> Hey Doug,
>>
>> The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
>>
>> But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
>
> I have an idea for how we could implement a replay function that would
> retain timestamps, I just haven't had the time to implement it.
I just whipped up a quick and dirty EXPERIMENTAL script that will
> Hi Jay,
>
> Replies inline.
>
> On Sun, Feb 18, 2018 at 8:02 PM, Jay Hawk <id1010...@gmail.com> wrote:
>> Hey Doug,
>>
>> The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
>>
>> But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
>
> I have an idea for how we could implement a replay function that would
> retain timestamps, I just haven't had the time to implement it.
import a pcap file into Security Onion Elastic Stack Release Candidate
2 (14.04.5.8 ISO) and retain the original timestamps:
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
It will do the following:
- stop and disable Curator to avoid closing old indices
- generate Bro logs and store them in Elasticsearch with the
timestamps of the original events
- split pcap into separate daily pcaps and store them where sguil's
pcap_agent can find them
Requirements:
- You must be running at least Security Onion Elastic Stack Release
Candidate 2 (14.04.5.8 ISO).
- You must have a sniffing interface defined (you can choose
Evaluation Mode in the Setup wizard).
TODO:
- generate IDS alerts using Snort or Suricata
Please let us know what you think.
Thanks!
--
Doug Burks
Jay Hawk
Feb 19, 2018, 5:02:52 PM2/19/18
to security-onion
Hey Doug,
Replies inline :D
On Monday, February 19, 2018 at 6:53:54 AM UTC-5, Doug Burks wrote:
> Hi Jay,
>
> Replies inline.
>
> > Another big thing is correlating data points back to PCAP easily, there are several instances where quickly transitioning to the correct PCAP files related to an event don’t seem possible via kibana and require manual command-line searches of that PCAP data. This happens for example with:
> > IPV6
> > Port not specified
> > Not TCP or UDP
>
> How often does this happen?
This happens every time I've tried to transition to CapMe via an _id value on IPV6 Traffic and there is a fair amount of IPV6 in the network I monitor. Capme says: "Destination IP is IPV6! CapMe currently only supports IPV4"
This also happens when attempting to view packet data related to ICMP or other non-TCP/UDP protocols. Giving the "CapMe Currently only supports TCP and UDP".
This also has happened when attempting to view events related to and "unknown_protocol..." from bro_weird where I receive a "Missing destination port"
If I were to give you a percentage of total traffic that causes this issue I'd say 25% at most, however traffic that generates these types of issue tend to be a bit more interesting.. having traffic with unknown protocols for example is something I'd want to investigate.
> > Additionally, while I definitely prefer the UX and UI of SecurityOnion at this point, there are one of two features in Moloch that I haven’t seen SecurityOnion Elastic take advantage of (bear in mind I haven’t implemented your latest release yet, so I could just be missing some things):
> >
> > Graph View - https://www.elastic.co/products/x-pack/graph
>
> X-Pack is not open source and so we can't include it.
>
> > This is implemented in Moloch as seen in the demo here: https://molo.ch/#demo
> > It reminds me of the maltego link analysis charts that could help us better visualize relationships in traffic.
>
> There is an open source plugin called kbn_network
> (https://github.com/dlumbrer/kbn_network) that you can install as
> follows:
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Kibana
Not bad.
> Additionally, Kibana 6.2 includes Vega for additional visualization
> capabilities:
> https://github.com/nyurik/kibana-vega-vis
>
> We hope to include Kibana 6.2 in our Release Candidate 3.
These network diagram visuals look great and go way beyond what I was expecting. https://vega.github.io/vega/examples/
The Kibana community is really giving Splunk a run for their money. Looking forward to RC3.
> > I also feel the help page inside of SecurityOnion could be fleshed out, I think Molochs use of the Markdown/Help page is on point. As it acts as a mobile wiki/reference all by itself. Building this out could make the transition from ELSA to Elastic a lot easier. I plan on building out this type of help page for my team as a reference
> > Their implementation: https://demo.molo.ch/help?date=-1#sessions
>
> We recently rewrote our Help page and have plans on extending it as well.
Couldn't locate the file in your github, but I look forward to seeing it after the update.
> >
> > There may be something to their implimentation of cyberchef when performing network analysis. https://gchq.github.io/CyberChef/
> >
> > While it clearly has more applications in malware analysis the Entropy detection, deobfuscation, regex searching, exif extraction, hashing, etc still have applications in network analysis and could make life easier in some instances.
>
> Yes, I've seen cyberchef and we may add it at some point as well.
>
> > That said, you guys are doing amazing work, and I look forward to setting up the latest release.
>
> RC2 is scheduled for release tomorrow, but you can test it today if you like!
> https://groups.google.com/d/topic/security-onion-testing/_NuzYTnN38c/discussion
>
> Thanks!
Nice, installing now! Thanks for everything you guys are doing!
>
> --
> Doug Burks
Replies inline :D
On Monday, February 19, 2018 at 6:53:54 AM UTC-5, Doug Burks wrote:
> Hi Jay,
>
> Replies inline.
>
> On Sun, Feb 18, 2018 at 8:02 PM, Jay Hawk <> wrote:
> > Hey Doug,
> >
> > The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
> >
> > But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
>
> I have an idea for how we could implement a replay function that would
> retain timestamps, I just haven't had the time to implement it.
This is awesome and I'm eager to see what you have in mind, please let me know if you need any testers when you find the time to work on this.
> > Hey Doug,
> >
> > The biggest tool gap for me right now is being able to readily process large volumes of previously collected PCAP data and import that into Kibana without negatively affecting timestamps. While this can be accomplished the methods I've seen for this seem hackish as this just isn’t how SecurityOnion was built to function.
> >
> > But there would be a HUGE benefit to utilizing SecurityOnion in such a “mobile analysis” mode. If you have many customers in many different locations who collect PCAP but utilize it differently than you then this would be a solid middle ground.
>
> I have an idea for how we could implement a replay function that would
> retain timestamps, I just haven't had the time to implement it.
> > Another big thing is correlating data points back to PCAP easily, there are several instances where quickly transitioning to the correct PCAP files related to an event don’t seem possible via kibana and require manual command-line searches of that PCAP data. This happens for example with:
> > IPV6
> > Port not specified
> > Not TCP or UDP
>
> How often does this happen?
This also happens when attempting to view packet data related to ICMP or other non-TCP/UDP protocols. Giving the "CapMe Currently only supports TCP and UDP".
This also has happened when attempting to view events related to and "unknown_protocol..." from bro_weird where I receive a "Missing destination port"
If I were to give you a percentage of total traffic that causes this issue I'd say 25% at most, however traffic that generates these types of issue tend to be a bit more interesting.. having traffic with unknown protocols for example is something I'd want to investigate.
> > Additionally, while I definitely prefer the UX and UI of SecurityOnion at this point, there are one of two features in Moloch that I haven’t seen SecurityOnion Elastic take advantage of (bear in mind I haven’t implemented your latest release yet, so I could just be missing some things):
> >
> > Graph View - https://www.elastic.co/products/x-pack/graph
>
> X-Pack is not open source and so we can't include it.
>
> > This is implemented in Moloch as seen in the demo here: https://molo.ch/#demo
> > It reminds me of the maltego link analysis charts that could help us better visualize relationships in traffic.
>
> There is an open source plugin called kbn_network
> (https://github.com/dlumbrer/kbn_network) that you can install as
> follows:
> https://github.com/Security-Onion-Solutions/security-onion/wiki/Kibana
> Additionally, Kibana 6.2 includes Vega for additional visualization
> capabilities:
> https://github.com/nyurik/kibana-vega-vis
>
> We hope to include Kibana 6.2 in our Release Candidate 3.
The Kibana community is really giving Splunk a run for their money. Looking forward to RC3.
> > I also feel the help page inside of SecurityOnion could be fleshed out, I think Molochs use of the Markdown/Help page is on point. As it acts as a mobile wiki/reference all by itself. Building this out could make the transition from ELSA to Elastic a lot easier. I plan on building out this type of help page for my team as a reference
> > Their implementation: https://demo.molo.ch/help?date=-1#sessions
>
> We recently rewrote our Help page and have plans on extending it as well.
> >
> > There may be something to their implimentation of cyberchef when performing network analysis. https://gchq.github.io/CyberChef/
> >
> > While it clearly has more applications in malware analysis the Entropy detection, deobfuscation, regex searching, exif extraction, hashing, etc still have applications in network analysis and could make life easier in some instances.
>
> Yes, I've seen cyberchef and we may add it at some point as well.
>
> > That said, you guys are doing amazing work, and I look forward to setting up the latest release.
>
> RC2 is scheduled for release tomorrow, but you can test it today if you like!
> https://groups.google.com/d/topic/security-onion-testing/_NuzYTnN38c/discussion
>
> Thanks!
>
> --
> Doug Burks
Jay Hawk
Feb 19, 2018, 5:05:52 PM2/19/18
to security-onion
Awesome! I'll test this out tonight after installing the latest RC.
Thanks,
Jay
On Monday, February 19, 2018 at 4:59:19 PM UTC-5, Doug Burks wrote:
Jay Hawk
Feb 19, 2018, 10:24:28 PM2/19/18
to security-onion
Hey Doug,
The import script seems to be working flawlessly for Bro.
Thanks,
Jay
On Monday, February 19, 2018 at 4:59:19 PM UTC-5, Doug Burks wrote:
The import script seems to be working flawlessly for Bro.
Thanks,
Jay
On Monday, February 19, 2018 at 4:59:19 PM UTC-5, Doug Burks wrote:
Doug Burks
Feb 20, 2018, 6:04:28 AM2/20/18
to securit...@googlegroups.com
Thanks, Jay!
Doug Burks
Feb 21, 2018, 3:41:29 PM2/21/18
to securit...@googlegroups.com
I've just updated so-import-pcap with the following features:
- allows you to specify multiple pcap files at once
- generates IDS alerts using Snort or Suricata (based on $ENGINE in
/etc/nsm/securityonion.conf)
If you view the IDS alerts in Sguil or Squert, they will show the
original timestamps. However, if you view the IDS alerts in Kibana,
the timestamps will be for the time of import. We can fix these
Kibana timestamps, but it will require a little more plumbing work.
In the meantime, please check out the current version and let us know
what you think:
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
Thanks!
- allows you to specify multiple pcap files at once
- generates IDS alerts using Snort or Suricata (based on $ENGINE in
/etc/nsm/securityonion.conf)
If you view the IDS alerts in Sguil or Squert, they will show the
original timestamps. However, if you view the IDS alerts in Kibana,
the timestamps will be for the time of import. We can fix these
Kibana timestamps, but it will require a little more plumbing work.
In the meantime, please check out the current version and let us know
what you think:
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
Thanks!
Jay Hawk
Feb 21, 2018, 4:34:06 PM2/21/18
to security-onion
Hey Doug,
This is great, from what I can see it's working exactly how you say it should.
This is great, from what I can see it's working exactly how you say it should.
Obviously having the alerts with their creation time in kibana will be best, but being able to align them using squert will still work out for my needs.
I've been watching your commits since about 4 hours ago when you added support for multiple PCAPs. To see how how quick you threw this script together in just a couple of days? Awesome.
Thanks again Doug!
Jay
Doug Burks
Feb 23, 2018, 5:33:31 PM2/23/18
to securit...@googlegroups.com
I just added initial support for setting timestamps properly for IDS alerts:
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
Please try it out and let us know how it goes.
Thanks!
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
Please try it out and let us know how it goes.
Thanks!
Jay Hawk
Feb 23, 2018, 5:47:35 PM2/23/18
to securit...@googlegroups.com
Thanks Doug I'll try this out this weekend.
Another thought... if I were to use an instance of securityonion to only perform this offline analysis. What do you think would be the easiest way to stop the unneeded background processes?
I'm still having issues with exceeding memory usage. I havent been able to play with the system today but would so-stop and so-elastic-start do the trick? It almost seems too easy.
Thanks,
Jay
Doug Burks
Feb 24, 2018, 7:30:46 AM2/24/18
to securit...@googlegroups.com
The latest version of so-import-pcap now stops and disables the
following processes:
- Bro
- Snort
- Suricata
- netsniff-ng
- ossec_agent
The remaining processes are needed for Kibana, Squert, and Sguil to
function properly.
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
Please try it out and let us know what you think.
Thanks!
following processes:
- Bro
- Snort
- Suricata
- netsniff-ng
- ossec_agent
The remaining processes are needed for Kibana, Squert, and Sguil to
function properly.
https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-import-pcap
Please try it out and let us know what you think.
Thanks!
Jay Hawk
Feb 24, 2018, 5:37:09 PM2/24/18
to security-onion
Hey Doug,
Been playing around with so-import-pcap, stopping the services seems to have solved the issues I'd been having previously with memory consumption.
Been playing around with so-import-pcap, stopping the services seems to have solved the issues I'd been having previously with memory consumption.
With that said, am I right to assume the so-setup (or some other mechanism) will not revert the changes made to:
/etc/nsm/$SENSOR/barnyard*.conf
/etc/logstash/conf.d/1033_preprocess_snort.conf
/etc/nsm/securityonion.conf
/etc/nsm/$SENSOR/sensor.conf
and reverting back will require manual modification or re-installation? Would simply creating backups of these files and running so-setup/restart be the easiest way for a user to reset this type of configuration?
Thanks,
Jay
>
>
>
> --
> Doug Burks
Doug Burks
Feb 25, 2018, 6:53:33 AM2/25/18
to securit...@googlegroups.com
Hi Jay,
Re-running Setup *should* overwrite all those files and return the
system to "normal" sniffing and output. (Note that the changes in
/etc/logstash/conf.d/1033_preprocess_snort.conf don't hinder normal
operation and so they will be included in RC3 by default.)
If you're running in a VM with snapshot capability, then another
option would be to create your VM, run Setup, and snapshot before
running so-import-pcap, so that you can then revert back to that
snapshot.
I've updated the usage statement in so-import-pcap to reflect this:
https://github.com/Security-Onion-Solutions/securityonion-elastic/commit/ef5e581ebfb2bab915d99221738cf40c4671bab0
Re-running Setup *should* overwrite all those files and return the
system to "normal" sniffing and output. (Note that the changes in
/etc/logstash/conf.d/1033_preprocess_snort.conf don't hinder normal
operation and so they will be included in RC3 by default.)
If you're running in a VM with snapshot capability, then another
option would be to create your VM, run Setup, and snapshot before
running so-import-pcap, so that you can then revert back to that
snapshot.
I've updated the usage statement in so-import-pcap to reflect this:
https://github.com/Security-Onion-Solutions/securityonion-elastic/commit/ef5e581ebfb2bab915d99221738cf40c4671bab0
Jay Hawk
Feb 25, 2018, 5:02:22 PM2/25/18
to security-onion
Hi Doug,
After rerunning so-setup my system had issues with logstash forwarding logs to kibana, but -Turning it off and on again- fixed whatever the issue was.
After rerunning so-setup my system had issues with logstash forwarding logs to kibana, but -Turning it off and on again- fixed whatever the issue was.
That said, you're right - a snapshot is still the easiest method.
Everything is working great.
Thanks,
Jay
Reply all
Reply to author
Forward
0 new messages