Add tag with Logstash when using so-import

[Matthew Robison]

I figured out how to add tags with Logstash using the 8200_postprocess_tagging.conf in the /etc/logstash directory.  We are looking to add a tag if a specific IP address is in the source_ip or destination_ip fields.  This worked when doing live network monitoring, but once we start using the so-import-pcap script it turns off the live network monitoring and changes something within Logstash that tagging does not work anymore.

I see that after the so-import config scripts runs it creates a conf.d.ingest.output directory within /etc/logstash.  When data is imported using the so-import script I see that the 0000_input_sguild_nids.conf adds the nids and import tags and 0007_input_import.conf adds the bro and import tags.  I've tried to add another .conf file in the directory for postprocess but it never fires.  Is there another item that gets modified when the so-import config scripts runs that changes how Logstash ingests data to get data tagging working again?

This is the filter string I put in the postprocess .conf file:

filter {

  if [destination_ip] {

    if [destination_ip] == "<IP>" or [source_ip] == "<IP>" {

      mutate {

        add_tag => [ "Match" ]

      }

    }

  }

}

[Doug Burks]

Hi Matthew,

As you noticed, so-import-pcap configures Logstash to use the /etc/logstash/conf.d.ingest.output/ directory.  This is a minimal Logstash config where Logstash doesn't really do any parsing, it simply sends logs to Elasticsearch and that's where the parsing occurs.  This is what allows so-import-pcap to run in a much smaller amount of RAM with a quicker Logstash initialization time than traditional Logstash parsing config.  So you have a couple of options:

1.  remove the LOGSTASH_OUTPUT_INGEST option from /etc/nsm/securityonion.conf.  You should be able to configure Logstash as you would with live network monitoring, but keep in mind that it will require more RAM and Logstash initialization will be slower.

2.  the better option would probably be to perform your tagging via Elasticsearch ingest node.  See our existing parsers in /etc/elasticsearch/ingest/ for examples.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/98f5df0d-8d24-49ce-af3f-88c851495a46%40googlegroups.com.

--

Doug Burks
CEO
Security Onion Solutions, LLC

[Matthew Robison]

Hi Doug

I will check into both solutions and report back the findings.  For our use case we may need Logstash to the the parsing as we will have to do lots of IF statements to tie specific IPs to specific tags on the PCAP data we are ingesting.  I did look at the Elastic ingest files and it would seem that we would need to add our IP to tag matching within each file and going through the Logstash parsing we would only need to add the filter in the post processing file in one location.

Thanks for the response.

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/98f5df0d-8d24-49ce-af3f-88c851495a46%40googlegroups.com.

[Doug Burks]

I don't think you would need to update each ingest file.  The individual parser files flow into common files (example: bro_conn --> bro_common --> common), so you should be able to add your tagging to the proper common file.  

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/0d59d123-0dae-45a0-9888-bde84a9c829f%40googlegroups.com.