Suricata for offline mode

[emre....@btpsec.com]

Hi,

I want to use suricata for offline pcap analsis.

I have just installed SELKS-3.0rc1-nodesktop.iso.

I only changed the configuration in /etc/suricata/suricata.yaml

runmode: autofp 

Then restarted with 

sudo service suricata restart

Then I tested my pcap

suricata –c /etc/suricata/suricata.yml –r malicious.pcap

The result

[15323] 14/4/2016 -- 09:05:43 - (suricata.c:1079) <Notice> (SCPrintVersion) -- This is Suricata version 3.1dev (rev 50b33ad)

[15323] 14/4/2016 -- 09:05:48 - (detect.c:3411) <Warning> (RulesGroupByPorts) -- [ERRCODE: SC_WARN_POOR_RULE(276)] - rule 2405000: SYN-only to port(s) 22:22 w/o direction specified, disabling for toclient direction

[15323] 14/4/2016 -- 09:05:48 - (detect.c:3411) <Warning> (RulesGroupByPorts) -- [ERRCODE: SC_WARN_POOR_RULE(276)] - rule 2405001: SYN-only to port(s) 23:23 w/o direction specified, disabling for toclient direction

[15323] 14/4/2016 -- 09:05:48 - (detect.c:3411) <Warning> (RulesGroupByPorts) -- [ERRCODE: SC_WARN_POOR_RULE(276)] - rule 2405002: SYN-only to port(s) 80:80 w/o direction specified, disabling for toclient direction

...

[15323] 14/4/2016 -- 09:05:48 - (detect.c:3411) <Warning> (RulesGroupByPorts) -- [ERRCODE: SC_WARN_POOR_RULE(276)] - rule 2405042: SYN-only to port(s) 10324:10324 w/o direction specified, disabling for toclient direction

...

[15323] 14/4/2016 -- 09:05:48 - (detect.c:3411) <Warning> (RulesGroupByPorts) -- [ERRCODE: SC_WARN_POOR_RULE(276)] - rule 2405048: SYN-only to port(s) 33333:33333 w/o direction specified, disabling for toclient direction

[15323] 14/4/2016 -- 09:05:58 - (tm-threads.c:2050) <Notice> (TmThreadWaitOnThreadInit) -- all 2 packet processing threads, 4 management threads initialized, engine started.

[15323] 14/4/2016 -- 09:05:58 - (suricata.c:2475) <Notice> (main) -- Signal Received.  Stopping engine.

[15325] 14/4/2016 -- 09:06:00 - (source-pcap-file.c:390) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 507 packets, 445887 bytes

There is no activity in https://myip/suricata

Thanks for your help,

[Peter Manev]

On Thu, Apr 14, 2016 at 8:12 AM, <emre....@btpsec.com> wrote:
> Hi,
>
> I want to use suricata for offline pcap analsis.
> I have just installed SELKS-3.0rc1-nodesktop.iso.

Thanks for trying it out and giving us feedback !

This is a rule load warning - rather than the result itself that you
are after I believe. The result of the pcap run itself is in the
/var/log/suricata/eve.json

For example the warning for rule with sid - 2405048 is that in the
rule itself there is no flow direction specified - toclient/toserver -
poor rule in terms of performance in this case so Suricata actually
fixes that for you.

> [15323] 14/4/2016 -- 09:05:58 - (tm-threads.c:2050) <Notice>
> (TmThreadWaitOnThreadInit) -- all 2 packet processing threads, 4 management
> threads initialized, engine started.
> [15323] 14/4/2016 -- 09:05:58 - (suricata.c:2475) <Notice> (main) -- Signal
> Received. Stopping engine.
> [15325] 14/4/2016 -- 09:06:00 - (source-pcap-file.c:390) <Notice>
> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 507 packets,
> 445887 bytes
>

At the end of the run it should also give you some stats with how many
alerts were generated and other.
You could try adding the "-v" switch to your command line.

> There is no activity in https://myip/suricata

Try https://myip/rules

>
> Thanks for your help,

Thanks

>
> --
> IRC: Let's talk about SELKS on Freenode IRC on the #SELKS channel
> Wiki: https://github.com/StamusNetworks/SELKS/wiki
> GitHub: https://github.com/StamusNetworks/SELKS
> Blog: https://www.stamus-networks.com/theblog/
> Twitter: @StamusN
> g+: Stamus Networks
> ---
> You received this message because you are subscribed to the Google Groups
> "SELKS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to selks+un...@googlegroups.com.
> To post to this group, send email to se...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Regards,
Peter Manev

[emre....@btpsec.com]

Thank you very much for your very quick response.

You are right, the issue is about Suricata.

But I want to catch your attention to the critical point.

I am not able to replay a pcap file, simple thing, is not it weird?

Your product is based on Suricata and your are not opening an issue, but advising it to me, is not it weird?

Sincerely,

[Peter Manev]

On Thu, Apr 14, 2016 at 3:44 PM, <emre....@btpsec.com> wrote:
> Thank you very much for your very quick response.
>

Thanks for the feedback and trying it out!

> You are right, the issue is about Suricata.
> But I want to catch your attention to the critical point.
> I am not able to replay a pcap file, simple thing, is not it weird?
> Your product is based on Suricata and your are not opening an issue, but
> advising it to me, is not it weird?

What i am referring to is that Suricata detects the problem and
automatically fixes it for you :) with regards to the rule loading.
You have a warning ( (detect.c:3411) <Warning> ), then Suricata
explains the issue ( ex: SYN-only to port(s) 23:23 w/o direction
specified ) then it fixes it for you - "disabling for toclient
direction".

The warning from Suricata is because it detects that this particular
rule is not optimal (and subsequently fixes it automatically). I have
let rule writer folks at ET (Emerging Threats) about that and they are
looking into optimizing those particular rules :)

With regards to not being able to replay a pcap file:

Are you trying to replay or read a pcap? "-r" on the command line is
used to read pcaps not replay.
Can you try the following -

suricata –c /etc/suricata/suricata.yml –r malicious.pcap -v
and paste the output here - using pastebin or something similar? (Lets
see if we could get more helpful info)

(I think I requested feedback about the bellow as well in the
previous mail, not sure if you seen it)
After you read the pcap - do you have output in
/var/log/suricata/eve.json with regards to the malicious.pcap?


From your feedback in your previous mail - it seems the pcap was read -

(suricata.c:2475) <Notice> (main) -- Signal Received. Stopping engine.

(source-pcap-file.c:390) <Notice> (ReceivePcapFileThreadExitStats) --
Pcap-file module read 507 packets, 445887 bytes

How packets do you have in the malicious.pcap ?

Thanks

[emre....@btpsec.com]

Here is the http://pastebin.com/Lf81x2ap
Here is the pcap: http://malware-traffic-analysis.net/2015/08/31/2015-08-31-traffic-analysis-exercise.pcap

Yes I have output in /var/log/suricata/eve.json (do you also need it?)

Thanks,

[Peter Manev]

On Thu, Apr 14, 2016 at 4:29 PM, <emre....@btpsec.com> wrote:
> Here is the http://pastebin.com/Lf81x2ap

Judging from the output the pcap was read successfully -

[15763] 14/4/2016 -- 10:36:19 - (source-pcap-file.c:232) <Info>
(ReceivePcapFileLoop) -- pcap file end of file reached (pcap err code
0)
[15762] 14/4/2016 -- 10:36:20 - (suricata.c:2475) <Notice> (main) --
Signal Received. Stopping engine.
[15766] 14/4/2016 -- 10:36:20 - (flow-manager.c:697) <Info>
(FlowManager) -- 0 new flows, 0 established flows were timed out, 0
flows in closed state
[15762] 14/4/2016 -- 10:36:20 - (tmqh-packetpool.c:398) <Info>
(PacketPoolInit) -- preallocated 1024 packets. Total memory 3311616
[15762] 14/4/2016 -- 10:36:20 - (suricata.c:1101) <Info>
(SCPrintElapsedTime) -- time elapsed 0.256s
[15767] 14/4/2016 -- 10:36:21 - (flow-manager.c:853) <Info>
(FlowRecycler) -- 6 flows processed
[15763] 14/4/2016 -- 10:36:21 - (source-pcap-file.c:390) <Notice>

(ReceivePcapFileThreadExitStats) -- Pcap-file module read 507 packets,
445887 bytes

[15762] 14/4/2016 -- 10:36:21 - (tmqh-flow.c:223) <Info>
(TmqhOutputFlowFreeCtx) -- AutoFP - Total flow handler queues - 1
[15762] 14/4/2016 -- 10:36:21 - (tmqh-flow.c:227) <Info>
(TmqhOutputFlowFreeCtx) -- AutoFP - Queue 0 - pkts: 507
flows: 6
[15764] 14/4/2016 -- 10:36:21 - (stream-tcp.c:5173) <Info>
(StreamTcpExitPrintStats) -- Stream TCP processed 510 TCP packets
[15764] 14/4/2016 -- 10:36:21 - (alert-fastlog.c:237) <Info>
(AlertFastLogExitPrintStats) -- Fast log output wrote 5 alerts
[15762] 14/4/2016 -- 10:36:21 - (ippair.c:251) <Info>
(IPPairPrintStats) -- ippair memory usage: 398144 bytes, maximum:
16777216
[15762] 14/4/2016 -- 10:36:21 - (host.c:255) <Info> (HostPrintStats)
-- host memory usage: 398144 bytes, maximum: 16777216
[15762] 14/4/2016 -- 10:36:21 - (detect.c:3937) <Info>
(SigAddressCleanupStage1) -- cleaning up signature grouping
structure... complete

-
"pcap file end of file reached (pcap err code 0)"
and it seems you have 5 alerts on that - "Fast log output wrote 5 alerts".
So it looks like the pcap (2016-03-31-Rig-EK-after-pavtube.com.pcap)
was read and alerted on.

How many suricata's you have running on the system - just the one that
reads the pcaps right? Or you have another instance running as well?

> Here is the pcap:
> http://malware-traffic-analysis.net/2015/08/31/2015-08-31-traffic-analysis-exercise.pcap

In the output you have -
sudo suricata -c /etc/suricata/suricata.yaml -r
2016-03-31-Rig-EK-after-pavtube.com.pcap
are those two the same pcaps that you are referring to ? (since the
link you gave above is for -
2015-08-31-traffic-analysis-exercise.pcap )


Thanks

[emre....@btpsec.com]

I have just one suricata.

I have just installed new version of SELKS and did not change any settings.

There are 5 alerts on the logs but not on my web interface.

Yes the same pcap.

I am trying lots of samples, sorry for confusion.

[Peter Manev]

On Thu, Apr 14, 2016 at 4:57 PM, <emre....@btpsec.com> wrote:
> I have just one suricata.
> I have just installed new version of SELKS and did not change any settings.
> There are 5 alerts on the logs but not on my web interface.

Redoing your tests with the following command line -

root@SELKS:/opt/test# suricata -c
/etc/suricata/pcap-test-suricata.yaml -v -r
2015-08-31-traffic-analysis-exercise.pcap


I get alerts and other metadata on the SELKS Kibana dashboards ...some
alerts are :

ET CURRENT_EVENTS Possible Evil Redirector Leading to EK June 10 2015
ET POLICY Vulnerable Java Version 1.8.x Detected
ET TROJAN AlphaCrypt CnC Beacon 5
ET CURRENT_EVENTS Job314/Neutrino Flash Exploit M1 Aug 02 2015 (IE)
ET CURRENT_EVENTS Job314/Neutrino Flash Exploit M2 Aug 02 2015
ET CURRENT_EVENTS Job314/Neutrino Reboot EK Landing Aug 02 2015
ET CURRENT_EVENTS Job314/Neutrino Reboot EK Payload Aug 19 2015
ET CURRENT_EVENTS SUSPICIOUS Likely Neutrino EK or other EK IE Flash
request to DYNDNS set non-standard filename
ET POLICY Possible External IP Lookup ipinfo.io


So most likely the reason why you dont see anything on the dashboards
(and hence assuming you dont read the pcap I suspect) is because the
by default the dashboard looks at the last 24 hrs - and the pcap's
packets time stamps are from August 31st 2015

For this particular pcap example -
2015-08-31-traffic-analysis-exercise.pcap - you should set the time
span search in the Kibana dashboards (click on the upper right corner
of the dashboard - default is "Last 24 hours") to "Absolute" and
choose dates between 30 August 2015 and 01 September 2015 since the
time stamp in the packets inside the pcap is from August 31 2015 - if
you look at the alert time stamps in /var/log/suricata/eve.json you
should see something similar for timestamps -
{"timestamp":"2015-08-31T13:59:15.803549-0400","flow_id":2301385500,"pcap_cnt":10573,"event_type":"alert"........


Thanks

[emre....@btpsec.com]

pevma, I am very appreciated for your support.

I am also able to get the same alerts on my Kibana Dashboard, no problem in here.

Problem is that, there is no a activity on http://myip/suricata dashboard.

"Rules activity" and "Alerts activity" panels say "No data available".

This dashboard shows "last 7" days at maximum, I know.

However, I also read pcap file that is recorded in last 7 days, namely timestamp is greater than 2016-04-08.

Should not it populate those two panels rather than Kibana?

For example you can grab http://malware-traffic-analysis.net/2016/04/11/2016-04-11-pseudo-Darkleech-Angler-EK-after-condocosmetics.com.pcap.zip

[Peter Manev]

On Fri, Apr 15, 2016 at 10:50 AM, <emre....@btpsec.com> wrote:
> pevma, I am very appreciated for your support.
>
> I am also able to get the same alerts on my Kibana Dashboard, no problem in
> here.
> Problem is that, there is no a activity on http://myip/suricata dashboard.
> "Rules activity" and "Alerts activity" panels say "No data available".
> This dashboard shows "last 7" days at maximum, I know.

I think you mean - the Scirius interface right? That woudl be max back
7 days but not the Kibana/Evebox dashboards.
There are three different web interfaces:

Scirius - https://your.selks.ip.here/suricata/
Evebox - https://your.selks.ip.here/evebox/
11 Kibana Dashboards accessed from the (i think you already know
that) upper left corner drop down menu in Scirius. Just to avoid any
doubt in my previous mail I was referring to the Kibana dashboards
timespan selection - there you can select any timespan you would like.

> However, I also read pcap file that is recorded in last 7 days, namely
> timestamp is greater than 2016-04-08.

The reason for that could be that today is 15 april and the last 7
days (including today - 15 ) should bring you back to 09 April. Can
you try with a time stamp form 09 April for example to confirm ?

> Should not it populate those two panels rather than Kibana?
> For example you can grab
> http://malware-traffic-analysis.net/2016/04/11/2016-04-11-pseudo-Darkleech-Angler-EK-after-condocosmetics.com.pcap.zip
>

if it is from 11th it should appear in Scirius yes.
Please feel free(if you would like) to mail me the credentials that
you use for this download - off the list here.

[emre....@btpsec.com]

Yes I mean Scirius interface.

No problem with the timespan in Kibana.

Password is infected

Do not be afraid, it is just a pcap file.
timestamp is 11.04.2016 (it is inside one week period)

[Peter Manev]

On Fri, Apr 15, 2016 at 12:56 PM, <emre....@btpsec.com> wrote:
> Yes I mean Scirius interface.
> No problem with the timespan in Kibana.
>
> Password is infected
> Do not be afraid, it is just a pcap file.
> timestamp is 11.04.2016 (it is inside one week period)

Yes - I have them on the dashboard in Scirius (2 counts of each alert)
inside the 7 day timespan.

2021724 ET TROJAN Alphacrypt/TeslaCrypt Ransomware CnC Beacon
Responseemerging-trojan
2022502 ET TROJAN Suspicious Accept in HTTP POST - Possible
Alphacrypt/TeslaCryptemerging-trojan
2022504 ET TROJAN Alphacrypt/TeslaCrypt Ransomware CnC Beaconemerging-trojan

command used -

root@SELKS:/opt/test# suricata -c
/etc/suricata/pcap-test-suricata.yaml -v -r

2016-04-11-pseudo-Darkleech-Angler-EK-after-condocosmetics.com.pcap

When I run the command again I get 4 counts(in total) of each/per of
the alerts above - you just need to refresh/reload the page.

Thanks

[emre....@btpsec.com]

Is your installation SELKS-3.0rc1-nodesktop.iso?

Did you make any configuration change after installation?

[Peter Manev]

On Fri, Apr 15, 2016 at 1:44 PM, <emre....@btpsec.com> wrote:
> Is your installation SELKS-3.0rc1-nodesktop.iso?

It is the regular - desktop.iso (that part should not matter actually)

> Did you make any configuration change after installation?

nope - just the runmode change in /etc/suricata/suricata.yaml

[emre....@btpsec.com]

I assume you changed to autof, is it correct?

By the way

https:/myip/evebox/

gives this error message

URLError at /evebox/

<urlopen error [Errno 111] Connection refused>

Request Method:	GET
Request URL:	https://172.16.1.129/evebox/
Django Version:	1.8.7
Exception Type:	URLError
Exception Value:	<urlopen error [Errno 111] Connection refused>
Exception Location:	/usr/lib/python2.7/urllib2.py in do_open, line 1197
Python Executable:	/usr/bin/python
Python Version:	2.7.9
Python Path:	['/usr/local/lib/python2.7/dist-packages/git/ext/gitdb', '/opt/selks/scirius', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7', '/usr/local/lib/python2.7/dist-packages/gitdb/ext/smmap']
Server time:	Fri, 15 Apr 2016 12:11:18 +0000

[Peter Manev]

On Fri, Apr 15, 2016 at 2:42 PM, <emre....@btpsec.com> wrote:
>
> I assume you changed to autof, is it correct?

yes.

>
> By the way
> https:/myip/evebox/
> gives this error message
>

Did you have the latest packages installed?
On the SELKS box you just do (accept the changes if/when asked) -
apt-get update && apt-get dist-upgrade


then restart these services:
systemctl restart evebox.service
/etc/init.d/scirius restart

[emre....@btpsec.com]

The following package will be upgraded

evebox

Very interesting, upgraded and I restarted the services, however it is not solved the problem neither evebox nor scirius dashboard

[Peter Manev]

On Fri, Apr 15, 2016 at 3:24 PM, <emre....@btpsec.com> wrote:
> The following package will be upgraded
> evebox
>
>
> Very interesting, upgraded and I restarted the services, however it is not
> solved the problem neither evebox nor scirius dashboard
>

What does
dpkg -l |grep scirius

and

dpkg -l |grep evebox

show?

[emre....@btpsec.com]

dpkg -l |grep scirius

ii  scirius                          1.1.6-9                    all          Django application to manage Suricata ruleset

dpkg -l |grep evebox

ii  evebox                           1:0.5.0~dev20160414055646  amd64        no description given

[Peter Manev]

On Fri, Apr 15, 2016 at 4:03 PM, <emre....@btpsec.com> wrote:
> dpkg -l |grep scirius
> ii scirius 1.1.6-9 all
> Django application to manage Suricata ruleset
> dpkg -l |grep evebox
> ii evebox 1:0.5.0~dev20160414055646 amd64
> no description given
>
>

Yes - those look ok.
So you can open the Scirius dashboard but not the Evebox one -
correct? (just so we are on the same page)?

Not sure if you have restarted the nginx service (if not go ahead and
restart it) -
systemctl restart nginx.service

Please confirm if that fixes or not the issue?

[emre....@btpsec.com]

I restarted nginx, but the issues still continue.

My real problem;

Dashboards (for ex: Rules activity (last 7d) and Alerts activity (last 7d)) under https://myip/suricata/ says "No data available"

still continues

Dashboard under https://myip/rules/ and https://myip/rules/ruleset/ shows correct content, no problem in here.

After you gave me /evebox url in your previous responses, I tried it and realized second problem

> For more options, visit https://groups.google...

[Peter Manev]

On Fri, Apr 15, 2016 at 4:32 PM, <emre....@btpsec.com> wrote:
> I restarted nginx, but the issues still continue.
>
> My real problem;
> Dashboards (for ex: Rules activity (last 7d) and Alerts activity (last 7d))
> under https://myip/suricata/ says "No data available"
> still continues
>

Can you please share a screenshot?

> Dashboard under https://myip/rules/ and https://myip/rules/ruleset/ shows
> correct content, no problem in here.
>

Can you please share a screenshot for these as well?

(I will download the nondesktop iso and try to specifically reproduce that case)

> After you gave me /evebox url in your previous responses, I tried it and
> realized second problem
>

so no problems now with Evebox and/or Kibana dashboards - correct?

[emre....@btpsec.com]

http://prnt.sc/aslg5w

http://prnt.sc/aslga2

http://prnt.sc/aslgcy

http://prnt.sc/aslfyn

>> >> >> >> >> >> >> > https:...

[Peter Manev]

On Fri, Apr 15, 2016 at 4:58 PM, <emre....@btpsec.com> wrote:
> http://prnt.sc/aslg5w
> http://prnt.sc/aslga2
> http://prnt.sc/aslgcy
> http://prnt.sc/aslfyn
>

I downloaded the nondesktop iso and tried to reproduce your case - but
could not. I get the dashboards populated and everything looks as it
should.

This is rather strange indeed :)
Can you try to change multiple times the timespan in Scirius - from
6hrs to 7d to 24 hrs etc..etc.. and back , reload the page a few times
- if they are there they will be displayed (especially if the
alerts/events appear in the Kibana dashboards for the same timespan).

Thank you

[emre....@btpsec.com]

Just now I have reşnstalled.

In this case suricata dashboard is populated.

But you previously gave me a url /evebox it is still not working.

What about you?

>> >> >> >> >> >> >> >> >> >> >> > email to...

[Peter Manev]

On Mon, Apr 18, 2016 at 8:39 AM, <emre....@btpsec.com> wrote:
> Just now I have reşnstalled.
> In this case suricata dashboard is populated.
> But you previously gave me a url /evebox it is still not working.
> What about you?

It is working - however please make sure you have done (there are
some packages that need upgrading) -

apt-get update && apt-get dist-upgrade

(accept all changes) and restart the needed services -

systemctl restart evebox.service
systemctl restart nginx.service
/etc/init.d/scirius restart


Also make sure the elasticsearch.yaml config has "http.cors.enabled:
true" - commented out ("#" at the beginning of the line).

Thanks



--
Regards,
Peter Manev

[emre....@btpsec.com]

Thank you

Lessons learnt:

Re-instal

apt-get update && apt-get dist-upgrade