HUP causes snort to segfault

[jason lytle]

I've been running snort 2.9.0.4 with pulledpork 0.5.0 for a while and things were good - pulledpork was able to HUP the running snort processes ok.

I upgraded snort to 2.9.0.5 and the HUP was working ok still with PP0.5.0

I have just upgrade PP to 0.6.0 and now when PP tries to HUP snort, I get a kernel segfault:
kernel: snort[415]: segfault at 10c48 ip 0000000000470350 sp 00007fff6759aff0 error 4 in snort[400000+eb000]

if I start snort myself after this, it starts up ok and is detecting traffic based on the alerts it wrote into snort.rules.

so i switched back to PP 0.5.0 and now the HUP also causes snort to segfault! Since I made no changes to PP0.5.0 I can't figure out why I segfault now...

anyone else hit this before? it must be something I'm missing...
thanks

[JJC]

Are you using shared object rules, or did you just switch over to use them?

JJC

--
You received this message because you are subscribed to the Google Groups "pulledpork users" group.
To post to this group, send email to pulledpo...@googlegroups.com.
To unsubscribe from this group, send email to pulledpork-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pulledpork-users?hl=en.

[jason lytle]

sorry, this is long...
I was using so_rules previously yes but I did make a change in snort.conf related to them (then changed it back):

I had this commented in 2.9.0.4 snort.conf:
#dynamicdetection directory /usr/local/lib/snort_dynamicrules

I forgot to comment it when I installed 2.9.0.5 and snort would not start because the dir did not exist. Joel Esler form snort advised me to either create the dir empty or use pulledpork to manage so_rules. I tried creating the dir but I left it empty and snort did start ok.

I then realized that the pulledpork.conf setting was the same thing while I was upgrading to pulledpork 0.6.0:
sorule_path=/etc/snort/so_rules/

thinking it is redundant (as well as working previously), so I again commented out the line from 2.9.0.5 snort.conf:
#dynamicdetection directory /usr/local/lib/snort_dynamicrules

maybe it's related?

another observation - I ran pulledpork with the -v:
HangUP Time....
        Sent kill signal to 9212 from /var/log/snort.eth0/snort_eth0.pid with result 1
        Sent kill signal to 9215 from /var/log/snort.eth1/snort_eth1.pid with result 1
        Sent kill signal to 19060 from /var/log/snort.eth0/barnyard2_eth0.pid with result 1
        Sent kill signal to 19070 from /var/log/snort.eth1/barnyard2_eth1.pid with result 1

so as far as pulledpork knows, the HUP was successful.
I am back now on PP0.5.0 and I still have the problem so I don't think it is PP directly but it only happens when PP tries to HUP.

another change i made to snort.conf:
I get inundated with these snort alerts:
[120:3:1] http_inspect: NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE[Priority: 3]

to quiet this, I removed these lines from snort.conf http_inspect section:
extended_response_inspection \
inspect_gzip \
unlimited_decompress \

after I started to get the segfaults, it also seemed that snort stopped alerting when running. I added these lines back into snort.conf to confirm that I am still getting alerts and it starts just fine (and detects that alert about 100 times /sec) so it is working ok when I start it manually.
but now when I run PP0.5.0 with the -H, I don't get a segfault but rather:
>> http_inspect:  Changing decompress_depth requires a restart.
>> Reload via Signal HUP does not work if you aren't root or are chroot'ed.

why does PP think the config has changed? I restart snort manually and then try again but this error is thrown again every time. I think this error is superseding the segfault.

I tried to delete snort.rules and sid-msg.map to have PP recreate them but it did not help.
I am now going to try deleting the files in the so_rules dir and have PP re-populate that dir.
if that doesn't work, I guess I am going to rebuild the whole box from scratch, I can't figure out what's screwed up but I know if I follow the build doc i created, everything works.

[jason lytle]

i might not have been clear above:

I only get this when trying to HUP snort with PP currently:

http_inspect:  Changing decompress_depth requires a restart

when I start snort manually, the error does not appear.

[christoph_murauer]

Hy !

The reload option of Snort is very strict and if you use so_rules it
don´t work. The information from Joel was correct. If you run Snort
the first time with the basic snort.conf you get the error about /usr/
local/lib/snort_dynamicrules and you have to out comment it in
snort.conf or create it with mkdir /usr/local/lib/snort_dynamicrules.

If you build Snort from source you have to add the option --enable-
reload otherwise it don´t work. If you use a binary version it must be
enabled. See also the Snort documentation here (point 2.9 page 128 and
following).

If you match the requirements the reload with SIGHUP shouldn´t be a
problem. Some people report also that it don´t work if you run Snort
by a dedicated user like snort and not by root (haven´t checked that -
I run it under root on Mac OS).

If you don´t match the requirements Snort restarts and don´t reload
the configuration. I can´t say more for now because I don´t know what
you change exactly - please see the doc for details and upgrade also
PP.

C.M.

[Christoph Roland Murauer]

Hy !

The reload option of Snort is very strict and if you use so_rules it don´t work. The information from Joel was correct. If you run Snort the first time with the basic snort.conf you get the error about /usr/local/lib/snort_dynamicrules and you have to out comment it in snort.conf or create it with mkdir /usr/local/lib/snort_dynamicrules. 

If you build Snort from source you have to add the option --enable-reload otherwise it don´t work. If you use a binary version it must be enabled. See also the Snort documentation here (point 2.9 page 128 and following). 

If you match the requirements the reload with SIGHUP shouldn´t be a problem. Some people report also that it don´t work if you run Snort by a dedicated user like snort and not by root (haven´t checked that - I run it under root on Mac OS).

If you don´t match the requirements Snort restarts and don´t reload the configuration. I can´t say more for now because I don´t know what you change exactly - please see the doc for details and upgrade also PP.

C.M.

[JJ Cummings]

Part of the issue may also be with the way that snort handles SO rules, and ultimately the root of why I had originally asked if you were running them. The short of it is that any time you touch the SO binary files, snort must be stopped and started, a HUP signal will not work and will cause unexpected results. I will be building in a function to PP that will handle this case and restart Snort appropriately, in an upcoming release.

Make sense?

Sent from the iRoad

[jason lytle]

It does make sense thanks

I did build snort from source and with the reload option- the hups have been working great until I played with snort.conf. what im confused about is why reversing my changes or deleting the files that PP generate doesn't fix it. Im confident that I can get it all working again if I start from a clean slate.

Always looking forward to the new releases, thanks for all the good work.

[JJ Cummings]

The point is, unless you run PP with the -T flag, it touches the SO binary files, as such a HUP becomes unusable and snort must be stopped and started.  You should see the same results if you run PP without sending a HUP, then manually HUP snort (assuming you didnt use -T).

Sent from the iRoad

[jason lytle]

yeah when I say 'confused', I mean that I am too much of a noob with this that I probably screwed it up inadvertently.
I'm sure you are right about the so_rules being the root cause but I don't understand it well enough to fix it.

however, I do run PP with the -T (runs from cron twice a month) and I don't believe I ever omitted that when running it manually:

/etc/snort/pulledpork/pulledpork.pl -l -T -c /etc/snort/pulledpork/etc/pulledpork.conf -i /etc/snort/pulledpork/etc/disablesid.conf -H

I'll get it sorted one way or another but as always, thanks for the tips and info!

[JJC]

If running with -T then it's not an so_rule problem... that forces to run only GID:1 rules...

[jason lytle]

maybe I ran it once without the -T and it touched the so_stuff?

I just tried the following and I am still getting the snort segfaults only when PP tries to HUP - snort runs ok if I kick it off when it is not running:

- moved the entire snort dir and PP dir out of the way (renamed)
- built from source snort 2.9.0.5 and put the conf files in /etc/snort
- created rules dirs: mkdir /etc/snort/rules -and- mkdir /etc/snort/so_rules
- untar'd PP 0.6.0 and moved it all to: /etc/snort/pulledpork
- edited pulledpork.conf for my setup (everything is the same from my PP0.5.0 conf)
- edited snort.conf for my setup (looks at snort.rules file that PP generates and all other rules commented)
- untar'd snort rules: snortrules-snapshot-2904.tar.gz
- copied extracted rules to snort dirs:
-- cp /rules/* /etc/snort/rules
-- cp -r /preproc_rules/ /etc/snort/
-- cp /so_rules/precompiled/Centos-5-4/i386/2.9.0.4/* /etc/snort/so_rules/
- so I am putting the precompiled so_rules in place before I run PP for the first time
- I then run PP without the HUP so that it can create the snort.rules file:
-- /etc/snort/pulledpork/pulledpork.pl -l -T -c /etc/snort/pulledpork/etc/pulledpork.conf -i /etc/snort/pulledpork/etc/disablesid.conf
- I then start snort manually and everything is ok
- I then run PP with the -H and it causes snort to segfault

something must be left over from the previous installation either with snort or with PP - this should be a clean slate but apparently it is not.
I will do this again but will wipe the box completely first which will surely catch whatever it is I am missing.

otherwise, is there something obvious I am missing in order to have a fresh, new snort and PP? While I admit that I don't fully understand shared object rules, I thought I had a fairly good handle on installation - I've installed snort/barnyard/PP/Base on clean boxes quite a few times now and I've never had this problem.

[jason lytle]

and to confirm your theory, if I send the HUP signal to snort process myself, it segfaults as well so it's nothing to do with PP as far as I can tell.