[ossec-list] All UNIX/LINUX agents disconnecting

[Griffith, Robert]

  We have been running the new version of Ossec 2.4 in our environment for 3 weeks.  Yesterday all of our UNIX/LINUX client agents started disconnecting.  None of our Windows Server client agents have disconnected.  Has anyone experienced this and/or found a resolution for this issue.

 

Thank you,

Robert

 

[Griffith, Robert]

[dan (ddp)]

Anything in the logs (both server and agents)?

[Pendergrast, Michael L]

Yes we have

 

although we have v1.6

 

I don't have the details as my team has worked the problem and is currently deployed. 

 

What we did find is that there is a counter in the agent and in the manager and if they get out of sequence the agent will stop (basicaqlly they get out of sequence).  We also found that at startup of the UNIX agents that if multiple agents all start at the same time, the agents will stop.  In this case, for initial startup we had to sequence the startup in about 10 min increments.

 

Mike

---

From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Griffith, Robert
Sent: Tuesday, May 11, 2010 12:26 PM
To: 'ossec...@ossec.net'
Subject: [ossec-list] All UNIX/LINUX agents disconnecting
Importance: High

[Tate Hansen]

I'm having issues with disconnecting agents as well.
- all are linux based agents
- ossec server & agents are all v2.4. (performed clean install)

On one agent, I set agent.debug=2 in internal_options.conf and observed:

[...] ossec-syscheckd: INFO: Finished creating syscheck database (pre-scan
completed).
[...] ossec-syscheckd: INFO: Starting syscheck scan (forwarding database).
[...] ossec-agentd: INFO: Event count after '20000': 3969001->3232384 (81%)
[...] ossec-agentd: WARN: Server unavailable. Setting lock.
[...] ossec-logcollector: WARN: Process locked. Waiting for permission...
[...] ossec-agentd(4102): INFO: Connected to the server
(xxx.xxx.xxx.xxx:1514).
[...] ossec-agentd: INFO: Server responded. Releasing lock.
[...] ossec-logcollector: INFO: Lock free. Continuing...
[...] ossec-agentd: INFO: Event count after '20000': 4207989->3383792 (80%)
[...] ossec-syscheckd: INFO: Ending syscheck scan (forwarding database).
[...] ossec-syscheckd: INFO: Starting real time file monitoring.
[...] ossec-rootcheck: INFO: Starting rootcheck scan.
[...] ossec-rootcheck: INFO: Ending rootcheck scan.
[...] ossec-agentd: WARN: Server unavailable. Setting lock.
[...] ossec-agentd(4102): INFO: Connected to the server
(xxx.xxx.xxx.xxx:1514).
[...] ossec-agentd: INFO: Server responded. Releasing lock.
[...] ossec-agentd: INFO: Event count after '20000': 4069230->3331752 (81%)
[...] ossec-syscheckd: INFO: Starting syscheck scan.
[...] ossec-agentd: WARN: Server unavailable. Setting lock.
[...] ossec-agentd(4102): INFO: Connected to the server
(xxx.xxx.xxx.xxx:1514).
[...] ossec-agentd: INFO: Server responded. Releasing lock.

I set remoted.debug=2 on the server, but nothing interesting appeared.
Agents will reconnect without intervention, sometimes after a few minutes
and other times it takes 20 minutes or much longer.

[Lucio Emanuel Soldo]

Hi Mike, how are you? Could you post the final solution your team has produced in order to fix its problem?

Thanx alot!

[Daniel Cid]

Hi Lucio,

There is two issues in this thread. One, the agent disconnects and
then reconnects
by itself. That's fine and can happen on high load environment or when a message
gets dropped.

The second issue that Mike mentioned happens when the counters get out of
sync and the agent never reconnects. For this problem, you have to either clean
the "rids" directory on the manager or disable the counters. To disable it, set
verify_msg_id to 0 on the internal_options.conf file:

# Verify msg id (set to 0 to disable it)
remoted.verify_msg_id=0

Thanks,

--
Daniel B. Cid
dcid ( at ) ossec.net

[Swartz, Patrick H]

Hi Daniel,
Could you expand on the effects of disabling the counters? Understand the consequences would help us decide the best path to follow.

Thank you for all you do!

Patrick Swartz
UNIX Planning & Engineering (DSUSSE)
First Data
402-777-7337 desk
402-871-8981 cell

-----------------------------------------
The information in this message may be proprietary and/or
confidential, and protected from disclosure. If the reader of this
message is not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient,
you are hereby notified that any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify First Data
immediately by replying to this message and deleting it from your
computer.

[dan (ddp)]

Not Daniel, but... The counters help protect against replay attacks.
The counter should be incremented after every message sent from the
agent to the server. If the server gets a message with the counter set
lower than the current counter on the server it will reject the
message.

[Daniel Cid]

Hi Patrick,

Yes, that's basically what Dan explained. Removing the counters would allow for
someone inside your network to replay the events into ossec.

However, if you are using syslog internally, you will have this
problem anyway... So
even using this option would not protect you.

I disable this many times on internal networks because it makes easy to manage
and use with multiple servers (they don't have to be in sync anymore).

Thanks,

--
Daniel B. Cid
dcid ( at ) ossec.net

[Griffith, Robert]

We continue to observe the Ossec Client disconnects to our Ossec Master Server over the network. We started receiving these disconnects again on Wednesday July 7th 2010. The clients disconnected daily and failed to reconnect for hours (some clients took days or never reconnected again). This issue was also observed in May on 5/10/2010. That issue lasted for two weeks and then suddenly stopped without any Ossec configuration changes.

We implemented the fix you provided below after we encountered the issue again on Wednesday July 7th 2010. We cleaned the "rids" directory and disabled the counters on both the Ossec Master Server and all UNIX/Linux Clients (set verify_msg_id to 0 on the internal_options.conf). The "fix" you provided cleared the issue for 5 days and then the Client disconnect issue re-emerged. We re-applied the "fix" again without success.

We are again experiencing disconnects and failed re-connects on all UNIX/LINUX Ossec agents.

FYI: We are using Ossec Version 2.4. Counters are disabled.

Thank you,
Robert

-----Original Message-----
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Daniel Cid
Sent: Friday, May 14, 2010 9:43 AM
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] RE: All UNIX/LINUX agents disconnecting

[Griffith, Robert]

Also, on our Ossec Master Server, we are observing that the "ossec-analysisd" uses ~100% of a single CPU (4 CPU's are available). Could this be causing any issues?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2779 ossec 25 0 8236 2932 704 R 93.2 0.1 119:14.70 ossec-analysisd

[Jeremy Rossi]

Both should not be happening. More inline. This is a hard problem to just
jump right into via email. Might want hop on to IRC and get the collective
to talk with (irc.freenode.net room #ossec).

> Also, on our Ossec Master Server, we are observing that the
> "ossec-analysisd" uses ~100% of a single CPU (4 CPU's are available).
> Could this be causing any issues?
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2779 ossec 25 0 8236 2932 704 R 93.2 0.1 119:14.70
> ossec-analysisd

> -----Original Message-----

[Griffith, Robert]

Attached is a portion of the Client Log that shows the exact time when an OSSEC client disconnects & fails to reconnect. Also is snswmsta1.pcap which contains the network trace of that particular instance and the communication between both servers. It shows the conversation between the client (Source) and the OSSEC server (Destination).

[Daniel Cid]

Hi Robert,

How many events per second is the manager processing right now?

Such high CPU utilization is not normal.. You can see these values at
/var/ossec/stats . Also, anything special in the manager log when the
disconnect happen?

Thanks,

--
Daniel B. Cid
dcid ( at ) ossec.net

[Griffith, Robert]

Daniel,

On August 3rd 2010, we observed that all of the Ossec Clients suddenly connected again and continue to stay connected. We did not make any changes to the environment on that day. The Ossec Client-Master Disconnect issue appears to be cleared for the last couple of days. We will continue to monitor the environment to see if the issue re-occurs.

Just a re-cap ... We started receiving these disconnects again on Wednesday July 7th 2010. The clients disconnected daily and failed to reconnect. This issue was also observed in May on 5/10/2010. That issue lasted for two weeks and then suddenly stopped. Cleaning out the "rids" directory and disabling the counters on both the Ossec Master Server and all Clients (set verify_msg_id to 0 on the internal_options.conf) cleared the issue for a couple of days and then the Client disconnect issue re-emerged again on July 15th 2010. As stated above, the issue observed then cleared again without any intervention on August 3rd 2010 .

Below is what I think you are looking for:

[root@nydcossec01 hourly-average]# pwd
/opt/ossec/stats/hourly-average

[root@nydcossec01 hourly-average]# ls -ltar
total 108
drwxr-x--- 5 ossec ossec 4096 Apr 17 16:03 ..
drwxr-x--- 2 ossec ossec 4096 Apr 18 00:00 .
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 9
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 8
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 7
-rw-r----- 1 ossec ossec 6 Aug 6 00:00 6
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 5
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 4
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 3
-rw-r----- 1 ossec ossec 3 Aug 6 00:00 24
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 23
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 22
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 21
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 20
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 2
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 19
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 18
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 17
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 16
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 15
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 14
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 13
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 12
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 11
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 10
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 1
-rw-r----- 1 ossec ossec 5 Aug 6 00:00 0

[root@nydcossec01 hourly-average]# cat 0
30662

Thanks,

[Doug Burks]

Was there ever any conclusion on this problem?  I have an OSSEC 2.5.1 server with 43 agents.  ossec-analysisd is using 99% CPU!  Unix agents periodically disconnect and will eventually reconnect.  What can I do to troubleshoot this further?

Thanks,

Doug Burks

[dan (ddp)]

I'd start by trying to find out why analysisd is at 99% cpu.

[Doug Burks]

Agreed. Any ideas on how to find out why analysisd is at 99% cpu? :)

Thanks,
Doug Burks

[Doug Burks]

41 agents total.

Here are the stats from /var/ossec/stats/hourly-average:
for i in *; do echo -n "$i "; cat $i; echo ""; done |sort -n
0 144467
1 135681
2 143439
3 139292
4 143869
5 139974
6 143945
7 156203
8 179020
9 199613
10 220229
11 199679
12 235240
13 200294
14 171326
15 173679
16 165433
17 116530
18 94434
19 88046
20 105235
21 98339
22 93802
23 104293
24 1124

Most of the alerts are Windows events coming from domain controllers.

Thanks,
--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

On Mon, Mar 28, 2011 at 3:25 PM, dan (ddp) <ddp...@gmail.com> wrote:
> How many agents? How many events per second? What kind of alerts are
> you seeing most of?

[dan (ddp)]

How many agents? How many events per second? What kind of alerts are
you seeing most of?

[dan (ddp)]

It doesn't look like a very busy system. I'm not sure how else to
figure out what's going on. You could go high tech and use ktrace or
something to see where it's spending its time, or low tech by turning
on debugging (run it with -d).
Sorry I can't be much help with this one.

[Rob Brooks]

I'm having the same exact problem with disconnecting Linux agents as well.  I'm at the point that I'm running tcpdump to see what the issue is because it's baffling.  We have around 30 registered agents and only around 5-10 are active at any given time (which seem to rotate in an out)...the rest of the time, logging in the client shows it's can't connect. 

I am wondering if there is a UDP with the F5 that the packets are traversing.

Here's an example from the agent:

2011/04/11 08:47:34 ossec-agentd: INFO: Event count after '20000': 2226760->2255200 (101%) <---What does this mean?
2011/04/11 08:55:47 ossec-agentd: WARN: Server unavailable. Setting lock.
2011/04/11 08:56:08 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'xxx.xxx.xxx.54'.
2011/04/11 08:56:10 ossec-agentd: INFO: Trying to connect to server (xxx.xxx.xxx.54:1514).
2011/04/11 08:56:31 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'xxx.xxx.xxx.54'.
2011/04/11 08:56:51 ossec-agentd: INFO: Trying to connect to server (xxx.xxx.xxx.54:1514).

Kind Regards,
Rob

[Jason Frisvold]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Apr 11, 2011, at 3:32 PM, Rob Brooks wrote:
> I'm having the same exact problem with disconnecting Linux agents as well. I'm at the point that I'm running tcpdump to see what the issue is because it's baffling. We have around 30 registered agents and only around 5-10 are active at any given time (which seem to rotate in an out)...the rest of the time, logging in the client shows it's can't connect.
>
> I am wondering if there is a UDP with the F5 that the packets are traversing.
>
> Here's an example from the agent:
>
> 2011/04/11 08:47:34 ossec-agentd: INFO: Event count after '20000': 2226760->2255200 (101%) <---What does this mean?
> 2011/04/11 08:55:47 ossec-agentd: WARN: Server unavailable. Setting lock.
> 2011/04/11 08:56:08 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'xxx.xxx.xxx.54'.
> 2011/04/11 08:56:10 ossec-agentd: INFO: Trying to connect to server (xxx.xxx.xxx.54:1514).
> 2011/04/11 08:56:31 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'xxx.xxx.xxx.54'.
> 2011/04/11 08:56:51 ossec-agentd: INFO: Trying to connect to server (xxx.xxx.xxx.54:1514).

Do you restart the ossec server to fix this, or do they come back on their own? I've noticed a few times that remoted has stopped functioning and apparently crashed. This generally seems to happen when I'm monkeying about with config changes. I haven't had a chance to dig into the cause as of yet.

> Kind Regards,
> Rob

- ---------------------------
Jason 'XenoPhage' Frisvold
xeno...@godshell.com
- ---------------------------
"Any sufficiently advanced magic is indistinguishable from technology."
- - Niven's Inverse of Clarke's Third Law

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.16 (Darwin)

iEYEARECAAYFAk2jrKYACgkQ8CjzPZyTUTS0+ACfW+ZjYYTMAJMnWxA64cJ/pK0C
/v8AoJLeu2SCWLLqg6/41LZTl9CWz8+i
=sFWB
-----END PGP SIGNATURE-----

[Brooks, Rob]

No, whether or not we restart the ossec service via init.d (I'm guessing this is what you meant and not "reboot"), it makes no difference. They all disconnect and periodically are able to check in when not having the errors listed below. I had previously changed the verify_msg_id=1 on the server to "0" and didn't see a difference. A few minutes ago it dawned on me to check on the agent version of internal_options.conf. I then changed it to "0" as well. Bounced both server and client...no differences.

No ideas on this one, tcpdumps aren't very revealing so far.

Thx,
Rob

-----Original Message-----
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Jason Frisvold
Sent: Monday, April 11, 2011 6:37 PM
To: ossec...@googlegroups.com
Cc: dan (ddp)
Subject: Re: [ossec-list] All UNIX/LINUX agents disconnecting and failing to reconnect

[Doug Burks]

I had two servers that were exhibiting this behavior (ossec-analysisd using 99% CPU resulting in agents disconnecting).  They were both running CentOS 5.5 and I had verified that rebooting the server didn't help.  As soon as CentOS 5.6 became available, I upgraded and rebooted, and have not seen this issue since.  This could have been a bad interaction with the kernel or some other part of the OS that has been fixed now.  

For anybody else who has experienced this, were you running CentOS/RHEL 5.5?  Can you try updating to 5.6 and see if that helps?

Thanks,

Doug

[jjennings]

how many agents was the host monitoring? I'm monitoring about 20 agents running OSSEC on a virtualized machine with Centos5.5 with only 1 cpu and  1 GB ram and it's hardly breaking 1.0 in cpu utilization.

[Doug Burks]

One of my OSSEC servers has about 40 agents and sees about 3 million
events/day. Now that the issue seems to have been resolved, it's CPU
utilization is quite low just like yours and is what I'm expecting.

I actually had 5 different OSSEC servers running RHEL/CentOS 5.5 and
only 2 of them experienced this particular issue, so I'm not saying it
happens to everybody or that it's normal. But I know there were
others in the thread who seemed to experience the same issue, so I was
asking them to see if they were perhaps running 5.5 and if the upgrade
to 5.6 resolved it for them like it seems to have resolved it for me.

Thanks,
--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

[Martin Gottlieb]

Hi,

Is OSSEC capable of triggering an active response on Windows events? In
particular, I am frequently
seeing event 18152, "Multiple Windows Logon Failures", but no active
response is ever triggered.
There are 2 (at least) different variations on the events, 1 for Windows
log-in failures and another
for SQL Server log-in failures.

I added the null_cmd command mentioned in the docs, but I'd be happy if
it just triggered the firewall drop script.

Am I missing something in the configuration?

thanks.

Martin

[Tanishk Lakhaani]

Hey martin,
All these default active response scripts are written for a specific event. Read these scripts to understand these scripts.

For the event of ur interest -- multiple logon failures...for linux, there is a default active response script -- for locking the account. But for windows there is no such script. What u can do is that u can create your own customised script and use it for active response purposes.

Regards
Tanishk lakhaani
Sent from BlackBerry® on Airtel

[Martin Gottlieb]

Thanks, Tanishk. I'm really surprised nothing has been written for
windows yet. Am I correct
in assuming the script would reside on the Windows agent machine?

Obviously, the windows agent communicates with the Linux server. Is it
not possible to have
an active response script triggered on the server side as happens with
Linux agents?

Thanks.

Martin

On 4/22/2011 3:28 PM, Tanishk Lakhaani wrote:
> Hey martin,
> All these default active response scripts are written for a specific event. Read these scripts to understand these scripts.
>
> For the event of ur interest -- multiple logon failures...for linux, there is a default active response script -- for locking the account. But for windows there is no such script. What u can do is that u can create your own customised script and use it for active response purposes.
>
> Regards
> Tanishk lakhaani

> Sent from BlackBerry� on Airtel

[dan (ddp)]

Hi Martin,

On Fri, Apr 22, 2011 at 8:22 AM, Martin Gottlieb <mar...@axion-it.net> wrote:
> Hi,
>
> Is OSSEC capable of triggering an active response on Windows events?  In
> particular, I am frequently
> seeing event 18152, "Multiple Windows Logon Failures", but no active
> response is ever triggered.
> There are 2 (at least) different variations on the events, 1 for Windows
> log-in failures and another
> for SQL Server log-in failures.
>

Yes, it's possible.

> I added the null_cmd command mentioned in the docs, but I'd be happy if it
> just triggered the firewall drop script.
>
> Am I missing something in the configuration?
>

Don't know, your configuration didn't come through.

> thanks.
>
> Martin
>

[Tanishk Lakhaani]

Hey martin,
See, the active response related scripts will be placed at the server side, executed at the server/client side (depending upon the way it is configured in ossec.conf using the location tab) and the commands written in these scripts will actually take an action on the agent side. This is the basic of active response.
Sent from BlackBerry® on Airtel

-----Original Message-----
From: Martin Gottlieb <mar...@axion-it.net>
Sender: ossec...@googlegroups.com
Date: Fri, 22 Apr 2011 16:04:14
To: <ossec...@googlegroups.com>
Reply-To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Active Response on Windows events


Thanks, Tanishk. I'm really surprised nothing has been written for
windows yet. Am I correct
in assuming the script would reside on the Windows agent machine?

Obviously, the windows agent communicates with the Linux server. Is it
not possible to have
an active response script triggered on the server side as happens with
Linux agents?

Thanks.

Martin

On 4/22/2011 3:28 PM, Tanishk Lakhaani wrote:
> Hey martin,
> All these default active response scripts are written for a specific event. Read these scripts to understand these scripts.
>
> For the event of ur interest -- multiple logon failures...for linux, there is a default active response script -- for locking the account. But for windows there is no such script. What u can do is that u can create your own customised script and use it for active response purposes.
>
> Regards
> Tanishk lakhaani

> Sent from BlackBerry® on Airtel

[dan (ddp)]

Hi Tanishk,
The active response scripts should exist on the systems (agents and
servers) they need to be run on.

[Martin Gottlieb]

I guess what I'm trying to understand is this:

When an event is triggered from a Linux agent, the firewall drop script
is run on the
OSSEC server (in addition to the hosts deny script being called on the
agent). I don't recall
doing anything special to make this happen when I installed OSSEC, I
assume it is part of
the default behavior.

When an event is triggered on a Windows agent, the firewall drop script
is NOT called on the server,
but I would like it to be. I would like the default behavior on Windows
agents to be the same
as Linux agents, at least as far as what happens on the OSSEC server.
The Windows agent is
obviously reporting the event to the server as it logs it and reports it
to me.

Am I understanding the responses so far to mean that I have to write a
script to make this
happen, and that the script needs to reside on the Windows agent?

Thanks again.

Martin

[dan (ddp)]

Hi Martin,

On Fri, Apr 22, 2011 at 4:37 PM, Martin Gottlieb <mar...@axion-it.net> wrote:
>
> I guess what I'm trying to understand is this:
>
> When an event is triggered from a Linux agent, the firewall drop script is
> run on the
> OSSEC server (in addition to the hosts deny script being called on the
> agent).  I don't recall
> doing anything special to make this happen when I installed OSSEC, I assume
> it is part of
> the default behavior.
>

The default actions (if I'm reading
https://bitbucket.org/dcid/ossec-hids/src/4908b28513b0/etc/ossec-server.conf
correctly) is that the script is run on the system where the log
message originated.
Unless you changed the configurations the scripts shouldn't be running
on both the server and the agents.

> When an event is triggered on a Windows agent, the firewall drop script is
> NOT called on the server,
> but I would like it to be.  I would like the default behavior on Windows
> agents to be the same
> as Linux agents, at least as far as what happens on the OSSEC server.  The
> Windows agent is
> obviously reporting the event to the server as it logs it and reports it to
> me.
>
> Am I understanding the responses so far to mean that I have to write a
> script to make this
> happen, and that the script needs to reside on the Windows agent?
>
> Thanks again.
>
> Martin
>

The script would have to reside on all of the systems you want it to
run on. Having it run on both Windows and Linux systems may be
difficult.

[Martin Gottlieb]

Shouldn't this block from the config on the OSSEC server:

  <active-response>
    <!-- Firewall Drop response. Block the IP for
       - 600 seconds on the firewall (iptables,
       - ipfilter, etc).
      -->
    <command>firewall-drop</command>
    <location>as</location>
    <level>6</level>
    <timeout>3600</timeout>
  </active-response>

cause the firewall drop script to be run on the server for any event that is level 6 or higher, regardless of
which agent it came from?  That's all I'm trying to accomplish, I don't need anything to run on the Windows
agent if I can get the firewall drop script to run on the server.

Thanks.

Martin

[dan (ddp)]

Hi Martin,

On Fri, Apr 22, 2011 at 5:08 PM, Martin Gottlieb <mar...@axion-it.net> wrote:
>
> Shouldn't this block from the config on the OSSEC server:
>
>   <active-response>
>     <!-- Firewall Drop response. Block the IP for
>        - 600 seconds on the firewall (iptables,
>        - ipfilter, etc).
>       -->
>     <command>firewall-drop</command>
>     <location>as</location>
>     <level>6</level>
>     <timeout>3600</timeout>
>   </active-response>
>
> cause the firewall drop script to be run on the server for any event that is
> level 6 or higher, regardless of
> which agent it came from?  That's all I'm trying to accomplish, I don't need
> anything to run on the Windows
> agent if I can get the firewall drop script to run on the server.
>
> Thanks.
>
> Martin
>

Oh, I get it now. Your <location> field looks wrong. It should be
<location>server</location>
http://www.ossec.net/doc/syntax/head_ossec_config.active-responce.html#element-active-response.location

[Martin Gottlieb]

Thanks! I'll give that a try. Sorry if I wasn't entirely clear about this.

Martin

[AndiC]

The problem I found was that the Windows decoder in the server /dev/
ossec/etc/decoder.xml does not extract the "srcip", so you have
nothing to work with to block

Now this is what I replaced mine with:

<decoder name="windows">
<type>windows</type>
<prematch>^WinEvtLog: </prematch>
<regex offset="after_prematch">^\.+: (\w+)\((\d+)\): (\.+): </regex>
<regex>(\.+): \.+: (\S+):</regex>
<regex> \.+: \.+: \.+: \.+: \.+: \.+: </regex>
<regex>\.+: \.+: \.+: \.+: \.+: \.+: \.+: \.+:</regex>
<regex>\.(\S+)</regex>
<order>status, id, extra_data, user, system_name, srcip</order>
<fts>name, location, user, system_name</fts> </decoder>

Then, in /dev/ossec/rules/msauth.xml, I replaced rule 18152 with:

<rule id="181521" level="10" frequency="$MS_FREQ" timeframe="240">
<if_matched_group>win_authentication_failed</if_matched_group>
<same_source_ip />
<description>Multiple Windows Logon Failures Same IP.</
description>
<group>authentication_failures,</group>
</rule>
<rule id="181522" level="10" frequency="$MS_FREQ" timeframe="240">
<if_matched_group>win_authentication_failed</if_matched_group>
<description>Multiple Windows Logon Failures.</description>
<group>authentication_failures,</group>
</rule>

I also dropped $MS_FREQ (start of msauth.xml) to 3

This works for me, and my Windows clients are well protected.

I am sure someone could write a far more eloquent decode Regex - sorry
I'm just coming to grips with that. I'm also uncertain if this will
work against anything other than Server 2003 for which it is written

But this is only the decoder that needs some tuning, the rest seems
fine

Regards

Andy

> >https://bitbucket.org/dcid/ossec-hids/src/4908b28513b0/etc/ossec-serv...

> > correctly) is that the script is run on the system where the log
> > message originated.
> > Unless you changed the configurations the scripts shouldn't be running
> > on both the server and the agents.
>
> >> When an event is triggered on a Windows agent, the firewall drop script is
> >> NOT called on the server,
> >> but I would like it to be.  I would like the default behavior on Windows
> >> agents to be the same
> >> as Linux agents, at least as far as what happens on the OSSEC server.  The
> >> Windows agent is
> >> obviously reporting the event to the server as it logs it and reports it to
> >> me.
>
> >> Am I understanding the responses so far to mean that I have to write a
> >> script to make this
> >> happen, and that the script needs to reside on the Windows agent?
>
> >> Thanks again.
>
> >> Martin
>
> > The script would have to reside on all of the systems you want it to
> > run on. Having it run on both Windows and Linux systems may be

> > difficult.- Hide quoted text -
>
> - Show quoted text -

[Andy Cockroft (andic)]

The problem I found was that the Windows decoder in the server /dev/ossec/etc/decoder.xml does not extract the "srcip", so you have nothing to work with to block

Now this is what I replaced mine with:

<decoder name="windows">
<type>windows</type>
<prematch>^WinEvtLog: </prematch>
<regex offset="after_prematch">^\.+: (\w+)\((\d+)\): (\.+): </regex>
<regex>(\.+): \.+: (\S+):</regex>
<regex> \.+: \.+: \.+: \.+: \.+: \.+: </regex>
<regex>\.+: \.+: \.+: \.+: \.+: \.+: \.+: \.+:</regex>
<regex>\.(\S+)</regex>
<order>status, id, extra_data, user, system_name, srcip</order>
<fts>name, location, user, system_name</fts>
</decoder>

Then, in /dev/ossec/rules/msauth.xml, I replaced rule 18152 with:

<rule id="181521" level="10" frequency="$MS_FREQ" timeframe="240">
<if_matched_group>win_authentication_failed</if_matched_group>
<same_source_ip />

<description>Multiple Windows Logon Failures Same IP.</description>

<group>authentication_failures,</group>
</rule>
<rule id="181522" level="10" frequency="$MS_FREQ" timeframe="240">
<if_matched_group>win_authentication_failed</if_matched_group>
<description>Multiple Windows Logon Failures.</description>
<group>authentication_failures,</group>
</rule>

I also dropped $MS_FREQ (start of msauth.xml) to 3

This works for me, and my Windows clients are well protected.

I am sure someone could write a far more eloquent decode Regex - sorry I'm just coming to grips with that. I'm also uncertain if this will work against anything other than Server 2003 for which it is written

But this is only the decoder that needs some tuning, the rest seems fine

Regards

Andy
 

-----Original Message-----
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Martin Gottlieb
Sent: Saturday, 23 April 2011 9:14 a.m.
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Active Response on Windows events

[Martin Gottlieb]

Awesome, thanks!  The events I'm seeing generally take 2 forms:

SQL Server Events:

WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: (no user): no domain: WINSERVER: Login failed for user 'admin'. [CLIENT: 203.81.30.248]


And general Windows Events:

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:    	Reason: 	Unknown user name or bad password    	User Name: admin    	Domain: 	WINSERVER    	Logon Type: 10    	Logon Process: User32      	Authentication Package: Negotiate    	Workstation Name: WINSERVER    	Caller User Name: WINSERVER$    	Caller Domain: WINDOMAIN    	Caller Logon ID: (0x0,0x3E7)    	Caller Process ID: 532    	Transited Services: -    	Source Network Address: 118.126.5.109    	Source Port: 3041    

Would these work as the corresponding decoders:

<decoder name="mssql">
  <prematch>^WinEvtLog: Application: AUDIT_FAILURE\(\d+\): MSSQLSERVER: \.* Login failed for user</prematch>
  <regex offset="after_prematch">'(\w+)'. [CLIENT: (\d+.\d+.\d+.\d+)]</regex>
  <order>user,srcip</order>
</decoder>

<decoder name="winevt">
  <prematch>^WinEvtLog: Security: AUDIT_FAILURE\(\d+\): Security\.* Logon Failure: </prematch>
  <regex offset="after_prematch">User Name: (\w+) \.* Source Network Address: (\d+.\d+.\d+.\d+)</regex>
  <order>user,srcip</order>
</decoder>

Thanks.

Martin

[Rob Brooks]

I've seen the issues and we're running CentOS 5.2.  We're testing on around 50 servers (plans to possibly roll out onto thousands) and ALL of them disconnect and reconnect, some of them never being able to connect at all.

When I have time to devote to work on it, I'm going to attempt more troubleshooting.

--Rob

[Tanishk Lakhaani]

Hey martin,
These scripts contain some commands that are OS specific. So when the server sees events, it executes the script whihc execute commands on the client be it to deny a host or to add a rule to firewall (iptables).

Also, the scripts can be used as intended. It means u cannot use a unix related AR script on windows, as the commands in that script will be unix specific. Howveer, u can try understand the logic , and create ur own scripts.

[Daniel Cid]

I would love to get to the bottom of it. If anyone is experiencing
this issue and can either give me SSH access
to the manager to debug or is willing to debug it for me (will take at
least a few hours of work), let me know (just
email me directly dc...@ossec.net).

I have heard multiple reports of it, but was never able to duplicate
anywhere (or get full access to a box with
this behavior).

thanks,

[Andy Cockroft (andic)]

Hi

 

I didn’t have that much success with a Regex similar to the one you wrote, I ended up having to specify everything in a very long-handed way – as I said perhaps someone could write the decoder far more eloquently than I – especially constructs such as \.* in the middle of the Regex

 

However, what I did do, is make my changes to the decoder and  run ossec-logtest – this makes checking the decoder and rules so much easier without actually affecting production operation

 

Best I can do for now – hope you have your Rules sorted as well – ossec-logtest will check these at the same time

 

Andy

 

 

From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Martin Gottlieb

Sent: Sunday, 24 April 2011 3:16 a.m.
To: ossec...@googlegroups.com

I guess what I'm trying to understand is this:

When an event is triggered from a Linux agent, the firewall drop script is

run on the

OSSEC server (in addition to the hosts deny script being called on the

agent).  I don't recall

doing anything special to make this happen when I installed OSSEC, I assume

it is part of

the default behavior.

The default actions (if I'm reading

https://bitbucket.org/dcid/ossec-hids/src/4908b28513b0/etc/ossec-serv...

correctly) is that the script is run on the system where the log

message originated.

Unless you changed the configurations the scripts shouldn't be running

on both the server and the agents.

When an event is triggered on a Windows agent, the firewall drop script is

NOT called on the server,

but I would like it to be.  I would like the default behavior on Windows

agents to be the same

as Linux agents, at least as far as what happens on the OSSEC server.  The

Windows agent is

obviously reporting the event to the server as it logs it and reports it to

me.

Am I understanding the responses so far to mean that I have to write a

script to make this

happen, and that the script needs to reside on the Windows agent?

Thanks again.

Martin

The script would have to reside on all of the systems you want it to

run on. Having it run on both Windows and Linux systems may be

difficult.- Hide quoted text -

[Martin Gottlieb]

Hi Andy,

Thanks again for another great piece of advice.  ossec-logtest seems to confirm that the
regexes are good.  The SQL Server decoder triggers rule 2501, level 5.  I had to add the
following to my local rules to get the winevt decoder to also trigger 2501:

  <rule id="100245" level="5">
    <match>Logon Failure</match>
    <group>authentication_failed,</group>
    <description>User authentication failure.</description>
  </rule>

I think this should to the trick.  Thanks again for your help.

Martin

[Scott VR]

It is important to undertstand that firewall-drop.sh script executes unix/linux commands and the only way that invoking it on the server will serve any function to protect your windows hosts is if your ossec server is *also* running as a router/firewall in front of your windows boxes. If this is the case, it's a pretty major piece of the design that you left off of your description. If it's not the case, you are going to need something similar (which I think you alluded to in your inital email) to the null-route.cmd setup outlined in http://www.ossec.net/main/manual/manual-active-response-on-windows.

In summary, if your ossec server is also a router for your network, then running the ipfilters/ipchains/ipsec commands in the firewall-drop.sh script will work, with the proper regex to obtain srcip. If it is not, the running this command on the ossec server will have no effect and you need to run the command on the windows box through its agent.

Cheers,

[Martin Gottlieb]

Thanks, my ossec server is a router/firewall, my apologies for omitting this detail.� I was really
just trying to figure out how to get the server to trigger the script(s) in the first place on the
windows events, since it was clearly getting notified about the events.

With help from Andy, I believe I have found that the issue boils down to the decoders.� I think I
have a fix i place now and will be posting a "RESOLVED" message once I have verified this (just waiting
for someone to attack the server).

Thanks again to everyone who offered help on this.

Martin

On 4/25/2011 11:23 AM, Scott VR wrote:

It is important to undertstand that firewall-drop.sh script executes unix/linux commands and the only way that invoking it on the server will serve any function to protect your windows hosts is if your ossec server is *also* running as a router/firewall in front of your windows boxes. If this is the case, it's a pretty major piece of the design that you left off of your description. If it's not the case, you are going to need something similar (which I think you alluded to in your inital email) to the null-route.cmd setup outlined in http://www.ossec.net/main/manual/manual-active-response-on-windows.

In summary, if your ossec server is also a router for your network, then running the ipfilters/ipchains/ipsec commands in the firewall-drop.sh script will work, with the proper regex to obtain srcip. If it is not, the running this command on the ossec server will have no effect and you need to run the command on the windows box through its agent.

Cheers,

On Sat, Apr 23, 2011 at 10:27 PM, Martin Gottlieb <mar...@axion-it.net> wrote:

Hi Andy,

Thanks again for another great piece of advice.� ossec-logtest seems to confirm that the
regexes are good.� The SQL Server decoder triggers rule 2501, level 5.� I had to add the

following to my local rules to get the winevt decoder to also trigger 2501:

� <rule id="100245" level="5">
��� <match>Logon Failure</match>
��� <group>authentication_failed,</group>
��� <description>User authentication failure.</description>
� </rule>

I think this should to the trick.� Thanks again for your help.

Martin

On 4/23/2011 5:26 PM, Andy Cockroft (andic) wrote:

Hi

�

I didn�t have that much success with a Regex similar to the one you wrote, I ended up having to specify everything in a very long-handed way � as I said perhaps someone could write the decoder far more eloquently than I � especially constructs such as \.* in the middle of the Regex

�

However, what I did do, is make my changes to the decoder and �run ossec-logtest � this makes checking the decoder and rules so much easier without actually affecting production operation

�

Best I can do for now � hope you have your Rules sorted as well � ossec-logtest will check these at the same time

�

Andy

�

�

From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Martin Gottlieb
Sent: Sunday, 24 April 2011 3:16 a.m.
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Re: Active Response on Windows events

�

Awesome, thanks!� The events I'm seeing generally take 2 forms:

SQL Server Events:

WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: (no user): no domain: WINSERVER: Login failed for user 'admin'. [CLIENT: 203.81.30.248]


And general Windows Events:

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:��� �� Reason: ������ Unknown user name or bad password���� User Name: admin��� ������� Domain: ������ WINSERVER��� � Logon Type: 10��� ���� Logon Process: User32����� ������� Authentication Package: Negotiate� �� Workstation Name: WINSERVER��� ������ Caller User Name: WINSERVER$��� ����� Caller Domain: WINDOMAIN��� �� Caller Logon ID: (0x0,0x3E7)��� ������� Caller Process ID: 532��� ���� Transi
ted Services: -��� ����� Source Network Address: 118.126.5.109��� ����� Source Port: 3041��� 

Would these work as the corresponding decoders:

<decoder name="mssql">
� <prematch>^WinEvtLog: Application: AUDIT_FAILURE\(\d+\): MSSQLSERVER: \.* Login failed for user</prematch>
� <regex offset="after_prematch">'(\w+)'. [CLIENT: (\d+.\d+.\d+.\d+)]</regex>
� <order>user,srcip</order>
</decoder>

<decoder name="winevt">
� <prematch>^WinEvtLog: Security: AUDIT_FAILURE\(\d+\): Security\.* Logon Failure: </prematch>
� <regex offset="after_prematch">User Name: (\w+) \.* Source Network Address: (\d+.\d+.\d+.\d+)</regex>
� <order>user,srcip</order>
</decoder>

Thanks.

Martin


On 4/22/2011 7:28 PM, AndiC wrote:

The problem I found was that the Windows decoder in the server /dev/

ossec/etc/decoder.xml does not extract the "srcip", so you have

nothing to work with to block

�

Now this is what I replaced mine with:

�

<decoder name="windows">

� <type>windows</type>

� <prematch>^WinEvtLog: </prematch>

� <regex offset="after_prematch">^\.+: (\w+)\((\d+)\): (\.+): </regex>

� <regex>(\.+): \.+: (\S+):</regex>

� <regex> \.+: \.+: \.+: \.+: \.+: \.+: </regex>

� <regex>\.+: \.+: \.+: \.+: \.+: \.+: \.+: \.+:</regex>

� <regex>\.(\S+)</regex>

� <order>status, id, extra_data, user, system_name, srcip</order>

� <fts>name, location, user, system_name</fts> </decoder>

�

Then, in /dev/ossec/rules/msauth.xml, I replaced rule 18152 with:

�

� <rule id="181521" level="10" frequency="$MS_FREQ" timeframe="240">

��� <if_matched_group>win_authentication_failed</if_matched_group>

��� <same_source_ip />

��� <description>Multiple Windows Logon Failures Same IP.</

description>

��� <group>authentication_failures,</group>

� </rule>

� <rule id="181522" level="10" frequency="$MS_FREQ" timeframe="240">

��� <if_matched_group>win_authentication_failed</if_matched_group>

��� <description>Multiple Windows Logon Failures.</description>

��� <group>authentication_failures,</group>

� </rule>

�

I also dropped $MS_FREQ (start of msauth.xml) to 3

�

This works for me, and my Windows clients are well protected.

�

I am sure someone could write a far more eloquent decode Regex - sorry

I'm just coming to grips with that. I'm also uncertain if this will

work against anything other than Server 2003 for which it is written

�

But this is only the decoder that needs some tuning, the rest seems

fine

�

Regards

�

Andy

�

�

On Apr 23, 9:08�am, Martin Gottlieb <mar...@axion-it.net> wrote:

Shouldn't this block from the config on the OSSEC server:

�

<active-response>

<!-- Firewall Drop response. Block the IP for

� � � � - 600 seconds on the firewall (iptables,

� � � � - ipfilter, etc).

� � � �-->

<command>firewall-drop</command>

<location>as</location>

<level>6</level>

<timeout>3600</timeout>

</active-response>

�

cause the firewall drop script to be run on the server for any event

that is level 6 or higher, regardless of

which agent it came from? �That's all I'm trying to accomplish, I don't

need anything to run on the Windows

agent if I can get the firewall drop script to run on the server.

�

Thanks.

�

Martin

�

On 4/22/2011 4:58 PM, dan (ddp) wrote:

�

�

�

Hi Martin,

�

On Fri, Apr 22, 2011 at 4:37 PM, Martin Gottlieb<mar...@axion-it.net> �wrote:

I guess what I'm trying to understand is this:

�

When an event is triggered from a Linux agent, the firewall drop script is

run on the

OSSEC server (in addition to the hosts deny script being called on the

agent). �I don't recall

doing anything special to make this happen when I installed OSSEC, I assume

it is part of

the default behavior.

�

The default actions (if I'm reading

https://bitbucket.org/dcid/ossec-hids/src/4908b28513b0/etc/ossec-serv...

correctly) is that the script is run on the system where the log

message originated.

Unless you changed the configurations the scripts shouldn't be running

on both the server and the agents.

�

When an event is triggered on a Windows agent, the firewall drop script is

NOT called on the server,

but I would like it to be. �I would like the default behavior on Windows

agents to be the same

as Linux agents, at least as far as what happens on the OSSEC server. �The

Windows agent is

obviously reporting the event to the server as it logs it and reports it to

me.

�

Am I understanding the responses so far to mean that I have to write a

script to make this

happen, and that the script needs to reside on the Windows agent?

�

Thanks again.

�

Martin

�

The script would have to reside on all of the systems you want it to

run on. Having it run on both Windows and Linux systems may be

difficult.- Hide quoted text -

�

- Show quoted text -

�

[Martin Gottlieb]

Well, I thought I was making progress, but now I'm not so sure.� My MSSQL decoder has triggered a couple
of active responses, so I believe it is working properly. But, I am not getting any alerts on windows logon
failures (I did previously), much less an active response.

I found the following event in my ossec alert log (identifying info modified):

** Alert 1303837130.3865847: - syslog,false_positivesauthentication_failed,
2011 Apr 26 12:58:50 (win3) 2.1.1.2->WinEvtLog
Rule: 100245 (level 5) -> 'User authentication failure.'
Src IP: (none)
User: (none)
WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:���������� Reason:�������� Unknown user name or bad password������ User Name: ryan�������� Domain:�������� WINSERVER����������� Logon Type: 10��������� Logon Process: User32���������� Authentication Package: Negotiate������ Workstation Name: WINSERVER��������� Caller User Name: WINSERVER$�������� Caller Domain: WINDOMAIN �������� Caller Logon ID: (0x0,0x3E7)����������� Caller Process ID: 5716�������� Transited Services: -���������� Source Network Address: 7.7.7.226��������� Source Port: 51287

What's strange is that it does not match the SrcIP or User fields.� When I run this log entry through ossec-logtest, I get the
following results:

**Phase 2: Completed decoding.
������ decoder: 'winevt'
������ dstuser: 'ryan'
������ srcip: '7.7.7.226'

**Phase 3: Completed filtering (rules).
������ Rule id: '100245'
������ Level: '5'
������ Description: 'User authentication failure.'
**Alert to be generated.

So clearly the winevt decoder is working correctly.� Any ideas as to why it works in test mode, but not "live"?
Here's the winevt decoder:

<decoder name="winevt">
� <prematch>^WinEvtLog:\s*Security:\s*AUDIT_FAILURE\(\d+\):\s*Security\.* Logon Failure: </prematch>
� <regex offset="after_prematch">User Name:\s+(\w+) \.* Source Network Address:\s+(\d+.\d+.\d+.\d+)</regex>

� <order>user,srcip</order>
</decoder>

I did make a few minor changes since my previous posts, mainly replacing spaces with "\s*" to allow for multiple white-space characters.

Thanks.

Martin

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:��� �� Reason: ������ Unknown user name or bad password���� User Name: admin��� ������� Domain: ������ WINSERVER��� � Logon Type: 10��� ���� Logon Process: User32����� ������� Authentication Package: Negotiate� �� Workstation Name: WINSERVER��� ������ Caller User Name: WINSERVER$��� ����� Caller Domain: WINDOMAIN��� �� Caller Logon ID: (0x0,0x3E7)��� ������� Caller Process ID: 532��� ���� Transi
ted Services: -��� ����� Source Network Address: 7.7.7.109��� ����� Source Port: 3041��� 

[Andy Cockroft (andic)]

Hi

 

This is triggering a level 5 alert – will that actually do anything on your system? Or do you have another rule for multiple occurrences?

 

Certainly for mine, I have a level 10 alert for multiple occurrences (more than 3) which then activates the response on the windows agent

 

Just a random thought

 

Andy

 

 

From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Martin Gottlieb
Sent: Thursday, 28 April 2011 1:23 a.m.
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Re: Active Response on Windows events

 

Well, I thought I was making progress, but now I'm not so sure.  My MSSQL decoder has triggered a couple

of active responses, so I believe it is working properly. But, I am not getting any alerts on windows logon
failures (I did previously), much less an active response.

I found the following event in my ossec alert log (identifying info modified):

** Alert 1303837130.3865847: - syslog,false_positivesauthentication_failed,
2011 Apr 26 12:58:50 (win3) 2.1.1.2->WinEvtLog
Rule: 100245 (level 5) -> 'User authentication failure.'
Src IP: (none)
User: (none)

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:           Reason:         Unknown user name or bad password       User Name: ryan         Domain:         WINSERVER            Logon Type: 10          Logon Process: User32           Authentication Package: Negotiate       Workstation Name: WINSERVER          Caller User Name: WINSERVER$         Caller Domain: WINDOMAIN          Caller Logon ID: (0x0,0x3E7)            Caller Process ID: 5716         Transited Services: -           Source Network Address: 7.7.7.226          Source Port: 51287

What's strange is that it does not match the SrcIP or User fields.  When I run this log entry through ossec-logtest, I get the
following results:

**Phase 2: Completed decoding.
       decoder: 'winevt'
       dstuser: 'ryan'

       srcip: '7.7.7.226'

**Phase 3: Completed filtering (rules).

       Rule id: '100245'
       Level: '5'

       Description: 'User authentication failure.'
**Alert to be generated.

So clearly the winevt decoder is working correctly.  Any ideas as to why it works in test mode, but not "live"?

Here's the winevt decoder:

<decoder name="winevt">

  <prematch>^WinEvtLog:\s*Security:\s*AUDIT_FAILURE\(\d+\):\s*Security\.* Logon Failure: </prematch>

  <regex offset="after_prematch">User Name:\s+(\w+) \.* Source Network Address:\s+(\d+.\d+.\d+.\d+)</regex>

  <order>user,srcip</order>
</decoder>

I did make a few minor changes since my previous posts, mainly replacing spaces with "\s*" to allow for multiple white-space characters.

Thanks.

Martin

On 4/25/2011 11:43 AM, Martin Gottlieb wrote:

Thanks, my ossec server is a router/firewall, my apologies for omitting this detail.  I was really

just trying to figure out how to get the server to trigger the script(s) in the first place on the
windows events, since it was clearly getting notified about the events.

With help from Andy, I believe I have found that the issue boils down to the decoders.  I think I

have a fix i place now and will be posting a "RESOLVED" message once I have verified this (just waiting
for someone to attack the server).

Thanks again to everyone who offered help on this.

Martin

On 4/25/2011 11:23 AM, Scott VR wrote:

It is important to undertstand that firewall-drop.sh script executes unix/linux commands and the only way that invoking it on the server will serve any function to protect your windows hosts is if your ossec server is *also* running as a router/firewall in front of your windows boxes. If this is the case, it's a pretty major piece of the design that you left off of your description. If it's not the case, you are going to need something similar (which I think you alluded to in your inital email) to the null-route.cmd setup outlined in http://www.ossec.net/main/manual/manual-active-response-on-windows.

In summary, if your ossec server is also a router for your network, then running the ipfilters/ipchains/ipsec commands in the firewall-drop.sh script will work, with the proper regex to obtain srcip. If it is not, the running this command on the ossec server will have no effect and you need to run the command on the windows box through its agent.

Cheers,

On Sat, Apr 23, 2011 at 10:27 PM, Martin Gottlieb <mar...@axion-it.net> wrote:

Hi Andy,

Thanks again for another great piece of advice.  ossec-logtest seems to confirm that the
regexes are good.  The SQL Server decoder triggers rule 2501, level 5.  I had to add the

following to my local rules to get the winevt decoder to also trigger 2501:

  <rule id="100245" level="5">

    <match>Logon Failure</match>
    <group>authentication_failed,</group>

    <description>User authentication failure.</description>

  </rule>

I think this should to the trick.  Thanks again for your help.

Martin

On 4/23/2011 5:26 PM, Andy Cockroft (andic) wrote:

Hi

 

I didn’t have that much success with a Regex similar to the one you wrote, I ended up having to specify everything in a very long-handed way – as I said perhaps someone could write the decoder far more eloquently than I – especially constructs such as \.* in the middle of the Regex

 

However, what I did do, is make my changes to the decoder and  run ossec-logtest – this makes checking the decoder and rules so much easier without actually affecting production operation

 

Best I can do for now – hope you have your Rules sorted as well – ossec-logtest will check these at the same time

 

Andy

 

 

From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Martin Gottlieb
Sent: Sunday, 24 April 2011 3:16 a.m.
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Re: Active Response on Windows events

 

Awesome, thanks!  The events I'm seeing generally take 2 forms:

SQL Server Events:

WinEvtLog: Application: AUDIT_FAILURE(18456): MSSQLSERVER: (no user): no domain: WINSERVER: Login failed for user 'admin'. [CLIENT: 203.81.30.248]


And general Windows Events:

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:       Reason:        Unknown user name or bad password     User Name: admin            Domain:        WINSERVER      Logon Type: 10         Logon Process: User32              Authentication Package: Negotiate     Workstation Name: WINSERVER           Caller User Name: WINSERVER$          Caller Domain: WINDOMAIN       Caller Logon ID: (0x0,0x3E7)            Caller Process ID: 532         Transi

ted Services: -          Source Network Address: 7.7.7.109          Source Port: 3041    

Would these work as the corresponding decoders:

<decoder name="mssql">

  <prematch>^WinEvtLog: Application: AUDIT_FAILURE\(\d+\): MSSQLSERVER: \.* Login failed for user</prematch>

  <regex offset="after_prematch">'(\w+)'. [CLIENT: (\d+.\d+.\d+.\d+)]</regex>

  <order>user,srcip</order>
</decoder>

<decoder name="winevt">

  <prematch>^WinEvtLog: Security: AUDIT_FAILURE\(\d+\): Security\.* Logon Failure: </prematch>

  <regex offset="after_prematch">User Name: (\w+) \.* Source Network Address: (\d+.\d+.\d+.\d+)</regex>

  <order>user,srcip</order>
</decoder>

Thanks.

Martin


On 4/22/2011 7:28 PM, AndiC wrote:

The problem I found was that the Windows decoder in the server /dev/

ossec/etc/decoder.xml does not extract the "srcip", so you have

nothing to work with to block

Now this is what I replaced mine with:

<decoder name="windows">

  <type>windows</type>

  <prematch>^WinEvtLog: </prematch>

  <regex offset="after_prematch">^\.+: (\w+)\((\d+)\): (\.+): </regex>

  <regex>(\.+): \.+: (\S+):</regex>

  <regex> \.+: \.+: \.+: \.+: \.+: \.+: </regex>

  <regex>\.+: \.+: \.+: \.+: \.+: \.+: \.+: \.+:</regex>

  <regex>\.(\S+)</regex>

  <order>status, id, extra_data, user, system_name, srcip</order>

  <fts>name, location, user, system_name</fts> </decoder>

Then, in /dev/ossec/rules/msauth.xml, I replaced rule 18152 with:

  <rule id="181521" level="10" frequency="$MS_FREQ" timeframe="240">

    <if_matched_group>win_authentication_failed</if_matched_group>

    <same_source_ip />

    <description>Multiple Windows Logon Failures Same IP.</

description>

    <group>authentication_failures,</group>

  </rule>

  <rule id="181522" level="10" frequency="$MS_FREQ" timeframe="240">

    <if_matched_group>win_authentication_failed</if_matched_group>

    <description>Multiple Windows Logon Failures.</description>

    <group>authentication_failures,</group>

  </rule>

I also dropped $MS_FREQ (start of msauth.xml) to 3

This works for me, and my Windows clients are well protected.

I am sure someone could write a far more eloquent decode Regex - sorry

I'm just coming to grips with that. I'm also uncertain if this will

work against anything other than Server 2003 for which it is written

But this is only the decoder that needs some tuning, the rest seems

fine

Regards

Andy

On Apr 23, 9:08 am, Martin Gottlieb <mar...@axion-it.net> wrote:

Shouldn't this block from the config on the OSSEC server:

<active-response>

<!-- Firewall Drop response. Block the IP for

        - 600 seconds on the firewall (iptables,

        - ipfilter, etc).

       -->

<command>firewall-drop</command>

<location>as</location>

<level>6</level>

<timeout>3600</timeout>

</active-response>

cause the firewall drop script to be run on the server for any event

that is level 6 or higher, regardless of

which agent it came from?  That's all I'm trying to accomplish, I don't

need anything to run on the Windows

agent if I can get the firewall drop script to run on the server.

Thanks.

Martin

On 4/22/2011 4:58 PM, dan (ddp) wrote:

Hi Martin,

On Fri, Apr 22, 2011 at 4:37 PM, Martin Gottlieb<mar...@axion-it.net>  wrote:

I guess what I'm trying to understand is this:

When an event is triggered from a Linux agent, the firewall drop script is

run on the

OSSEC server (in addition to the hosts deny script being called on the

agent).  I don't recall

doing anything special to make this happen when I installed OSSEC, I assume

it is part of

the default behavior.

The default actions (if I'm reading

https://bitbucket.org/dcid/ossec-hids/src/4908b28513b0/etc/ossec-serv...

correctly) is that the script is run on the system where the log

message originated.

Unless you changed the configurations the scripts shouldn't be running

on both the server and the agents.

When an event is triggered on a Windows agent, the firewall drop script is

NOT called on the server,

but I would like it to be.  I would like the default behavior on Windows

agents to be the same

as Linux agents, at least as far as what happens on the OSSEC server.  The

Windows agent is

obviously reporting the event to the server as it logs it and reports it to

me.

Am I understanding the responses so far to mean that I have to write a

script to make this

happen, and that the script needs to reside on the Windows agent?

Thanks again.

Martin

The script would have to reside on all of the systems you want it to

run on. Having it run on both Windows and Linux systems may be

difficult.- Hide quoted text -

- Show quoted text -

 

 

 

 

 

[Martin Gottlieb]

good point, I should not be expecting email alerts on the level 5 rule.  But since it's not recording the SrcIP
value, it never triggers the level 10 rule, which I did also create:

  <rule id="100245" level="5">
    <match>Logon Failure</match>
    <group>authentication_failed,</group>
    <description>User authentication failure.</description>
  </rule>

  <rule id="100246" level="10" frequency="5" timeframe="120" ignore="60">
    <if_matched_sid>100245</if_matched_sid>
    <description>Windows brute force trying to get access to </description>
    <description>the system.</description>
    <same_source_ip />
    <group>authentication_failures,</group>
  </rule>

So my original question remains, why is it not able to extract the SrcIP address using the decoder that I created
and verified using ossec-logtest?

Thanks.

Martin

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:       Reason:        Unknown user name or bad password     User Name: admin            Domain:        WINSERVER      Logon Type: 10         Logon Process: User32              Authentication Package: Negotiate     Workstation Name: WINSERVER           Caller User Name: WINSERVER$          Caller Domain: WINDOMAIN       Caller Logon ID: (0x0,0x3E7)            Caller Process ID: 532       &nb
sp; Transi

[Andy Cockroft (andic)]

Hi

 

You should be able to run ossec-logtest repeatedly (ie 6 times at least) with the same data, and you should see what it does in triggering the level 10 rule

[Martin Gottlieb]

Thanks, that does work.  The problem is that when a real intruder is triggering my level 5 rule (100245),
it is not recording the source IP, so it has no way of ever triggering the level 10 rule.  That is what I am
trying to figure out, why the decoder is not working properly "live" when it works fine using ossec-logtest.

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:       Reason:        Unknown user name or bad password     User Name: admin            Domain:        WINSERVER      Logon Type: 10         Logon Process: User32              Authentication Package: Negotiate     Workstation Name: WINSERVER           Caller User Name: WINSERVER$          Caller Domain: WINDOMAIN       Caller Logon ID: (0x0,0x3E7)            Caller Process ID: 532       &am
p;nb

[Andy Cockroft (andic)]

Sorry Martin, got to go out now, but one last suggestion

 

Are you running the web-gui? If so, can you see the level 5 alerts as they arise? And again can you determine if those alerts are logging the IP?

[Martin Gottlieb]

Hi Andy,

No problem, thanks again for all of your help. 

I'm not running the web-gui, but I'm pretty certain the IPs are not being logged based on the
log entry I posted from /var/ossec/logs/alerts/2011/Apr/ossec-alerts-27.log (below in this thread).

I'll look into installing the web-gui...

Martin

WinEvtLog: Security: AUDIT_FAILURE(529): Security: SYSTEM: NT AUTHORITY: WINSERVER: Logon Failure:       Reason:        Unknown user name or bad password     User Name: admin            Domain:        WINSERVER      Logon Type: 10         Logon Process: User32              Authentication Package: Negotiate     Workstation Name: WINSERVER           Caller User Name: WINSERVER$          Caller Domain: WINDOMAIN       Caller Logon ID: (0x0,0x3E7)            Caller Process ID: 532       &am
p;am

[Kat]

Has anyone found anything with this - I have the exact same problem -
there has got to be something that is known about this. All my Windoze
agents work fine, but I have lost every single UNIX/Linux agent and
for no reason other than the same silly

WARN: Waiting for server reply (not started).

Why are we not figuring this out??? :-(

so frustrating....

[dan (ddp)]

What OS/distro/revision are you using on your manager system?
Daniel Cid has offered to help track it down, but he needs access to a system showing this issue.
dan

[Kat]

RHEL 5.3

Only "special" update is PHP 5.3, which would have nothing to do with
OSSEC, but mentioning it.

I would be happy to supply some debug info.

It was working flawlessly when first installed, then they just started
dropping off. Agents are a mixture of AIX 6.1 , RHEL 5.3 and Solaris
10
The only agents that have never exhibited any problems are the Windoze
boxes.

-k

[Kat]

PS - I can packet capture on both ends - what would you want to see???

[Doug Burks]

Kat,

Is ossec-analysisd using a high percentage of CPU (more than 5%)?
That was what I experienced. Since I upgraded to CentOS (RHEL) 5.6, I
haven't seen the issue again.

Thanks,
--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

[dan (ddp)]

I'm trying to find a CentOS 5.2 or 5.3 ISO right now to see if I can
reproduce this. No luck so far.

I don't think it's a packet thing, I think one of the components in
ossec-analysisd is interacting poorly with something in CentOS that
was updated (to a version that doesn't have a problem with what's in
OSSEC) between 5.3 and 5.6.
I haven't had time to track down the CentOS changelogs for clues though.

[Kat]

Perhaps this is the kicker to help figure this out:

tcpdump on the ossec-server - watching the system agent attempt to
connect. But there are no firewalls in place anyway, just a router.
And the weird part is - another box, 10.15.58.62 works - and has been
- but I know if I restart it, it will fail - that is the symptom.
(both Solaris)

# tcpdump -ni bond0 host 10.15.58.60

13:01:37.848570 IP 10.15.58.60.47102 > 10.15.40.45.ossec-agent: UDP,
length 81
13:01:37.848851 IP 10.15.40.100.ossec-agent > 10.15.58.60.47102: UDP,
length 73
13:01:37.849118 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47102 unreachable, length 92
13:01:42.848771 arp who-has 10.15.58.60 tell 10.15.40.100
13:01:42.849372 arp reply 10.15.58.60 is-at 00:00:0c:01:00:40
13:01:43.849593 IP 10.15.58.60.47102 > 10.15.40.45.ossec-agent: UDP,
length 81
13:01:43.849877 IP 10.15.40.100.ossec-agent > 10.15.58.60.47102: UDP,
length 73
13:01:43.850150 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47102 unreachable, length 92
13:01:47.850439 IP 10.15.58.60.47102 > 10.15.40.45.ossec-agent: UDP,
length 81
13:01:47.850695 IP 10.15.40.100.ossec-agent > 10.15.58.60.47102: UDP,
length 73
13:01:47.850955 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47102 unreachable, length 92
13:01:52.851341 IP 10.15.58.60.47102 > 10.15.40.45.ossec-agent: UDP,
length 81
13:01:52.851653 IP 10.15.40.100.ossec-agent > 10.15.58.60.47102: UDP,
length 73
13:01:52.851894 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47102 unreachable, length 92
13:01:58.852222 IP 10.15.58.60.47102 > 10.15.40.45.ossec-agent: UDP,
length 81
13:01:58.852477 IP 10.15.40.100.ossec-agent > 10.15.58.60.47102: UDP,
length 73
13:01:58.852644 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47102 unreachable, length 92
13:02:00.853995 IP 10.15.58.60.47103 > 10.15.40.45.ossec-agent: UDP,
length 81
13:02:00.854262 IP 10.15.40.100.ossec-agent > 10.15.58.60.47103: UDP,
length 73
13:02:00.854487 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47103 unreachable, length 92
13:02:05.855020 arp who-has 10.15.58.60 tell 10.15.40.100
13:02:05.855765 arp reply 10.15.58.60 is-at 00:00:0c:01:00:40
13:02:06.855025 IP 10.15.58.60.47103 > 10.15.40.45.ossec-agent: UDP,
length 81
13:02:06.855281 IP 10.15.40.100.ossec-agent > 10.15.58.60.47103: UDP,
length 73
13:02:06.855586 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47103 unreachable, length 92
13:02:10.855908 IP 10.15.58.60.47103 > 10.15.40.45.ossec-agent: UDP,
length 81
13:02:10.856173 IP 10.15.40.100.ossec-agent > 10.15.58.60.47103: UDP,
length 73
13:02:10.856502 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47103 unreachable, length 92
13:02:15.856776 IP 10.15.58.60.47103 > 10.15.40.45.ossec-agent: UDP,
length 81
13:02:15.857057 IP 10.15.40.100.ossec-agent > 10.15.58.60.47103: UDP,
length 73
13:02:15.857359 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47103 unreachable, length 92
13:02:21.857679 IP 10.15.58.60.47103 > 10.15.40.45.ossec-agent: UDP,
length 73
13:02:21.857941 IP 10.15.40.100.ossec-agent > 10.15.58.60.47103: UDP,
length 73
13:02:21.858196 IP 10.15.58.60 > 10.15.40.100: ICMP 10.15.58.60 udp
port 47103 unreachable, length 92

[Doug Burks]

I experienced the issue with CentOS 5.5, which may be easier to find
than 5.2 or 5.3.

Thanks,
--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

[dan (ddp)]

Thanks for the heads up. I think I may have a copy of 5.5. I don't
remember having an issue like that, but it's been a while.

[Doug Burks]

It's entirely possible you never experienced it on 5.5. I had a four
or five different servers on RHEL/CentOS 5.5 and only 2 of them
exhibited this behavior. These 2 were the busiest OSSEC servers I
had, so it could be related to number of agents and/or alerts. But
again, both of these servers have been upgraded to 5.6 and I haven't
seen the issue since.

--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

[Martin Gottlieb]

Finally figured out what was going on with the Windows Event log decoder:

I originally had this:

<decoder name="winevt">
  <prematch>^WinEvtLog: Security: AUDIT_FAILURE\(\d+\): Security\.* Logon Failure: </prematch>
  <regex offset="after_prematch">User Name: (\w+) \.* Source Network Address: (\d+.\d+.\d+.\d+)</regex>
  <order>user,srcip</order>
</decoder>

When I copied and pasted an event from the alert log (see WinEvtLog in thread below) into ossec-logtest, everything matched and
it worked fine.  The problem with that approach is that tabs and other characters get converted to spaces.

When I ran the command:   sed -n 16741p logs/alerts/2011/May/ossec-alerts-04.log | bin/ossec-logtest
from within /var/ossec, the decoder did not extract the user and srcip fields.  I then ran:
sed -n 16741p logs/alerts/2011/May/ossec-alerts-04.log | od -a  and found that each Name: Value pair was
preceded by a variable number of spaces and then a TAB character.

It turns out that \s only matches spaces, not any white-space character.  So I changed my regex to this:

<regex offset="after_prematch">User\s+Name:\s*(\w+)\s+\.*Source Network Address:\s+(\d+.\d+.\d+.\d+)\s+</regex>

and it  now works.

Thanks to everyone who offered suggestions, especially Andy who pointed me to ossec-logtest.

Martin


On 4/23/2011 5:26 PM, Andy Cockroft (andic) wrote:

[Michael Starks]

On 05/04/2011 08:32 PM, Martin Gottlieb wrote:

> When I ran the command: sed -n 16741p
> logs/alerts/2011/May/ossec-alerts-04.log | bin/ossec-logtest
> from within /var/ossec, the decoder did not extract the user and srcip
> fields. I then ran:
> sed -n 16741p logs/alerts/2011/May/ossec-alerts-04.log | od -a and found
> that each Name: Value pair was

> preceded by a variable number of spaces and then a *TAB *character.
>
> It turns out that *\s* only matches spaces, not any white-space

> character. So I changed my regex to this:
>
> <regex offset="after_prematch">User\s+Name:\s*(\w+)\s+\.*Source Network
> Address:\s+(\d+.\d+.\d+.\d+)\s+</regex>
>
> and it now works.

Thanks for sharing your solution and for introducing me to od. That
looks pretty useful. I guess the lesson here is to feed the log sample
directly from archives.log into ossec-logtest to avoid any translation
issues. That's something that I certainly haven't been doing.

So is the number of spaces variable depending on the Windows version, or
in general? I am surprised we haven't seen this before..

[Martin Gottlieb]

i was referring specifically to the Windows log when mentioning the # of
spaces. I think you'd have to take any
given log record on a case-by-case basis.

[Doug Burks]

My CentOS 5.6 server is now displaying this behavior again.  ossec-analysisd is at 99% CPU usage and causing agents to disconnect.  It's been a few weeks since performing the upgrade to CentOS 5.6 and I haven't seen the issue until today.  Any ideas on how to troubleshoot ossec-analysisd?

Thanks,

Doug

[Daniel Cid]

Awesome! :) Can you run strace in there so we can get an idea on what
it is doing? It is probably
in a lock/loop somewhere....

thanks,

[Doug Burks]

I've verified this issue on two CentOS 5.6 servers now:
1. OSSEC Server installation with ~40 agents. Attaching strace to
the ossec-analysisd process shows that it's receiving syscheck info
(filenames and hashes) from some of the OSSEC agents.
2. OSSEC local installation. Attaching strace to the ossec-analysisd
process shows that it's receiving syscheck info (filenames and hashes)
from some of the local files. (Of course, this doesn't cause the
agents to disconnect since it is a local installation and there are no
agents.)

Thanks,
--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

--

[Doug Burks]

I ran strace in count mode for 10 seconds on both servers:

Server #1
strace -c -p 9773
Process 9773 attached - interrupt to quit
Process 9773 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.68 0.028147 0 165935 read
0.17 0.000048 0 157 recvfrom
0.08 0.000022 0 126 _llseek
0.07 0.000019 0 157 time
0.00 0.000000 0 114 write
------ ----------- ----------- --------- --------- ----------------
100.00 0.028236 166489 total

Server #2
strace -c -p 855
Process 855 attached - interrupt to quit
Process 855 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.017206 0 292706 read
0.00 0.000000 0 3 lseek
0.00 0.000000 0 46 recvfrom
0.00 0.000000 0 46 time
------ ----------- ----------- --------- --------- ----------------
100.00 0.017206 292801 total

What else would you like to see?

Thanks,
--
Doug Burks, GSE, CISSP
President, Greater Augusta ISSA
http://augusta.issa.org
http://securityonion.blogspot.com

[Phil Erlenbeck]

Just in case anyone runs into this on OpenBSD I had an issue with agents disconnecting and it was related to system limits.
I exceeded the system default for open files and needed to adjust with umlimit -n "integer".

On Thu, May 19, 2011 at 1:12 PM, Doug Burks <doug....@gmail.com> wrote