Detecting scrapper activity

[frwa onto]

Dear All,

            I am new to ossec. I am still learning how it works just wondering can it detect scraper activities because I have banned directory traversing but I notice yet the scrapper manage to get to some of the directories but got this error Directory index forbidden by Options directive: 

[dan (ddp)]

Are these logs being monitored by OSSEC? You should be able to create
a rule looking for the log message.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ossec-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[frwa onto]

Dear Dan,

              If I look into my ossec.conf I can see this both these apache_rules.xml and web_appsec_rules.xml and I can see it monitors the /var/log/httpd/error_log. What else do I need to check on ? Is monitoring just fine or must I still create rules sorry I am newbie into this. Besides that when will the rootkit check will be done on a period basic or launch manually ?

[dan (ddp)]

On Wed, Nov 6, 2013 at 9:54 AM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,
> If I look into my ossec.conf I can see this both these
> apache_rules.xml and web_appsec_rules.xml and I can see it monitors the
> /var/log/httpd/error_log. What else do I need to check on ? Is monitoring
> just fine or must I still create rules sorry I am newbie into this. Besides

You didn't provide a log sample, so I cannot determine whether the log
will be identified by OSSEC or not.

> that when will the rootkit check will be done on a period basic or launch
> manually ?
>

It should scan periodically.

[frwa onto]

Dear Dan,

               Which log sample you prefer to have the apache error log or the ossec log ? Are the rules need tweaking too? How can I be sure the rootkit is running any log to check on it?

You received this message because you are subscribed to a topic in the Google Groups "ossec-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ossec-list/O551cLvYKrs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ossec-list+...@googlegroups.com.

[dan (ddp)]

On Wed, Nov 6, 2013 at 10:39 AM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,

> Which log sample you prefer to have the apache error log or

Which log messasge do you want to trigger an alert? That is the
important one here, right? In your original message you mentioned a
log message containing "Directory index forbidden by Options
directive:," but did not include the entire log message. I assume this
is the log message you want an alert on?

> the ossec log ? Are the rules need tweaking too? How can I be sure the
> rootkit is running any log to check on it?
>

Check the ossec.log. If there is no mention of it, try turning on
debug for syscheckd.

[frwa onto]

Dear Dan,

               The log message is from the httpd error log. Here is the part of the log where I notice.

[Sun Oct 13 12:33:29 2013] [error] [client 103.246.38.196] Directory index forbidden by Options directive: /var/www/html/*******/
[Sun Oct 13 12:33:30 2013] [error] [client 103.246.38.196] File does not exist: /var/www/html/images
[Sun Oct 13 12:33:30 2013] [error] [client 103.246.38.196] Directory index forbidden by Options directive: /var/www/html/******/images/

What should I look for the ossec.log for the syscheckd ? What is the command to turning the debug for syscheckd ?

[dan (ddp)]

On Thu, Nov 7, 2013 at 10:20 AM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,

> The log message is from the httpd error log. Here is the part
> of the log where I notice.
>
> [Sun Oct 13 12:33:29 2013] [error] [client 103.246.38.196] Directory index
> forbidden by Options directive: /var/www/html/*******/
> [Sun Oct 13 12:33:30 2013] [error] [client 103.246.38.196] File does not
> exist: /var/www/html/images
> [Sun Oct 13 12:33:30 2013] [error] [client 103.246.38.196] Directory index
> forbidden by Options directive: /var/www/html/******/images/
>

The rules that match these log messages won't trigger an email or
anything. So you'll have to create better rules for them.

/tmp/xxx contains the log messages above.

# cat /tmp/xxx | /var/ossec/bin/ossec-logtest
2013/11/07 10:23:20 ossec-testrule: INFO: Reading local decoder file.
2013/11/07 10:23:21 ossec-testrule: INFO: Started (pid: 16416).
ossec-testrule: Type one log per line.



**Phase 1: Completed pre-decoding.
full event: '[Sun Oct 13 12:33:29 2013] [error] [client

103.246.38.196] Directory index forbidden by Options directive:

/var/www/html/*******/'
hostname: 'arrakis'
program_name: '(null)'
log: '[error] [client 103.246.38.196] Directory index forbidden
by Options directive: /var/www/html/*******/'

**Phase 2: Completed decoding.
decoder: 'apache-errorlog'
srcip: '103.246.38.196'

**Phase 3: Completed filtering (rules).
Rule id: '30101'
Level: '0'
Description: 'Apache error messages grouped.'


**Phase 1: Completed pre-decoding.
full event: '[Sun Oct 13 12:33:30 2013] [error] [client
103.246.38.196] File does not exist: /var/www/html/images'
hostname: 'arrakis'
program_name: '(null)'
log: '[error] [client 103.246.38.196] File does not exist:
/var/www/html/images'

**Phase 2: Completed decoding.
decoder: 'apache-errorlog'
srcip: '103.246.38.196'

**Phase 3: Completed filtering (rules).
Rule id: '30112'
Level: '0'
Description: 'Attempt to access an non-existent file (those are
reported on the access.log).'

> What should I look for the ossec.log for the syscheckd ? What is the command
> to turning the debug for syscheckd ?
>

Kill ossec-syscheckd, then run `/var/ossec/bin/ossec-syscheckd -d`

[frwa onto]

Dear Dan,

              So meaning that the existing apache rules wont trigger this as an alert is it ? How to create new rules and are we allowed to add existing rules? Does it need any compilation or just some xml documents? You added these what are these actually

*Phase 1: Completed pre-decoding.
       full event: '[Sun Oct 13 12:33:29 2013] [error] [client

103.246.38.196] Directory index forbidden by Options directive:

/var/www/html/*******/'
       hostname: 'arrakis'
       program_name: '(null)'
       log: '[error] [client 103.246.38.196] Directory index forbidden
by Options directive: /var/www/html/*******/'

**Phase 2: Completed decoding.
       decoder: 'apache-errorlog'
       srcip: '103.246.38.196'

**Phase 3: Completed filtering (rules).
       Rule id: '30101'
       Level: '0'
       Description: 'Apache error messages grouped.'

[dan (ddp)]

On Thu, Nov 7, 2013 at 1:22 PM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,
> So meaning that the existing apache rules wont trigger this as
> an alert is it ? How to create new rules and are we allowed to add existing

I provided the output from ossec-logtest. That output tells you
exactly what OSSEC does with the log messages you provided.

> rules? Does it need any compilation or just some xml documents? You added

Just xml documents. You'll have to restart the ossec processes on the
server after modifying the rules.

> these what are these actually

Those are not rules, that is the output from ossec-logtest

[frwa onto]

Dear Dan,

               Ok so to test it you created a file called xxx and you let the ossec engine to run through it to decode the message rite. Please correct me if my understanding is wrong here. Do you think for the two rules Rule id: '30101' and '30112' should I increase the Level: '0' for the email trigger as I have set now to 5 for email triggering?

[dan (ddp)]

On Thu, Nov 7, 2013 at 1:35 PM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,

> Ok so to test it you created a file called xxx and you let
> the ossec engine to run through it to decode the message rite. Please

Essentially, yes.

> correct me if my understanding is wrong here. Do you think for the two rules
> Rule id: '30101' and '30112' should I increase the Level: '0' for the email
> trigger as I have set now to 5 for email triggering?
>

30101 is very generic. I would not increase the level of that rule,
only use it as a parent rule for more specific rules.
If 30112 is something you want to be notified of, you should increase the level.

[frwa onto]

Dear Dan,

              Thank you for your guidance I am really learning a lots of new things in ossec now. So I can increase that rule and restart ossec. How will I know exactly where this 30112 rule is stored ? Thank you once again.

[dan (ddp)]

On Thu, Nov 7, 2013 at 1:46 PM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,

> Thank you for your guidance I am really learning a lots of new
> things in ossec now. So I can increase that rule and restart ossec. How will
> I know exactly where this 30112 rule is stored ? Thank you once again.
>

The rules are kept in the rules files, which are in /var/ossec/rules
(on a default install)

[frwa onto]

Dear Dan,

              Ok yes I found these both rules are in the apache_rules.xml. What is in the log_entries I see some things which are not even related to my system with some very old time stamp ?

[dan (ddp)]

On Fri, Nov 8, 2013 at 10:47 PM, frwa onto <frwa...@gmail.com> wrote:
> Dear Dan,

> Ok yes I found these both rules are in the apache_rules.xml.
> What is in the log_entries I see some things which are not even related to
> my system with some very old time stamp ?
>

I know nothing about your log files. I'm not sure how I would be
expected to know what you are asking about.