tried ftwrunner, most tests failing

36 views
Skip to first unread message

Mike Melo

unread,
Aug 13, 2020, 3:44:06 PM8/13/20
to ModSecurity Core Rule Set project
Hi Airween -

I followed instructions https://github.com/digitalwave/ftwrunner - and built ftwrunner (I also have libmodsecurity built from source, tag v3.0.4).  

Most all the CRS tests fail when run from ftwrunner.  From looking at modsec debug log, it seems this is because modsec is in anomaly mode and the rule that should be forcing the block is being skipped due to SecMarker from previous rule

I am using modsecurity.conf from: https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/modsecurity.conf-recommended, using OWASP CRS 3.2.0 direct from github with crs-setup.conf from example modified to be in traditional mode  (not anomaly mode).  I run a single YAML test:

root@t430sDebian:/etc/nginx/modsec/ftwrunner# ./src/ftwrunner -d -m /etc/nginx/modsec/main.conf -f /etc/nginx/modsec/owasp-modsecurity-crs/util/regression-tests/tests/REQUEST-933-APPLICATION-ATTACK-PHP/933131Mike.yaml
933131-3-Mike: FAILED

SUMMARY:
===============================
PASSED:                   0
FAILED:                   1
FAILED (whitelisted):     0
SKIPPED:                  0
===============================
TOTAL:                    1
===============================
FAILED TESTS:             933131-3-Mike
===============================


and debug log shows:

[159734634119.448168] [/?x=$_%53%20ERVER['REQUEST_URI'];] [9] Skipped rule id '933131' due to a SecMarker: END-REQUEST-933-APPLICATION-ATTACK-PHP
[159734634119.448168] [/?x=$_%53%20ERVER['REQUEST_URI'];] [9] Rule:

Is anomaly mode required?  Will traditional mode work?  Do you have a modsecurity.conf and crs-setup.conf that are known to work?   I'm using the YAML from OWASP CRS 3.2.0 and similarly the rules from 3.2.0.

I attached all relevant files...  any tips much appreciated....

Mike,

modsec_debug.log
modsecurity.conf
crs-setup.conf
933131Mike.yaml
main.conf

Mike Melo

unread,
Aug 13, 2020, 4:47:04 PM8/13/20
to ModSecurity Core Rule Set project, Mike Melo
think it could be as simple as the logging is not getting back to the framework via modsec->setServerLogCb(logCbText); -- is this known to work for v3.0.4 libmodsecurity (which i used to build ftwrunner as well). if there is no logging at all then no_log_contains will all pass and log_contains will all fail

Ervin Hegedüs

unread,
Aug 13, 2020, 4:51:56 PM8/13/20
to Mike Melo, ModSecurity Core Rule Set project
Hi Mike,


On Thu, Aug 13, 2020 at 12:44:06PM -0700, Mike Melo wrote:
>
> I followed instructions https://github.com/digitalwave/ftwrunner - and
> built ftwrunner (I also have libmodsecurity built from source, tag
> v3.0.4).

oh, it's nice to see that somebody uses this tool... :D

> Most all the CRS tests fail when run from ftwrunner. From looking at
> modsec debug log, it seems this is because modsec is in anomaly mode and
> the rule that should be forcing the block is being skipped due to SecMarker
> from previous rule
>
> I am using modsecurity.conf from:
> https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/modsecurity.conf-recommended,
> using OWASP CRS 3.2.0 direct from github with crs-setup.conf from example
> modified to be in traditional mode (not anomaly mode). I run a single
> YAML test:
>
> root@t430sDebian:/etc/nginx/modsec/ftwrunner# *./src/ftwrunner -d -m
> /etc/nginx/modsec/main.conf -f
> /etc/nginx/modsec/owasp-modsecurity-crs/util/regression-tests/tests/REQUEST-933-APPLICATION-ATTACK-PHP/933131Mike.yaml*
> 933131-3-Mike: FAILED

well, looks like everything is almost fine - except one thing.
You forgot to swith on the required rule - see this:

https://github.com/coreruleset/coreruleset/blob/v3.2/master/util/regression-tests/README.md#requirements

(I've copied this link from v3.2/master).

There is a very important thing in case of *EVERY* formation
(Nginx/Apache with mod_sec2/libmodsecurity3), if you want to run
the regression tests for CRS.

This FAILED tests had triggered by wrong CRS setup - may be your
CRS setup is well, but you can't run the test cases. It needs to
add the extra rule.

I installed your comfig to a separate directory, and here is the
only modificítion what I made:

$ git diff
diff --git a/owasp-modsecurity-crs/crs-setup.conf b/owasp-modsecurity-crs/crs-setup.conf
index d27d51e..c7f0852 100644
--- a/owasp-modsecurity-crs/crs-setup.conf
+++ b/owasp-modsecurity-crs/crs-setup.conf
@@ -798,3 +798,15 @@ SecAction \
pass,\
t:none,\
setvar:tx.crs_setup_version=320"
+
+SecAction "id:900005,\
+ phase:1,\
+ nolog,\
+ pass,\
+ ctl:ruleEngine=DetectionOnly,\
+ ctl:ruleRemoveById=910000,\
+ setvar:tx.paranoia_level=4,\
+ setvar:tx.crs_validate_utf8_encoding=1,\
+ setvar:tx.arg_name_length=100,\
+ setvar:tx.arg_length=400"
+

as you can see, there is only the necessary rule with id 900005,
as it stands in documentation.

> SUMMARY:
> ===============================
> PASSED: 0
> FAILED: 1
> FAILED (whitelisted): 0
> SKIPPED: 0
> ===============================
> TOTAL: 1
> ===============================
> FAILED TESTS: 933131-3-Mike
> ===============================

and here is the result:

$ ./ftwrunner -d -m /home/airween/tmp/mikemelo/main.conf -f /home/airween/tmp/mikemelo/933131Mike.yaml
933131-3-Mike: PASSED

SUMMARY:
===============================
PASSED: 1
FAILED: 0
FAILED (whitelisted): 0
SKIPPED: 0
===============================
TOTAL: 1
===============================


Now I unpack the regression tests into the working directory from
v3.2/master, and ran all tests:


$ time ./ftwrunner -d -m /home/airween/tmp/mikemelo/main.conf -f /home/airween/tmp/mikemelo/regression-tests/tests/
...
...
SUMMARY:
===============================
PASSED: 2181
FAILED: 6
FAILED (whitelisted): 35
SKIPPED: 31
===============================
TOTAL: 2253
===============================
FAILED TESTS: 920450-1, 920450-2, 920450-3, 920450-4, 920450-6, 920450-7
===============================

real 1m31.545s
user 0m42.068s
sys 0m40.312s

(looks like I forgot to clear the whitelisted tests - what I
showed you before)

As you can see, 6 new tests were failed, and 35 whitelisted.

I assume the reason is the incompability of libmodsecurity3. As
you can see, all failed tests are from rule 920450. This is a bug
in libmodsecurity3, and there is a fix for that:
https://github.com/SpiderLabs/ModSecurity/pull/2297

Also note, that the CRS was fixed here:
https://github.com/SpiderLabs/owasp-modsecurity-crs/pull/1745/files

so the 3.4 isn't affected.



> I attached all relevant files... any tips much appreciated....

thanks, that was a big help - and hope now you can run the tests.



regards,



a.

Ervin Hegedüs

unread,
Aug 13, 2020, 4:55:37 PM8/13/20
to Mike Melo, ModSecurity Core Rule Set project
Hi Mike,

On Thu, Aug 13, 2020 at 01:47:04PM -0700, Mike Melo wrote:
> think it could be as simple as the logging is not getting back to the
> framework via modsec->setServerLogCb(logCbText); -- is this known to work
> for v3.0.4 libmodsecurity (which i used to build ftwrunner as well). if
> there is no logging at all then no_log_contains will all pass and
> log_contains will all fail

no, this callback needs for the correct work.

The reason is different - see my another answer.



regards,


a.

Mike Melo

unread,
Aug 13, 2020, 5:03:36 PM8/13/20
to ModSecurity Core Rule Set project, air...@gmail.com, ModSecurity Core Rule Set project, Mike Melo
SUMMARY:
===============================
PASSED:                   2184
FAILED:                   41
FAILED (whitelisted):     0
SKIPPED:                  31
===============================
TOTAL:                    2256
===============================


airween - thanks for repro'ing this and finding the issue!!!

how do i buy you a beer (or three) ???

Ervin Hegedüs

unread,
Aug 13, 2020, 5:14:50 PM8/13/20
to Mike Melo, ModSecurity Core Rule Set project
On Thu, Aug 13, 2020 at 02:03:36PM -0700, Mike Melo wrote:
> SUMMARY:
> ===============================
> PASSED: 2184
> FAILED: 41
> FAILED (whitelisted): 0
> SKIPPED: 31
> ===============================
> TOTAL: 2256
> ===============================
>

cool,

> airween - thanks for repro'ing this and finding the issue!!!

yw,

> how do i buy you a beer (or three) ???

:D:D

hope once you can at some way... :)



a.

ps: just one last note - there are 31 skipped test. Here you can
find more info about SKIPPED tests:

https://github.com/digitalwave/ftwrunner#output

but "Note, that this tests will not counts anywhere at final
summary."

If you see there are more than 0 skipped test, it means the test
expects some HTTP status or error code - but there is no status
(because there isn't any HTTP daemon). Or the test should sent
reauest as raw_request or encoded_request - these aren't
implemented yet.


Mike Melo

unread,
Aug 13, 2020, 5:29:03 PM8/13/20
to ModSecurity Core Rule Set project, air...@gmail.com, ModSecurity Core Rule Set project, Mike Melo
ok thanks for that, will check out the skipped test details...
Reply all
Reply to author
Forward
0 new messages