CRS regression tests

[Mike Melo]

Do all CRS regression tests pass on all setups (ie 2.9-apache, 3.0-apache, 3.0-nginx)?

I am running libmodsecurity v3.0.4 with nginx 1.19.0 (connector 1.0.1) and CRS 3.2.0 and get 94 failed, 2227 passed in 1047.87 seconds without changing any rules.

If all setups dont pass 100% is there a list of tests that fail for each setup?

Thanks!

[Ervin Hegedüs]

Hi Mike,

On Tue, Aug 11, 2020 at 12:59:27PM -0700, Mike Melo wrote:
> Do all CRS regression tests pass on all setups (ie 2.9-apache, 3.0-apache,
> 3.0-nginx)?

this isn't a simple topic :).

The fact: Apache2 + mod_security2 2.9.3 *MUST* pass *ALL*
regression tests.

Other fact: Apache2 + libModSecurity3 *ISN'T* production ready,
there are few missing things from code. The best advice what can
I give you that *DON'T* use that.

> I am running libmodsecurity v3.0.4 with nginx 1.19.0 (connector 1.0.1) and
> CRS 3.2.0 and get 94 failed, 2227 passed in 1047.87 seconds without
> changing any rules.

In case of Nginx + libModSecurity3 the failed 4% of tests is
about realistic, but the 1047 seconds is a bit more - but it
depends on your HW, so no worry.

There are two classes of reasons (of failed tests):
* Nginx handles the request as different way than Apache (eg.
some header in test case forces Nginx that drops the request)
* libModSecurity3 different behavior

Now I don't know exactly which test cases belongs to which classes from
the 97 failed, but when I checked them at last, there were about
40 failed tests caused by libModSecurity3. I assume the others
occurres by webserver mismatch.

> If all setups dont pass 100% is there a list of tests that fail for each
> setup?

As I wrote above, the Apache2 + modsec2 must pass 100% of tests.

Once I played with modsec3 with Apache, and with some custom
patch (for libmodsecurity3) I reached the 100%, with CRS 3.1 (as
I remember... that was in March of 2019). But again: that was
just a playground, "mod_security3" is so far from production
ready state.

There is a small tool to check the "pure" library against CRS and
its test cases: "ftwrunner". You can find it here:

https://github.com/digitalwave/ftwrunner

As I see that's a year old, so I assume I have to make some
update (the 3.0.4 API is changed if I'm right), but may be you
can use that as is.

This tool uses your modsecurity.conf file, like your webserver
uses it: loads and parses it, create rules, starts the engine,
and so on... then it loads the CRS test cases, and starts to
emulate the requests. But there isn't any HTTP restrictions, it
just focuses to the behavior, so you can exclude the failed tests
caused by different HTTP server (Nginx vs Apache). And that's a
bit faster - the whole test takes about 150 seconds (depends on
your HW too) :).

Here is a sample list from a test case (a bit old, I don't
remember which CRS version produced this - with which libmodsecurity3
version):

- 920450-1 # know MSC bug - PR #2107
- 920450-2
- 920450-3
- 920450-4
- 920450-6
- 920450-7
- 920460-1 # know MSC bug - #2148 (double escape)
- 920460-2
- 920460-3
- 920460-4
- 941330-1 # know MSC bug - #2148 (double escape)
- 944100-11 # known MSC bug - PR #2045, ISSUE #2146
- 944100-12
- 944100-15
- 944100-16
- 944110-11 # known MSC bug - PR #2045, ISSUE #2146
- 944110-12
- 944120-5 # known MSC bug - PR #2045, ISSUE #2146
- 944120-6
- 944120-22
- 944120-23
- 944120-39
- 944120-40
- 944120-56
- 944120-57
- 944120-73
- 944120-74
- 944120-90
- 944120-91
- 944120-107
- 944120-108
- 944120-124
- 944120-125
- 944210-5 # known MSC bug - PR #2045, ISSUE #2146
- 944210-6
- 944210-22
- 944210-23
- 944210-39
- 944210-40

Where you see a comment, then it means all next cases caused by
same reason till the next comment.

Try to run ftwrunner, and let see what happens!


Hope this helped you.



a.


ps: I've updated the source tree, now the tool can be used with
both "old" and "new" API of libmodsecurity3 (see
https://github.com/digitalwave/ftwrunner/commit/2fa26e)

[Mike Melo]

Thank you!!!! 

this information is -hugely- helpful, appreciate it!

[Christian Folini]

On Tue, Aug 11, 2020 at 02:22:06PM -0700, Mike Melo wrote:
> Thank you!!!!
>
> this information is -hugely- helpful, appreciate it!

Ervin is a huge asset for our project!

Christian

> --
> You received this message because you are subscribed to the Google Groups "ModSecurity Core Rule Set project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to modsecurity-core-rule-...@owasp.org.
> To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/modsecurity-core-rule-set-project/1455a43e-600f-43ce-9adf-04151ec378dbn%40owasp.org.