Why MULTIPART_STRICT_ERROR does not include REQBODY_PROCESSOR_ERROR as specification?

[Phan Thanh Bình]

Something weird with this SecRule REQBODY_PROCESSOR_ERROR, I'm trying to follow this: https://github.com/SpiderLabs/ModSecurity/issues/2267 .

When I leave as original, as below:

SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:406, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

The request is fine. But when I split a bit:

SecRule REQBODY_PROCESSOR_ERROR "!@eq 0" \
"id:'200007',phase:2,t:none,log,deny,status:406, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}'"

=> error, exactly at this rule 200007 (I already commented out the 200003 before)

So I wonder if MULTIPART_STRICT_ERROR includes REQBODY_PROCESSOR_ERROR or not? https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#multipart_strict_error 

Thanks.

[Ervin Hegedüs]

Hi Phan,

On Tue, Jan 11, 2022 at 12:09:49AM -0800, Phan Thanh Bình wrote:
> Something weird with this SecRule REQBODY_PROCESSOR_ERROR, I'm trying to
> follow this: https://github.com/SpiderLabs/ModSecurity/issues/2267 .
>
> When I leave as original, as below:
>
> SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
> "id:'200003',phase:2,t:none,log,deny,status:406, \
> msg:'Multipart request body failed strict validation: \
> PE %{REQBODY_PROCESSOR_ERROR}, \

> ...

> IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
> FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"
>
> The request is fine. But when I split a bit:
>
> SecRule REQBODY_PROCESSOR_ERROR "!@eq 0" \
> "id:'200007',phase:2,t:none,log,deny,status:406, \
> msg:'Multipart request body failed strict validation: \
> PE %{REQBODY_PROCESSOR_ERROR}'"
>
> => error, exactly at this rule 200007 (I already commented out the 200003
> before)
> So I wonder if MULTIPART_STRICT_ERROR includes REQBODY_PROCESSOR_ERROR or
> not? https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#multipart_strict_error

As I wrote my previous e-mail, the documentation says it does.

But if you check the source code, you can see, the
REQBODY_PROCESSOR_ERROR not set if there is any kind of multipart
error occurs.

Multipart strict error handling:

v2:
https://github.com/SpiderLabs/ModSecurity/blob/v2/master/apache2/re_variables.c#L1584-L1595

v3:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/request_body_processor/multipart.cc#L1078-L1085

Request body processor error variable can set in many but
different places, eg. in v2: msc_requbody.c, in v3 in
transactions.cc.

Looks like there is no common part of these variables.


a.

[Phan Thanh Bình]

Thanks for your information, so when I split  MULTIPART_STRICT_ERROR into smaller part, don't need the REQBODY_PROCESSOR_ERROR, right?

[Ervin Hegedüs]

Hi,

On Tue, Jan 11, 2022 at 12:33:45AM -0800, Phan Thanh Bình wrote:
> Thanks for your information, so when I split MULTIPART_STRICT_ERROR into
> smaller part, don't need the REQBODY_PROCESSOR_ERROR, right?

I don't understand your question exactly - what do you want to
check?

I think REQUEST_BODY_PROCESSOR error is more important than the
MULTIPART_STRICT_ERROR. The first one can occur in many cases,
eg. the payload is XML or JSON, but with some syntax error.

Multipart strict error occurs only when the CT is multipart, and
one of the listed reasons occurred. I think this can lead more
FP's.



a.

[Phan Thanh Bình]

Context: we have an issue with the quote in file name when uploading, related to MULTIPART_STRICT_ERROR, I see as advise from modsecurity team, should split this one into smaller pieces: https://github.com/SpiderLabs/ModSecurity/issues/2267 .

But when splitting this MULTIPART_STRICT_ERROR into the first piece  REQBODY_PROCESSOR_ERROR , encounter error which does not happen before. As you point it out the  REQBODY_PROCESSOR_ERROR is not included in MULTIPART_STRICT_ERROR, so now I don't need to care the  REQBODY_PROCESSOR_ERROR.

Even as you said the  REQBODY_PROCESSOR_ERROR is more important but it's not in the recommendation: https://github.com/SpiderLabs/ModSecurity/blob/v3/master/modsecurity.conf-recommended and when I've tested with  REQBODY_PROCESSOR_ERROR, it occurs a lot, so better to exclude it for now.

My purpose is splitting MULTIPART_STRICT_ERROR into smaller part, not enhance security.

Thanks.

[Hegedüs Ervin]

Hi,


before we continue this topic, could you describe *EXACTLY* what
is your error message, and could you show us a payload example
(eg. with curl).


Thanks,



a.

[Phan Thanh Bình]

Hi,

Example when GET request, it's very simple, no request body, I think this is the error:

"[9] Saving msg: Multipart request body failed strict validation: \

PE 1"

"(Rule: 200007) Executing operator "Eq" with param "0" against REQBODY_PROCESSOR_ERROR."

"Target value: "1" (Variable: REQBODY_PROCESSOR_ERROR)"

Executing operator "Eq" with param "0": what does it mean? Is there something wrong with "1"

I think with this small info, it's hard to find out what's wrong huh?

I think possibility:

- maybe in our system there is intercepter that modify, adding something?

- the other request (POST request) violates, not the simple GET request?

[Phan Thanh Bình]

Found the root cause, probably because of 'Content-Type': 'application/json' is passed as httpOptions to the GET request, can you check?

Below is info in debug.log

"(Rule: 200001) Executing operator "Rx" with param "application/json" against REQUEST_HEADERS:Content-Type."

[9]  T (0) t:lowercase: "application/json"

Target value: "application/json" (Variable: REQUEST_HEADERS:Content-Type)

Matched vars updated.

[4] Rule returned 1."

[Phan Thanh Bình]

So this is as the recommended:

SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \

msg:'Multipart request body failed strict validation: \

PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \

IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

Will equal to below (separated MULTIPART_STRICT_ERROR)

SecRule MULTIPART_BOUNDARY_QUOTED "!@eq 0" \
"id:'200008',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

BQ %{MULTIPART_BOUNDARY_QUOTED}'"

SecRule MULTIPART_BOUNDARY_WHITESPACE "!@eq 0" \
"id:'200009',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

BW %{MULTIPART_BOUNDARY_WHITESPACE}'"

SecRule MULTIPART_DATA_BEFORE "!@eq 0" \
"id:'200010',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

DB %{MULTIPART_DATA_BEFORE}'"

SecRule MULTIPART_DATA_AFTER "!@eq 0" \
"id:'200011',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

DA %{MULTIPART_DATA_AFTER}'"

SecRule MULTIPART_HEADER_FOLDING "!@eq 0" \
"id:'200012',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

HF %{MULTIPART_HEADER_FOLDING}'"

SecRule MULTIPART_LF_LINE "!@eq 0" \
"id:'200013',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

LF %{MULTIPART_LF_LINE}'"

SecRule MULTIPART_MISSING_SEMICOLON "!@eq 0" \
"id:'200014',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

SM %{MULTIPART_MISSING_SEMICOLON}'"

SecRule MULTIPART_INVALID_QUOTING "!@eq 0" \
"id:'200015',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

IQ %{MULTIPART_INVALID_QUOTING}'"

SecRule MULTIPART_INVALID_PART "!@eq 0" \
"id:'200016',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

IP %{MULTIPART_INVALID_PART}'"

SecRule MULTIPART_INVALID_HEADER_FOLDING "!@eq 0" \
"id:'200017',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

IH %{MULTIPART_INVALID_HEADER_FOLDING}'"

SecRule MULTIPART_FILE_LIMIT_EXCEEDED "!@eq 0" \
"id:'20001',phase:2,t:none,log,deny,status:406, \

msg:'Multipart request body failed strict validation: \

FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

Please let me know if my understanding is incorrect, thanks a lot.

[Ervin Hegedüs]

hi,

On Wed, Jan 12, 2022 at 01:43:15AM -0800, Phan Thanh Bình wrote:
> Example when GET request, it's very simple, no request body, I think this
> is the error:
> "[9] Saving msg: Multipart request body failed strict validation: \
> PE 1"
> "(Rule: 200007) Executing operator "Eq" with param "0" against
> REQBODY_PROCESSOR_ERROR."
> "Target value: "1" (Variable: REQBODY_PROCESSOR_ERROR)"

can you show us the `curl` request?

If your request is GET, you can't get this message, because - as
you wrote - the request body is empty.

> Executing operator "Eq" with param "0": what does it mean? Is there
> something wrong with "1"

Yes, the 1 is wrong.

'Executing operator "Eq" with param "0"' means the engine
evaulates the @eq operator, which is a two operand operator. One
of them is 0, the other one is the variable
REQBODY_PROCESSOR_ERROR, which is unfortunately (for some reason) 1.

> I think with this small info, it's hard to find out what's wrong huh?
>
> I think possibility:
> - maybe in our system there is intercepter that modify, adding something?

I don't think so,

> - the other request (POST request) violates, not the simple GET request?

what "other request"? ModSecurity checks transactions: one
transaction is one request (with headers and body) and it's
response (with headers and body). Ther is no "other request".
Other request is an other transaction.


Btw yesterday I checked the source code of mod_security2, and
found this:

https://github.com/SpiderLabs/ModSecurity/blob/v2/master/apache2/re_variables.c#L3344

https://github.com/SpiderLabs/ModSecurity/blob/v2/master/apache2/re_variables.c#L3355

As you can see these variables are marked as "deprecated", even
though theye are exist. I wouldn't use them, rather the
REQBODY_ERROR (and REQBODY_ERROR_MSG):

https://github.com/SpiderLabs/ModSecurity/blob/v2/master/apache2/re_variables.c#L527

https://github.com/SpiderLabs/ModSecurity/blob/v2/master/apache2/re_variables.c#L541

Note, that libmodsecurity3 also supports the deprecated variables
above, but as I saw they work together, I mean if the
REQBODY_ERROR has set then REQBODY_PROCESSOR_ERROR set too, and
vice versa.




a.

[Ervin Hegedüs]

hi,

On Wed, Jan 12, 2022 at 02:06:42AM -0800, Phan Thanh Bình wrote:
> Found the root cause, probably because of 'Content-Type':
> 'application/json' is passed as httpOptions to the GET request, can you
> check?

yes, it's 100% that this is the problem.

Well, altough CT header is not prohibited if the request method
is GET (see https://www.rfc-editor.org/rfc/rfc7231#section-3.1.1.5),
the rule 200001 turns on the body processor:

https://github.com/SpiderLabs/ModSecurity/blob/v3/master/modsecurity.conf-recommended#L29-L30

In this case the choosed body processor *EXPECTS* a JSON
formatted body. If the body is empty, then it generates an error.

That's it.

> Below is info in debug.log
> "(Rule: 200001) Executing operator "Rx" with param "application/json"
> against REQUEST_HEADERS:Content-Type."
> [9] T (0) t:lowercase: "application/json"
> Target value: "application/json" (Variable: REQUEST_HEADERS:Content-Type)
> Matched vars updated.
> [4] Rule returned 1."

yes.

The possible solutions:

* don't send CT header if your request is GET
* choose another (valid) content type, not application/json
* disable rule 200001 if the method is GET




a.

[Phan Thanh Bình]

Thanks for your advise "don't send CT header if your request is GET", I will note it but this is another topic and I won't touch on that since it's working fine now.

Regarding the 'Content-Type': 'application/json', it violates rule 200001, right? But in the log why it points at rule id:200007, our log below:

"Matched "Operator `Eq' with parameter `0' against variable `REQBODY_PROCESSOR_ERROR' (Value: `1' ) [file "xxx/modsecurity.conf"] [line "71"] [id "200007"] [rev ""] [msg "Multipart request body failed strict validation: \\x0aPE 1"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [unique_id "164197572686.699967"]"

Maybe because the message our log is wrong (already modify)? Where and how to get the error correctly? Or the 200001 and 200007 are linked together?

I really don't understand how to find out the incorrect value, as example above "Operator `Eq' with parameter `0' against variable `REQBODY_PROCESSOR_ERROR' (Value: `1' ) ": does this mean it will compare "0" with 1, and it's false so this rule is violated? So this message does not mention any evidence/what is wrong, right?

Where to find out it in the debug.log? (I guess this text helps "Rule returned 1.", 1 means violation?)

p/s: sometimes I see the "Value: xxx" is useful as it shows the incorrect value, sometimes it just show as Value: `1'

Thanks.

[Ervin Hegedüs]

Hi,

On Wed, Jan 12, 2022 at 03:41:41AM -0800, Phan Thanh Bình wrote:
> Regarding the 'Content-Type': 'application/json', it violates rule 200001,
> right?

no, it does not - it just triggers. The rule does not have any
disruptive action, just turns on the right body processor.

> But in the log why it points at rule id:200007, our log below:
> "Matched "Operator `Eq' with parameter `0' against variable
> `REQBODY_PROCESSOR_ERROR' (Value: `1' ) [file "xxx/modsecurity.conf"] [line
> "71"] [id "200007"] [rev ""] [msg "Multipart request body failed strict
> validation: \\x0aPE 1"] [data ""] [severity "0"] [ver ""] [maturity "0"]
> [accuracy "0"] [unique_id "164197572686.699967"]"

Rule 200001 turns on the JSON body processor.

Then the engine at the beginning of the phase 2 runs the body
processor to generate the variables (ARGS, ARGS_NAMES, ... and of
course REQBODY_PROCESSOR_ERROR).

Rule 200007 in phase 2 checks the REQBODY_PROCESSOR_ERROR
variable, which already had set, because the JSON body processor
can't parse the empty body.

*THIS* rule blocks your request.

> Maybe because the message our log is wrong (already modify)? Where and how
> to get the error correctly? Or the 200001 and 200007 are linked together?

I think every log messages are correct.

> I really don't understand how to find out the incorrect value, as example
> above "Operator `Eq' with parameter `0' against variable
> `REQBODY_PROCESSOR_ERROR' (Value: `1' ) ": does this mean it will compare
> "0" with 1, and it's false so this rule is violated? So this message does
> not mention any evidence/what is wrong, right?

Please see my explanation above.




a.