Where is inbound anomaly score ?

[Barnoux]

Hello,

Based on this super article : https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/#step_3_the_first_batch_of_rule_exclusions

How can we have the total of an anomaly score for a request, like the autor of the blog work with it in access of the Apache's log ?

I would like to have it in the caddy logs. i think it will be so much easier to analyse the data.

For the exemple bellow, i have for one request, 5 coraza messages with each of a critical severity. But in the access log of caddy for this request, i don't have the total of the inbound anomaly score.

[jcchavezs]

Ping @M4tteoP

…

On Fri, 17 Feb 2023, 21:20 Barnoux, ***@***.***> wrote: Hello, Based on this super article : https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/#step_3_the_first_batch_of_rule_exclusions How can we have the total of an anomaly score for a request, like the autor of the blog work with it in acess of the Apache's log ? I would like to have it in the caddy logs. i think it will be so much easier to analyse the data. For the exemple bellow, i have for one request, 5 coraza messages with each of a critical severity. But in the access log of caddy for this request, i don't have the total of the inbound anomaly score. [image: image] <https://user-images.githubusercontent.com/47791676/219783868-7ec00b85-aa4f-4c4a-ba13-ab94f641a33d.png> — Reply to this email directly, view it on GitHub <#662>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAWRUZESIHW66YFAPN3WX7MRDANCNFSM6AAAAAAU72U5PE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[M4tteoP]

Hi @Barnoux! Please, take a look at #684. It might not completely solve your question (To the best of my knowledge customizing access logs would require a proper support for each connector), but I think that fixed correlation rules with a high reporting level may come handy printing in the error logs, ,among other things, the total of anomaly scores

[Barnoux]

Hey @M4tteoP 😄 Now, i have something by enabling log by default log on phase 5!
I configure "tx.reporting_level=5" in crs-setup.conf. but i don't have the msg doesn't gave us the total of anomaly score in the msg.

{
  "level": "error",
  "ts": 1680024189.2647123,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. HTTP Parameter Pollution (utf-8) [file \"/ruleset/coreruleset/rules/REQUEST-921-PROTOCOL-ATTACK.conf\"] [line \"0\"] [id \"921180\"] [rev \"\"] [msg \"HTTP Parameter Pollution (utf-8)\"] [data \"Matched Data: sec-fetch-dest found within TX:paramcounter_args_names: 3\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/137/15/460\"] [tag \"paranoia-level/3\"] [hostname \"\"] [uri \"/identity/connect/token\"] [unique_id \"SJCANpvmLJlOelUj\"]\n[client \"192.168.1.1\"] Coraza: Warning. HTTP Parameter Pollution (utf-8) [file \"/ruleset/coreruleset/rules/REQUEST-921-PROTOCOL-ATTACK.conf\"] [line \"0\"] [id \"921180\"] [rev \"\"] [msg \"\"] [data \"\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/137/15/460\"] [tag \"paranoia-level/3\"] [hostname \"\"] [uri \"/identity/connect/token\"] [unique_id \"SJCANpvmLJlOelUj\"]\n"
}

{
  "level": "error",
  "ts": 1680024189.2974505,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning.  [file \"/ruleset/coreruleset/rules/RESPONSE-980-CORRELATION.conf\"] [line \"0\"] [id \"980170\"] [rev \"\"] [msg \"\"] [data \"\"] [severity \"emergency\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"reporting\"] [hostname \"\"] [uri \"/identity/connect/token\"] [unique_id \"SJCANpvmLJlOelUj\"]\n"
}

the rule that is triggered is this one

SecAction \
    "id:980170,\
    phase:5,\
    pass,\
    t:none,\
    noauditlog,\
    msg:'Anomaly Scores: \
(Inbound Scores: blocking=%{tx.blocking_inbound_anomaly_score}, detection=%{tx.detection_inbound_anomaly_score}, per_pl=%{tx.inbound_anomaly_score_pl1}-%{tx.inbound_anomaly_score_pl2}-%{tx.inbound_anomaly_score_pl3}-%{tx.inbound_anomaly_score_pl4}, threshold=%{tx.inbound_anomaly_score_threshold}) - \
(Outbound Scores: blocking=%{tx.blocking_outbound_anomaly_score}, detection=%{tx.detection_outbound_anomaly_score}, per_pl=%{tx.outbound_anomaly_score_pl1}-%{tx.outbound_anomaly_score_pl2}-%{tx.outbound_anomaly_score_pl3}-%{tx.outbound_anomaly_score_pl4}, threshold=%{tx.outbound_anomaly_score_threshold}) - \
(SQLI=%{tx.sql_injection_score}, XSS=%{tx.xss_score}, RFI=%{tx.rfi_score}, LFI=%{tx.lfi_score}, RCE=%{tx.rce_score}, PHPI=%{tx.php_injection_score}, HTTP=%{tx.http_violation_score}, SESS=%{tx.session_fixation_score}, COMBINED_SCORE=%{tx.anomaly_score})',\
    tag:'reporting',\
    ver:'OWASP_CRS/4.0.0-rc1'"

[M4tteoP]

We are getting closer :D. The last step is using a caddy connector with an updated Coraza. I think you are facing the empty message problem that I recently fixed in #684 (see here). The corazawaf/coraza-caddy#49 open PR should fix it. Hopefully we will manage to merge it soon 🤞

[Barnoux]

Alright thanks for the hard work, i need to be patient now 😄

[M4tteoP]

Hey! We merged a pretty big pr with the middleware fully refactored and pointing to a recent Coraza version. Any early feedback and updates about this issue are super welcome!
There is not a release yet, be sure to build the specific latest commit 34daaf87f9ddaca2833461de59ebada21c902598
Thanks :)

[Barnoux]

Hey 😄 with the command:
xcaddy build --with github.com/corazawaf/coraza-caddy@master

i got the following error:
Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: invalid WAF config from file: failed to readfile: open path: invalid argument
i don't have it with the coraza-caddy version v1.2.2

[jcchavezs]

Could you please try with the specific commit?

…

On Fri, 31 Mar 2023, 20:46 Barnoux, ***@***.***> wrote: Hey 😄 with the command: xcaddy build --with ***@***.*** i got the following error: Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: invalid WAF config from file: failed to readfile: open path: invalid argument i don't have it with the coraza-caddy version v1.2.2 — Reply to this email directly, view it on GitHub <#662 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAQPWYK72AZR4Y7VDW3W64RBPANCNFSM6AAAAAAU72U5PE> . You are receiving this because you commented.Message ID: ***@***.***>

[Barnoux]

Hey 😄
I used this command xcaddy build --with github.com/corazawaf/coraza-caddy@34daaf87f9ddaca2833461de59ebada21c902598
and i got this error:
Error: loading initial config: loading new config: loading http app module: provision http: server srv0: setting up route handlers: route 0: loading handler modules: position 0: loading module 'subroute': provision http.handlers.subroute: setting up subroutes: route 0: loading handler modules: position 0: loading module 'waf': provision http.handlers.waf: invalid WAF config from file: failed to readfile: open path: invalid argument

[jcchavezs]

How does your caddy config look like?

[Barnoux]

There you go! 😸

# Set up configuration for coraza_waf middleware
# `order coraza_waf first` ensures that the coraza_waf middleware is executed before any other middleware
# `persist_config off` disables configuration persistence across restarts
# `log` sets up logging configuration for Caddy
# `servers` sets up server-level settings for all servers defined in this file
{
        order coraza_waf first
        persist_config off
        log {
                level ERROR    # sets logging level to ERROR
                format json    # logs in JSON format
                output file /var/log/caddy/access.log {    # logs to file
                        roll_uncompressed    # ensures log files are uncompressed
                }
        }
        servers {
                timeouts {    # sets timeouts for various actions
                        idle 10s    # timeout for idle connections
                        read_body 10s    # timeout for reading request body
                        read_header 10s    # timeout for reading request header
                }
                max_header_size 4KB    # sets maximum header size for requests
        }
}

# Default server block that will respond with a "Not found" message for all requests to port 443
:443 {
        respond "Not found" 404
}

# Server block for {$DOMAIN} that includes the coraza_waf middleware, sets up TLS with {$EMAIL}, and defines various headers
{$DOMAIN}:443 {
        coraza_waf {
                include /ruleset/coraza.conf    # includes coraza_waf configuration file
                include /ruleset/sites/*.conf    # includes configuration files for individual sites
                include /ruleset/coreruleset/rules/*.conf    # includes core ruleset files
        }
        request_body {
                max_size 100KB    # sets maximum request body size
        }
        tls {$EMAIL}    # sets up TLS with email address
        header {    # sets various headers
                Strict-Transport-Security "max-age=15768000;"    # sets HSTS header
                X-XSS-Protection "1; mode=block"    # sets XSS protection header
                Referrer-Policy "no-referrer"    # sets referrer policy header
                X-Content-Type-Options "nosniff"    # sets content type options header
                X-Frame-Options "SAMEORIGIN"    # sets frame options header
                X-Robots-Tag "none"    # sets robots tag header
                -Server    # disables server header
        }
        @insecureadmin {    # defines named matcher for insecure admin requests
                not remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8    # matches requests not coming from specified IP ranges
                path /admin*    # matches requests to /admin* path
        }
        redir @insecureadmin /    # redirects insecure admin requests to root path
        encode gzip    # enables gzip compression
        reverse_proxy /notifications/hub vaultwarden:3012    # sets up reverse proxy for /notifications/hub path to vaultwarden:3012
        reverse_proxy vaultwarden:60278 {    # sets up reverse proxy for vaultwarden:60278
                trusted_proxies private_ranges    # trusts private IP ranges for X-Real-IP header
                header_up X-Real-IP {remote_host}    # sets X-Real-IP header to remote host
        }
}

[Barnoux]

Do you have any update about my problem ?

[jcchavezs]

Could you please try again with latest coraza-caddy version?

…

On Thu, 1 Jun 2023, 12:52 Barnoux, ***@***.***> wrote: Do you have any update about my problem ? — Reply to this email directly, view it on GitHub <#662 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAVKUEMP7IZWZ5ISVKTXJBX7FANCNFSM6AAAAAAU72U5PE> . You are receiving this because you commented.Message ID: ***@***.***>

[Barnoux]

I think it's working :) I need to test by triggering th e WAF with a XSS or OS command injection to see if the total inbound score of the request is logged and correct 👍

[Barnoux]

I tested with an OS command injection and it's working!! i have the total inbound score :) But one things is weird, i have duplicated coraza log for each line.

PL4 alert duplicated - score calculeted by correlation rule = 10

{
  "level": "error",
  "ts": 1685797070.720458,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. Invalid character in request (outside of very strict set) [file \"/ruleset/coreruleset/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"3479\"] [id \"920273\"] [rev \"\"] [msg \"Invalid character in request (outside of very strict set)\"] [data \"ARGS:foo=/etc/passwd\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/210/272\"] [tag \"paranoia-level/4\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n[client \"192.168.1.1\"] Coraza: Warning. Invalid character in request (outside of very strict set) [file \"/ruleset/coreruleset/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"3479\"] [id \"920273\"] [rev \"\"] [msg \"Invalid character in request (outside of very strict set)\"] [data \"ARGS:bar=/bin/sh\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/210/272\"] [tag \"paranoia-level/4\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

PL1 alert not duplicated - score calculeted by correlation rule = 5

{
  "level": "error",
  "ts": 1685797070.7224216,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. OS File Access Attempt [file \"/ruleset/coreruleset/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"4063\"] [id \"930120\"] [rev \"\"] [msg \"OS File Access Attempt\"] [data \"Matched Data: etc/passwd found within ARGS:foo: /etc/passwd\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [tag \"PCI/6.5.4\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

PL1 alert duplicated - score calculeted by correlation rule = 10

{
  "level": "error",
  "ts": 1685797070.723283,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. Remote Command Execution: Unix Shell Code Found [file \"/ruleset/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"4759\"] [id \"932160\"] [rev \"\"] [msg \"Remote Command Execution: Unix Shell Code Found\"] [data \"Matched Data: etc/passwd found within ARGS:foo: /etc/passwd\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/248/88\"] [tag \"PCI/6.5.2\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n[client \"192.168.1.1\"] Coraza: Warning. Remote Command Execution: Unix Shell Code Found [file \"/ruleset/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"4759\"] [id \"932160\"] [rev \"\"] [msg \"Remote Command Execution: Unix Shell Code Found\"] [data \"Matched Data: bin/sh found within ARGS:bar: /bin/sh\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/248/88\"] [tag \"PCI/6.5.2\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

Correlation alert - total inbound score = 25, without the duplication it should be = 15

{
  "level": "error",
  "ts": 1685797070.7382822,
  "logger": "http.handlers.waf",
  "msg": "[client \"192.168.1.1\"] Coraza: Warning. Anomaly Scores: (Inbound Scores: blocking=25, detection=25, per_pl=15-0-0-10, threshold=1000) - (Outbound Scores: blocking=0, detection=0, per_pl=0-0-0-0, threshold=1000) - (SQLI=0, XSS=0, RFI=0, LFI= [file \"/ruleset/coreruleset/rules/RESPONSE-980-CORRELATION.conf\"] [line \"11046\"] [id \"980170\"] [rev \"\"] [msg \"Anomaly Scores: (Inbound Scores: blocking=25, detection=25, per_pl=15-0-0-10, threshold=1000) - (Outbound Scores: blocking=0, detection=0, per_pl=0-0-0-0, threshold=1000) - (SQLI=0, XSS=0, RFI=0, LFI=\"] [data \"\"] [severity \"emergency\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy \"0\"] [tag \"reporting\"] [hostname \"\"] [uri \"/?foo=/etc/passwd&bar=/bin/sh\"] [unique_id \"DyldtRcMRMBWPrdM\"]\n"
}

[Barnoux]

Thanks a lot for the help. The initial question that i have is now resolved.

The duplicated modsecurity fields's in caddy logs's will be followed here: corazawaf/coraza-caddy#32

[jcchavezs]

Awesome!

…

On Sat, 3 Jun 2023, 17:37 Barnoux, ***@***.***> wrote: Closed #662 <#662> as resolved. — Reply to this email directly, view it on GitHub <#662>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAUPSWSRTCM7CZTOEU3XJNK4ZANCNFSM6AAAAAAU72U5PE> . You are receiving this because you commented.Message ID: ***@***.***>