detect/entropy: Unique flowvar names

[jlucovsky]

Continuation of #14008

Use unique variable names for each flowvar as they come from a global
namespace. The chosen name is: sid:<signature_id>;buffer:<buffer-name>;instance:<instance#>

Describe changes:

• Use a unique name for flowvar by appending the signature id to the name and the occurrence value. The occurrence values start from 1 and are significant if multiple entropy values are used in a single rule. They start from 1.

Entropy values will be logged into an array named entropy.

Each array element will look like this:

    {"sid":2,"buffer":"file_data","instance":1,"value":4.137370175000773}

Updates:

• Add a config setting to control whether unique names are generated: logging.entropy.make-unique
• Document new configuration setting and how output is affected.
• Disambiguate entropy output by adding the instance number.
• Changed variable name to sid:<signature_id>;buffer:<buffer-name>;instance:<instance#>
• Represent multiple entropy values in an array.
• Output log format changed
• Removed jsonbuilder changes
• Fixed ASAN issue
• Updated JSON schema -- sorted keys properly
• Removed atoi; replaced with SCStringParseU32RangeCheck
• Fixed double-free issue (cocci detected)

Issue: 7814

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7814

Provide values to any of the below to override the defaults.

• To use a Suricata-Verify or Suricata-Update pull request,
  link to the pull request in the respective _BRANCH variable.
• Leave unused overrides blank or remove.

SV_REPO=
SV_BRANCH=OISF/suricata-verify#2588
SU_REPO=
SU_BRANCH=

[codecov]

Codecov Report

❌ Patch coverage is 88.52459% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.85%. Comparing base (16d124c) to head (ef54e20).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #14009      +/-   ##
==========================================
- Coverage   83.87%   83.85%   -0.02%     
==========================================
  Files        1011     1011              
  Lines      275671   275728      +57     
==========================================
+ Hits       231207   231214       +7     
- Misses      44464    44514      +50     

Flag	Coverage Δ
fuzzcorpus	63.38% <4.91%> (-0.13%)	⬇️
livemode	19.34% <0.00%> (-0.11%)	⬇️
pcap	44.80% <0.00%> (+0.03%)	⬆️
suricata-verify	65.19% <88.52%> (+0.03%)	⬆️
unittests	59.13% <0.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[suricata-qa]

WARNING:

field	baseline	test	%
	SURI_TLPR1_stats_chk
.uptime	654	630	96.33%

Pipeline = 27963

[catenacyber]

Was still unclear to me :-/

So, I have been adding to test.rules in entropy-02 SV
alert http any any -> any any (msg:"entropy multi-buf test"; http.request_header; content: "ccept"; entropy: value <4; http.request_header; content: "gent"; entropy: value >5; sid: 20;)

I get

{
  "pcap_cnt": 8,
  "event_type": "alert",
  "metadata": {
    "entropy": [
      {
        "sid": 20,
        "buffer": "http_request_header",
        "instance": 2,
        "value": 5.061669357173793
      },
      {
        "sid": 20,
        "buffer": "http_request_header",
        "instance": 1,
        "value": 3.8136606896881853
      }
    ]
  },

And with some logging, I see that the entropy 5.06 is about the user agent (http.request_header; content: "gent"; entropy: value >5; ) and the entropy 3.8 is for Accept-Language: en-US,en after Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 was tried and had entropy 4.466285 which did not match

[catenacyber]

So, this is now good for uniqueness, but it will not be if you add something like entropy: all, value >5; see https://docs.suricata.io/en/latest/rules/integer-keywords.html#multi-integers

[victorjulien]

There is a sid field in the key, but the record is part of an alert which is locked to a sid too. Is there value in having it in the key then?

[jlucovsky]

There is a sid field in the key, but the record is part of an alert which is locked to a sid too. Is there value in having it in the key then?

Yes -- because all of the flow's flowbits are included in the alert; other rules may cause the entropy to be calculated -- see s-v's entropy-02 test for an example