[Security Solution][Detection Engine] skips processing gaps in auto backfill scheduler after number of retries reaches configured value

[vitaliidm]

Summary

• addresses https://github.com/elastic/security-team/issues/14888

UI

To test

Enable in kibana.dev.yml

xpack.alerting.gapAutoFillScheduler.enabled: true

1. Ensure you have rules with gaps

There are two ways to create gaps:

• Manual method:
  Create and enable a security rule that fails(can be ES|QL query that accesses non-existing field or so on) with a 1-minute interval and 0-second lookback.
  After the first run, disable the rule, wait 5 minutes, and then enable it again you should execution error about gaps, and see the gap in the gaps table in the execution tab.

2. Create and enable the scheduler

Run the following request (adjust as needed for your environment):

curl --location --request POST 'http://localhost:5601/internal/alerting/rules/gaps/auto_fill_scheduler' \
--header 'kbn-xsrf: kibana' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: kibana' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YOUR_AUTHRIZATION' \
--data-raw '{
       "id": "gap-scheduler",
       "name": "gap-scheduler",
       "enabled": true,
       "max_backfills": 1000,
       "num_retries": 3,
       "gap_fill_range": "now-90d",
       "schedule": { "interval": "1m" },
       "scope": ["security"],
       "rule_types": [
         { "type": "siem.queryRule", "consumer": "siem" },
         { "type": "siem.savedQueryRule", "consumer": "siem" },
         { "type": "siem.eqlRule", "consumer": "siem" },
         { "type": "siem.esqlRule", "consumer": "siem" },
         { "type": "siem.thresholdRule", "consumer": "siem" },
         { "type": "siem.newTermsRule", "consumer": "siem" },
         { "type": "siem.mlRule", "consumer": "siem" },
         { "type": "siem.indicatorRule", "consumer": "siem" }
       ]
     }'

3. Verify that it works

Go to rule details page and observe number of failed attempts in table by hovering over error health indicator in status column. When number of failures becomes greater than number of retries, this gap won't be getting processed anymore

Followup

https://github.com/elastic/security-team/issues/14943

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: bc110c6

Failed CI Steps

• Check Types

History

• 💔 Build #368354 failed 90dcb2b
• 💔 Build #367902 failed b13dfbe
• 💔 Build #367900 failed b13dfbe
• 💔 Build #367877 failed 7c24c30
• 💔 Build #367784 failed 314daf2

cc @vitaliidm

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8fc3991

Failed CI Steps

• Scout: [ observability / observability ] plugin
• Scout: [ observability / profiling ] plugin

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.1MB	11.1MB	+443.0B

History

• 💔 Build #368605 failed d0f33c0
• 💔 Build #368573 failed bc110c6
• 💔 Build #368354 failed 90dcb2b
• 💔 Build #367902 failed b13dfbe
• 💔 Build #367900 failed b13dfbe
• 💔 Build #367877 failed 7c24c30

cc @vitaliidm