Already sent alert is sent again after restart #1683

assimdarwish · 2025-07-02T11:16:22Z

assimdarwish
Jul 2, 2025

Hey!

I have an elastalert2 rule, that should suppress already alerted alerts for 14 days, if the data.upn is the same. My problem is, as soon as I restart the elastalert2 container, these already sent alerts, are sent again or if the rule hits again (so new event with same data.upn) - e.g. after x days and in meantime the container was restarted - an alert is sent, although there was an alert before the restart.. and I cannot understand this behaviour!? There's also entry in the _silence index.

The start of my rule:

name: ElastAlert Rule

type: frequency

index: wazuh-alerts-*-%Y.*
use_strftime_index: true

timestamp_field: "@timestamp"
timestamp_type: "iso"

num_events: 1

timeframe:
  minutes: 1

realert:
  days: 14

query_key: data.upn

include:
...

Most of the config.yaml is default (run every 1min, buffer_time=15min)

Anybody any idea? Thank you!

jertel · 2025-07-02T11:41:01Z

jertel
Jul 2, 2025
Maintainer

I just tried to replicate this and it's not resending alerts after the restart. Instead, I see it query the silence index and confirms that the alert is still silenced:

2025-07-02 07:40:21,196     INFO           elastalert Ignoring match for silenced rule my_test_frequency_rule

You will need to enable debug logging and slowly retrace what's happening in order to locate the issue.

8 replies

jertel Jul 2, 2025
Maintainer

Clear out the elastalert.handlers value: handlers: []

assimdarwish Jul 2, 2025
Author

like that?

logging:
  version: 1
  disable_existing_loggers: false
  formatters:
    logline:
      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
  handlers:
    console:
      class: logging.StreamHandler
      formatter: logline
      level: DEBUG
      stream: ext://sys.stderr
  loggers:
    elastalert:
      level: DEBUG
      handlers: []
      propagate: true
  root:
    level: DEBUG
    handlers: []

I cannot see docker logs anymore.

jertel Jul 2, 2025
Maintainer

No. I did not suggest removing it from root.handlers. Only elastalert.handlers.

assimdarwish Jul 2, 2025
Author

ah ok got it.

Just got new alerts, although I already were notified. I also have entries in the _silence index for the users.

2025-07-02 13:50:03,697     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:01:00], next run at: 2025-07-02 13:51:03 UTC)" (scheduled at 2025-07-02 13:50:03.696432+00:00)
2025-07-02 13:50:03,699     INFO           elastalert Background configuration change check run at 2025-07-02 13:50 UTC
2025-07-02 13:50:03,699     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:01:00], next run at: 2025-07-02 13:51:03 UTC)" executed successfully
2025-07-02 13:50:03,703    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert/_search?size=1000 HTTP/1.1" 200 160
2025-07-02 13:50:03,703     INFO           elastalert Background alerts thread 0 pending alerts sent at 2025-07-02 13:50 UTC
2025-07-02 13:50:03,704     INFO apscheduler.executors.default Job "Internal: Handle Pending Alerts (trigger: interval[0:01:00], next run at: 2025-07-02 13:51:03 UTC)" executed successfully
2025-07-02 13:50:03,706     INFO           elastalert Disabled rules are: []
2025-07-02 13:50:03,706     INFO           elastalert Sleeping for 59.999884 seconds
2025-07-02 13:50:46,574    DEBUG apscheduler.scheduler Looking for jobs to run
2025-07-02 13:50:46,575    DEBUG apscheduler.scheduler Next wakeup is due at 2025-07-02 13:50:58.671568+00:00 (in 12.096114 seconds)
2025-07-02 13:50:46,575     INFO apscheduler.executors.default Running job "Rule: ElastAlert mail   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:51:47 UTC)" (scheduled at 2025-07-02 13:50:46.574365+00:00)
2025-07-02 13:50:46,647    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /wazuh-alerts-*-2025.*/_search?_source_includes=data.upn%2CGeoLocation.country_name%2Cdata.srcip%2Cdata.vpn_ip%2C%40timestamp&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 1309
2025-07-02 13:50:46,647    DEBUG           elastalert {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHGkWX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdxZibFFjZENId1RRMnE0cXM0MGJTcVJAAAAAAKQNkWUFlwUEMza0NTOFcyU2RJeG9hcWFCZxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHGoWX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdw==', 'took': 64, 'timed_out': False, '_shards': {'total': 549, 'successful': 549, 'skipped': 546, 'failed': 0}, 'hits': {'total': {'value': 3, 'relation': 'eq'}, 'max_score': None, 'hits': [{'_index': 'wazuh-alerts-4.x-2025.07.02', '_id': 'G8Zky5cB2SWYGdswkmCi', '_score': None, '_source': {'@timestamp': '2025-07-02T13:47:49.092Z', 'data': {'srcip': '1.1.1.1', 'upn': '[email protected]', 'vpn_ip': '10.10.10.10'}, 'GeoLocation': {'country_name': 'Germany'}}, 'sort': [1751464069092]}, {'_index': 'wazuh-alerts-4.x-2025.07.02', '_id': 'oPFny5cBTQFWR9uML09D', '_score': None, '_source': {'@timestamp': '2025-07-02T13:50:15.240Z', 'data': {'srcip': '2.2.2.2', 'upn': '[email protected]', 'vpn_ip': '10.10.10.11'}, 'GeoLocation': {'country_name': 'Germany'}}, 'sort': [1751464215240]}, {'_index': 'wazuh-alerts-4.x-2025.07.02', '_id': 'ofFny5cBTQFWR9uML09D', '_score': None, '_source': {'@timestamp': '2025-07-02T13:50:15.240Z', 'data': {'srcip': '1.1.1.1', 'upn': '[email protected]', 'vpn_ip': '10.252.124.2'}, 'GeoLocation': {'country_name': 'Germany'}}, 'sort': [1751464215240]}]}}
2025-07-02 13:50:46,648     INFO           elastalert Queried rule ElastAlert mail   Rule from 2025-07-02 13:35 UTC to 2025-07-02 13:50 UTC: 3 / 3 hits
2025-07-02 13:50:46,652    DEBUG urllib3.connectionpool https://opensearch:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2025-07-02 13:50:46,657    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 160
2025-07-02 13:50:46,662    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 160
2025-07-02 13:50:46,670    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_silence/_doc HTTP/1.1" 201 171
2025-07-02 13:50:46,671    DEBUG urllib3.connectionpool Starting new HTTPS connection (1): hooks.slack.com:443
2025-07-02 13:50:46,951    DEBUG urllib3.connectionpool https://hooks.slack.com:443 "POST /services/1234ASDF HTTP/1.1" 200 2
2025-07-02 13:50:46,952     INFO           elastalert Alert 'ElastAlert mail   Rule' sent to Slack
2025-07-02 13:50:46,953  WARNING          py.warnings /usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'opensearch'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2025-07-02 13:50:46,961    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert/_doc HTTP/1.1" 201 163
2025-07-02 13:50:46,965    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_silence/_search?_source_includes=until%2Cexponent&size=1 HTTP/1.1" 200 160
2025-07-02 13:50:46,966     INFO           elastalert Ignoring match for silenced rule ElastAlert mail   [email protected]
2025-07-02 13:50:46,974    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_status/_doc HTTP/1.1" 201 171
2025-07-02 13:50:46,974     INFO           elastalert Ran ElastAlert mail   Rule from 2025-07-02 13:35 UTC to 2025-07-02 13:50 UTC: 3 query hits (1 already seen), 2 matches, 1 alerts sent
2025-07-02 13:50:46,974     INFO           elastalert ElastAlert mail   Rule range 900
2025-07-02 13:50:46,974     INFO apscheduler.executors.default Job "Rule: ElastAlert mail   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:51:47 UTC)" executed successfully
2025-07-02 13:50:58,671    DEBUG apscheduler.scheduler Looking for jobs to run
2025-07-02 13:50:58,672    DEBUG apscheduler.scheduler Next wakeup is due at 2025-07-02 13:51:03.696212+00:00 (in 5.023974 seconds)
2025-07-02 13:50:58,672     INFO apscheduler.executors.default Running job "Rule: ElastAlert   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:01 UTC)" (scheduled at 2025-07-02 13:50:58.671568+00:00)
2025-07-02 13:50:58,756    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /wazuh-alerts-*-2025.*/_search?_source_includes=data.srcip%2C%40timestamp%2Cdata.dstuser%2CGeoLocation.country_name&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 451
2025-07-02 13:50:58,756    DEBUG           elastalert {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHGwWX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdxZfaFkzdVRXZlJGbU9ybWZrcTNUZW9nAAAAAAAKTX0WcUhzbkdYd3hTQlNDVUNzdmJpVU9adxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHG0WX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdw==', 'took': 75, 'timed_out': False, '_shards': {'total': 549, 'successful': 549, 'skipped': 546, 'failed': 0}, 'hits': {'total': {'value': 0, 'relation': 'eq'}, 'max_score': None, 'hits': []}}
2025-07-02 13:50:58,756     INFO           elastalert Queried rule ElastAlert   Rule from 2025-07-02 13:35 UTC to 2025-07-02 13:50 UTC: 0 / 0 hits
2025-07-02 13:50:58,759    DEBUG urllib3.connectionpool https://opensearch:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2025-07-02 13:50:58,768    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_status/_doc HTTP/1.1" 201 171
2025-07-02 13:50:58,768     INFO           elastalert Ran ElastAlert   Rule from 2025-07-02 13:35 UTC to 2025-07-02 13:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
2025-07-02 13:50:58,768     INFO           elastalert ElastAlert   Rule range 900
2025-07-02 13:50:58,768     INFO apscheduler.executors.default Job "Rule: ElastAlert   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:01 UTC)" executed successfully
2025-07-02 13:51:03,696    DEBUG apscheduler.scheduler Looking for jobs to run
2025-07-02 13:51:03,696    DEBUG apscheduler.scheduler Next wakeup is due at 2025-07-02 13:51:47.331807+00:00 (in 43.634976 seconds)
2025-07-02 13:51:03,697     INFO apscheduler.executors.default Running job "Internal: Handle Pending Alerts (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:03 UTC)" (scheduled at 2025-07-02 13:51:03.696212+00:00)
2025-07-02 13:51:03,697     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:03 UTC)" (scheduled at 2025-07-02 13:51:03.696432+00:00)
2025-07-02 13:51:03,698     INFO           elastalert Background configuration change check run at 2025-07-02 13:51 UTC
2025-07-02 13:51:03,698     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:03 UTC)" executed successfully
2025-07-02 13:51:03,702    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert/_search?size=1000 HTTP/1.1" 200 160
2025-07-02 13:51:03,702     INFO           elastalert Background alerts thread 0 pending alerts sent at 2025-07-02 13:51 UTC
2025-07-02 13:51:03,702     INFO apscheduler.executors.default Job "Internal: Handle Pending Alerts (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:03 UTC)" executed successfully
2025-07-02 13:51:03,706     INFO           elastalert Disabled rules are: []
2025-07-02 13:51:03,707     INFO           elastalert Sleeping for 59.999842 seconds
2025-07-02 13:51:47,332    DEBUG apscheduler.scheduler Looking for jobs to run
2025-07-02 13:51:47,332    DEBUG apscheduler.scheduler Next wakeup is due at 2025-07-02 13:52:01.502043+00:00 (in 14.169464 seconds)
2025-07-02 13:51:47,332     INFO apscheduler.executors.default Running job "Rule: ElastAlert mail   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:48 UTC)" (scheduled at 2025-07-02 13:51:47.331807+00:00)
2025-07-02 13:51:47,427    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /wazuh-alerts-*-2025.*/_search?_source_includes=data.upn%2CGeoLocation.country_name%2Cdata.srcip%2Cdata.vpn_ip%2C%40timestamp&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 1309
2025-07-02 13:51:47,427    DEBUG           elastalert {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHHAWX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdxZibFFjZENId1RRMnE0cXM0MGJTcVJAAAAAAKQNwWUFlwUEMza0NTOFcyU2RJeG9hcWFCZxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHG8WX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdw==', 'took': 84, 'timed_out': False, '_shards': {'total': 549, 'successful': 549, 'skipped': 546, 'failed': 0}, 'hits': {'total': {'value': 3, 'relation': 'eq'}, 'max_score': None, 'hits': [{'_index': 'wazuh-alerts-4.x-2025.07.02', '_id': 'G8Zky5cB2SWYGdswkmCi', '_score': None, '_source': {'@timestamp': '2025-07-02T13:47:49.092Z', 'data': {'srcip': '1.1.1.1', 'upn': '[email protected]', 'vpn_ip': '10.10.10.10'}, 'GeoLocation': {'country_name': 'Germany'}}, 'sort': [1751464069092]}, {'_index': 'wazuh-alerts-4.x-2025.07.02', '_id': 'oPFny5cBTQFWR9uML09D', '_score': None, '_source': {'@timestamp': '2025-07-02T13:50:15.240Z', 'data': {'srcip': '2.2.2.2', 'upn': '[email protected]', 'vpn_ip': '10.10.10.11'}, 'GeoLocation': {'country_name': 'Germany'}}, 'sort': [1751464215240]}, {'_index': 'wazuh-alerts-4.x-2025.07.02', '_id': 'ofFny5cBTQFWR9uML09D', '_score': None, '_source': {'@timestamp': '2025-07-02T13:50:15.240Z', 'data': {'srcip': '1.1.1.1', 'upn': '[email protected]', 'vpn_ip': '10.10.10.10'}, 'GeoLocation': {'country_name': 'Germany'}}, 'sort': [1751464215240]}]}}
2025-07-02 13:51:47,427     INFO           elastalert Queried rule ElastAlert mail   Rule from 2025-07-02 13:36 UTC to 2025-07-02 13:51 UTC: 3 / 3 hits
2025-07-02 13:51:47,431    DEBUG urllib3.connectionpool https://opensearch:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2025-07-02 13:51:47,440    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_status/_doc HTTP/1.1" 201 171
2025-07-02 13:51:47,441     INFO           elastalert Ran ElastAlert mail   Rule from 2025-07-02 13:36 UTC to 2025-07-02 13:51 UTC: 3 query hits (3 already seen), 0 matches, 0 alerts sent
2025-07-02 13:51:47,441     INFO           elastalert ElastAlert mail   Rule range 900
2025-07-02 13:51:47,441     INFO apscheduler.executors.default Job "Rule: ElastAlert mail   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:52:48 UTC)" executed successfully
2025-07-02 13:52:01,502    DEBUG apscheduler.scheduler Looking for jobs to run
2025-07-02 13:52:01,502    DEBUG apscheduler.scheduler Next wakeup is due at 2025-07-02 13:52:03.696212+00:00 (in 2.193371 seconds)
2025-07-02 13:52:01,503     INFO apscheduler.executors.default Running job "Rule: ElastAlert   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:53:05 UTC)" (scheduled at 2025-07-02 13:52:01.502043+00:00)
2025-07-02 13:52:01,593    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /wazuh-alerts-*-2025.*/_search?_source_includes=data.srcip%2C%40timestamp%2Cdata.dstuser%2CGeoLocation.country_name&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 451
2025-07-02 13:52:01,594    DEBUG           elastalert {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZibFFjZENId1RRMnE0cXM0MGJTcVJAAAAAAKQN0WUFlwUEMza0NTOFcyU2RJeG9hcWFCZxZfaFkzdVRXZlJGbU9ybWZrcTNUZW9nAAAAAAAKTX8WcUhzbkdYd3hTQlNDVUNzdmJpVU9adxZHeExOVVAwSFFZR18yaS1famhFanpnAAAAAAAKHHEWX3M2Ml9CUkxSbUtIRGpGeVNXRlFtdw==', 'took': 82, 'timed_out': False, '_shards': {'total': 549, 'successful': 549, 'skipped': 546, 'failed': 0}, 'hits': {'total': {'value': 0, 'relation': 'eq'}, 'max_score': None, 'hits': []}}
2025-07-02 13:52:01,594     INFO           elastalert Queried rule ElastAlert   Rule from 2025-07-02 13:37 UTC to 2025-07-02 13:52 UTC: 0 / 0 hits
2025-07-02 13:52:01,599    DEBUG urllib3.connectionpool https://opensearch:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2025-07-02 13:52:01,608    DEBUG urllib3.connectionpool https://opensearch:9200 "POST /elastalert_status/_doc HTTP/1.1" 201 171
2025-07-02 13:52:01,609     INFO           elastalert Ran ElastAlert   Rule from 2025-07-02 13:37 UTC to 2025-07-02 13:52 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
2025-07-02 13:52:01,609     INFO           elastalert ElastAlert   Rule range 900
2025-07-02 13:52:01,609     INFO apscheduler.executors.default Job "Rule: ElastAlert   Rule (trigger: interval[0:01:00], next run at: 2025-07-02 13:53:05 UTC)" executed successfully
```

ptobis Sep 2, 2025

@assimdarwish : do you have any update on this issue? I noticed a similar behavior on my instance. BTW, I'm using Elastic v.8.18.4, if that matters. However, this may be a problem with my elastalert_status_silence index mapping.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Already sent alert is sent again after restart #1683

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 8 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Already sent alert is sent again after restart #1683

Uh oh!

Uh oh!

assimdarwish Jul 2, 2025

Replies: 1 comment · 8 replies

Uh oh!

jertel Jul 2, 2025 Maintainer

Uh oh!

jertel Jul 2, 2025 Maintainer

Uh oh!

assimdarwish Jul 2, 2025 Author

Uh oh!

jertel Jul 2, 2025 Maintainer

Uh oh!

assimdarwish Jul 2, 2025 Author

Uh oh!

Uh oh!

ptobis Sep 2, 2025

assimdarwish
Jul 2, 2025

Replies: 1 comment 8 replies

jertel
Jul 2, 2025
Maintainer

jertel Jul 2, 2025
Maintainer

assimdarwish Jul 2, 2025
Author

jertel Jul 2, 2025
Maintainer

assimdarwish Jul 2, 2025
Author