Alerting when elasticsearch process is being killed.

[RpArt1]

Could you help me with elastalert2 setup ? I'm trying to configure elastalert2 in such a way that, if elastic cluster is down, I will be notified.
I thought setting up notify_email will do the job but unfortunately it doesn't work.
Error which is occurring during cluster downtime - which I thought would trigger alert :

elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPSConnectionPool(host='####', port=####): Max retries exceeded with url: /elastalert_status_error/_doc (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x732622c2d400>: Failed to establish a new connection: [Errno 111] Connection refused'))) caused by: ConnectionError(HTTPSConnectionPool(host='####', port=####): Max retries exceeded with url: /elastalert_status_error/_doc (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x732622c2d400>: Failed to establish a new connection: [Errno 111] Connection refused')))

config.yml:

rules_folder: custom_rules
run_every:
  minutes: 3
buffer_time:
  minutes: 15

start_time: "now"

  #### ELASTIC & SSL
es_host: #### 
es_hosts:  ####

es_port: ####
es_username: ####
es_password: ####
use_ssl: True
ca_certs: ####
verify_certs: True

writeback_index: elastalert_status
writeback_alias: elastalert_alerts
alert_time_limit:
  days: 2
es_conn_timeout: 60

## EMAIL SETTINGS

smtp_host: ####
smtp_port: ####
smtp_auth_file: "smtp_auth_file"
notify_email: ####


from_addr: ####
alert:
- "email"
email:   
- ####

flatline rule :

name: "Elasticsearch Process Missing Alert"
type: flatline

index: metricbeat-*
timestamp_field: "@timestamp"

threshold: 1

timeframe:
  minutes: 6

filter:
- query:
    query_string:
      query: "process.command_line : *Elasticsearch*"

query_key: host.hostname

doc_type: _doc

alert:
  - "email"

realert:
  minutes: 20

alert_subject_args:
- key

alert_subject: "[ELASTALERT]: Elasticsearch process doesn't exist on {0}"
alert_text: " |
  No elasticsearch process

  Hostname: {0}
  
  Please take appropriate action.

  - ElastAlert2"

[jertel]

This is a known issue with the notify mechanism. It currently only alerts about rule-specific/alert-specific problems, not about system level outages. I have some other changes I'm planning to make in this area so I'll see what I can do for a future release.

[jertel]

#1546