401 unauthorized

[GitUseDeveloper]

ISSUE:

Sep 01 11:26:45 bash[2040371]: WARNING:elasticsearch:DELETE https://:9200/_search/scroll [status:401 request:0.017s] Sep 01 11:26:45 bash[2040371]: WARNING:elasticsearch:Undecodable raw error response from server: Expecting value: line 1 column 1 (char 0) Sep 01 11:26:45 bash[2040371]: ERROR:elastalert:Traceback (most recent call last): Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elastalert/elastalert.py", line 1283, in handle_rule_execution Sep 01 11:26:45 bash[2040371]: num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime')) Sep 01 11:26:45 bash[2040371]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elastalert/elastalert.py", line 915, in run_rule Sep 01 11:26:45 bash[2040371]: if not self.run_query(rule, rule['starttime'], endtime): Sep 01 11:26:45 bash[2040371]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elastalert/elastalert.py", line 671, in run_query Sep 01 11:26:45 bash[2040371]: self.thread_data.current_es.clear_scroll(scroll_id=scroll_id) Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped Sep 01 11:26:45 bash[2040371]: return func(*args, params=params, headers=headers, **kwargs) Sep 01 11:26:45 bash[2040371]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elasticsearch/client/init.py", line 481, in clear_scroll Sep 01 11:26:45 bash[2040371]: return self.transport.perform_request( Sep 01 11:26:45 bash[2040371]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elasticsearch/transport.py", line 392, in perform_request Sep 01 11:26:45 bash[2040371]: raise e Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elasticsearch/transport.py", line 358, in perform_request Sep 01 11:26:45 bash[2040371]: status, headers_response, data = connection.perform_request( Sep 01 11:26:45 bash[2040371]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elasticsearch/connection/http_requests.py", line 199, in perform_request Sep 01 11:26:45 bash[2040371]: self._raise_error(response.status_code, raw_data) Sep 01 11:26:45 bash[2040371]: File "lib/python3.12/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error Sep 01 11:26:45 bash[2040371]: raise HTTP_EXCEPTIONS.get(status_code, TransportError)( Sep 01 11:26:45 bash[2040371]: elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, 'Unauthorized') Sep 01 11:26:45 bash[2040371]: ERROR:elastalert:Uncaught exception running rule Multiple Success Logins for User : AuthenticationException(401, 'Unauthorized')

Additional details:
OpenSearch: 2.15.0
System: Ubuntu
Implementation: Systemd service
ElastAlert2 version: 2.23.0

This issue occurred only once, the cluster is running smoothly, no configuration changes and no creds changed. I also verified the audit logs, there is only one failed_login for this specific elastalert user. I don't understand why this error occurred. Great if anyone answer quickly.

[jertel]

You'll need to check the OpenSearch logs to see what reason it had for rejecting the credentials.

[GitUseDeveloper]

Hi, Jertel. Ok I will check.

[GitUseDeveloper]

I have checked the OpenSearch side logs at the time the error occurred, the log says the following

Error: Authentication finally failed for "xxxuseryyy" from {elastalert instance:some number}.

It's basically tells like the user is not from LDAP, because the user created internally. But it didn't fallback to internal authentication once failed for LDAP auth. Failed only once, unable to reproduce, no network issues at that time. The error did not came back in ElastAlert2. That's my worry. Can you help me with the issue @jertel .

[jertel]

Since this appears to be an OpenSearch authentication glitch, it would make more sense to contact an OpenSearch discussion forum for help.

[GitUseDeveloper]

Ok Jertel.