ACL token leakage: Consul agents fail to logout, leading to ~20k stale tokens and Raft performance impact #22613
Description
Overview of the Issue
A large number of ACL tokens accumulate due to Consul agents not performing a proper logout.
This results in increased Raft commit latency and resource consumption.
On a customer cluster, ~20,000 stale tokens had accumulated. After manual cleanup, Raft commit latency was reduced, but not significantly.
The problem originates from Consul itself rather than the application layer.
Reproduction Steps
- Run Consul agents integrated with ESO discovery.
- Perform repeated login/auth flows.
- Observe that tokens are not properly released/logged out.
- Over time, thousands of stale tokens accumulate.
Consul info for both Client and Server
There is a noticeable degradation in commit performance (1–4 seconds per commit), resulting in increased load on the Consul servers.
Operating system and Environment details
Consul version: 1.17.3
Kubernetes version: 1.31