Open
Description
Description of the problem including expected versus actual behavior
Logstash has node and pipeline level metric collection control signal. As of now (after #16864 fix), users will be able to disable pipeline level metric collection. However, when node level metric collection disable is impossible which leads endpoint failure.
Extra notes: not to confuse with api.enabled which disables the LS web server, it has broader context than metrics that loggin CRUD operations can be proceed.
Acceptance Criteria
- Fix the current bug;
- End users should have a meaningful guidance (reasoning and guide to enable back) when metric collection disabled and hit the endpoints
- Documentation to highlight metric collection enable/disable
Steps to reproduce:
- Use Logstash built from
mainor 8.17.0 - Set
metric.collect: falseinlogstash.yml - Run Logstash
- Hit the
localhost:9600/_node/statsendpoint
which results error in both LS console/log and endpoint result.
Provide logs
# Logstash log
[2025-01-09T12:51:02,844][ERROR][logstash.agent ] Internal API server error {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats/flow", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Instrument::MetricStore::MetricNotFound", :message=>"For path: jvm. Map keys: []", :backtrace=>["/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:249:in `block in get_recursively'", "org/jruby/RubyArray.java:1981:in `each'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:248:in `get_recursively'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:120:in `block in get'", "org/jruby/ext/thread/Mutex.java:174:in `synchronize'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:119:in `get'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:133:in `get_shallow'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:181:in `block in extract_metrics'", "org/jruby/RubyArray.java:1981:in `each'", "org/jruby/RubyEnumerable.java:1084:in `inject'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:157:in `extract_metrics'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/service.rb:45:in `extract_metrics'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/commands/base.rb:37:in `extract_metrics'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/commands/stats.rb:52:in `jvm'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:70:in `jvm_payload'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:34:in `block in NodeStats'", "org/jruby/RubyMethod.java:119:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1807:in `block in compile!'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1074:in `block in route!'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1092:in `route_eval'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1074:in `block in route!'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1123:in `block in process_route'", "org/jruby/RubyKernel.java:1426:in `catch'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1121:in `process_route'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1072:in `block in route!'", "org/jruby/RubyArray.java:1981:in `each'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1069:in `route!'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1193:in `block in dispatch!'", "org/jruby/RubyKernel.java:1426:in `catch'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1164:in `invoke'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1188:in `dispatch!'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1004:in `block in call!'", "org/jruby/RubyKernel.java:1426:in `catch'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1164:in `invoke'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1004:in `call!'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:993:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/base.rb:53:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/xss_header.rb:20:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/path_traversal.rb:18:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/json_csrf.rb:28:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/base.rb:53:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/base.rb:53:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/frame_options.rb:33:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/middleware/logger.rb:17:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/head.rb:15:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:227:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:2138:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/urlmap.rb:76:in `block in call'", "org/jruby/RubyArray.java:1981:in `each'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/urlmap.rb:60:in `call'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/rack_app.rb:75:in `call'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/rack_app.rb:49:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/builder.rb:277:in `call'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/request.rb:99:in `block in handle_request'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/thread_pool.rb:389:in `with_force_shutdown'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/request.rb:98:in `handle_request'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/server.rb:468:in `process_client'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/server.rb:249:in `block in run'", "/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/thread_pool.rb:166:in `block in spawn_thread'"]}
[2025-01-09T12:51:02,863][ERROR][logstash.agent ] API HTTP Request {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats/flow", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
# API call response
{
"status": 500,
"request_method": "GET",
"path_info": "/_node/stats",
"query_string": "",
"http_version": "HTTP/1.1",
"http_accept": "*/*",
"error": "Unexpected Internal Error",
"class": "LogStash::Instrument::MetricStore::MetricNotFound",
"message": "For path: jvm. Map keys: []",
"backtrace": [
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:249:in `block in get_recursively'",
"org/jruby/RubyArray.java:1981:in `each'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:248:in `get_recursively'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:120:in `block in get'",
"org/jruby/ext/thread/Mutex.java:174:in `synchronize'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:119:in `get'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:133:in `get_shallow'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:181:in `block in extract_metrics'",
"org/jruby/RubyArray.java:1981:in `each'",
"org/jruby/RubyEnumerable.java:1084:in `inject'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:157:in `extract_metrics'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/service.rb:45:in `extract_metrics'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/commands/base.rb:37:in `extract_metrics'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/commands/stats.rb:52:in `jvm'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:70:in `jvm_payload'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:34:in `block in NodeStats'",
"org/jruby/RubyMethod.java:119:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1807:in `block in compile!'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1074:in `block in route!'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1092:in `route_eval'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1074:in `block in route!'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1123:in `block in process_route'",
"org/jruby/RubyKernel.java:1426:in `catch'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1121:in `process_route'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1072:in `block in route!'",
"org/jruby/RubyArray.java:1981:in `each'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1069:in `route!'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1193:in `block in dispatch!'",
"org/jruby/RubyKernel.java:1426:in `catch'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1164:in `invoke'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1188:in `dispatch!'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1004:in `block in call!'",
"org/jruby/RubyKernel.java:1426:in `catch'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1164:in `invoke'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:1004:in `call!'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:993:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/base.rb:53:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/xss_header.rb:20:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/path_traversal.rb:18:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/json_csrf.rb:28:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/base.rb:53:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/base.rb:53:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-protection-4.1.1/lib/rack/protection/frame_options.rb:33:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/middleware/logger.rb:17:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/head.rb:15:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:227:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/sinatra-4.1.1/lib/sinatra/base.rb:2138:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/urlmap.rb:76:in `block in call'",
"org/jruby/RubyArray.java:1981:in `each'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/urlmap.rb:60:in `call'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/rack_app.rb:75:in `call'",
"/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/api/rack_app.rb:49:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/rack-3.1.8/lib/rack/builder.rb:277:in `call'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/request.rb:99:in `block in handle_request'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/thread_pool.rb:389:in `with_force_shutdown'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/request.rb:98:in `handle_request'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/server.rb:468:in `process_client'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/server.rb:249:in `block in run'",
"/Users/mashhur/Dev/elastic/logstash/vendor/bundle/jruby/3.1.0/gems/puma-6.5.0-java/lib/puma/thread_pool.rb:166:in `block in spawn_thread'"
]
}