Skip to content

Support trigger config reloading via HTTP request #13178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

chenlujjj
Copy link

@chenlujjj chenlujjj commented Jun 8, 2025

Description

Add a /-/reload HTTP API to reload config

Link to tracking issue

Fixes #10264

Testing

Manaul test:

  1. start an otel-collector locally
./bin/otelcorecol_darwin_amd64 --config ./examples/local/otel-config.yaml
  1. send a HTTP request
curl -XPOST  localhost:8888/-/reload
  1. observe the logs
2025-06-08T11:11:14.274+0800    info    [email protected]/service.go:282 Everything is ready. Begin running and processing data.       {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.315+0800    info    [email protected]/collector.go:277       Received reload config request  {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.315+0800    warn    [email protected]/collector.go:285       Config updated, restart service {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.315+0800    info    [email protected]/service.go:324 Starting shutdown...    {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.315+0800    info    extensions/extensions.go:69     Stopping extensions...  {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.315+0800    info    [email protected]/zpagesextension.go:112 Unregistered zPages span processor on tracer provider {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "zpages", "otelcol.component.kind": "extension"}
2025-06-08T11:11:29.315+0800    info    [email protected]/service.go:338 Shutdown complete.      {"resource": {"service.instance.id": "7e055a57-4ac5-4baa-9cb6-0783de26392c", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.318+0800    info    [email protected]/service.go:199 Setting up own telemetry...     {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.319+0800    info    builders/builders.go:26 Development component. May change in the future.     {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-06-08T11:11:29.319+0800    info    [email protected]/memorylimiter.go:74      Memory limiter configured    {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.kind": "processor", "limit_mib": 1536, "spike_limit_mib": 512, "check_interval": 5}
2025-06-08T11:11:29.319+0800    info    builders/builders.go:26 Development component. May change in the future.     {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-06-08T11:11:29.319+0800    info    builders/builders.go:26 Development component. May change in the future.     {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}
2025-06-08T11:11:29.320+0800    info    [email protected]/service.go:259 Starting otelcorecol... {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "Version": "0.127.0-dev", "NumCPU": 12}
2025-06-08T11:11:29.320+0800    info    extensions/extensions.go:41     Starting extensions...  {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}
2025-06-08T11:11:29.320+0800    info    extensions/extensions.go:45     Extension is starting...        {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "zpages", "otelcol.component.kind": "extension"}
2025-06-08T11:11:29.320+0800    info    [email protected]/zpagesextension.go:56  Registered zPages span processor on tracer provider   {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "zpages", "otelcol.component.kind": "extension"}
2025-06-08T11:11:29.320+0800    info    [email protected]/zpagesextension.go:71  Registered Host's zPages     {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "zpages", "otelcol.component.kind": "extension"}
2025-06-08T11:11:29.320+0800    info    [email protected]/zpagesextension.go:83  Starting zPages extension    {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "zpages", "otelcol.component.kind": "extension", "config": {"Endpoint":"localhost:55679","TLS":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Middlewares":null,"Expvar":{"Enabled":false}}}
2025-06-08T11:11:29.320+0800    info    extensions/extensions.go:62     Extension started.      {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "zpages", "otelcol.component.kind": "extension"}
2025-06-08T11:11:29.321+0800    info    [email protected]/otlp.go:117       Starting GRPC server    {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "localhost:4317"}
2025-06-08T11:11:29.321+0800    info    [email protected]/otlp.go:175       Starting HTTP server    {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "localhost:4318"}
2025-06-08T11:11:29.321+0800    info    [email protected]/service.go:282 Everything is ready. Begin running and processing data.       {"resource": {"service.instance.id": "1e9357f5-ddd6-418a-8e21-9e451c091261", "service.name": "otelcorecol", "service.version": "0.127.0-dev"}}

Documentation


This is just an initial implementation to demonstrate the purpose. I'm not sure if it should be implemented as a separate extension like zpages, healthcheck. Please correct me if it's not the right way.

@chenlujjj chenlujjj requested a review from a team as a code owner June 8, 2025 03:18
@chenlujjj chenlujjj requested a review from atoulme June 8, 2025 03:18
@chenlujjj
Copy link
Author

Hi @atoulme , Could you please have a look at this issue? Thank you.

Copy link
Member

@dmathieu dmathieu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this needs more discussion.
It would also need tests, and to be configurable.
I wonder if starting this as an extension wouldn't be better.

If we do move forward, this needs a changelog entry.

@@ -247,6 +256,31 @@ func (col *Collector) setupConfigurationComponents(ctx context.Context) error {
return nil
}

func (col *Collector) newReloadServer() (*http.Server, error) {
server := &http.Server{
Addr: ":8888",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be configurable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would also need to be authenticated. We can't expose an endpoint on all interfaces like this without some kind of security.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if authentication is a must-to-have thing here. Usually only maintainers of the otel-collector deployment can have access to this API.
Or in the 1st version, we can make this feature disabled by default, and users should enable it explicitly.

@@ -140,7 +143,13 @@ func NewCollector(set CollectorSettings) (*Collector, error) {
configProvider: configProvider,
bc: bc,
updateConfigProviderLogger: cc.SetCore,
}, nil
}
reloadServer, err := col.newReloadServer()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assigning a variable generated by the struct into the struct is a bit weird.
Why isn't the struct directly setting the value on itself?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can move the implementation of newReloadServer into the NewCollector function

Addr: ":8888",
}

http.HandleFunc("/-/reload", col.reloadHandler)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there a consensus somewhere on the endpoint's path? I'm only seeing one issue where you're the only commenter.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I follow the reload path of prometheus and alertmanager as mentioned in the issue.
I think we can make it configurable if needed.

@chenlujjj
Copy link
Author

@dmathieu Thanks for the review. The implementation of the PR is a bit incomplete, I planned to add tests and change log after the direction is aligned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support trigger config reloading via HTTP request besides SIGHUP
2 participants