Database metrics only appear after deployment

[jairhenrique]

Component(s)

auto-instrumentation, collector

What happened?

Description

Hi 👋

We are experiencing an issue with our application deployed on Kubernetes. The application utilizes FastAPI, Redis, and Elasticsearch. We have automatic OTEL instrumentation configured via annotations in the deployment.

While the FastAPI transaction/throughput metrics are successfully reaching our data visualization tool, the Redis and Elasticsearch database metrics are not being displayed.

Interestingly, when we perform a new deployment of the application, the historical metrics for Redis and Elasticsearch appear briefly, but then stop being collected.

We have not identified any error messages in the application logs.
We can see the Redis and Elasticsearch metrics in the logs when we set OTEL_METRICS_EXPORTER and OTEL_TRACES_EXPORTER to console.

Kubernetes Version

1.30

Operator version

0.119.0

Collector version

0.120.0

Environment information

Environment

python: 3:13:1
autoinstrumentation-python: 0.51b0
fastapi: 0.115.12

Log output

Additional context

No response

[iblancasa]

1. Can you share the Instrumentation manifest?
2. Can you double check if the issue happens without the operator?

[ramosbianca]

Hello @iblancasa,
Currently our default is to use the OpenTelemetry Operator (As Daemonset) as it auto-injects the instrumentation according to the application language and controls the OpenTelemetry Collector (As deployment). The test we can do is perform manual instrumentation, but it is not our reality to do this.
I can't share the manifesto because it has internal company endpoints.
But OTEL_EXPORTER_OTLP_ENDPOINT sends metrics to the internal endpoint and OTEL_LOGS_EXPORTER has a value of none, propagators (tracecontext and baggage).

Have you encountered any cases similar to this where metrics disappear after a new deployment?

Is there any other test you recommend we do?

[iblancasa]

Hello @iblancasa, Currently our default is to use the OpenTelemetry Operator (As Daemonset) as it auto-injects the instrumentation according to the application language and controls the OpenTelemetry Collector (As deployment). The test we can do is perform manual instrumentation, but it is not our reality to do this.

I know. I'm not asking you to switch to perform manual instrumentation but to test if injecting the instrumentation without the operator (installing the libraries directly) works. That will help us to know if the issue is in the instrumentation libraries or ir the operator itself.

I can't share the manifesto because it has internal company endpoints. But OTEL_EXPORTER_OTLP_ENDPOINT sends metrics to the internal endpoint and OTEL_LOGS_EXPORTER has a value of none, propagators (tracecontext and baggage).

Can you share a reproducer?

[ramosbianca]

I tested it by changing the otlp endpoint so that it sent directly to new-relic, the behavior was the same. When I restart the deployment, it returns all the metrics about the database. After 2 to 3 minutes, these metrics go to zero. When I restart it again, the behavior is as if it had never stopped collecting.

[ramosbianca]

It's as if there was a loss of shipping and it only returned to normal when I restarted the entire deployment.

[pavolloffay]

Could you please paste here pod yaml of the app that has the auto-instrumentation injected and also logs of that app?

It seems that the injection works, however, the instrumentation or exporter fails (probably) after some time.

The operator is responsible only for injecting the auto-instrumentation therefore this seems like an issue with the python auto-instrumentation and the bug should be reported there https://github.com/open-telemetry/opentelemetry-python-contrib

[iblancasa]

I tested it by changing the otlp endpoint so that it sent directly to new-relic,...

Did you test without using the operator?

[jairhenrique]

@iblancasa by removing auto-injection, installing the instrumentation and export dependencies, and calling the opentelemetry-instrument command with our application server, the database instrumentation data started to appear.

[iblancasa]

Ok, in that case it seems to be a problem in the operator. Wanted to double check if there was any issue with the instrumentation.

It is interesting because of this:

Interestingly, when we perform a new deployment of the application, the historical metrics for Redis and Elasticsearch appear briefly, but then stop being collected.

@iblancasa by removing auto-injection, installing the instrumentation and export dependencies, and calling the opentelemetry-instrument command with our application server, the database instrumentation data started to appear.

What libraries did you install during this step? I want to double check that the operator is not missing any dependencies or something that can be important.

[jairhenrique]

What libraries did you install during this step? I want to double check that the operator is not missing any dependencies or something that can be important.

@iblancasa this is my project dependencies pyproject.toml

dependencies = [
  "configcat-client==9.0.4",
  "cryptography==44.0.3",
  "dumb-init==1.2.5.post1",
  "elasticsearch-dsl==8.18.0",
  "fakeredis==2.28.1",
  "fastapi==0.115.12",
  "google-cloud-pubsub==2.29.0",
  "granian==2.2.5",
  "gunicorn==23.0.0",
  "hiredis==3.1.0",
  "httpx==0.28.1",
  "newrelic==10.11.0",
  "orjson==3.10.18",
  "pydantic-settings==2.9.1",
  "pydantic==2.11.4",
  "pyjwt==2.10.1",
  "redis==5.2.1",
  "structlog==25.3.0",
  "tzdata==2025.2",
  "uvicorn-worker==0.3.0",
  "uvicorn==0.34.2",
  "uvloop==0.21.0",
  "zstandard==0.23.0",
  "opentelemetry-instrumentation-redis>=0.53b1",
  "opentelemetry-instrumentation-fastapi>=0.53b1",
  "opentelemetry-instrumentation-threading>=0.53b1",
  "opentelemetry-instrumentation-urllib3>=0.53b1",
  "opentelemetry-instrumentation-system-metrics>=0.53b1",
  "opentelemetry-instrumentation-requests>=0.53b1",
  "opentelemetry-instrumentation-httpx>=0.53b1",
  "opentelemetry-api>=1.32.1",
  "opentelemetry-sdk>=1.32.1",
  "opentelemetry-distro>=0.53b1",
  "opentelemetry-exporter-otlp-proto-grpc>=1.32.1",
  "opentelemetry-instrumentation-elasticsearch>=0.53b1",
]

uv pip freeze | grep open                                                                                                                                                         

opentelemetry-api==1.32.1
opentelemetry-distro==0.53b1
opentelemetry-exporter-otlp-proto-common==1.32.1
opentelemetry-exporter-otlp-proto-grpc==1.32.1
opentelemetry-instrumentation==0.53b1
opentelemetry-instrumentation-asgi==0.53b1
opentelemetry-instrumentation-elasticsearch==0.53b1
opentelemetry-instrumentation-fastapi==0.53b1
opentelemetry-instrumentation-httpx==0.53b1
opentelemetry-instrumentation-redis==0.53b1
opentelemetry-instrumentation-requests==0.53b1
opentelemetry-instrumentation-system-metrics==0.53b1
opentelemetry-instrumentation-threading==0.53b1
opentelemetry-instrumentation-urllib3==0.53b1
opentelemetry-proto==1.32.1
opentelemetry-sdk==1.32.1
opentelemetry-semantic-conventions==0.53b1
opentelemetry-util-http==0.53b1