Python auto-instrumentation is not working

[haihh05]

I cannot see any traces with my simple lab.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://localhost:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://localhost:4318

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: sidecar
spec:
  mode: sidecar
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      logging:
        # verbosity: detailed
      prometheus:
        endpoint: 0.0.0.0:1234
    service:
      telemetry:
        logs:
          level: debug
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheus]
        logs:
          receivers: [otlp]
          processors: []
          exporters: [logging]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment-with-sidecar
spec:
  selector:
    matchLabels:
      app: my-pod-with-sidecar
  replicas: 1
  template:
    metadata:
      labels:
        app: my-pod-with-sidecar
      annotations:
        sidecar.opentelemetry.io/inject: "true"
        instrumentation.opentelemetry.io/inject-python: "true"
    spec:
      containers:
      - name: myapp
        image: python:3.11
        command: [ "/bin/bash", "-c"]
        args:
          - python -m http.server 8080
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

OTEL logs

2023-09-07T09:51:54.866Z	info	service/telemetry.go:84	Setting up own telemetry...
2023-09-07T09:51:54.867Z	info	service/telemetry.go:201	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2023-09-07T09:51:54.868Z	debug	exporter@v0.83.0/exporter.go:273	Beta component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-09-07T09:51:54.868Z	info	exporter@v0.83.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-09-07T09:51:54.868Z	debug	receiver@v0.83.0/receiver.go:294	Beta component. May change in the future.	{"kind": "receiver", "name": "otlp", "data_type": "logs"}
2023-09-07T09:51:54.868Z	info	exporter@v0.83.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2023-09-07T09:51:54.868Z	debug	receiver@v0.83.0/receiver.go:294	Stable component.	{"kind": "receiver", "name": "otlp", "data_type": "traces"}
2023-09-07T09:51:54.868Z	debug	receiver@v0.83.0/receiver.go:294	Stable component.	{"kind": "receiver", "name": "otlp", "data_type": "metrics"}
2023-09-07T09:51:54.869Z	info	service/service.go:138	Starting otelcol-contrib...	{"Version": "0.83.0", "NumCPU": 2}
2023-09-07T09:51:54.869Z	info	extensions/extensions.go:31	Starting extensions...
2023-09-07T09:51:54.869Z	warn	internal@v0.83.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-09-07T09:51:54.869Z	warn	internal@v0.83.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "receiver", "name": "otlp", "data_type": "logs", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-09-07T09:51:54.870Z	info	zapgrpc/zapgrpc.go:178	[core] [Server #1] Server created	{"grpc_log": true}
2023-09-07T09:51:54.870Z	info	otlpreceiver@v0.83.0/otlp.go:83	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4317"}
2023-09-07T09:51:54.870Z	info	zapgrpc/zapgrpc.go:178	[core] [Server #1 ListenSocket #2] ListenSocket created	{"grpc_log": true}
2023-09-07T09:51:54.870Z	warn	internal@v0.83.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "receiver", "name": "otlp", "data_type": "logs", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-09-07T09:51:54.870Z	info	otlpreceiver@v0.83.0/otlp.go:101	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4318"}
2023-09-07T09:51:54.870Z	info	service/service.go:161	Everything is ready. Begin running and processing data.
2023-09-07T09:52:55.427Z	debug	prometheusexporter@v0.83.0/accumulator.go:79	accumulating metric: system.cpu.time	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-09-07T09:52:55.428Z	debug	prometheusexporter@v0.83.0/accumulator.go:79	accumulating metric: system.cpu.utilization	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-09-07T09:52:55.429Z	debug	prometheusexporter@v0.83.0/accumulator.go:79	accumulating metric: system.memory.usage	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-09-07T09:52:55.429Z	debug	prometheusexporter@v0.83.0/accumulator.go:79	accumulating metric: system.memory.utilization	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
2023-09-07T09:52:55.429Z	debug	prometheusexporter@v0.83.0/accumulator.go:79	accumulating metric: system.swap.usage	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}

App Logs

127.0.0.1 - - [07/Sep/2023 09:52:29] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [07/Sep/2023 09:52:30] code 404, message File not found
127.0.0.1 - - [07/Sep/2023 09:52:30] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [07/Sep/2023 09:52:30] "GET /run/ HTTP/1.1" 200 -
127.0.0.1 - - [07/Sep/2023 09:52:46] "GET /run/lock/ HTTP/1.1" 200 -
127.0.0.1 - - [07/Sep/2023 09:52:49] "GET /sys/ HTTP/1.1" 200 -
127.0.0.1 - - [07/Sep/2023 09:53:17] "GET /sys/block/ HTTP/1.1" 200 -
127.0.0.1 - - [07/Sep/2023 09:53:20] "GET /otel-auto-instrumentation/ HTTP/1.1" 200 -

Auto-instrumentation Logs (nothing)

[TylerHelmuth]

@haihh05 can you see that the operator added the autoinstrumentation initContainer and the collector sidecare to your pods?

[haihh05]

Yes, of course, I can see logs of them as above.

[ad01bl]

I'm having very similiar configuration, but the python application is based on fastapi. And from auto-instrumentation I'm only receiving metrics. No traces are recorded by opentelemetry. Have you maybe found any solution for that?

[frzifus]

Does the auto-instr work without kubernetes?

[azunna1]

Same issue i'm facing for flask apps