Agent Interaction Tracing

[bibryam]

Context

Both agents and agentic workflows perform actions like LLM calls, tool executions, and MCP calls. Right now, there's no built-in tracing to understand what happened during these workflows. Adding tracing will help track every step an agent takes, using standard GenAI trace formats, so teams can debug and monitor agent behavior more easily.

Acceptance Criteria

• Generate traces for:
  • LLM calls
  • Tool calls
  • MCP calls
  • Calls to various storage, embeddings APIs
• Use OpenTelemetry GenAI semantic conventions(https://opentelemetry.io/docs/specs/semconv/gen-ai/) for trace attributes
• Support exporting traces using OpenTelemetry Collector

Resources

See similar traces here https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/e2cf380ffc8d47d28da98f136140642b?timestamp=2025-02-05T15%3A12%3A02.717Z&observation=be604a8a494bcd6e

[CasperGN]

@yaron2 this is sort of related to the functionality of cross tracing bigger setups and may be beneficially solved together.

Edit:
One use-case for us is both having LLMOrchestrator with multiple AssistantAgent's and individual Agent's in a combined ecosystem. Being able to trace calls across will greatly increase the visibility into what's happening under the hood which currently requires manual inspection of the redis store.

[CasperGN]

@yaron2, @Cyb3rWard0g what's your take on starting out with zero code auto-instrumentation while we build out more specific emitting of telemetry?

It'll give consumers an easy pluggable (albeit chatty..) setup until we have purpose built emitting.

[CasperGN]

@bibryam I've been working with our internal AI team and the marketplace and we run langfuse too as an option.
I have an OTel wrapper as part of #107 and with that I can test out how to use OTel overlay to send traces to langfuse and drop a quickstarter for that as well.

So far I've been able to just @observe() the main function where the agent gets instantiated and can see that trace represented in langfuse albeit with little to no value as no traces are emitted yet.

[Cyb3rWard0g]

@yaron2, @Cyb3rWard0g what's your take on starting out with zero code auto-instrumentation while we build out more specific emitting of telemetry?

It'll give consumers an easy pluggable (albeit chatty..) setup until we have purpose built emitting.

It would be amazing! thank you for taking a look at it and sharing an initial implementation, This would be game changer!

[Cyb3rWard0g]

@bibryam I've been working with our internal AI team and the marketplace and we run langfuse too as an option. I have an OTel wrapper as part of #107 and with that I can test out how to use OTel overlay to send traces to langfuse and drop a quickstarter for that as well.

So far I've been able to just @observe() the main function where the agent gets instantiated and can see that trace represented in langfuse albeit with little to no value as no traces are emitted yet.

That's awesome! Were you able to test the zero code auto-instrumentation? I can see your progress in PR 107 ⚡️

[CasperGN]

That's awesome! Were you able to test the zero code auto-instrumentation? I can see your progress in PR 107 ⚡️

Yeah, I didn't get much out of it and my initial assumption would be that there's not enough relevant libraries to do the monkey-patching (what the zero code auto-instrumentation does) with. I'll do more testing on this the coming days/week though!

[CasperGN]

@bibryam I've got a very early draft up in my fork.

You can access this through the following requirements.txt:

dapr_agents @ git+https://github.com/CasperGN/dapr-agents@5be92d9021228f7e9a1cb3ba7ff35cd1e9ee1a3a#egg=dapr-agents
python-dotenv==1.1.0
requests==2.32.3
mcp==1.7.1
chainlit
starlette==0.41.3
fastapi==0.115.12
uvicorn==0.34.2
langfuse==2.60.3
setuptools # you'll only need this if running in venv since python 3.12 don't ship with distutils in venv (for whatever odd reason that is...)
opentelemetry-distro
opentelemetry-exporter-otlp
opentelemetry-instrumentation-requests
opentelemetry-instrumentation-fastapi
opentelemetry-instrumentation-starlette
opentelemetry-instrumentation-asyncio

You'll need the following in your .env:

OTEL_EXPORTER_OTLP_ENDPOINT=otelcol.default.svc.cluster.local:4318

If you use the below collector:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otelcol
  namespace: default
spec:
  selector:
    matchLabels:
      app: otelcol
  replicas: 1
  template:
    metadata:
      annotations:
        dapr.io/enabled: "false"
      labels:
        app: otelcol
    spec:
      containers:
      - image: otel/opentelemetry-collector:latest
        command:
          - "/otelcol"
          - "--config=/conf/otel-agent-config.yaml"
        env:
        imagePullPolicy: Always
        name: otelcol
        ports:
        - containerPort: 4317
        volumeMounts:
        - name: otel-agent-config-vol
          mountPath: /conf
      volumes:
        - configMap:
            name: otel-agent-conf
            items:
              - key: otel-agent-config
                path: otel-agent-config.yaml
          name: otel-agent-config-vol
---
apiVersion: v1
kind: Service
metadata:
  name: otel-agent-conf
  namespace: default
spec:
  ports:
  - port: 4317
    name: otlp-grpc
    targetPort: 4317
    protocol: TCP
  - port: 4318
    name: otlp-http
    targetPort: 4318
    protocol: TCP
  selector:
    app: otelcol
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-conf
  labels:
    app: opentelemetry
    component: otel-agent-conf
data:
  otel-agent-config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      debug:
        verbosity: normal
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [debug]
        metrics:
          receivers: [otlp]
          exporters: [debug]
        logs:
          receivers: [otlp]
          exporters: [debug]

You can reuse the k8s example to get it running.

I'm seeing the follow in our internal setup (example span taken):

On the current TODO's in the above implementation:

• Ensure propagation of Cloud Event headers (CC @yaron2, here's an example to work from/with)
• Ensure semantic conventions for GenAI
• Ensure enrichment of spans (currently it's just a span with little to no content)
• Ensure all the relevant methods are decorated

---

The traces in langfuse are still not particularly helpful:

However it is possible to send stuff by decorating the agent instantiation:

from langfuse.decorators import observe

@observe()
async def main():
    try:
        workflow_service = LLMOrchestrator(
          ...