feat(llmobs): support manual instrumentation of prompts #7257

sabrenner · 2026-01-15T18:47:01Z

What does this PR do?

Adds support for manually instrumenting prompts from the LLM Observability SDK. Additionally, uses this new logic in the tagger in the existing OpenAI auto-instrumentation, which previously annotated prompts manually.

Motivation

Feature parity with the Python SDK.

MLOB-5073

Testing

Ran these agains recently-merged system tests locally:

dd-trace-js git:(sabrenner/llmobs-prompts-support) cd ../system-tests 
system-tests git:(sabrenner/llmobs-prompts) ./run.sh PARAMETRIC -L nodejs -vv tests/parametric/test_llm_observability.py::Test_Prompts
Build framework test container...
Build complete
==================================================================== test context =====================================================================
Scenario: PARAMETRIC
Logs folder: ./logs_parametric
Library: [email protected]
================================================================= test session starts =================================================================
[gw0] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw1] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw2] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw3] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw4] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw5] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw6] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw7] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw8] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw9] darwin Python 3.12.7 cwd: /Users/sam.brenner/dd/system-tests           
[gw3] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw2] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw0] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw6] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw5] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw8] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw9] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]
[gw1] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)] 
[gw4] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]   
[gw7] Python 3.12.7 (main, Oct 21 2024, 09:45:23) [Clang 15.0.0 (clang-1500.3.9.4)]     
gw0 [8] / gw1 [8] / gw2 [8] / gw3 [8] / gw4 [8] / gw5 [8] / gw6 [8] / gw7 [8] / gw8 [8] / gw9 [8]
scheduling tests via LoadScheduling

tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_default_id 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_supports_hallucinations 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_with_string_template 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_supports_tags 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_with_non_llm_span_does_not_annotate 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_in_annotation_context 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation 
tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_updates_existing_prompt 
[gw8] [ 12%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_in_annotation_context 
[gw9] [ 25%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_default_id 
[gw1] [ 37%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_updates_existing_prompt 
[gw5] [ 50%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_supports_hallucinations 
[gw3] [ 62%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation 
[gw6] [ 75%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_supports_tags 
[gw0] [ 87%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_with_string_template 
[gw2] [100%] XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_with_non_llm_span_does_not_annotate 

------------------------------- generated xml file: /Users/sam.brenner/dd/system-tests/logs_parametric/reportJunit.xml --------------------------------
=============================================================== short test summary info ===============================================================
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_in_annotation_context missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_default_id missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_updates_existing_prompt missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_supports_hallucinations missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_supports_tags missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_with_string_template missing_feature
XPASS tests/parametric/test_llm_observability.py::Test_Prompts::test_prompt_annotation_with_non_llm_span_does_not_annotate missing_feature
================================================================= 8 xpassed in 19.23s =================================================================

…r/llmobs-prompts-support

github-actions · 2026-01-15T18:48:26Z

Overall package size

Self size: 4.4 MB
Deduped: 5.23 MB
No deduping: 5.23 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 2.0.0 | 68.46 kB | 797.03 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2026-01-15T18:48:49Z

Codecov Report

❌ Patch coverage is 57.30337% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.08%. Comparing base (39c85a4) to head (63c3724).

Files with missing lines	Patch %	Lines
packages/dd-trace/src/llmobs/tagger.js	55.00%	36 Missing ⚠️
packages/dd-trace/src/llmobs/sdk.js	66.66%	1 Missing ⚠️
packages/dd-trace/src/llmobs/span_processor.js	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7257      +/-   ##
==========================================
- Coverage   85.19%   85.08%   -0.12%     
==========================================
  Files         532      532              
  Lines       22778    22863      +85     
==========================================
+ Hits        19405    19452      +47     
- Misses       3373     3411      +38

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2026-01-15T18:55:31Z

Benchmarks

Benchmark execution time: 2026-01-16 18:49:08

Comparing candidate commit 63c3724 in PR branch sabrenner/llmobs-prompts-support with baseline commit 39c85a4 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 230 metrics, 30 unstable metrics.

sabrenner added 9 commits November 7, 2025 13:22

prompt support for hallucinations + basic tests

967f2d8

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

2050664

…r/llmobs-prompts-support

update typedocs

97dc716

add more tests

1dcca26

updates from shared tests

aa2171e

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

25119d9

…r/llmobs-prompts-support

fmt

975ef69

fmt

710bba8

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

8bead9d

…r/llmobs-prompts-support

sabrenner added the semver-minor label Jan 15, 2026

Merge branch 'master' into sabrenner/llmobs-prompts-support

a5aa551

This comment has been minimized.

Sign in to view

sabrenner and others added 3 commits January 16, 2026 09:55

change template annotation

56e955d

update tests

00cac35

Merge branch 'master' into sabrenner/llmobs-prompts-support

63c3724

sabrenner marked this pull request as ready for review January 16, 2026 18:54

sabrenner requested review from a team as code owners January 16, 2026 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(llmobs): support manual instrumentation of prompts #7257

feat(llmobs): support manual instrumentation of prompts #7257

Uh oh!

sabrenner commented Jan 15, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 15, 2026 •

edited

Loading

Uh oh!

codecov bot commented Jan 15, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

pr-commenter bot commented Jan 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(llmobs): support manual instrumentation of prompts #7257

Are you sure you want to change the base?

feat(llmobs): support manual instrumentation of prompts #7257

Uh oh!

Conversation

sabrenner commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Testing

Uh oh!

github-actions bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

codecov bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment has been minimized.

pr-commenter bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sabrenner commented Jan 15, 2026 •

edited

Loading

github-actions bot commented Jan 15, 2026 •

edited

Loading

codecov bot commented Jan 15, 2026 •

edited

Loading

pr-commenter bot commented Jan 15, 2026 •

edited

Loading