Skip to content

[test-optimization] [SDTEST-1821] Implement impacted tests detection #5543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 17 commits into
base: master
Choose a base branch
from

Conversation

Mariovido
Copy link
Contributor

@Mariovido Mariovido commented Apr 7, 2025

What does this PR do?

Implements the Impacted Tests Detection feature for CI Visibility (milestones 1.X):

  • Identifies test cases that have been modified or impacted, i.e., any test with changes to its code lines.
  • Marks impacted tests with the test.is_modified:true tag.
  • Enables Early Flake Detection for tests flagged as impacted.

Frameworks that get support with this PR are:

  • Jest
  • Mocha
  • Cucumber
  • Cypress

Frameworks that not get supported with this PR and why:

  • Playwright → We cannot get the original fn, the one that we can retrieve is formatted. Only will work with API.
  • Vitest → We cannot get the start line of the method, neither the fn isolated. Only will work with API.

Motivation

We want to give support in the Node framework to this new feature.

Plugin Checklist

Additional Notes

Copy link

github-actions bot commented Apr 7, 2025

Overall package size

Self size: 9.4 MB
Deduped: 102.64 MB
No deduping: 103.16 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.5.1 | 29.73 MB | 29.73 MB | | @datadog/native-appsec | 8.5.2 | 19.33 MB | 19.34 MB | | @datadog/native-iast-taint-tracking | 3.3.1 | 13.99 MB | 13.99 MB | | @datadog/pprof | 5.7.1 | 9.51 MB | 9.88 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.4.0 | 2.77 MB | 5.42 MB | | @datadog/wasm-js-rewriter | 4.0.0 | 2.85 MB | 3.58 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.13.1 | 117.64 kB | 839.26 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | dc-polyfill | 0.1.8 | 25.08 kB | 25.08 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.2 | 23.54 kB | 23.54 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

Copy link

codecov bot commented Apr 7, 2025

Codecov Report

Attention: Patch coverage is 35.63218% with 112 lines in your changes missing coverage. Please review.

Project coverage is 79.05%. Comparing base (38f62e9) to head (ce58841).
Report is 13 commits behind head on master.

Files with missing lines Patch % Lines
packages/datadog-instrumentations/src/cucumber.js 0.00% 49 Missing ⚠️
...ackages/datadog-instrumentations/src/mocha/main.js 0.00% 21 Missing ⚠️
...ckages/datadog-instrumentations/src/mocha/utils.js 0.00% 11 Missing ⚠️
packages/datadog-plugin-mocha/src/index.js 0.00% 10 Missing ⚠️
packages/dd-trace/src/plugins/util/test.js 78.26% 10 Missing ⚠️
packages/datadog-plugin-cucumber/src/index.js 0.00% 5 Missing ⚠️
...ci-visibility/impacted-tests/get-modified-tests.js 80.95% 4 Missing ⚠️
packages/dd-trace/src/plugins/util/git.js 50.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5543      +/-   ##
==========================================
- Coverage   79.38%   79.05%   -0.34%     
==========================================
  Files         516      515       -1     
  Lines       23403    23578     +175     
==========================================
+ Hits        18579    18640      +61     
- Misses       4824     4938     +114     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Apr 7, 2025

Datadog Report

Branch report: mario.vidal/impacted_tests
Commit report: 5a2082d
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 1009 Passed, 0 Skipped, 20m 29.98s Total Time

@Mariovido Mariovido changed the title [test-optimization] [SDTEST-1821] Implement impacted tests detection [test-optimization] [SDTEST-1821] Implement impacted tests detection in Jest Apr 8, 2025
@pr-commenter
Copy link

pr-commenter bot commented Apr 8, 2025

Benchmarks

Benchmark execution time: 2025-04-15 16:27:35

Comparing candidate commit ce58841 in PR branch mario.vidal/impacted_tests with baseline commit 38f62e9 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 947 metrics, 16 unstable metrics.

@Mariovido Mariovido marked this pull request as ready for review April 8, 2025 08:40
@Mariovido Mariovido requested review from a team as code owners April 8, 2025 08:40
daniel-mohedano
daniel-mohedano previously approved these changes Apr 8, 2025
@Mariovido Mariovido force-pushed the mario.vidal/impacted_tests branch from 84ee004 to d471892 Compare April 14, 2025 08:29
@Mariovido Mariovido force-pushed the mario.vidal/impacted_tests branch from d471892 to f1bb7b8 Compare April 14, 2025 08:30
@Mariovido Mariovido marked this pull request as draft April 14, 2025 09:24
@Mariovido Mariovido changed the title [test-optimization] [SDTEST-1821] Implement impacted tests detection in Jest [test-optimization] [SDTEST-1821] Implement impacted tests detection Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants