Skip to content

Setup auto update system-tests #8922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Setup auto update system-tests #8922

wants to merge 1 commit into from

Conversation

TonyCTHsu
Copy link
Contributor

@TonyCTHsu TonyCTHsu commented Jun 4, 2025 •
edited
Loading

Motivation

Test with HEAD of main from system tests is inconsistent.

What Does This Do

Setup automation to upgrade the version with commit SHA periodically. It would create an automated pull requests to be review and merged.

The script is validated with

$ export TARGET=".github/workflows/run-system-tests.yaml"
$ export PATTERN='(\s*system-tests.yml@)(\S+)(\s+# Automated:.*)'
$ export REF=007f43b50f720be299f0dae27f4fcbabdd623591
$ .github/scripts/update_reference.sh

Target: .github/workflows/run-system-tests.yaml
Ref: 007f43b50f720be299f0dae27f4fcbabdd623591
Pattern: (\s*system-tests.yml@)(\S+)(\s+# Automated:.*)
✓ Updated references in .github/workflows/run-system-tests.yaml

$ git diff
diff --git a/.github/workflows/run-system-tests.yaml b/.github/workflows/run-system-tests.yaml
index bb320a46dd..feceb83dbb 100644
--- a/.github/workflows/run-system-tests.yaml
+++ b/.github/workflows/run-system-tests.yaml
@@ -50,7 +50,7 @@ jobs:
   main:
     needs:
     - build
-    uses: DataDog/system-tests/.github/workflows/system-tests.yml@main # Automated: This reference is automatically updated.
+    uses: DataDog/system-tests/.github/workflows/system-tests.yml@007f43b50f720be299f0dae27f4fcbabdd623591 # Automated: This reference is automatically updated.
     secrets: inherit
     permissions:
       contents: read

Additional Notes

The workflow requires to allow peter-evans/create-pull-request@* in the repository settings.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@pr-commenter
Copy link

pr-commenter bot commented Jun 4, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master tonycthsu/pin-system-tests
git_commit_date 1749044153 1749130405
git_commit_sha 7787af7 3532e4a
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~3532e4a780
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749132770 1749132770
ci_job_id 969867938 969867938
ci_pipeline_id 67058912 67058912
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-guhm2hol-project-304-concurrent-0-jmg2oh10 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-guhm2hol-project-304-concurrent-0-jmg2oh10 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 55 metrics, 16 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.02 s) : 0, 1020332
Total [baseline] (11.076 s) : 0, 11075824
Agent [candidate] (1.03 s) : 0, 1030083
Total [candidate] (11.135 s) : 0, 11134909
section appsec
Agent [baseline] (1.164 s) : 0, 1163808
Total [baseline] (11.217 s) : 0, 11216631
Agent [candidate] (1.165 s) : 0, 1165481
Total [candidate] (11.213 s) : 0, 11213490
section iast
Agent [baseline] (1.158 s) : 0, 1157643
Total [baseline] (11.397 s) : 0, 11396515
Agent [candidate] (1.151 s) : 0, 1150698
Total [candidate] (11.336 s) : 0, 11336421
section profiling
Agent [baseline] (1.279 s) : 0, 1278920
Total [baseline] (11.46 s) : 0, 11459541
Agent [candidate] (1.267 s) : 0, 1266978
Total [candidate] (11.523 s) : 0, 11522609
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.02 s -
Agent appsec 1.164 s 143.477 ms (14.1%)
Agent iast 1.158 s 137.311 ms (13.5%)
Agent profiling 1.279 s 258.588 ms (25.3%)
Total tracing 11.076 s -
Total appsec 11.217 s 140.806 ms (1.3%)
Total iast 11.397 s 320.691 ms (2.9%)
Total profiling 11.46 s 383.717 ms (3.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.03 s -
Agent appsec 1.165 s 135.398 ms (13.1%)
Agent iast 1.151 s 120.615 ms (11.7%)
Agent profiling 1.267 s 236.894 ms (23.0%)
Total tracing 11.135 s -
Total appsec 11.213 s 78.581 ms (0.7%)
Total iast 11.336 s 201.512 ms (1.8%)
Total profiling 11.523 s 387.699 ms (3.5%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.158 ms) : 0, 682158
BytebuddyAgent [candidate] (687.557 ms) : 0, 687557
GlobalTracer [baseline] (239.879 ms) : 0, 239879
GlobalTracer [candidate] (241.756 ms) : 0, 241756
AppSec [baseline] (57.179 ms) : 0, 57179
AppSec [candidate] (57.986 ms) : 0, 57986
Debugger [baseline] (6.204 ms) : 0, 6204
Debugger [candidate] (6.228 ms) : 0, 6228
Remote Config [baseline] (724.207 µs) : 0, 724
Remote Config [candidate] (744.993 µs) : 0, 745
Telemetry [baseline] (10.546 ms) : 0, 10546
Telemetry [candidate] (12.076 ms) : 0, 12076
section appsec
BytebuddyAgent [baseline] (701.08 ms) : 0, 701080
BytebuddyAgent [candidate] (702.56 ms) : 0, 702560
GlobalTracer [baseline] (238.222 ms) : 0, 238222
GlobalTracer [candidate] (238.226 ms) : 0, 238226
IAST [baseline] (21.664 ms) : 0, 21664
IAST [candidate] (21.955 ms) : 0, 21955
AppSec [baseline] (176.241 ms) : 0, 176241
AppSec [candidate] (176.001 ms) : 0, 176001
Debugger [baseline] (5.915 ms) : 0, 5915
Debugger [candidate] (6.009 ms) : 0, 6009
Remote Config [baseline] (618.249 µs) : 0, 618
Remote Config [candidate] (636.584 µs) : 0, 637
Telemetry [baseline] (7.272 ms) : 0, 7272
Telemetry [candidate] (7.414 ms) : 0, 7414
section iast
BytebuddyAgent [baseline] (806.86 ms) : 0, 806860
BytebuddyAgent [candidate] (802.454 ms) : 0, 802454
GlobalTracer [baseline] (231.979 ms) : 0, 231979
GlobalTracer [candidate] (230.631 ms) : 0, 230631
IAST [baseline] (27.097 ms) : 0, 27097
IAST [candidate] (28.537 ms) : 0, 28537
AppSec [baseline] (53.363 ms) : 0, 53363
AppSec [candidate] (51.06 ms) : 0, 51060
Debugger [baseline] (6.044 ms) : 0, 6044
Debugger [candidate] (5.941 ms) : 0, 5941
Remote Config [baseline] (622.112 µs) : 0, 622
Remote Config [candidate] (592.534 µs) : 0, 593
Telemetry [baseline] (8.044 ms) : 0, 8044
Telemetry [candidate] (7.923 ms) : 0, 7923
section profiling
BytebuddyAgent [baseline] (681.379 ms) : 0, 681379
BytebuddyAgent [candidate] (675.162 ms) : 0, 675162
GlobalTracer [baseline] (363.951 ms) : 0, 363951
GlobalTracer [candidate] (359.617 ms) : 0, 359617
AppSec [baseline] (62.215 ms) : 0, 62215
AppSec [candidate] (61.734 ms) : 0, 61734
Debugger [baseline] (6.212 ms) : 0, 6212
Debugger [candidate] (6.099 ms) : 0, 6099
Remote Config [baseline] (655.257 µs) : 0, 655
Remote Config [candidate] (648.579 µs) : 0, 649
Telemetry [baseline] (8.21 ms) : 0, 8210
Telemetry [candidate] (8.168 ms) : 0, 8168
ProfilingAgent [baseline] (104.724 ms) : 0, 104724
ProfilingAgent [candidate] (104.777 ms) : 0, 104777
Profiling [baseline] (104.748 ms) : 0, 104748
Profiling [candidate] (104.801 ms) : 0, 104801
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.028 s) : 0, 1028404
Total [baseline] (8.51 s) : 0, 8510438
Agent [candidate] (1.029 s) : 0, 1028601
Total [candidate] (8.527 s) : 0, 8527336
section iast
Agent [baseline] (1.149 s) : 0, 1149351
Total [baseline] (9.145 s) : 0, 9144760
Agent [candidate] (1.151 s) : 0, 1150521
Total [candidate] (9.181 s) : 0, 9181163
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.15 s) : 0, 1150354
Total [baseline] (9.139 s) : 0, 9139154
Agent [candidate] (1.157 s) : 0, 1156842
Total [candidate] (9.163 s) : 0, 9163307
section iast_TELEMETRY_OFF
Agent [baseline] (1.147 s) : 0, 1147233
Total [baseline] (9.2 s) : 0, 9200109
Agent [candidate] (1.146 s) : 0, 1145956
Total [candidate] (9.19 s) : 0, 9190266
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.028 s -
Agent iast 1.149 s 120.948 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.15 s 121.951 ms (11.9%)
Agent iast_TELEMETRY_OFF 1.147 s 118.829 ms (11.6%)
Total tracing 8.51 s -
Total iast 9.145 s 634.323 ms (7.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.139 s 628.716 ms (7.4%)
Total iast_TELEMETRY_OFF 9.2 s 689.672 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.029 s -
Agent iast 1.151 s 121.92 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.157 s 128.242 ms (12.5%)
Agent iast_TELEMETRY_OFF 1.146 s 117.356 ms (11.4%)
Total tracing 8.527 s -
Total iast 9.181 s 653.827 ms (7.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.163 s 635.971 ms (7.5%)
Total iast_TELEMETRY_OFF 9.19 s 662.93 ms (7.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.678 ms) : 0, 686678
BytebuddyAgent [candidate] (687.504 ms) : 0, 687504
GlobalTracer [baseline] (240.785 ms) : 0, 240785
GlobalTracer [candidate] (241.219 ms) : 0, 241219
AppSec [baseline] (59.235 ms) : 0, 59235
AppSec [candidate] (57.246 ms) : 0, 57246
Debugger [baseline] (6.153 ms) : 0, 6153
Debugger [candidate] (6.204 ms) : 0, 6204
Remote Config [baseline] (729.583 µs) : 0, 730
Remote Config [candidate] (730.774 µs) : 0, 731
Telemetry [baseline] (11.206 ms) : 0, 11206
Telemetry [candidate] (12.12 ms) : 0, 12120
section iast
BytebuddyAgent [baseline] (802.126 ms) : 0, 802126
BytebuddyAgent [candidate] (802.088 ms) : 0, 802088
GlobalTracer [baseline] (230.375 ms) : 0, 230375
GlobalTracer [candidate] (230.79 ms) : 0, 230790
IAST [baseline] (26.84 ms) : 0, 26840
IAST [candidate] (27.105 ms) : 0, 27105
AppSec [baseline] (52.137 ms) : 0, 52137
AppSec [candidate] (52.58 ms) : 0, 52580
Debugger [baseline] (5.954 ms) : 0, 5954
Debugger [candidate] (5.945 ms) : 0, 5945
Remote Config [baseline] (611.012 µs) : 0, 611
Remote Config [candidate] (586.803 µs) : 0, 587
Telemetry [baseline] (7.858 ms) : 0, 7858
Telemetry [candidate] (7.883 ms) : 0, 7883
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (801.882 ms) : 0, 801882
BytebuddyAgent [candidate] (806.55 ms) : 0, 806550
GlobalTracer [baseline] (230.61 ms) : 0, 230610
GlobalTracer [candidate] (231.69 ms) : 0, 231690
IAST [baseline] (26.293 ms) : 0, 26293
IAST [candidate] (28.894 ms) : 0, 28894
AppSec [baseline] (53.491 ms) : 0, 53491
AppSec [candidate] (51.545 ms) : 0, 51545
Debugger [baseline] (5.983 ms) : 0, 5983
Debugger [candidate] (5.937 ms) : 0, 5937
Remote Config [baseline] (620.349 µs) : 0, 620
Remote Config [candidate] (594.352 µs) : 0, 594
Telemetry [baseline] (7.913 ms) : 0, 7913
Telemetry [candidate] (8.044 ms) : 0, 8044
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (798.553 ms) : 0, 798553
BytebuddyAgent [candidate] (798.285 ms) : 0, 798285
GlobalTracer [baseline] (231.004 ms) : 0, 231004
GlobalTracer [candidate] (230.665 ms) : 0, 230665
IAST [baseline] (27.549 ms) : 0, 27549
IAST [candidate] (28.094 ms) : 0, 28094
AppSec [baseline] (51.452 ms) : 0, 51452
AppSec [candidate] (50.292 ms) : 0, 50292
Debugger [baseline] (5.969 ms) : 0, 5969
Debugger [candidate] (5.965 ms) : 0, 5965
Remote Config [baseline] (590.172 µs) : 0, 590
Remote Config [candidate] (596.551 µs) : 0, 597
Telemetry [baseline] (7.842 ms) : 0, 7842
Telemetry [candidate] (7.846 ms) : 0, 7846
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-06-05T13:42:17 2025-06-05T13:55:28
git_branch master tonycthsu/pin-system-tests
git_commit_date 1749044153 1749130405
git_commit_sha 7787af7 3532e4a
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~3532e4a780
start_time 2025-06-05T13:42:02 2025-06-05T13:55:14
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749132355 1749132355
ci_job_id 969867939 969867939
ci_pipeline_id 67058912 67058912
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-nxkl4h5j-project-304-concurrent-0-b38d56le 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-nxkl4h5j-project-304-concurrent-0-b38d56le 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results results results
variant iast iast

Summary

Found 3 performance improvements and 5 performance regressions! Performance is the same for 5 metrics, 17 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:appsec better
[-9.566ms; -9.435ms] or [-90.841%; -89.592%]		 unstable
[+171.419op/s; +6506.452op/s] or [+36.427%; +1382.621%]		 1.030ms 3809.524op/s 10.531ms 470.588op/s
scenario:load:petclinic:appsec_no_iast better
[-1.670ms; -1.542ms] or [-86.012%; -79.444%]		 unstable
[-373.033op/s; +11928.589op/s] or [-16.786%; +536.786%]		 0.335ms 8000.000op/s 1.941ms 2222.222op/s
scenario:load:petclinic:iast worse
[+8.160ms; +8.314ms] or [+469.175%; +478.051%]		 unstable
[-4081.986op/s; +467.856op/s] or [-177.428%; +20.336%]		 9.976ms 493.575op/s 1.739ms 2300.640op/s
scenario:load:petclinic:no_agent worse
[+1.045ms; +1.094ms] or [+13.649%; +14.291%]		 worse
[-92.142op/s; -53.106op/s] or [-14.397%; -8.298%]		 8.727ms 567.376op/s 7.657ms 640.000op/s
scenario:load:petclinic:profiling better
[-6.503ms; -6.389ms] or [-79.492%; -78.100%]		 unstable
[-212.755op/s; +3858.232op/s] or [-35.371%; +641.431%]		 1.735ms 2424.242op/s 8.180ms 601.504op/s
scenario:load:petclinic:tracing worse
[+2.859ms; +2.955ms] or [+38.506%; +39.807%]		 worse
[-212.537op/s; -151.694op/s] or [-32.146%; -22.944%]		 10.331ms 479.042op/s 7.424ms 661.157op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (7.657 ms) : 7635, 7679
.   : milestone, 7657,
appsec (10.531 ms) : 10467, 10595
.   : milestone, 10531,
appsec_no_iast (1.941 ms) : 1866, 2017
.   : milestone, 1941,
code_origins (550.66 µs) : 503, 598
.   : milestone, 551,
iast (1.739 ms) : 1648, 1831
.   : milestone, 1739,
profiling (8.18 ms) : 8151, 8210
.   : milestone, 8180,
tracing (7.424 ms) : 7398, 7451
.   : milestone, 7424,
section candidate
no_agent (8.727 ms) : 8703, 8751
.   : milestone, 8727,
appsec (1.03 ms) : 972, 1088
.   : milestone, 1030,
appsec_no_iast (335.317 µs) : 300, 371
.   : milestone, 335,
code_origins (29.863 ms) : 29764, 29961
.   : milestone, 29863,
iast (9.976 ms) : 9932, 10020
.   : milestone, 9976,
profiling (1.735 ms) : 1666, 1803
.   : milestone, 1735,
tracing (10.331 ms) : 10274, 10389
.   : milestone, 10331,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 7.657 ms [7.635 ms, 7.679 ms] -
appsec 10.531 ms [10.467 ms, 10.595 ms] 2.874 ms (37.5%)
appsec_no_iast 1.941 ms [1.866 ms, 2.017 ms] -5.716 ms (-74.6%)
code_origins 550.66 µs [502.964 µs, 598.357 µs] -7.107 ms (-92.8%)
iast 1.739 ms [1.648 ms, 1.831 ms] -5.918 ms (-77.3%)
profiling 8.18 ms [8.151 ms, 8.21 ms] 523.253 µs (6.8%)
tracing 7.424 ms [7.398 ms, 7.451 ms] -232.998 µs (-3.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 8.727 ms [8.703 ms, 8.751 ms] -
appsec 1.03 ms [972.341 µs, 1.088 ms] -7.697 ms (-88.2%)
appsec_no_iast 335.317 µs [299.585 µs, 371.048 µs] -8.392 ms (-96.2%)
code_origins 29.863 ms [29.764 ms, 29.961 ms] 21.136 ms (242.2%)
iast 9.976 ms [9.932 ms, 10.02 ms] 1.249 ms (14.3%)
profiling 1.735 ms [1.666 ms, 1.803 ms] -6.992 ms (-80.1%)
tracing 10.331 ms [10.274 ms, 10.389 ms] 1.604 ms (18.4%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (379.954 µs) : 360, 400
.   : milestone, 380,
iast (523.501 µs) : 502, 545
.   : milestone, 524,
iast_FULL (743.575 µs) : 720, 767
.   : milestone, 744,
iast_GLOBAL (578.507 µs) : 555, 602
.   : milestone, 579,
iast_HARDCODED_SECRET_DISABLED (519.526 µs) : 498, 541
.   : milestone, 520,
iast_INACTIVE (469.362 µs) : 447, 492
.   : milestone, 469,
iast_TELEMETRY_OFF (517.558 µs) : 494, 541
.   : milestone, 518,
tracing (464.871 µs) : 442, 487
.   : milestone, 465,
section candidate
no_agent (379.683 µs) : 360, 399
.   : milestone, 380,
iast (530.065 µs) : 507, 553
.   : milestone, 530,
iast_FULL (744.254 µs) : 722, 766
.   : milestone, 744,
iast_GLOBAL (558.676 µs) : 536, 582
.   : milestone, 559,
iast_HARDCODED_SECRET_DISABLED (518.98 µs) : 497, 540
.   : milestone, 519,
iast_INACTIVE (461.458 µs) : 439, 484
.   : milestone, 461,
iast_TELEMETRY_OFF (516.956 µs) : 494, 540
.   : milestone, 517,
tracing (458.799 µs) : 437, 481
.   : milestone, 459,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 379.954 µs [360.107 µs, 399.801 µs] -
iast 523.501 µs [501.733 µs, 545.27 µs] 143.547 µs (37.8%)
iast_FULL 743.575 µs [720.09 µs, 767.06 µs] 363.621 µs (95.7%)
iast_GLOBAL 578.507 µs [555.201 µs, 601.814 µs] 198.553 µs (52.3%)
iast_HARDCODED_SECRET_DISABLED 519.526 µs [497.819 µs, 541.234 µs] 139.572 µs (36.7%)
iast_INACTIVE 469.362 µs [446.975 µs, 491.75 µs] 89.408 µs (23.5%)
iast_TELEMETRY_OFF 517.558 µs [494.461 µs, 540.654 µs] 137.603 µs (36.2%)
tracing 464.871 µs [442.378 µs, 487.364 µs] 84.917 µs (22.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 379.683 µs [360.007 µs, 399.358 µs] -
iast 530.065 µs [506.888 µs, 553.241 µs] 150.382 µs (39.6%)
iast_FULL 744.254 µs [722.418 µs, 766.09 µs] 364.572 µs (96.0%)
iast_GLOBAL 558.676 µs [535.747 µs, 581.605 µs] 178.994 µs (47.1%)
iast_HARDCODED_SECRET_DISABLED 518.98 µs [497.46 µs, 540.499 µs] 139.297 µs (36.7%)
iast_INACTIVE 461.458 µs [439.104 µs, 483.811 µs] 81.775 µs (21.5%)
iast_TELEMETRY_OFF 516.956 µs [493.768 µs, 540.145 µs] 137.274 µs (36.2%)
tracing 458.799 µs [436.865 µs, 480.733 µs] 79.116 µs (20.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master tonycthsu/pin-system-tests
git_commit_date 1749044153 1749130405
git_commit_sha 7787af7 3532e4a
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~3532e4a780
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1749132388 1749132388
ci_job_id 969867941 969867941
ci_pipeline_id 67058912 67058912
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-guhm2hol-project-304-concurrent-1-ymcxadjf 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-guhm2hol-project-304-concurrent-1-ymcxadjf 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.536 s) : 15536000, 15536000
.   : milestone, 15536000,
appsec (14.812 s) : 14812000, 14812000
.   : milestone, 14812000,
iast (19.181 s) : 19181000, 19181000
.   : milestone, 19181000,
iast_GLOBAL (17.831 s) : 17831000, 17831000
.   : milestone, 17831000,
profiling (15.166 s) : 15166000, 15166000
.   : milestone, 15166000,
tracing (15.017 s) : 15017000, 15017000
.   : milestone, 15017000,
section candidate
no_agent (15.664 s) : 15664000, 15664000
.   : milestone, 15664000,
appsec (15.068 s) : 15068000, 15068000
.   : milestone, 15068000,
iast (18.912 s) : 18912000, 18912000
.   : milestone, 18912000,
iast_GLOBAL (17.688 s) : 17688000, 17688000
.   : milestone, 17688000,
profiling (15.606 s) : 15606000, 15606000
.   : milestone, 15606000,
tracing (14.497 s) : 14497000, 14497000
.   : milestone, 14497000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.536 s [15.536 s, 15.536 s] -
appsec 14.812 s [14.812 s, 14.812 s] -724.0 ms (-4.7%)
iast 19.181 s [19.181 s, 19.181 s] 3.645 s (23.5%)
iast_GLOBAL 17.831 s [17.831 s, 17.831 s] 2.295 s (14.8%)
profiling 15.166 s [15.166 s, 15.166 s] -370.0 ms (-2.4%)
tracing 15.017 s [15.017 s, 15.017 s] -519.0 ms (-3.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.664 s [15.664 s, 15.664 s] -
appsec 15.068 s [15.068 s, 15.068 s] -596.0 ms (-3.8%)
iast 18.912 s [18.912 s, 18.912 s] 3.248 s (20.7%)
iast_GLOBAL 17.688 s [17.688 s, 17.688 s] 2.024 s (12.9%)
profiling 15.606 s [15.606 s, 15.606 s] -58.0 ms (-0.4%)
tracing 14.497 s [14.497 s, 14.497 s] -1.167 s (-7.5%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~3532e4a780, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1462, 1486
.   : milestone, 1474,
appsec (2.389 ms) : 2341, 2438
.   : milestone, 2389,
iast (2.175 ms) : 2114, 2236
.   : milestone, 2175,
iast_GLOBAL (2.217 ms) : 2156, 2278
.   : milestone, 2217,
profiling (2.015 ms) : 1967, 2064
.   : milestone, 2015,
tracing (1.989 ms) : 1942, 2036
.   : milestone, 1989,
section candidate
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.389 ms) : 2341, 2437
.   : milestone, 2389,
iast (2.168 ms) : 2107, 2228
.   : milestone, 2168,
iast_GLOBAL (2.218 ms) : 2156, 2280
.   : milestone, 2218,
profiling (2.021 ms) : 1972, 2070
.   : milestone, 2021,
tracing (1.998 ms) : 1951, 2046
.   : milestone, 1998,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.474 ms [1.462 ms, 1.486 ms] -
appsec 2.389 ms [2.341 ms, 2.438 ms] 915.349 µs (62.1%)
iast 2.175 ms [2.114 ms, 2.236 ms] 701.363 µs (47.6%)
iast_GLOBAL 2.217 ms [2.156 ms, 2.278 ms] 743.217 µs (50.4%)
profiling 2.015 ms [1.967 ms, 2.064 ms] 541.486 µs (36.7%)
tracing 1.989 ms [1.942 ms, 2.036 ms] 514.963 µs (34.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.461 ms, 1.484 ms] -
appsec 2.389 ms [2.341 ms, 2.437 ms] 916.204 µs (62.2%)
iast 2.168 ms [2.107 ms, 2.228 ms] 695.331 µs (47.2%)
iast_GLOBAL 2.218 ms [2.156 ms, 2.28 ms] 745.411 µs (50.6%)
profiling 2.021 ms [1.972 ms, 2.07 ms] 548.548 µs (37.3%)
tracing 1.998 ms [1.951 ms, 2.046 ms] 525.981 µs (35.7%)

@TonyCTHsu TonyCTHsu force-pushed the tonycthsu/pin-system-tests branch from 2fdd417 to 3532e4a Compare June 5, 2025 13:33
@TonyCTHsu TonyCTHsu marked this pull request as ready for review June 5, 2025 13:43
@TonyCTHsu TonyCTHsu requested a review from a team as a code owner June 5, 2025 13:43
@TonyCTHsu TonyCTHsu requested review from sarahchen6 and removed request for a team June 5, 2025 13:43
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 5, 2025

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@TonyCTHsu TonyCTHsu added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling dependencies Pull requests that update a dependency file labels Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sarahchen6 sarahchen6 Awaiting requested review from sarahchen6 sarahchen6 is a code owner automatically assigned from DataDog/apm-lang-platform-java

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
comp: tooling Build & Tooling dependencies Pull requests that update a dependency file tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@TonyCTHsu