Implement Failed Test Replay #9214

daniel-mohedano · 2025-07-22T14:22:37Z

What Does This Do

Implements Test Optimization's Failed Test Replay using Live Debugger's Exception Replay. When the feature is enabled and a test is retried due to Auto Test Retries, Exception Replay's logic will create a probe for the exception thrown (in the case of the test probably an assertion error, but not limited to it). When the test is retried, the probe captures debugging information if the exception is encountered again, creating a snapshot of the variables. If the snapshot is captured, it is send as a log to Datadog. The following modifications were made to Exception Replay's original implementation:

Exception Replay is enabled if Failed Test Replay is enabled by the user.
- If the build system (Maven or Gradle) is instrumented, the property is propagated to the child process
- If running in headless mode, without build system instrumentation, Failed Test Replay is marked as active through Config. The enabling of Exception Replay now checks for either the property to be marked as enabled or Failed Test Replay being marked as active. This works due to CiVisibility's system being initialized before Live Debugger's.
DefaultExceptionDebugger was modified to support Failed Test Replay:
- Will not skip instrumenting errors if Failed Test Replay is marked as active, in order to avoid ignoring test assertion errors.
- If Failed Test Replay is active, the instrumentation will be applied synchronously. Failed test retries can be performed in rapid succession and the async approach to the instrumentation meant that most of the times the instrumentation was not performed before the next test failure.
Adds a product field to snapshots, populated with test_optimization if Failed Test Replay was marked as active. This allows us to have the option of not billing customers for logs generated by the product.
Removed Live Debugger's dependency on Remote Config being enabled for its configuration to be initialized.
Exception Replay now supports Agentless mode when Failed Test Replay is enabled. If DD_CIVISIBILITY_AGENTLESS_ENABLED is set, Live Debugger's logic for Exception Replay will use the logs API instead of the agent's.
If Failed Test Replay is enabled, a TestListener is registered to flush DebuggerSink on test suite end, to avoid unsent snapshots.

Additional changes:

Refactored BackendApiFactory.Intake to a standalone Intake, given that it is useful in order to compute agentless mode URLs.
Updated libraries capabilities to add failed_test_replay in test frameworks that support Auto Test Retries.
Other changes related to adding di_enabled to the Settings response and telemetry.

Validation:

MavenSmokeTest now has an additional test for Failed Test Replay, validating the feature when build system instrumentation is present.
Implemented JUnitConsoleSmokeTest to validate the feature in headless mode. This test should ensure that the ordering dependency between CiVisibility's system and Live Debugger's is always accounted for.

Motivation

Test Optimization wants to improve the support for Failed Test Replay, implementing it in additional languages apart from JS.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-2242

pr-commenter · 2025-07-23T14:41:30Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1755592792	1755593137
end_time	2025-08-19T08:41:13	2025-08-19T08:46:58
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_sha	`8daf352`	`ffda9b3`
start_time	2025-08-19T08:39:53	2025-08-19T08:45:38

See matching parameters

	Baseline	Candidate
ci_job_id	1086409943	1086409943
ci_pipeline_id	74085193	74085193
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1755592188	1755592188

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 5 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-8.995µs; +25.244µs] or [-3.304%; +9.272%]	unstable [-21.301µs; +33.028µs] or [-6.823%; +10.579%]	unstable [-31.585µs; +42.762µs] or [-9.685%; +13.112%]	unstable [-159.437µs; +38.436µs] or [-16.243%; +3.916%]	same
scenario:basic	unsure [+2.020µs; +7.111µs] or [+0.821%; +2.890%]	same	same	unstable [-257.204µs; -34.420µs] or [-29.323%; -3.924%]	same
scenario:loop	unsure [+1.558µs; +9.376µs] or [+0.018%; +0.106%]	same	same	same	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (312.2 µs) : 290, 334
.   : milestone, 312,
basic (274.252 µs) : 268, 281
.   : milestone, 274,
loop (8.96 ms) : 8956, 8965
.   : milestone, 8960,
section candidate
noprobe (318.064 µs) : 289, 347
.   : milestone, 318,
basic (276.757 µs) : 271, 283
.   : milestone, 277,
loop (8.964 ms) : 8959, 8968
.   : milestone, 8964,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	312.2 µs [290.072 µs, 334.328 µs]
basic	274.252 µs [267.856 µs, 280.648 µs]
loop	8.96 ms [8.956 ms, 8.965 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	318.064 µs [288.946 µs, 347.181 µs]
basic	276.757 µs [270.727 µs, 282.787 µs]
loop	8.964 ms [8.959 ms, 8.968 ms]

pr-commenter · 2025-07-23T14:55:48Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_date	1755587881	1755592188
git_commit_sha	`8daf352`	`ffda9b3`
release_version	1.53.0-SNAPSHOT~8daf35212f	1.51.0-SNAPSHOT~ffda9b3272

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1755594046	1755594046
ci_job_id	1086409936	1086409936
ci_pipeline_id	74085193	74085193
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-rjpuwj2p 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-rjpuwj2p 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 13 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048589
Total [baseline] (10.653 s) : 0, 10652553
Agent [candidate] (1.058 s) : 0, 1057935
Total [candidate] (10.695 s) : 0, 10695467
section appsec
Agent [baseline] (1.219 s) : 0, 1218768
Total [baseline] (10.717 s) : 0, 10716728
Agent [candidate] (1.223 s) : 0, 1223211
Total [candidate] (10.788 s) : 0, 10787623
section iast
Agent [baseline] (1.18 s) : 0, 1180300
Total [baseline] (10.992 s) : 0, 10992392
Agent [candidate] (1.177 s) : 0, 1177192
Total [candidate] (10.965 s) : 0, 10964800
section profiling
Agent [baseline] (1.197 s) : 0, 1196617
Total [baseline] (10.875 s) : 0, 10874789
Agent [candidate] (1.206 s) : 0, 1205591
Total [candidate] (11.0 s) : 0, 10999810

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	appsec	1.219 s	170.179 ms (16.2%)
Agent	iast	1.18 s	131.711 ms (12.6%)
Agent	profiling	1.197 s	148.028 ms (14.1%)
Total	tracing	10.653 s	-
Total	appsec	10.717 s	64.175 ms (0.6%)
Total	iast	10.992 s	339.839 ms (3.2%)
Total	profiling	10.875 s	222.236 ms (2.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.223 s	165.276 ms (15.6%)
Agent	iast	1.177 s	119.257 ms (11.3%)
Agent	profiling	1.206 s	147.656 ms (14.0%)
Total	tracing	10.695 s	-
Total	appsec	10.788 s	92.156 ms (0.9%)
Total	iast	10.965 s	269.333 ms (2.5%)
Total	profiling	11.0 s	304.343 ms (2.8%)

gantt
    title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.451 ms) : 0, 1451
crashtracking [candidate] (1.488 ms) : 0, 1488
BytebuddyAgent [baseline] (733.651 ms) : 0, 733651
BytebuddyAgent [candidate] (740.228 ms) : 0, 740228
GlobalTracer [baseline] (242.459 ms) : 0, 242459
GlobalTracer [candidate] (243.462 ms) : 0, 243462
AppSec [baseline] (30.113 ms) : 0, 30113
AppSec [candidate] (30.405 ms) : 0, 30405
Debugger [baseline] (6.057 ms) : 0, 6057
Debugger [candidate] (6.117 ms) : 0, 6117
Remote Config [baseline] (680.623 µs) : 0, 681
Remote Config [candidate] (680.133 µs) : 0, 680
Telemetry [baseline] (13.11 ms) : 0, 13110
Telemetry [candidate] (14.32 ms) : 0, 14320
section appsec
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.442 ms) : 0, 1442
BytebuddyAgent [baseline] (753.112 ms) : 0, 753112
BytebuddyAgent [candidate] (755.487 ms) : 0, 755487
GlobalTracer [baseline] (234.336 ms) : 0, 234336
GlobalTracer [candidate] (235.388 ms) : 0, 235388
AppSec [baseline] (169.735 ms) : 0, 169735
AppSec [candidate] (168.227 ms) : 0, 168227
Debugger [baseline] (6.421 ms) : 0, 6421
Debugger [candidate] (6.513 ms) : 0, 6513
Remote Config [baseline] (657.309 µs) : 0, 657
Remote Config [candidate] (640.86 µs) : 0, 641
Telemetry [baseline] (8.414 ms) : 0, 8414
Telemetry [candidate] (10.777 ms) : 0, 10777
IAST [baseline] (23.662 ms) : 0, 23662
IAST [candidate] (23.584 ms) : 0, 23584
section iast
crashtracking [baseline] (1.451 ms) : 0, 1451
crashtracking [candidate] (1.564 ms) : 0, 1564
BytebuddyAgent [baseline] (851.469 ms) : 0, 851469
BytebuddyAgent [candidate] (849.347 ms) : 0, 849347
GlobalTracer [baseline] (233.122 ms) : 0, 233122
GlobalTracer [candidate] (232.829 ms) : 0, 232829
AppSec [baseline] (28.752 ms) : 0, 28752
AppSec [candidate] (28.998 ms) : 0, 28998
Debugger [baseline] (5.833 ms) : 0, 5833
Debugger [candidate] (6.642 ms) : 0, 6642
Remote Config [baseline] (623.576 µs) : 0, 624
Remote Config [candidate] (608.938 µs) : 0, 609
Telemetry [baseline] (8.614 ms) : 0, 8614
Telemetry [candidate] (8.444 ms) : 0, 8444
IAST [baseline] (29.276 ms) : 0, 29276
IAST [candidate] (27.693 ms) : 0, 27693
section profiling
crashtracking [baseline] (1.408 ms) : 0, 1408
crashtracking [candidate] (1.431 ms) : 0, 1431
BytebuddyAgent [baseline] (762.29 ms) : 0, 762290
BytebuddyAgent [candidate] (767.212 ms) : 0, 767212
GlobalTracer [baseline] (222.035 ms) : 0, 222035
GlobalTracer [candidate] (224.186 ms) : 0, 224186
AppSec [baseline] (30.165 ms) : 0, 30165
AppSec [candidate] (30.605 ms) : 0, 30605
Debugger [baseline] (6.234 ms) : 0, 6234
Debugger [candidate] (6.38 ms) : 0, 6380
Remote Config [baseline] (723.918 µs) : 0, 724
Remote Config [candidate] (734.518 µs) : 0, 735
Telemetry [baseline] (16.437 ms) : 0, 16437
Telemetry [candidate] (16.365 ms) : 0, 16365
ProfilingAgent [baseline] (107.719 ms) : 0, 107719
ProfilingAgent [candidate] (108.644 ms) : 0, 108644
Profiling [baseline] (108.364 ms) : 0, 108364
Profiling [candidate] (109.3 ms) : 0, 109300

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1052723
Total [baseline] (8.608 s) : 0, 8608212
Agent [candidate] (1.045 s) : 0, 1045309
Total [candidate] (8.562 s) : 0, 8562183
section iast
Agent [baseline] (1.182 s) : 0, 1182041
Total [baseline] (9.277 s) : 0, 9277167
Agent [candidate] (1.178 s) : 0, 1178106
Total [candidate] (9.293 s) : 0, 9292751

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	iast	1.182 s	129.318 ms (12.3%)
Total	tracing	8.608 s	-
Total	iast	9.277 s	668.955 ms (7.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	iast	1.178 s	132.797 ms (12.7%)
Total	tracing	8.562 s	-
Total	iast	9.293 s	730.568 ms (8.5%)

gantt
    title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.463 ms) : 0, 1463
crashtracking [candidate] (1.45 ms) : 0, 1450
BytebuddyAgent [baseline] (737.973 ms) : 0, 737973
BytebuddyAgent [candidate] (731.856 ms) : 0, 731856
GlobalTracer [baseline] (243.556 ms) : 0, 243556
GlobalTracer [candidate] (242.088 ms) : 0, 242088
AppSec [baseline] (30.393 ms) : 0, 30393
AppSec [candidate] (30.11 ms) : 0, 30110
Debugger [baseline] (6.08 ms) : 0, 6080
Debugger [candidate] (6.039 ms) : 0, 6039
Remote Config [baseline] (670.042 µs) : 0, 670
Remote Config [candidate] (663.053 µs) : 0, 663
Telemetry [baseline] (11.525 ms) : 0, 11525
Telemetry [candidate] (12.174 ms) : 0, 12174
section iast
crashtracking [baseline] (1.463 ms) : 0, 1463
crashtracking [candidate] (1.448 ms) : 0, 1448
BytebuddyAgent [baseline] (853.161 ms) : 0, 853161
BytebuddyAgent [candidate] (850.483 ms) : 0, 850483
GlobalTracer [baseline] (233.633 ms) : 0, 233633
GlobalTracer [candidate] (232.753 ms) : 0, 232753
AppSec [baseline] (26.873 ms) : 0, 26873
AppSec [candidate] (29.186 ms) : 0, 29186
Debugger [baseline] (7.56 ms) : 0, 7560
Debugger [candidate] (6.626 ms) : 0, 6626
Remote Config [baseline] (589.76 µs) : 0, 590
Remote Config [candidate] (594.488 µs) : 0, 594
Telemetry [baseline] (8.325 ms) : 0, 8325
Telemetry [candidate] (8.399 ms) : 0, 8399
IAST [baseline] (29.376 ms) : 0, 29376
IAST [candidate] (27.57 ms) : 0, 27570

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_date	1755587881	1755592188
git_commit_sha	`8daf352`	`ffda9b3`
release_version	1.53.0-SNAPSHOT~8daf35212f	1.51.0-SNAPSHOT~ffda9b3272

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1755593723	1755593723
ci_job_id	1086409937	1086409937
ci_pipeline_id	74085193	74085193
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-v2e7rudu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-v2e7rudu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling:high_load	worse [+1.644ms; +2.629ms] or [+3.516%; +5.623%]	unstable [-11.657op/s; +3.032op/s] or [-11.647%; +3.030%]	48.900ms	95.775op/s	46.763ms	100.088op/s
scenario:load:petclinic:tracing:high_load	worse [+1.286ms; +2.102ms] or [+2.989%; +4.886%]	unstable [-11.914op/s; +3.639op/s] or [-10.958%; +3.347%]	44.718ms	104.588op/s	43.024ms	108.725op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.312 ms) : 4263, 4362
.   : milestone, 4312,
iast (9.466 ms) : 9312, 9620
.   : milestone, 9466,
iast_FULL (14.302 ms) : 14015, 14588
.   : milestone, 14302,
iast_GLOBAL (10.656 ms) : 10459, 10854
.   : milestone, 10656,
profiling (8.733 ms) : 8599, 8867
.   : milestone, 8733,
tracing (7.635 ms) : 7526, 7744
.   : milestone, 7635,
section candidate
no_agent (4.335 ms) : 4282, 4387
.   : milestone, 4335,
iast (9.291 ms) : 9143, 9440
.   : milestone, 9291,
iast_FULL (14.23 ms) : 13946, 14514
.   : milestone, 14230,
iast_GLOBAL (10.328 ms) : 10126, 10530
.   : milestone, 10328,
profiling (8.997 ms) : 8858, 9137
.   : milestone, 8997,
tracing (7.7 ms) : 7584, 7817
.   : milestone, 7700,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.312 ms [4.263 ms, 4.362 ms]	-
iast	9.466 ms [9.312 ms, 9.62 ms]	5.154 ms (119.5%)
iast_FULL	14.302 ms [14.015 ms, 14.588 ms]	9.989 ms (231.6%)
iast_GLOBAL	10.656 ms [10.459 ms, 10.854 ms]	6.344 ms (147.1%)
profiling	8.733 ms [8.599 ms, 8.867 ms]	4.42 ms (102.5%)
tracing	7.635 ms [7.526 ms, 7.744 ms]	3.322 ms (77.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.335 ms [4.282 ms, 4.387 ms]	-
iast	9.291 ms [9.143 ms, 9.44 ms]	4.957 ms (114.3%)
iast_FULL	14.23 ms [13.946 ms, 14.514 ms]	9.895 ms (228.3%)
iast_GLOBAL	10.328 ms [10.126 ms, 10.53 ms]	5.993 ms (138.3%)
profiling	8.997 ms [8.858 ms, 9.137 ms]	4.663 ms (107.6%)
tracing	7.7 ms [7.584 ms, 7.817 ms]	3.366 ms (77.6%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.698 ms) : 36400, 36996
.   : milestone, 36698,
appsec (47.242 ms) : 46812, 47672
.   : milestone, 47242,
code_origins (45.973 ms) : 45564, 46382
.   : milestone, 45973,
iast (45.393 ms) : 44990, 45796
.   : milestone, 45393,
profiling (46.763 ms) : 46314, 47212
.   : milestone, 46763,
tracing (43.024 ms) : 42674, 43373
.   : milestone, 43024,
section candidate
no_agent (36.887 ms) : 36593, 37181
.   : milestone, 36887,
appsec (46.694 ms) : 46277, 47112
.   : milestone, 46694,
code_origins (45.359 ms) : 44960, 45759
.   : milestone, 45359,
iast (45.749 ms) : 45352, 46146
.   : milestone, 45749,
profiling (48.9 ms) : 48434, 49366
.   : milestone, 48900,
tracing (44.718 ms) : 44311, 45124
.   : milestone, 44718,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.698 ms [36.4 ms, 36.996 ms]	-
appsec	47.242 ms [46.812 ms, 47.672 ms]	10.544 ms (28.7%)
code_origins	45.973 ms [45.564 ms, 46.382 ms]	9.275 ms (25.3%)
iast	45.393 ms [44.99 ms, 45.796 ms]	8.695 ms (23.7%)
profiling	46.763 ms [46.314 ms, 47.212 ms]	10.065 ms (27.4%)
tracing	43.024 ms [42.674 ms, 43.373 ms]	6.325 ms (17.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.887 ms [36.593 ms, 37.181 ms]	-
appsec	46.694 ms [46.277 ms, 47.112 ms]	9.807 ms (26.6%)
code_origins	45.359 ms [44.96 ms, 45.759 ms]	8.472 ms (23.0%)
iast	45.749 ms [45.352 ms, 46.146 ms]	8.862 ms (24.0%)
profiling	48.9 ms [48.434 ms, 49.366 ms]	12.013 ms (32.6%)
tracing	44.718 ms [44.311 ms, 45.124 ms]	7.831 ms (21.2%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/failed-test-replay
git_commit_date	1755587881	1755592188
git_commit_sha	`8daf352`	`ffda9b3`
release_version	1.53.0-SNAPSHOT~8daf35212f	1.51.0-SNAPSHOT~ffda9b3272

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1755594379	1755594379
ci_job_id	1086409938	1086409938
ci_pipeline_id	74085193	74085193
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-h6xn9j2g 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-h6xn9j2g 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.491 ms) : 1479, 1502
.   : milestone, 1491,
appsec (2.727 ms) : 2646, 2809
.   : milestone, 2727,
iast (2.552 ms) : 2453, 2651
.   : milestone, 2552,
iast_GLOBAL (2.602 ms) : 2504, 2701
.   : milestone, 2602,
profiling (2.362 ms) : 2279, 2444
.   : milestone, 2362,
tracing (2.353 ms) : 2272, 2433
.   : milestone, 2353,
section candidate
no_agent (1.489 ms) : 1477, 1501
.   : milestone, 1489,
appsec (4.095 ms) : 3846, 4345
.   : milestone, 4095,
iast (2.548 ms) : 2450, 2646
.   : milestone, 2548,
iast_GLOBAL (2.596 ms) : 2497, 2694
.   : milestone, 2596,
profiling (2.396 ms) : 2311, 2480
.   : milestone, 2396,
tracing (2.349 ms) : 2268, 2430
.   : milestone, 2349,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.491 ms [1.479 ms, 1.502 ms]	-
appsec	2.727 ms [2.646 ms, 2.809 ms]	1.237 ms (83.0%)
iast	2.552 ms [2.453 ms, 2.651 ms]	1.062 ms (71.2%)
iast_GLOBAL	2.602 ms [2.504 ms, 2.701 ms]	1.112 ms (74.6%)
profiling	2.362 ms [2.279 ms, 2.444 ms]	871.061 µs (58.4%)
tracing	2.353 ms [2.272 ms, 2.433 ms]	862.357 µs (57.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.489 ms [1.477 ms, 1.501 ms]	-
appsec	4.095 ms [3.846 ms, 4.345 ms]	2.606 ms (175.0%)
iast	2.548 ms [2.45 ms, 2.646 ms]	1.059 ms (71.1%)
iast_GLOBAL	2.596 ms [2.497 ms, 2.694 ms]	1.106 ms (74.3%)
profiling	2.396 ms [2.311 ms, 2.48 ms]	906.396 µs (60.9%)
tracing	2.349 ms [2.268 ms, 2.43 ms]	859.754 µs (57.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~ffda9b3272, baseline=1.53.0-SNAPSHOT~8daf35212f
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.08 s) : 15080000, 15080000
.   : milestone, 15080000,
appsec (14.519 s) : 14519000, 14519000
.   : milestone, 14519000,
iast (17.613 s) : 17613000, 17613000
.   : milestone, 17613000,
iast_GLOBAL (17.765 s) : 17765000, 17765000
.   : milestone, 17765000,
profiling (14.783 s) : 14783000, 14783000
.   : milestone, 14783000,
tracing (14.646 s) : 14646000, 14646000
.   : milestone, 14646000,
section candidate
no_agent (14.309 s) : 14309000, 14309000
.   : milestone, 14309000,
appsec (14.175 s) : 14175000, 14175000
.   : milestone, 14175000,
iast (17.618 s) : 17618000, 17618000
.   : milestone, 17618000,
iast_GLOBAL (17.427 s) : 17427000, 17427000
.   : milestone, 17427000,
profiling (14.763 s) : 14763000, 14763000
.   : milestone, 14763000,
tracing (14.519 s) : 14519000, 14519000
.   : milestone, 14519000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.08 s [15.08 s, 15.08 s]	-
appsec	14.519 s [14.519 s, 14.519 s]	-561.0 ms (-3.7%)
iast	17.613 s [17.613 s, 17.613 s]	2.533 s (16.8%)
iast_GLOBAL	17.765 s [17.765 s, 17.765 s]	2.685 s (17.8%)
profiling	14.783 s [14.783 s, 14.783 s]	-297.0 ms (-2.0%)
tracing	14.646 s [14.646 s, 14.646 s]	-434.0 ms (-2.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.309 s [14.309 s, 14.309 s]	-
appsec	14.175 s [14.175 s, 14.175 s]	-134.0 ms (-0.9%)
iast	17.618 s [17.618 s, 17.618 s]	3.309 s (23.1%)
iast_GLOBAL	17.427 s [17.427 s, 17.427 s]	3.118 s (21.8%)
profiling	14.763 s [14.763 s, 14.763 s]	454.0 ms (3.2%)
tracing	14.519 s [14.519 s, 14.519 s]	210.0 ms (1.5%)

datadog-official · 2025-08-11T09:23:09Z

Code coverage: total 57.32%, base diff -0.03%, patch 45.45% (view details)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ffda9b3 | Docs | Was this helpful? Give us feedback!}

daniel-mohedano added 4 commits July 4, 2025 12:22

add di_enabled to settings response

2c219fd

add FTR related metrics

a911ae0

add FTR to execution settings

d20159e

add basic exception replay integration in agent mode

c4c84d6

daniel-mohedano added type: enhancement Enhancements and improvements tag: do not merge Do not merge changes comp: ci visibility Continuous Integration Visibility comp: debugger Dynamic Instrumentation labels Jul 22, 2025

daniel-mohedano added 3 commits July 22, 2025 16:27

feat: headless and agentless changes

ae31670

Merge branch 'master' into daniel.mohedano/failed-test-replay

4f2d6a9

fix: tests

c7eeac8

daniel-mohedano added 8 commits August 4, 2025 10:21

fix: testng capabilities

14cf11c

feat: refactor agentless intakes

461fb1f

Merge branch 'master' into daniel.mohedano/failed-test-replay

14b1054

chore: update smoke test fixtures

a87eff0

test: add unit test for new Intake enum

19b4edf

test: remove ftr from instrumentation tests (not used)

9e81cc7

test: introduce FTR smoke tests for headfull and headless modes

4e3deb4

style: spotless and codenarc

f931654

daniel-mohedano added 3 commits August 11, 2025 12:31

feat: add test event finished FTR telemetry

eacde60

feat: add product field to snapshots

0228f71

feat: implement SuiteEnd listener for sink flushing

88fd665

daniel-mohedano changed the title ~~Failed Test Replay~~ Implement Failed Test Replay Aug 12, 2025

feat: introduce new config variables for debugger

ec9c54d

daniel-mohedano removed the tag: do not merge Do not merge changes label Aug 13, 2025

daniel-mohedano added 2 commits August 19, 2025 10:29

chore: remove todo

d4d2432

Merge branch 'master' into daniel.mohedano/failed-test-replay

ffda9b3

daniel-mohedano marked this pull request as ready for review August 20, 2025 10:07

daniel-mohedano requested review from a team as code owners August 20, 2025 10:07

daniel-mohedano requested review from shatzi, Mariovido, bric3 and PerfectSlayer and removed request for a team August 20, 2025 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement Failed Test Replay #9214

Implement Failed Test Replay #9214

daniel-mohedano commented Jul 22, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

Uh oh!

datadog-official bot commented Aug 11, 2025 •

edited by datadog-datadog-prod-us1 bot

Loading

Uh oh!

Uh oh!

Implement Failed Test Replay #9214

Are you sure you want to change the base?

Implement Failed Test Replay #9214

Conversation

daniel-mohedano commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Contributor Checklist

Uh oh!

pr-commenter bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

pr-commenter bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

datadog-official bot commented Aug 11, 2025 • edited by datadog-datadog-prod-us1 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

daniel-mohedano commented Jul 22, 2025 •

edited

Loading

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

pr-commenter bot commented Jul 23, 2025 •

edited

Loading

datadog-official bot commented Aug 11, 2025 •

edited by datadog-datadog-prod-us1 bot

Loading