Skip to content

Commit 98a8616

Browse files
linconvidallinconvidal
authored
feat: enhance stability test with error threshold and additional metrics (#449)
- Added `--error-threshold` argument to specify acceptable non-2xx error rates. - Updated output parsing to include non-2xx responses and failed requests. - Enhanced CSV and markdown reporting to include new metrics: Non-2xx Responses, Error Rate, and Meets Error Threshold. - Modified test endpoint logic to evaluate both SLA and error thresholds, improving overall test accuracy and reporting.
1 parent 9926397 commit 98a8616

File tree

2 files changed

+119
-49
lines changed

2 files changed

+119
-49
lines changed

‎load-tests/README.MD‎

Lines changed: 28 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ This script performs stability testing of Cardano Rosetta API endpoints by:
1010
- Monitoring endpoint stability across multiple concurrent connections
1111
- Measuring response times (p95, p99) and validating against SLA thresholds
1212
- Identifying potential degradation during sustained operation
13+
- Tracking non-2xx error responses and validating against error thresholds
1314

1415
## Requirements
1516

@@ -75,9 +76,9 @@ usage: stability_test.py [-h] [--url BASE_URL] [--csv CSV_FILE]
7576
[--hardware-profile HARDWARE_PROFILE]
7677
[--machine-specs MACHINE_SPECS]
7778
[--concurrency CONCURRENCIES] [--duration TEST_DURATION]
78-
[--sla SLA_THRESHOLD] [--skip-header] [-v]
79-
[--cooldown COOLDOWN] [--endpoints SELECTED_ENDPOINTS]
80-
[--list-endpoints]
79+
[--sla SLA_THRESHOLD] [--error-threshold ERROR_THRESHOLD]
80+
[--skip-header] [-v] [--cooldown COOLDOWN]
81+
[--endpoints SELECTED_ENDPOINTS] [--list-endpoints]
8182
8283
Cardano Rosetta API Stability Testing Tool
8384
@@ -96,6 +97,8 @@ options:
9697
--duration TEST_DURATION
9798
Duration in seconds for each concurrency level test (default: 60)
9899
--sla SLA_THRESHOLD SLA threshold in milliseconds (default: 1000)
100+
--error-threshold ERROR_THRESHOLD
101+
Threshold for non-2xx errors as percentage (default: 1.0)
99102
--skip-header Skip the header row in the CSV file (default: False)
100103
-v, --verbose Enable verbose output (default: False)
101104
--cooldown COOLDOWN Cooldown period in seconds between endpoint tests (default: 60)
@@ -155,6 +158,12 @@ Set custom hardware profile and SLA:
155158
./load-tests/stability_test.py --hardware-profile high_performance --machine-specs "16 cores, 64GB RAM, SSD" --sla 500
156159
```
157160

161+
Test with custom SLA and error thresholds:
162+
163+
```bash
164+
./load-tests/stability_test.py --sla 500 --error-threshold 0.5
165+
```
166+
158167
## Output
159168

160169
The script creates a timestamped directory containing:
@@ -177,22 +186,22 @@ The script creates a timestamped directory containing:
177186

178187
### Detailed Results (excerpt)
179188

180-
| Hardware | Machine Specs | Endpoint | Concurrency | p95 (ms) | p99 (ms) | Meets SLA | Complete Reqs | Reqs/sec | Mean Time (ms) |
181-
| ----------- | ------------------------------------------------ | ---------------- | ----------- | -------- | -------- | --------- | ------------- | -------- | -------------- |
182-
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 1 | 1ms | 2ms | Yes | 9441 | 943.97 | 1.06ms |
183-
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 2 | 1ms | 1ms | Yes | 20538 | 2053.68 | 0.49ms |
184-
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 4 | 1ms | 1ms | Yes | 39249 | 3924.89 | 0.26ms |
185-
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /account/balance | 1 | 34ms | 40ms | Yes | 454 | 45.32 | 22.06ms |
186-
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /block | 4 | 120ms | 129ms | Yes | 361 | 36.02 | 27.76ms |
189+
| Hardware | Machine Specs | Endpoint | Concurrency | p95 (ms) | p99 (ms) | Meets SLA | Complete Reqs | Reqs/sec | Mean Time (ms) | Non-2xx | Error Rate (%) | Meets Error Threshold |
190+
| ----------- | ------------------------------------------------ | ---------------- | ----------- | -------- | -------- | --------- | ------------- | -------- | -------------- | ------- | -------------- | --------------------- |
191+
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 1 | 1ms | 2ms | Yes | 9441 | 943.97 | 1.06ms | 0 | 0.00% | Yes |
192+
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 2 | 1ms | 1ms | Yes | 20538 | 2053.68 | 0.49ms | 0 | 0.00% | Yes |
193+
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 4 | 1ms | 1ms | Yes | 39249 | 3924.89 | 0.26ms | 0 | 0.00% | Yes |
194+
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /account/balance | 1 | 34ms | 40ms | Yes | 454 | 45.32 | 22.06ms | 0 | 0.00% | Yes |
195+
| entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /block | 4 | 120ms | 129ms | Yes | 361 | 36.02 | 27.76ms | 0 | 0.00% | Yes |
187196

188197
### Summary Results
189198

190-
| ID | Release | Hardware | Machine Specs | Endpoint | Max Concurrency | p95 (ms) | p99 (ms) |
191-
| --- | ------- | ----------- | ------------------------------------------------ | ---------------------- | --------------- | -------- | -------- |
192-
| 1 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 4 | 1ms | 1ms |
193-
| 2 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /account/balance | 4 | 35ms | 39ms |
194-
| 3 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /account/coins | 4 | 33ms | 39ms |
195-
| 4 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /block | 4 | 120ms | 129ms |
196-
| 5 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /block/transaction | 4 | 84ms | 95ms |
197-
| 6 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /search/transactions | 4 | 4ms | 4ms |
198-
| 7 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /construction/metadata | 4 | 1ms | 1ms |
199+
| ID | Release | Hardware | Machine Specs | Endpoint | Max Concurrency | p95 (ms) | p99 (ms) | Non-2xx | Error Rate (%) | Reqs/sec |
200+
| --- | ------- | ----------- | ------------------------------------------------ | ---------------------- | --------------- | -------- | -------- | ------- | -------------- | -------- |
201+
| 1 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /network/status | 4 | 1ms | 1ms | 0 | 0.00% | 3924.89 |
202+
| 2 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /account/balance | 4 | 35ms | 39ms | 0 | 0.00% | 107.42 |
203+
| 3 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /account/coins | 4 | 33ms | 39ms | 0 | 0.00% | 102.51 |
204+
| 4 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /block | 4 | 120ms | 129ms | 0 | 0.00% | 36.02 |
205+
| 5 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /block/transaction | 4 | 84ms | 95ms | 0 | 0.00% | 24.75 |
206+
| 6 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /search/transactions | 4 | 4ms | 4ms | 0 | 0.00% | 1854.29 |
207+
| 7 | 1.2.6 | entry_level | 4 cores, 8 threads, 32GB RAM, SSD, Intel Core i3 | /construction/metadata | 4 | 1ms | 1ms | 0 | 0.00% | 4015.67 |

0 commit comments

Comments
 (0)