Skip to content

fix(benches): forge build benches #11036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jul 24, 2025
Merged

fix(benches): forge build benches #11036

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
cff5679
run.sh + test and build bench
yash-atreya Jul 16, 2025
bb445b7
Merge branch 'master' into v1.3.0-benches
yash-atreya Jul 17, 2025
d6796c7
bench seq: test, fuzz and build
yash-atreya Jul 17, 2025
52ba9a4
fix(`benches`): use prepare in forge build benches
yash-atreya Jul 17, 2025
be59190
coverage results
yash-atreya Jul 17, 2025
e2e8738
fix(`benches`): forge build prepare and cleanup
yash-atreya Jul 17, 2025
9457b84
latest run
yash-atreya Jul 18, 2025
bf349bb
rm intermediate results
yash-atreya Jul 18, 2025
81ef5d7
rm sequential run script
yash-atreya Jul 18, 2025
4f38187
LINT_ON_BUILD=false
yash-atreya Jul 18, 2025
a10514a
fix: FOUNDRY_LINT_LINT_ON_BUILD=false
yash-atreya Jul 18, 2025
0758a0d
latest build benches
yash-atreya Jul 19, 2025
d5cd8d7
Merge branch 'master' into v1.3.0-benches
yash-atreya Jul 24, 2025
62de47b
fix
yash-atreya Jul 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 50 additions & 43 deletions benches/LATEST.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,86 +1,93 @@
# 📊 Foundry Benchmark Results

**Generated at**: 2025-07-23 09:01:16 UTC
**Generated at**: 2025-07-18 23:05:00 UTC

## Forge Test

### Repositories Tested

1. [ithacaxyz/account](https://github.com/ithacaxyz/account)
2. [Vectorized/solady](https://github.com/Vectorized/solady)
3. [sparkdotfi/spark-psm](https://github.com/sparkdotfi/spark-psm)
4. [Uniswap/v4-core](https://github.com/Uniswap/v4-core)

### Foundry Versions

- **stable**: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
- **nightly**: forge Version: 1.3.0-nightly (cb8f3bf 2025-07-23)
- **v1.2.3**: forge Version: 1.2.3-v1.2.3 (a813a2c 2025-06-08)
- **nightly-05918765cb239024e9ca396825abb9f46257419a**: forge Version: 1.2.3-nightly (0591876 2025-07-15)
- **nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085**: forge Version: 1.3.0-nightly (0af4341 2025-07-17)

| Repository | stable | nightly |
|------------|----------|----------|
| ithacaxyz-account | 5.44 s | 4.91 s |
| solady | 3.02 s | 2.76 s |
| Repository | v1.2.3 | nightly-05918765cb239024e9ca396825abb9f46257419a | nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085 |
| -------------------- | ------- | ------------------------------------------------ | ------------------------------------------------ |
| ithacaxyz-account | 6.74 s | 3.24 s | 3.52 s |
| solady | 2.77 s | 2.76 s | 2.71 s |
| sparkdotfi-spark-psm | 1m 3.7s | 1m 1.2s | 1m 5.3s |
| Uniswap-v4-core | 8.04 s | 7.58 s | 8.44 s |

## Forge Fuzz Test

| Repository | stable | nightly |
|------------|----------|----------|
| ithacaxyz-account | 5.27 s | 4.90 s |
| solady | 2.89 s | 2.82 s |

## Forge Test (Isolated)

### Repositories Tested

1. [Vectorized/solady](https://github.com/Vectorized/solady)
### Foundry Versions

- **stable**: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
- **nightly**: forge Version: 1.3.0-nightly (cb8f3bf 2025-07-23)

| Repository | stable | nightly |
|------------|----------|----------|
| solady | 3.21 s | 2.88 s |
| Repository | v1.2.3 | nightly-05918765cb239024e9ca396825abb9f46257419a | nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085 |
| -------------------- | ------ | ------------------------------------------------ | ------------------------------------------------ |
| ithacaxyz-account | 3.92 s | 3.37 s | 3.52 s |
| solady | 2.96 s | 2.66 s | 2.82 s |
| sparkdotfi-spark-psm | 3.68 s | 3.52 s | 3.63 s |
| Uniswap-v4-core | 8.06 s | 7.82 s | 8.31 s |

## Forge Build

### Repositories Tested

1. [ithacaxyz/account](https://github.com/ithacaxyz/account)
2. [Vectorized/solady](https://github.com/Vectorized/solady)
3. [sparkdotfi/spark-psm](https://github.com/sparkdotfi/spark-psm)
4. [Uniswap/v4-core](https://github.com/Uniswap/v4-core)

### Foundry Versions

- **stable**: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
- **nightly**: forge Version: 1.3.0-nightly (cb8f3bf 2025-07-23)
- **v1.2.3**: forge Version: 1.2.3-v1.2.3 (a813a2c 2025-06-08)
- **nightly-05918765cb239024e9ca396825abb9f46257419a**: forge Version: 1.2.3-nightly (0591876 2025-07-15)
- **nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085**: forge Version: 1.3.0-nightly (0af4341 2025-07-17)

### No Cache

| Repository | stable | nightly |
|------------|----------|----------|
| ithacaxyz-account | 2.58 s | 2.79 s |
| solady | 3.79 s | 3.89 s |
| Repository | v1.2.3 | nightly-05918765cb239024e9ca396825abb9f46257419a | nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085 |
| -------------------- | -------- | ------------------------------------------------ | ------------------------------------------------ |
| ithacaxyz-account | 9.32 s | 9.32 s | 9.46 s |
| solady | 15.01 s | 14.97 s | 14.81 s |
| sparkdotfi-spark-psm | 13.42 s | 13.29 s | 13.28 s |
| Uniswap-v4-core | 2m 11.6s | 2m 7.4s | 2m 6.3s |

### With Cache

| Repository | stable | nightly |
|------------|----------|----------|
| ithacaxyz-account | 0.209 s | 0.458 s |
| solady | 0.096 s | 0.226 s |
| Repository | v1.2.3 | nightly-05918765cb239024e9ca396825abb9f46257419a | nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085 |
| -------------------- | ------- | ------------------------------------------------ | ------------------------------------------------ |
| ithacaxyz-account | 0.203 s | 0.205 s | 0.201 s |
| solady | 0.092 s | 0.090 s | 0.094 s |
| sparkdotfi-spark-psm | 0.170 s | 0.174 s | 0.173 s |
| Uniswap-v4-core | 0.135 s | 0.139 s | 0.135 s |

## Forge Coverage

### Repositories Tested

1. [ithacaxyz/account](https://github.com/ithacaxyz/account)
2. [sparkdotfi/spark-psm](https://github.com/sparkdotfi/spark-psm)
3. [Uniswap/v4-core](https://github.com/Uniswap/v4-core)

### Foundry Versions

- **stable**: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
- **nightly**: forge Version: 1.3.0-nightly (cb8f3bf 2025-07-23)
- **v1.2.3**: forge Version: 1.2.3-v1.2.3 (a813a2c 2025-06-08)
- **nightly-05918765cb239024e9ca396825abb9f46257419a**: forge Version: 1.2.3-nightly (0591876 2025-07-15)
- **nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085**: forge Version: 1.3.0-nightly (0af4341 2025-07-17)

| Repository | stable | nightly |
|------------|----------|----------|
| ithacaxyz-account | 21.88 s | 22.52 s |
| Repository | v1.2.3 | nightly-05918765cb239024e9ca396825abb9f46257419a | nightly-0af43412f809c998d8b2fe69a1c9a789b7ebd085 |
| -------------------- | -------- | ------------------------------------------------ | ------------------------------------------------ |
| ithacaxyz-account | 16.33 s | 17.31 s | 16.43 s |
| sparkdotfi-spark-psm | 3m 52.9s | 4m 12.8s | 4m 15.0s |
| Uniswap-v4-core | 1m 40.6s | 1m 42.7s | 1m 47.5s |

## System Information


- **OS**: linux
- **OS**: macos
- **CPU**: 8
- **Rustc**: rustc 1.88.0 (6b00bc388 2025-06-23)
- **Rustc**: rustc 1.90.0-nightly (3014e79f9 2025-07-15)
29 changes: 25 additions & 4 deletions benches/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,12 +195,14 @@ impl BenchmarkProject {
/// * `command` - The command to benchmark
/// * `runs` - Number of runs to perform
/// * `setup` - Optional setup command to run before the benchmark series (e.g., "forge build")
/// * `prepare` - Optional prepare command to run before each timing run (e.g., "forge clean")
/// * `conclude` - Optional conclude command to run after each timing run (e.g., cleanup)
/// * `verbose` - Whether to show command output
///
/// # Hyperfine flags used:
/// * `--runs` - Number of timing runs
/// * `--setup` - Execute before the benchmark series (not before each run)
/// * `--prepare` - Execute before each timing run
/// * `--conclude` - Execute after each timing run
/// * `--export-json` - Export results to JSON for parsing
/// * `--shell=bash` - Use bash for shell command execution
Expand All @@ -213,6 +215,7 @@ impl BenchmarkProject {
command: &str,
runs: u32,
setup: Option<&str>,
prepare: Option<&str>,
conclude: Option<&str>,
verbose: bool,
) -> Result<HyperfineResult> {
Expand Down Expand Up @@ -240,6 +243,11 @@ impl BenchmarkProject {
hyperfine_cmd.arg("--setup").arg(setup_cmd);
}

// Add optional prepare command
if let Some(prepare_cmd) = prepare {
hyperfine_cmd.arg("--prepare").arg(prepare_cmd);
}

// Add optional conclude command
if let Some(conclude_cmd) = conclude {
hyperfine_cmd.arg("--conclude").arg(conclude_cmd);
Expand Down Expand Up @@ -282,6 +290,7 @@ impl BenchmarkProject {
runs,
Some("forge build"),
None,
None,
verbose,
)
}
Expand All @@ -293,8 +302,16 @@ impl BenchmarkProject {
runs: u32,
verbose: bool,
) -> Result<HyperfineResult> {
// No setup needed, uses existing cache
self.hyperfine("forge_build_with_cache", version, "forge build", runs, None, None, verbose)
self.hyperfine(
"forge_build_with_cache",
version,
"FOUNDRY_LINT_LINT_ON_BUILD=false forge build",
runs,
None,
Some("forge build"),
None,
verbose,
)
}

/// Benchmark forge build without cache
Expand All @@ -304,14 +321,15 @@ impl BenchmarkProject {
runs: u32,
verbose: bool,
) -> Result<HyperfineResult> {
// Clean before the benchmark series
// Clean before each timing run
self.hyperfine(
"forge_build_no_cache",
version,
"forge build",
"FOUNDRY_LINT_LINT_ON_BUILD=false forge build",
runs,
Some("forge clean"),
None,
Some("forge clean"),
verbose,
)
}
Expand All @@ -331,6 +349,7 @@ impl BenchmarkProject {
runs,
Some("forge build"),
None,
None,
verbose,
)
}
Expand All @@ -351,6 +370,7 @@ impl BenchmarkProject {
runs,
None,
None,
None,
verbose,
)
}
Expand All @@ -370,6 +390,7 @@ impl BenchmarkProject {
runs,
Some("forge build"),
None,
None,
verbose,
)
}
Expand Down