Skip to content

feat(ci): Enable parallel builds for Ubuntu 20.04 & 24.04 base images #13964

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 44 commits into from
Oct 9, 2025
Merged

feat(ci): Enable parallel builds for Ubuntu 20.04 & 24.04 base images #13964

merged 44 commits into from
Oct 9, 2025

Conversation

hunsche
Copy link
Contributor

@hunsche hunsche commented Sep 8, 2025 •
edited
Loading

Part of: b/441792502

Description

This PR modifies the existing base image testing pipeline (trial_build.py) to support the Ubuntu 24.04 migration. It enables the parallel build and testing of base images for latest, ubuntu-20-04, and ubuntu-24-04.

Instead of introducing a new, separate CI pipeline, this change integrates the multi-version logic directly into the established trial_build.py workflow. This approach ensures that both local development builds and remote CI builds are consistent and leverage the existing, robust testing infrastructure.

Changes Made

  • build_and_push_test_images.py Refactored:

    • A BASE_IMAGE_VERSIONS list now serves as the single source of truth for all target Ubuntu versions.
    • The GCB build function (gcb_build_and_push_images) was updated to generate a single Google Cloud Build trigger with parallel steps for each image and version combination.
    • The local build function (build_and_push_images) was also updated to build all versions in parallel using multiprocessing, ensuring local and remote build parity.

  • Dynamic Dockerfile Selection: The script now intelligently selects the correct Dockerfile (e.g., base-image/ubuntu-24-04.Dockerfile) based on the target version, skipping any that do not exist.

  • Consistent Versioned Tagging: Helper functions were updated to apply the correct tags to the images (e.g., gcr.io/oss-fuzz-base/base-image:ubuntu-24-04-testing).

How to Verify

  1. Trigger the trial_build.py script against a set of projects.
  2. Observe the Google Cloud Build logs. A single build should be triggered containing a large number of parallel steps, one for each image/version combination (e.g., base-image:latest, base-image:ubuntu-20-04, etc.).
  3. Verify that the build completes successfully and that the test images are pushed to the Artifact Registry with the correct versioned tags.

DavidKorczynski
DavidKorczynski previously requested changes Sep 9, 2025
View reviewed changes
@DavidKorczynski
Copy link
Collaborator

rebasing this on top should solve the AFL issue

hunsche
hunsche commented Sep 10, 2025
View reviewed changes
hunsche
hunsche commented Sep 10, 2025
View reviewed changes
@hunsche hunsche force-pushed the feat/multi-ubuntu-builds branch 7 times, most recently from 2955804 to 3207b66 Compare September 16, 2025 14:20
@hunsche hunsche force-pushed the feat/multi-ubuntu-builds branch 2 times, most recently from 33593e5 to ddf13fb Compare September 22, 2025 21:53
@maflcko
Copy link
Contributor

maflcko commented Sep 24, 2025

I think the new images are still using the old llvm:

$ git grep OUR_LLVM_REVISION= 2d2bd6976f9197e812e5af1098c667b3b1085472 
2d2bd6976f9197e812e5af1098c667b3b1085472:infra/base-images/base-clang/checkout_build_install_llvm.sh:OUR_LLVM_REVISION=cb2f0d0a5f14
2d2bd6976f9197e812e5af1098c667b3b1085472:infra/base-images/base-clang/checkout_build_install_llvm_ubuntu_20_04.sh:OUR_LLVM_REVISION=llvmorg-18.1.8
2d2bd6976f9197e812e5af1098c667b3b1085472:infra/base-images/base-clang/checkout_build_install_llvm_ubuntu_24_04.sh:OUR_LLVM_REVISION=llvmorg-18.1.8

Not sure if this intentional, but this means the new images will be using the old rust version and projects would have difficutly switching from the "legacy" image (with newer versions) to the new images (with older toolchain versions).

@evverx evverx mentioned this pull request Sep 24, 2025
Merged
@hunsche hunsche force-pushed the feat/multi-ubuntu-builds branch 2 times, most recently from f04a5d9 to 0b00121 Compare September 25, 2025 20:27
@hunsche
Copy link
Contributor Author

hunsche commented Sep 26, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

2 similar comments
@hunsche
Copy link
Contributor Author

hunsche commented Sep 26, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

@hunsche
Copy link
Contributor Author

hunsche commented Sep 26, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

@hunsche hunsche force-pushed the feat/multi-ubuntu-builds branch 4 times, most recently from 97f2133 to 8e7f8eb Compare September 26, 2025 22:11
@hunsche
Copy link
Contributor Author

hunsche commented Sep 26, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

3 similar comments
@hunsche
Copy link
Contributor Author

hunsche commented Sep 26, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

@hunsche
Copy link
Contributor Author

hunsche commented Sep 26, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

@hunsche
Copy link
Contributor Author

hunsche commented Sep 27, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address

@hunsche
refactor(logs): Improve clarity and correctness in build reports
75b7e82 
Refactored trial_build.py and report_generator.py to provide clearer and more accurate build logs.

Key changes:
- Differentiated between 'projects' and 'builds' for precise counting.
- Consolidated and clarified the reporting of skipped builds.
- Updated the results JSON schema with descriptive keys.
- Updated the report generator to consume the new schema and display both build counts per version and a final summary of projects analyzed.
@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

1 similar comment
@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
fix(base-images): Add lcab package and improve Ubuntu sync process
2a7284a 
The 'lcab' package was removed from Ubuntu 24.04 repositories, causing build failures for projects like libarchive.

This change resolves the issue by installing 'lcab' from a .deb package into the Ubuntu 24.04 base-builder image. To support this while adhering to the image strategy, version-specific install_deps.sh scripts were created for Ubuntu 20.04 and 24.04.

Additionally, the CI workflow for checking synchronization was improved:
- Renamed to 'ubuntu_version_sync.yml' for clarity.
- Updated to monitor 'install_deps.sh' to prevent future drift.
@hunsche hunsche force-pushed the feat/multi-ubuntu-builds branch from a02a45b to 2a7284a Compare October 8, 2025 17:54
@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

hunsche added 2 commits October 8, 2025 19:04
@hunsche
fix(report_generator): Update separator line in final summary for con…
45b2fc8 
…sistency
@hunsche
fix(install_deps): Update apt configuration for non-interactive insta…
c65cdcb 
…lls and downgrades

fix(trial_build): Implement exponential backoff for build status checks with retry logic
@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py all --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py all --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 8, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 9, 2025

/gcbrun trial_build.py all --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 9, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche
Copy link
Contributor Author

hunsche commented Oct 9, 2025

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address --force-build

@hunsche hunsche merged commit 4b541c7 into master Oct 9, 2025
30 of 31 checks passed
@hunsche hunsche deleted the feat/multi-ubuntu-builds branch October 9, 2025 17:11
@evverx evverx mentioned this pull request Oct 10, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vitaliset vitaliset vitaliset approved these changes

@DavidKorczynski DavidKorczynski DavidKorczynski left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hunsche @DavidKorczynski @maflcko @vitaliset