Start building Clang runtimes runtimes on-demand

This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.

The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.

This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.

It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.

It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.

Note that I looked at whether we could split this into another layer
separate from the `ClangRunner`, but that proved frustratingly difficult
to manage. While we support building these on-demand as part of
a detected link, that doesn't seem feasible as we don't have the
necessary separation between compilation runs of Clang and link runs of
Clang. However, I have tried to factor the internals to provide as clear
of separation as I could across these.

Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.

The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Hopefully this is OK in the brief interim until I can
put caching in place.

Author: chandlerc

.codespell_ignore

@@ -2,6 +2,7 @@
 # Exceptions. See /LICENSE for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
+ArchType
 atleast
 circularly
 compiletime
@@ -14,6 +15,7 @@ forin
 groupt
 indext
 inout
+isELF
 parameteras
 pullrequest
 rightt

MODULE.bazel

@@ -125,6 +125,7 @@ http_archive(
         "@carbon//bazel/llvm_project:0001_Patch_for_mallinfo2_when_using_Bazel_build_system.patch",
         "@carbon//bazel/llvm_project:0002_Added_Bazel_build_for_compiler_rt_fuzzer.patch",
         "@carbon//bazel/llvm_project:0003_Comment_out_unloaded_proto_library_dependencies.patch",
+        "@carbon//bazel/llvm_project:0004_Add_support_for_custom_rules_to_build_builtins.patch",
     ],
     sha256 = "3d3621c8462e79713d0000778f1dc352c8596d904419c6a33a93a82e6dbe8b58",
     strip_prefix = "llvm-project-{0}".format(llvm_project_version),

bazel/llvm_project/0004_Add_support_for_custom_rules_to_build_builtins.patch

@@ -0,0 +1,26 @@
+From 0a46fc798170e76a0110bf118ae53db7a8c10b27 Mon Sep 17 00:00:00 2001
+From: Chandler Carruth <chandlerc@gmail.com>
+Date: Fri, 14 Feb 2025 01:08:54 +0000
+Subject: [PATCH 4/4] Add support for custom rules to build builtins
+
+This exposes the compiler-rt files for any custom rules that we need to
+write to build and install them for the Carbon toolchain.
+---
+ utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel b/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel
+index 00f2a3b9d7c0..a53a34cdc952 100644
+--- a/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel
++++ b/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel
+@@ -127,3 +127,7 @@ cc_library(
+     ],
+     includes = ["lib/fuzzer"],
+ )
++
++# Allow building custom rules for runtimes.
++exports_files(glob(["**"]))
++
+-- 
+2.48.1.601.g30ceb7b040-goog
+

scripts/fix_cc_deps.py

@@ -70,7 +70,8 @@ class RuleChoice(NamedTuple):
 IGNORE_SOURCE_FILE_REGEX = re.compile(
     r"^(third_party/clangd.*|common/version.*\.cpp"
     r"|.*_autogen_manifest\.cpp"
-    r"|toolchain/base/llvm_tools.def)$"
+    r"|toolchain/base/llvm_tools.def"
+    r"|toolchain/base/runtime_sources.h)$"
 )
 
 

toolchain/base/BUILD

@@ -4,6 +4,7 @@
 
 load("//bazel/cc_rules:defs.bzl", "cc_library", "cc_test")
 load("llvm_tools.bzl", "LLVM_MAIN_TOOLS", "generate_llvm_tools_def")
+load("runtime_sources.bzl", "generate_runtime_sources_cc_library")
 
 package(default_visibility = ["//visibility:public"])
 
@@ -152,6 +153,10 @@ cc_library(
     ] + [info.lib for info in LLVM_MAIN_TOOLS.values()],
 )
 
+generate_runtime_sources_cc_library(
+    name = "runtime_sources",
+)
+
 cc_library(
     name = "shared_value_stores",
     hdrs = ["shared_value_stores.h"],