Introduce CPUAffinity process property instead of execCPUAffinity #1296

kad · 2025-09-15T11:08:41Z

This change introduces more generic CPUAffinity property of Process to specify desired CPU affinities while performing operations on create, start and exec operations.

As it was originally discussed in PR#1253, the existing implementation covers only exec usecase, where setting affinity for OCI hooks and initial container process will benefit wider set of workloads.

This change introduces more generic `CPUAffinity` property of `Process` to specify desired CPU affinities while performing operations on create, start and exec operations. Signed-off-by: Alexander Kanevskiy <alexander.kanevskiy@intel.com>

giuseppe · 2025-09-15T11:24:41Z

config.md

+    ranges. For example, `0-3,7` represents CPUs 0,1,2,3, and 7. If omitted or empty for particular [lifecycle](runtime.md#lifecycle) stage,
+    the runtime SHOULD NOT change process' CPU affinity, and the affinity is determined by the Linux kernel.
+    The following properties are available:
+    * **`runtimeCreate`** (string, OPTIONAL) is a list of CPUs the runtime parent process should run on during the runtime creation stage,


shouldn't this just be configured by the process that runs the OCI runtime so that more of the startup runs on the correct CPUs?

For better or worse, this might be non-trivial to do if someone is using a tool that wraps an OCI runtime and doesn't provide a knob for it (basically the same argument as runtime hooks).

I thin matching the lifecycle stages (which also match the hook names) as done here is probably the simplest approach.

theoretically it is possible, e.g. via systemd to restrict container runtime to run on subset of CPUs, but this gives granularity on splitting CPU affinity for containerd/cri-o itself vs. per-container hook execution. In other words, do not affect higher level runtime' CPU quota with operations that might be local to subset of containers.

regarding names: currently it is in form "whereWhat", but OCI hooks are named "whatWhere". if we feel that OCI hook naming is better, I can rename proposed fields in this PR to have more consistency with the hooks.

I think having the same names as the hooks would be preferable if only for consistency.

giuseppe · 2025-09-15T11:25:13Z

config.md

+    * **`containerCreate`** (string, OPTIONAL) is a list of CPUs the process should run on during the container creation stage,
+    after entering the container's namespaces but before the user process or any of [`createContainer` hooks](#createContainer-hooks) are executed.
+    * **`containerStart`** (string, OPTIONAL) is a list of CPUs the process should be shoud run on during the container start stage,
+    inside the container's namespace during `start` operation. The affinity should be applied before executing [`startContainer` hooks](#startContainer-hooks).


why do we need to differentiate between these two?

You could argue that the runtime should have a particular set of affinities which then should be adjusted to be more or less restrictive as late as possible to avoid latency on those cores.

I suspect most people wouldn't set both at the same time but the distinction does make some sense. Now, whether this should be done before or after applying seccomp rules is a different question...

why do we need to differentiate between these two?

the difference is that "runc create" and "runc start" might be executed in different PIDs, thus those PIDs by default would inherit CPU affinity from parent process(es). Having above in the spec allows to make sure that OCI hooks are executed (and quota accounted) on the correct "infrastructure" CPUs.

giuseppe · 2025-09-15T11:25:42Z

@kolyshkin PTAL

cyphar · 2025-09-15T11:37:31Z

Given issues like golang/sys#259, I think it might be nice to have a way to indicate "all CPUs" as a way to reset affinity (but without doing 0-1024, which is ~300x slower than the memset approach). In runc we implicitly do this now, but users might want to reset affinity for other stages explicitly.

kad · 2025-09-15T12:16:01Z

Given issues like golang/sys#259, I think it might be nice to have a way to indicate "all CPUs" as a way to reset affinity (but without doing 0-1024, which is ~300x slower than the memset approach). In runc we implicitly do this now, but users might want to reset affinity for other stages explicitly.

theoretically, the higher-level runtime that generates OCI spec might be feeling it with right number based on detected system information. e.g. use result of sched_getaffinity(2) of parent process

haircommander · 2025-09-15T15:13:02Z

fyi @bitoku

kolyshkin · 2025-09-15T23:18:21Z

specs-go/config.go

@@ -94,8 +94,12 @@ type Process struct {
 	SelinuxLabel string `json:"selinuxLabel,omitempty" platform:"linux"`
 	// IOPriority contains the I/O priority settings for the cgroup.
 	IOPriority *LinuxIOPriority `json:"ioPriority,omitempty" platform:"linux"`
+	// ExecCPUAffinity is Deprecated.


If you want various linters to warn about using deprecated functionality (and other linters, that insist that description starts with the Id, to keep working fine), the deprecation notice should be added like this:

// ExecCPUAffinity specifies CPU affinity for exec processes. // // Deprecated: use [CPUAffinity] instead.

(I.e. after the description, on a separate paragraph, etc.)

cyphar · 2025-09-18T02:58:02Z

@kad

theoretically, the higher-level runtime that generates OCI spec might be feeling it with right number based on detected system information. e.g. use result of sched_getaffinity(2) of parent process

My experience is that this rarely happens -- usually higher-level runtimes either hide new knobs like this (requiring you to specify patch config.json through experimental or unsupported hacks) or they transparently forward the values to the lower-level runtime without adding new functionality. Even if they do implement it, there is no guarantee that the behaviour or syntax will be standardised between runtimes, which leads to more problems than it solves.

Given that we have had seen practical issues with container runtimes being spawned with suboptimal CPU affinity values, I would suggest that having an "all" or "max" special value would be a good idea.

Also you do not need to detect anything with sched_getaffinity(2) for this -- you just need to memset(&cpuset, 0xFF, sizeof(cpuset)) to reset the affinity to the maximum possible value. In fact, you don't want to use sched_getaffinity(2) or Go's runtime.NumCpu because they give you the current affinity which is precisely the value you don't want.

marquiz · 2025-09-18T07:03:03Z

config.md

+    will be affinitized to specified list of CPUs.
+    * **`containerCreate`** (string, OPTIONAL) is a list of CPUs the process should run on during the container creation stage,
+    after entering the container's namespaces but before the user process or any of [`createContainer` hooks](#createContainer-hooks) are executed.
+    * **`containerStart`** (string, OPTIONAL) is a list of CPUs the process should be shoud run on during the container start stage,


Suggested change

* **`containerStart`** (string, OPTIONAL) is a list of CPUs the process should be shoud run on during the container start stage,

* **`containerStart`** (string, OPTIONAL) is a list of CPUs the process should be run on during the container start stage,

marquiz · 2025-09-18T07:04:54Z

config.md

-        "final": "0-3,7"
+    "CPUAffinity": {
+        "runtimeCreate": "7",
+        "runtimeExec": "7",


For completeness, should we add alsoi containerStart in this example?

cyphar

Also, when it comes to formatting -- in the OCI specs we do not hard-wrap lines at N columns. Each complete sentence should be one line (this makes diffs easier to read).

cyphar · 2025-09-18T07:13:52Z

config.md

+* **`CPUAffinity`** (object, OPTIONAL) specifies CPU affinity used to execute processes in the runtime and the container.
+    All the properties in this setting expects as argument a string that contain a comma-separated list, with dashes to represent
+    ranges. For example, `0-3,7` represents CPUs 0,1,2,3, and 7. If omitted or empty for particular [lifecycle](runtime.md#lifecycle) stage,
+    the runtime SHOULD NOT change process' CPU affinity, and the affinity is determined by the Linux kernel.


I think it would be very useful to have an all special value which indicates that the CPU affinity should be reset to include as many CPUs as possible.

I think "max" makes that clearer symantically

I'm okay with either, but I guess you could say that "max" implies that it might not be all.

kad requested review from AkihiroSuda, crosbymichael, cyphar, dqminh, giuseppe, hqhq, kolyshkin, mrunalp, thaJeztah, tianon and utam0k as code owners September 15, 2025 11:08

kad force-pushed the cpuaffinity branch from bbfa49f to d5b1802 Compare September 15, 2025 11:14

giuseppe reviewed Sep 15, 2025

View reviewed changes

kolyshkin reviewed Sep 15, 2025

View reviewed changes

marquiz reviewed Sep 18, 2025

View reviewed changes

cyphar requested changes Sep 18, 2025

View reviewed changes

	* `containerStart` (string, OPTIONAL) is a list of CPUs the process should be shoud run on during the container start stage,
	* `containerStart` (string, OPTIONAL) is a list of CPUs the process should be run on during the container start stage,

Introduce CPUAffinity process property instead of execCPUAffinity #1296

Are you sure you want to change the base?

Introduce CPUAffinity process property instead of execCPUAffinity #1296

Conversation

kad commented Sep 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Sep 15, 2025

Uh oh!

cyphar commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kad commented Sep 15, 2025

Uh oh!

haircommander commented Sep 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cyphar commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cyphar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cyphar commented Sep 15, 2025 •

edited

Loading

cyphar commented Sep 18, 2025 •

edited

Loading