diff --git a/keps/prod-readiness/sig-scheduling/3094.yaml b/keps/prod-readiness/sig-scheduling/3094.yaml new file mode 100644 index 00000000000..5fee9089946 --- /dev/null +++ b/keps/prod-readiness/sig-scheduling/3094.yaml @@ -0,0 +1,3 @@ +kep-number: 3094 +alpha: + approver: "@wojtek-t" diff --git a/keps/sig-scheduling/3094-pod-topology-spread-considering-taints/README.md b/keps/sig-scheduling/3094-pod-topology-spread-considering-taints/README.md new file mode 100644 index 00000000000..3570de522c0 --- /dev/null +++ b/keps/sig-scheduling/3094-pod-topology-spread-considering-taints/README.md @@ -0,0 +1,837 @@ + +# KEP-3094: Take taints/tolerations into consideration when calculating PodTopologySpread skew + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [x] (R) KEP approvers have approved the KEP status as `implementable` +- [x] (R) Design details are appropriately documented +- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests for meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [x] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [x] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + +This KEP introduces an option for end-users to specify whether to take taints/tolerations +into consideration when calculating pod topology spread skew. + +## Motivation + + +Currently when calculating pod topology spread skew, tainted nodes are treated the same as +other regular nodes. This behavior may lead to unexpected Pending pods as the skew constraint +can only be satisfied on the tainted nodes.(See [issue](https://github.com/kubernetes/kubernetes/issues/106127)). + +Besides, given that we have already some node inclusion policies(nodeAffinity/nodeSelector) +plumbed into PodTopologySpread implicitly, we'd like to use this chance +to use a new API to represent the semantics explicitly. + +### Goals + + + +- Introduce a new struct to define all node inclusion policies explicitly +- Provide an option for end-users to specify whether to respect taints or not + +### Non-Goals + + +- Support customized taints in array + +## Proposal + + +Introduce a new field to `TopologySpreadConstraint` to define all node inclusion +policies including nodeAffinity and nodeTaint. + +### User Stories (Optional) + + + +#### Story 1 +When calculating pod topology spread skew, I want to exclude nodes that don't tolerate +all taints to prevent pods from falling into unexpected Pending state. + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + +- Checking nodeAffinity and nodeTaints the same time may lead to performance +problem, we need to verify this by adding performance tests. If performance problem +does exists, we'd like to add a toleration parser and cache the parsed object +during PreFilter. + +## Design Details + + + +A new field named `NodeInclusionPolicies` will be introduced to `TopologySpreadConstraint`: +```golang +type TopologySpreadConstraint struct { + // NodeInclusionPolicies includes several policies + // when calculating pod topology spread skew + NodeInclusionPolicies NodeInclusionPolicies +} +``` + +There are two policies now regarding to nodeAffinity and nodeTaint: +```golang +type NodeInclusionPolicies struct { + // NodeAffinity policy indicates how we will treat nodeAffinity/nodeSelector + // when calculating pod topology spread skew. The semantics vary by PolicyName: + // - Respect (default): nodes matching nodeAffinity/nodeSelector will be included. + // - Ignore: all nodes will be included. + NodeAffinity PolicyName + // NodeTaint policy indicates how we will treat node taints + // when calculating pod topology spread skew. The semantics vary by PolicyName: + // - Respect: tainted nodes that tolerate the incoming pod, along with regular nodes, will be included. + // - Ignore (default): all nodes will be included. + NodeTaint PolicyName +} +``` + +We will define two policyNames by default: +```golang +type PolicyName string + +const ( + // Ignore means ignore this policy in calculating. + Ignore PolicyName = "Ignore" + // Respect means use this policy in calculating. + Respect PolicyName = "Respect" +) +``` + +We will check these policies in the extension points of `PreFilter/PreScore` +in PodTopologySpread plugin, some refactoring works are also needed, but we will +not change the default behavior. + +### Test Plan + + +- Unit and integration tests: + - Defaulting and validation tests + - Feature gate enable/disable tests + - `NodeInclusionPolicies` works as expected +- Benchmark tests: + - Verify the performance of looping all toleration and taints in calculating skew is acceptable + +### Graduation Criteria + + +#### Alpha +- Feature implemented behind feature gate. +- Unit and integration tests passed as designed in [TestPlan](#test-plan). + +#### Beta +- Feature is enabled by default +- Benchmark tests passed, and there is no performance problem. +- Gather feedback from developers. + +#### GA +- No negative feedback. + +### Upgrade / Downgrade Strategy + + + +- Upgrade + - While the feature gate is enabled, `NodeInclusionPolicies` is allowed to use by end-users. + - While the feature gate is enabled, and we don't set this field, default values + will be configured, which will maintain previous behavior. +- Downgrade + - Previously configured values will be ignored. + +### Version Skew Strategy +N/A + + + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [x] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: PodTopologySpreadNodePolicies + - Components depending on the feature gate: kube-scheduler, kube-apiserver + +###### Does enabling the feature change any default behavior? + + +No. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + +Yes. + +###### What happens if we reenable the feature if it was previously rolled back? +The policies are respected again. + +###### Are there any tests for feature enablement/disablement? +No, appropriate unit tests will be added for Alpha. + + + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + +It's an opt-in feature for end-users and will maintain current behaviors if not set, so +it will not impact the running workloads. + +###### What specific metrics should inform a rollback? + + +- A spike on metric schedule_attempts_total{result="error|unschedulable"} when pods using this feature are added. +- Metric plugin_execution_duration_seconds{plugin="PodTopologySpread"} larger than 100ms on 90-percentile. +- A spike on failure events with keyword "failed spreadConstraint" in scheduler log. + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? +No. This will be tested upon beta graduation. + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? +No + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + +Operator can query `pod.spec.topologySpreadConstraints[].NodeInclusionPolicies` field +and identify if this is set to non-default values + +###### How can someone using this feature know that it is working for their instance? + + + + +N/A + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +- Metric plugin_execution_duration_seconds{plugin="PodTopologySpread"} <= 100ms on 90-percentile. +- Frequency of critical error keywords <= 2 times per minute. +- Frequency of regular scheduling failures < 10 times per minute. + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + + + +- Metric plugin_execution_duration_seconds{plugin="PodTopologySpread"} to indicate the scheduling latency for a pod using this feature. +- Frequency of critical error keywords in scheduler log: + - "PreFilterPodTopologySpread" + - "convert to podtopologyspread.preFilterState error" + - "hard topology spread constraints" + - "internal error: get paths from key" +- Frequency of regular scheduling failures (with keyword "failed spreadConstraint") in scheduler log. + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + +N/A + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + +No + +### Scalability + + +No + +###### Will enabling / using this feature result in any new API calls? + + +No + +###### Will enabling / using this feature result in introducing new API types? + + +No + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + +No + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + +No + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + +No + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + +No + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? +N/A + +###### What are other known failure modes? + + + +Configuration errors are logged to stderr. + +###### What steps should be taken if SLOs are not being met to determine the problem? +N/A + +## Implementation History + + + +- 2021.01.12: KEP proposed for review, including motivation, proposal, risks, +test plan and graduation criteria. + +## Drawbacks + + +N/A + +## Alternatives + + +- The community has discussed about changing the current behavior implicitly, +but considering this will introduce a break user-facing change, for backwards +compatibility, we decided to add a feature as switch for end-users. +- We have also discussed about whether to support specific taints, but considering +there's no strong demands from end-users, we will delay this until needed. + +## Infrastructure Needed (Optional) + + +N/A diff --git a/keps/sig-scheduling/3094-pod-topology-spread-considering-taints/kep.yaml b/keps/sig-scheduling/3094-pod-topology-spread-considering-taints/kep.yaml new file mode 100644 index 00000000000..497028fc7fe --- /dev/null +++ b/keps/sig-scheduling/3094-pod-topology-spread-considering-taints/kep.yaml @@ -0,0 +1,49 @@ +title: Take taints/tolerations into consideration when calculating PodTopologySpread skew +kep-number: 3094 +authors: + - "@kerthcet" +owning-sig: sig-scheduling +participating-sigs: +status: implementable +creation-date: 2021-12-30 +reviewers: + - "@alculquicondor" + - "@Huang-Wei" +approvers: + - "@alculquicondor" + - "@Huang-Wei" + +see-also: + - "/keps/sig-scheduling/895-pod-topology-spread" + - "/keps/sig-scheduling/1258-default-pod-topology-spread" + +##### WARNING !!! ###### +# prr-approvers has been moved to its own location +# You should create your own in keps/prod-readiness +# Please make a copy of keps/prod-readiness/template/nnnn.yaml +# to keps/prod-readiness/sig-xxxxx/00000.yaml (replace with kep number) +#prr-approvers: + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.24" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.24" + beta: "v1.25" + stable: "" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: PodTopologySpreadNodePolicies + components: + - kube-apiserver + - kube-scheduler + +disable-supported: false