AKO pod reconciliation loop interferes with cluster autoscaling

[rodrigorfk]

Problem

The Aerospike Kubernetes Operator (AKO) repeatedly deletes and recreates pods flagged as Unschedulable by the Kubernetes scheduler. This behavior prevents cluster autoscalers, such as Karpenter, from provisioning new nodes, leading to an infinite loop of pod creation and termination.

Context

In a cost-efficient cluster design, nodes are provisioned on demand rather than pre-provisioned as warm spare capacity. Cluster autoscalers like Karpenter or the Cluster Autoscaler are responsible for adding new nodes when they detect pods in a pending state. Karpenter, for example, typically provisions new nodes only after the Kubernetes scheduler marks a pod as Unschedulable, allowing a configurable --batch-idle-duration to consolidate multiple pending pods before acting.

Current Behavior

The AKO's reconciliation logic proactively deletes Aerospike pods almost immediately after the Kubernetes scheduler marks them as Unschedulable. This happens before a cluster autoscaler has sufficient time to observe the pending pod and provision new capacity. As a result, the autoscaler never sees a stable Unschedulable pod to trigger a scale-up event. This creates a destructive feedback loop where AKO continuously deletes pods, which are then immediately recreated by the StatefulSet controller, only to be deleted again by AKO.

Details

When creating a new Aerospike clusters, pods are created and deleted endlessly as following:

NAME               READY   STATUS    RESTARTS   AGE
firstcluster-0-0   0/1     Pending   0          0s
firstcluster-0-0   0/1     Pending   0          0s
firstcluster-0-2   0/1     Pending   0          0s
firstcluster-0-1   0/1     Pending   0          0s
firstcluster-0-2   0/1     Pending   0          0s
firstcluster-0-1   0/1     Pending   0          0s
firstcluster-0-2   0/1     Terminating   0          4s
firstcluster-0-2   0/1     Terminating   0          4s
firstcluster-0-1   0/1     Terminating   0          4s
firstcluster-0-1   0/1     Terminating   0          4s
firstcluster-0-0   0/1     Terminating   0          4s
firstcluster-0-0   0/1     Terminating   0          4s
firstcluster-0-2   0/1     Pending       0          0s
firstcluster-0-2   0/1     Pending       0          0s
firstcluster-0-0   0/1     Pending       0          0s
firstcluster-0-0   0/1     Pending       0          0s
firstcluster-0-1   0/1     Pending       0          0s
firstcluster-0-1   0/1     Pending       0          0s
firstcluster-0-2   0/1     Terminating   0          0s
firstcluster-0-2   0/1     Terminating   0          0s
firstcluster-0-1   0/1     Terminating   0          0s
firstcluster-0-1   0/1     Terminating   0          0s
firstcluster-0-0   0/1     Terminating   0          0s
firstcluster-0-0   0/1     Terminating   0          0s
firstcluster-0-2   0/1     Pending       0          0s
firstcluster-0-2   0/1     Pending       0          0s
firstcluster-0-0   0/1     Pending       0          0s
firstcluster-0-0   0/1     Pending       0          0s
firstcluster-0-1   0/1     Pending       0          0s
firstcluster-0-1   0/1     Pending       0          0s

In the AKO logs, you will find messages like the following:

2025-09-01T09:23:13Z	DEBUG	controller.AerospikeCluster	Check statefulSet pod running and ready	{"aerospikecluster": {"name":"firstcluster","namespace":"aerospike"}, "pod": "firstcluster-0-0"}
2025-09-01T09:23:13Z	ERROR	controller.AerospikeCluster	Failed to wait for statefulset to be ready	{"aerospikecluster": {"name":"firstcluster","namespace":"aerospike"}, "STS": {"name":"firstcluster-0","namespace":"aerospike"}, "error": "statefulSet pod firstcluster-0-0 failed: pod firstcluster-0-0 is in unschedulable state and reason is 0/23 nodes are available: 20 node(s) didn't match Pod's node affinity/selector. preemption: 0/23 nodes are available: 23 Preemption is not helpful for scheduling."}

Expected Behavior

AKO should not immediately delete Unschedulable pods. Instead, it should wait for a configurable timeout period. This would allow a cluster autoscaler to observe the pending pods and provision the necessary node capacity. Once new nodes are available, the pods can be scheduled successfully.

[abhishekdwivedi3060]

Hi @rodrigorfk, the current AKO implementation attempts to recover pods that fail or remain pending for any reason (e.g., failed state, scheduling errors, etc.). In the case of a fresh cluster deployment, if all pods end up in a pending or failed state, it is safe to re-create the entire cluster since no data exists yet. That seems to be what’s happening in your case.

In short, AKO currently assumes that the underlying infrastructure is available to provision an AerospikeCluster. I’ll check internally to see how we can improve the handling of this scenario.

[rodrigorfk]

Hey @abhishekdwivedi3060, thank you for providing that context. I understand the rationale behind the operator's current behavior, especially for new clusters where data persistence isn't a concern. However, I'd like to provide more detail on why this specific behavior, in the context of Unschedulable pods, is problematic for common Kubernetes operations.

From a Kubernetes platform perspective, the pod lifecycle for a lack of resources is very well-defined:

• A Pod enters a Pending state.
• The kube-scheduler attempts to find a suitable node.
• If no node can be found due to resource constraints, the scheduler adds a Unschedulable condition to the pod and continues retrying every few seconds on a schedule.

Cluster autoscalers like Karpenter or Cluster Autoscaler are specifically designed to monitor these Unschedulable events. They rely on the pod remaining in this state for a brief period to detect the need for a new node and provision it accordingly.

AKO's current action of deleting the pod breaks this standard workflow. Deleting and recreating the pod doesn't change the underlying scheduling problem; it only restarts the process. This creates an endless cycle and, as described above, prevents the autoscaler from ever initiating a scale-up event.

Beyond the technical conflict, this behavior also creates significant operational challenges for cluster administrators:

• Hard Troubleshooting: Deleting the pod removes all the standard Kubernetes events and information that an administrator would use to troubleshoot a scheduling issue. This forces them to dive into the operator's specific logs and understand its unique logic, which goes against the principle of using standard Kubernetes primitives.
• Lack of Cluster Scalability: In our specific use case, this issue is critical because it makes it impossible to provision new Aerospike clusters without manual intervention to scale the node group first. The operator's behavior essentially blocks a common and critical part of automated cluster management.

I believe that for Unschedulable events, a more resilient approach would be for AKO to either back off and wait for the cluster to scale or, at a minimum, not actively intervene. This would allow the built-in Kubernetes scheduler and autoscaler to handle the situation as they were designed to, making AKO more compatible with a broader range of cluster environments.