Skip to content

Reconcile errors on EKS cluster with Hybrid nodes #4315

@sanbyk

Description

@sanbyk

Bug Description
Logs and TargetGroupBinding objects are flooded with errors when target pods are deployed on hybrid nodes.

Steps to Reproduce

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        eks.amazonaws.com/compute-type: hybrid
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  labels:
    name: nginx
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: nginx
            port:
              number: 80
---
  • Controller logs/error messages while reproducing the issue:
{"level":"error","ts":"2025-08-21T13:17:46Z","msg":"Requesting network requeue due to error from ReconcileForPodEndpoints","tgb":{"name":"k8s-default-nginx-1234567890","namespace":"default"},"error":"providerID eks-hybrid:///ap-northeast-1/eks-test/mi-01c49bb1234567890 is invalid for EC2 instances, node: mi-01c49bb1234567890"}
{"level":"info","ts":"2025-08-21T13:17:46Z","msg":"Requeue for networking requeue","tgb":{"name":"k8s-default-nginx-1234567890","namespace":"default"}}

Expected Behavior

ALB forwarding traffic to Nginx pod, no errors in controller logs, TargetGroupBinding resource is healthy,

Actual Behavior

It works, but controller logs and k8s events are flooded with reconcile errors.

Current Workarounds

Environment

  • AWS Load Balancer controller version: v2.13.3
  • Kubernetes version: 1.32
  • Using EKS (yes/no), if so version?: Yes, 1.32
  • Using Service or Ingress: Ingress

Possible Solution (Optional)

Hybrid nodes use different ProviderID and InstanceID format (mi-)
Hybrid node: eks-hybrid:///ap-northeast-1/eks-test/mi-03a43a81234567890
EC2 node: aws:///ap-northeast-1c/i-05c8de21234567890
So this check is not working:

var awsInstanceIDRegex = regexp.MustCompile("^i-[^/]*$")

Hybrid pods and nodes don't have ENIs, so the resolver should account for this.

Contribution Intention (Optional)

  • Yes, I'm willing to submit a PR to fix this issue
  • No, I cannot work on a PR at this time
    Additional Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions