Skip to content

Add beta stage test for NodeLocalCRISocket feature gate #3212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 19, 2025
+162 −84
Merged

Add beta stage test for NodeLocalCRISocket feature gate #3212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 81 additions & 42 deletions kinder/ci/tools/update-workflows/templates/workflows/instance-config-tasks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ tasks:
- --name={{ .vars.clusterName }}
- --loglevel=debug
- --kubeadm-verbosity={{ .vars.kubeadmVerbosity }}
- --kubeadm-feature-gate="NodeLocalCRISocket=true"
- --kubeadm-feature-gate="NodeLocalCRISocket=false"
- --copy-certs=auto
timeout: 5m
- name: join
Expand All @@ -84,16 +84,16 @@ tasks:
set -x

docker exec {{ .vars.clusterName }}-control-plane-1 bash -c '
nodes=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o jsonpath='{.items[*].metadata.name}')
nodes=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o jsonpath="{.items[*].metadata.name}")

for node in $nodes; do
annotation=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get node "$node" -o jsonpath="{.metadata.annotations['kubeadm\.alpha\.kubernetes\.io/cri-socket']}")
annotation=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get node "$node" -o jsonpath="{.metadata.annotations.kubeadm\.alpha\.kubernetes\.io\/cri-socket}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wasn't ['...'] the correct way to handle this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also quite surprised. I spent a lot of time interacting with it, and it's possible that our previous mistakes may have masked it, preventing it from throwing errors.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think both are supposed to work - accessing a key by using [....] and by using ., no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested

# kubectl get node kind-control-plane -o jsonpath="{.metadata.annotations['kubeadm\.alpha\.kubernetes\.io/cri-socket']}"
OUTPUT:
unix:///run/containerd/containerd.sock

# kubectl get node kind-control-plane  -o jsonpath="{.metadata.annotations.kubeadm\.alpha\.kubernetes\.io\/cri-socket}"
OUTPUT:
unix:///run/containerd/containerd.sock


if [ -z "$annotation" ]; then
echo "Annotation not found on node $node"
exit 1
else
echo "Annotation found on node $node: $annotation"
exit 1
fi
done
' || {
Expand All @@ -118,8 +118,35 @@ tasks:
${CMD} bash -c '
if [[ -f "/var/lib/kubelet/instance-config.yaml" ]] && grep -q "containerRuntimeEndpoint:" /var/lib/kubelet/instance-config.yaml; then
echo "File exists and contains 'containerRuntimeEndpoint:' field."
exit 1
else
echo "File does not exist or does not contain 'containerRuntimeEndpoint:' field."
fi
' || {
echo "Command failed for $node. Exiting.";
exit 1;
}
done

exit 0
timeout: 5m
- name: check-flag
description: |
Check that the container-runtime-endpoint flag does not exist on each node.
cmd: /bin/bash
args:
- -c
- |
set -x

for node in control-plane-{1..3} worker-{1..2}; do
CMD="docker exec {{ .vars.clusterName }}-$node"

${CMD} bash -c '
if [[ -f "/var/lib/kubelet/kubeadm-flags.env" ]] && grep -q "container-runtime-endpoint" /var/lib/kubelet/kubeadm-flags.env; then
echo "File exists and contains 'container-runtime-endpoint' field."
else
echo "File does not contain 'container-runtime-endpoint' field."
exit 1
fi
' || {
Expand Down Expand Up @@ -157,9 +184,9 @@ tasks:
${CMD} bash -c '
if [[ -f "/var/lib/kubelet/instance-config.yaml" ]] && grep -q "containerRuntimeEndpoint:" /var/lib/kubelet/instance-config.yaml; then
echo "File exists and contains 'containerRuntimeEndpoint:' field."
exit 1
else
echo "File does not exist or does not contain 'containerRuntimeEndpoint:' field."
exit 1
fi
' || {
echo "Command failed for $node. Exiting.";
Expand All @@ -169,9 +196,9 @@ tasks:

exit 0
timeout: 5m
- name: remove-config
- name: check-flag-after-upgrade
description: |
Remove the kubelet instance configuration from work node.
Check that the container-runtime-endpoint flag does not exist on each node.
cmd: /bin/bash
args:
- -c
Expand All @@ -181,36 +208,41 @@ tasks:
for node in control-plane-{1..3} worker-{1..2}; do
CMD="docker exec {{ .vars.clusterName }}-$node"

${CMD} bash -c 'rm -f /var/lib/kubelet/instance-config.yaml' || {
${CMD} bash -c '
if [[ -f "/var/lib/kubelet/kubeadm-flags.env" ]] && grep -q "container-runtime-endpoint" /var/lib/kubelet/kubeadm-flags.env; then
echo "File exists and contains 'container-runtime-endpoint' field."
else
echo "File does not contain 'container-runtime-endpoint' field."
exit 1
fi
' || {
echo "Command failed for $node. Exiting.";
exit 1;
}
done

exit 0
timeout: 5m
- name: add-cri-socket-annotation
- name: check-annotation-after-upgrade
description: |
Add the cri-socket annotation to each node.
Check that the cri-socket annotation does not exist on each node.
cmd: /bin/bash
args:
- -c
- |
set -x

docker exec {{ .vars.clusterName }}-control-plane-1 bash -c '
ANNOTATION_KEY="kubeadm.alpha.kubernetes.io/cri-socket"
ANNOTATION_VALUE="unix:///var/run/containerd/containerd.sock"

nodes=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o jsonpath='{.items[*].metadata.name}')
nodes=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o jsonpath="{.items[*].metadata.name}")

for node in $nodes; do
KUBECONFIG=/etc/kubernetes/admin.conf kubectl annotate node "$node" "$ANNOTATION_KEY=$ANNOTATION_VALUE" --overwrite
if [ $? -eq 0 ]; then
echo "Successfully added annotation to node $node."
else
echo "Failed to add annotation to node $node."
annotation=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get node "$node" -o jsonpath="{.metadata.annotations.kubeadm\.alpha\.kubernetes\.io\/cri-socket}")

if [ -z "$annotation" ]; then
echo "Annotation not found on node $node"
exit 1
else
echo "Annotation found on node $node: $annotation"
fi
done
' || {
Expand All @@ -220,33 +252,39 @@ tasks:

exit 0
timeout: 5m
- name: check-and-add-flag

- name: enable-feature
description: |
Check and add the container-runtime-endpoint flag to each node's kubeadm-flags.env file
enable the nodeLocalCRISocket Feature gate.
cmd: /bin/bash
args:
- -c
- |
set -x

for node in control-plane-{1..3} worker-{1..2}; do
CMD="docker exec {{ .vars.clusterName }}-$node"

${CMD} bash -c '
FILE="/var/lib/kubelet/kubeadm-flags.env"
NEW_ARG="--container-runtime-endpoint=unix:///run/containerd/containerd.sock"

if grep -q "container-runtime-endpoint" "$FILE"; then
echo "flag container-runtime-endpoint already exists"
exit 1
fi

sed -i "/^KUBELET_KUBEADM_ARGS=/ s|\"$| $NEW_ARG\"|" "$FILE"
' || {
echo "Command failed for $node. Exiting.";
exit 1;
}
done
docker exec {{ .vars.clusterName }}-control-plane-1 bash -c '
KUBECONFIG=/etc/kubernetes/admin.conf kubectl get configmap kubeadm-config -n kube-system -o yaml > config.yaml
sed '\''s/^\( *\)\(NodeLocalCRISocket:\) false$/\1\2 true/'\'' config.yaml > updated_config.yaml

if grep -q "NodeLocalCRISocket: true" updated_config.yaml; then
echo "Updating ConfigMap YAML Successful."
else
echo "Updating ConfigMap YAML failed."
exit 1
fi

if KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f updated_config.yaml; then
echo "ConfigMap updated successfully."
else
echo "Failed to apply updated ConfigMap."
exit 1
fi

rm config.yaml updated_config.yaml
' || {
echo "Command failed for $node. Exiting.";
exit 1;
}

exit 0
timeout: 5m
Expand Down Expand Up @@ -299,10 +337,10 @@ tasks:
set -x

docker exec {{ .vars.clusterName }}-control-plane-1 bash -c '
nodes=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o jsonpath='{.items[*].metadata.name}')
nodes=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes -o jsonpath="{.items[*].metadata.name}")

for node in $nodes; do
annotation=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get node "$node" -o jsonpath="{.metadata.annotations['kubeadm\.alpha\.kubernetes\.io/cri-socket']}")
annotation=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get node "$node" -o jsonpath="{.metadata.annotations.kubeadm\.alpha\.kubernetes\.io\/cri-socket}")

if [ -z "$annotation" ]; then
echo "Annotation not found on node $node"
Expand All @@ -315,7 +353,6 @@ tasks:
echo "Command failed for $node. Exiting.";
exit 1;
}

exit 0
timeout: 5m
- name: check-flag-after-second-upgrade
Expand All @@ -334,6 +371,8 @@ tasks:
if [[ -f "/var/lib/kubelet/kubeadm-flags.env" ]] && grep -q "container-runtime-endpoint" /var/lib/kubelet/kubeadm-flags.env; then
echo "File exists and contains 'container-runtime-endpoint' field."
exit 1
else
echo "File does not contain 'container-runtime-endpoint' field."
fi
' || {
echo "Command failed for $node. Exiting.";
Expand Down
Loading