Skip to content

Commit 0612509

Browse files
venkatscvenkatsc
authored
Cleanup shared volume with rm command (#103)
* Add separate node selectors .. for csi driver pods and pod killer cache. * Bump CSI sidecar containers to recent versions ..update snapshot definitions (still need some more fine tuning - integrate into csi-driver-templates, refine roles rbacs and tie to csi service account, fix leader election etc) * Bump helm unittest file * Option to disable submission of delete files task .. to remove PV(C) directory from the shared volume. When disabled, rm -rf is used for cleanup via client mount point. * Bump configuration tests to latest modifyications * Fix typos and cleanup .. further move Quobyte version to 3 and 4 * Improve shared volume configuration plus configuration documentation. * Driver configuration Stringer * Fix helm tests
1 parent 73256a4 commit 0612509

File tree

16 files changed

+498
-345
lines changed

16 files changed

+498
-345
lines changed

‎csi-driver-templates/k8s-snapshot-controller.yaml‎

Lines changed: 3 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# https://github.com/kubernetes-csi/external-snapshotter/blob/v7.0.2/deploy/kubernetes/snapshot-controller
1+
# https://github.com/kubernetes-csi/external-snapshotter/blob/v8.3.0/deploy/kubernetes/snapshot-controller
22
---
33
apiVersion: v1
44
kind: ServiceAccount
@@ -32,7 +32,7 @@ rules:
3232
verbs: ["patch"]
3333
- apiGroups: ["snapshot.storage.k8s.io"]
3434
resources: ["volumesnapshots"]
35-
verbs: ["get", "list", "watch", "update", "patch", "delete"]
35+
verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
3636
- apiGroups: ["snapshot.storage.k8s.io"]
3737
resources: ["volumesnapshots/status"]
3838
verbs: ["update", "patch"]
@@ -82,20 +82,6 @@ rules:
8282
resources: ["leases"]
8383
verbs: ["get", "watch", "list", "delete", "update", "create"]
8484

85-
---
86-
kind: RoleBinding
87-
apiVersion: rbac.authorization.k8s.io/v1
88-
metadata:
89-
name: snapshot-controller-leaderelection
90-
namespace: kube-system
91-
subjects:
92-
- kind: ServiceAccount
93-
name: snapshot-controller
94-
roleRef:
95-
kind: Role
96-
name: snapshot-controller-leaderelection
97-
apiGroup: rbac.authorization.k8s.io
98-
9985
---
10086
kind: Deployment
10187
apiVersion: apps/v1
@@ -125,7 +111,7 @@ spec:
125111
serviceAccountName: snapshot-controller
126112
containers:
127113
- name: snapshot-controller
128-
image: registry.k8s.io/sig-storage/snapshot-controller:v6.3.1
114+
image: registry.k8s.io/sig-storage/snapshot-controller:v8.2.1
129115
args:
130116
- "--v=5"
131117
- "--leader-election=true"

‎csi-driver-templates/k8s-snapshot-crd.yaml‎

Lines changed: 323 additions & 226 deletions
Large diffs are not rendered by default.

‎csi-driver-templates/templates/pods/_quobyte_csi_controller_pod.tpl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,9 @@ spec:
2626
app: quobyte-csi-controller-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
2727
role: quobyte-csi-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
2828
spec:
29-
{{- if default "" .Values.quobyte.nodeSelector | trim }}
29+
{{- if default "" .Values.quobyte.csiDriverNodeSelector | trim }}
3030
nodeSelector:
31-
{{ .Values.quobyte.nodeSelector | trim }}
31+
{{ .Values.quobyte.csiDriverNodeSelector | trim }}
3232
{{- end }}
3333
priorityClassName: system-cluster-critical
3434
serviceAccount: quobyte-csi-controller-sa-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}

‎csi-driver-templates/templates/pods/_quobyte_csi_node_driver_pod.tpl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@ spec:
1919
app: quobyte-csi-node-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
2020
role: quobyte-csi
2121
spec:
22-
{{- if default "" .Values.quobyte.nodeSelector | trim }}
22+
{{- if default "" .Values.quobyte.csiDriverNodeSelector | trim }}
2323
nodeSelector:
24-
{{ .Values.quobyte.nodeSelector | trim }}
24+
{{ .Values.quobyte.csiDriverNodeSelector | trim }}
2525
{{- end }}
2626
priorityClassName: system-node-critical
2727
serviceAccount: quobyte-csi-node-sa-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}

‎csi-driver-templates/templates/pods/_quobyte_csi_pod_killer_cache.tpl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,9 @@ spec:
2121
app: quobyte-csi-pod-killer-cache-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
2222
role: quobyte-csi-pod-killer-cache-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
2323
spec:
24-
{{- if default "" .Values.quobyte.nodeSelector | trim }}
24+
{{- if default "" .Values.quobyte.podKillerCacheNodeSelector | trim }}
2525
nodeSelector:
26-
{{ .Values.quobyte.nodeSelector | trim }}
26+
{{ .Values.quobyte.podKillerCacheNodeSelector | trim }}
2727
{{- end }}
2828
priorityClassName: system-cluster-critical
2929
serviceAccount: quobyte-csi-pod-killer-cache-sa-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}

‎csi-driver-templates/templates/pods/containers/_quobyte_csi_controller_container.tpl‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@
2424
- "--quobyte_version={{ .Values.quobyte.version }}"
2525
- "--immediate_erase={{ .Values.quobyte.immediateErase }}"
2626
- "--use_k8s_namespace_as_tenant={{ .Values.quobyte.useK8SNamespaceAsTenant }}"
27-
- "--shared_volumes_list={{ .Values.quobyte.sharedVolumesList }}"
27+
- "--shared_volumes_list={{ join "," .Values.quobyte.sharedVolumesList }}"
28+
- "--use_delete_files_task={{ .Values.quobyte.useDeleteFilesTaskForSharedVolumeCleanup }}"
2829
- "--role=controller"
2930
env:
3031
- name: NODE_ID

‎csi-driver-templates/templates/pods/rbac/_sidecar_snapshotter_rbac.tpl‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,35 @@ roleRef:
4343
kind: ClusterRole
4444
name: external-snapshotter-runner-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
4545
apiGroup: rbac.authorization.k8s.io
46+
4647
---
48+
{{- if gt (.Values.quobyte.csiControllerReplicas | toString | atoi) 1 }}
49+
kind: Role
50+
apiVersion: rbac.authorization.k8s.io/v1
51+
metadata:
52+
namespace: kube-system
53+
name: external-snapshotter-leaderelection-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
54+
rules:
55+
- apiGroups: ["coordination.k8s.io"]
56+
resources: ["leases"]
57+
verbs: ["get", "watch", "list", "delete", "update", "create"]
58+
{{- end }}
59+
60+
---
61+
{{- if gt (.Values.quobyte.csiControllerReplicas | toString | atoi) 1 }}
62+
kind: RoleBinding
63+
apiVersion: rbac.authorization.k8s.io/v1
64+
metadata:
65+
name: external-snapshotter-leaderelection-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
66+
namespace: kube-system
67+
subjects:
68+
- kind: ServiceAccount
69+
name: quobyte-csi-controller-sa-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
70+
namespace: kube-system
71+
roleRef:
72+
kind: Role
73+
name: external-snapshotter-leaderelection-{{ .Values.quobyte.csiProvisionerName | replace "." "-" }}
74+
apiGroup: rbac.authorization.k8s.io
75+
{{- end }}
4776
{{- end }}
4877
{{- end }}

‎csi-driver-templates/tests/__snapshot__/csi_driver_test.yaml.snap‎

Lines changed: 22 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -302,7 +302,7 @@ should render when resource limits are provided:
302302
env:
303303
- name: ADDRESS
304304
value: /var/lib/csi/sockets/pluginproxy/csi.sock
305-
image: registry.k8s.io/sig-storage/csi-provisioner:v4.0.1
305+
image: registry.k8s.io/sig-storage/csi-provisioner:v5.3.0
306306
imagePullPolicy: IfNotPresent
307307
name: csi-provisioner
308308
resources:
@@ -318,7 +318,7 @@ should render when resource limits are provided:
318318
env:
319319
- name: ADDRESS
320320
value: /var/lib/csi/sockets/pluginproxy/csi.sock
321-
image: registry.k8s.io/sig-storage/csi-resizer:v1.8.1
321+
image: registry.k8s.io/sig-storage/csi-resizer:v1.14.0
322322
imagePullPolicy: IfNotPresent
323323
name: csi-resizer
324324
resources:
@@ -334,7 +334,7 @@ should render when resource limits are provided:
334334
env:
335335
- name: ADDRESS
336336
value: /var/lib/csi/sockets/pluginproxy/csi.sock
337-
image: registry.k8s.io/sig-storage/csi-attacher:v4.5.1
337+
image: registry.k8s.io/sig-storage/csi-attacher:v4.9.0
338338
imagePullPolicy: IfNotPresent
339339
name: csi-attacher
340340
resources:
@@ -350,7 +350,7 @@ should render when resource limits are provided:
350350
env:
351351
- name: ADDRESS
352352
value: /var/lib/csi/sockets/pluginproxy/csi.sock
353-
image: registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2
353+
image: registry.k8s.io/sig-storage/csi-snapshotter:v8.3.0
354354
imagePullPolicy: IfNotPresent
355355
name: csi-snapshotter
356356
resources:
@@ -368,10 +368,11 @@ should render when resource limits are provided:
368368
- --driver_name=csi.quobyte.com
369369
- --driver_version=v2.1.6
370370
- --enable_access_key_mounts=false
371-
- --quobyte_version=3
371+
- --quobyte_version=4
372372
- --immediate_erase=false
373373
- --use_k8s_namespace_as_tenant=false
374374
- --shared_volumes_list=
375+
- --use_delete_files_task=true
375376
- --role=controller
376377
env:
377378
- name: NODE_ID
@@ -514,7 +515,7 @@ should render when resource limits are provided:
514515
valueFrom:
515516
fieldRef:
516517
fieldPath: spec.nodeName
517-
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
518+
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.14.0
518519
imagePullPolicy: IfNotPresent
519520
lifecycle:
520521
preStop:
@@ -541,7 +542,7 @@ should render when resource limits are provided:
541542
- --driver_name=csi.quobyte.com
542543
- --driver_version=v2.1.6
543544
- --enable_access_key_mounts=false
544-
- --quobyte_version=3
545+
- --quobyte_version=4
545546
- --immediate_erase=false
546547
- --use_k8s_namespace_as_tenant=false
547548
- --enable_volume_metrics=true
@@ -988,7 +989,7 @@ should render when tolerations are provided:
988989
env:
989990
- name: ADDRESS
990991
value: /var/lib/csi/sockets/pluginproxy/csi.sock
991-
image: registry.k8s.io/sig-storage/csi-provisioner:v4.0.1
992+
image: registry.k8s.io/sig-storage/csi-provisioner:v5.3.0
992993
imagePullPolicy: IfNotPresent
993994
name: csi-provisioner
994995
volumeMounts:
@@ -1000,7 +1001,7 @@ should render when tolerations are provided:
10001001
env:
10011002
- name: ADDRESS
10021003
value: /var/lib/csi/sockets/pluginproxy/csi.sock
1003-
image: registry.k8s.io/sig-storage/csi-resizer:v1.8.1
1004+
image: registry.k8s.io/sig-storage/csi-resizer:v1.14.0
10041005
imagePullPolicy: IfNotPresent
10051006
name: csi-resizer
10061007
volumeMounts:
@@ -1012,7 +1013,7 @@ should render when tolerations are provided:
10121013
env:
10131014
- name: ADDRESS
10141015
value: /var/lib/csi/sockets/pluginproxy/csi.sock
1015-
image: registry.k8s.io/sig-storage/csi-attacher:v4.5.1
1016+
image: registry.k8s.io/sig-storage/csi-attacher:v4.9.0
10161017
imagePullPolicy: IfNotPresent
10171018
name: csi-attacher
10181019
volumeMounts:
@@ -1026,10 +1027,11 @@ should render when tolerations are provided:
10261027
- --driver_name=csi.quobyte.com
10271028
- --driver_version=v2.1.6
10281029
- --enable_access_key_mounts=false
1029-
- --quobyte_version=3
1030+
- --quobyte_version=4
10301031
- --immediate_erase=false
10311032
- --use_k8s_namespace_as_tenant=false
10321033
- --shared_volumes_list=
1034+
- --use_delete_files_task=true
10331035
- --role=controller
10341036
env:
10351037
- name: NODE_ID
@@ -1172,7 +1174,7 @@ should render when tolerations are provided:
11721174
valueFrom:
11731175
fieldRef:
11741176
fieldPath: spec.nodeName
1175-
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
1177+
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.14.0
11761178
imagePullPolicy: IfNotPresent
11771179
lifecycle:
11781180
preStop:
@@ -1195,7 +1197,7 @@ should render when tolerations are provided:
11951197
- --driver_name=csi.quobyte.com
11961198
- --driver_version=v2.1.6
11971199
- --enable_access_key_mounts=false
1198-
- --quobyte_version=3
1200+
- --quobyte_version=4
11991201
- --immediate_erase=false
12001202
- --use_k8s_namespace_as_tenant=false
12011203
- --enable_volume_metrics=true
@@ -1638,7 +1640,7 @@ should render with default values:
16381640
env:
16391641
- name: ADDRESS
16401642
value: /var/lib/csi/sockets/pluginproxy/csi.sock
1641-
image: registry.k8s.io/sig-storage/csi-provisioner:v4.0.1
1643+
image: registry.k8s.io/sig-storage/csi-provisioner:v5.3.0
16421644
imagePullPolicy: IfNotPresent
16431645
name: csi-provisioner
16441646
volumeMounts:
@@ -1650,7 +1652,7 @@ should render with default values:
16501652
env:
16511653
- name: ADDRESS
16521654
value: /var/lib/csi/sockets/pluginproxy/csi.sock
1653-
image: registry.k8s.io/sig-storage/csi-resizer:v1.8.1
1655+
image: registry.k8s.io/sig-storage/csi-resizer:v1.14.0
16541656
imagePullPolicy: IfNotPresent
16551657
name: csi-resizer
16561658
volumeMounts:
@@ -1662,7 +1664,7 @@ should render with default values:
16621664
env:
16631665
- name: ADDRESS
16641666
value: /var/lib/csi/sockets/pluginproxy/csi.sock
1665-
image: registry.k8s.io/sig-storage/csi-attacher:v4.5.1
1667+
image: registry.k8s.io/sig-storage/csi-attacher:v4.9.0
16661668
imagePullPolicy: IfNotPresent
16671669
name: csi-attacher
16681670
volumeMounts:
@@ -1676,10 +1678,11 @@ should render with default values:
16761678
- --driver_name=csi.quobyte.com
16771679
- --driver_version=v2.1.6
16781680
- --enable_access_key_mounts=false
1679-
- --quobyte_version=3
1681+
- --quobyte_version=4
16801682
- --immediate_erase=false
16811683
- --use_k8s_namespace_as_tenant=false
16821684
- --shared_volumes_list=
1685+
- --use_delete_files_task=true
16831686
- --role=controller
16841687
env:
16851688
- name: NODE_ID
@@ -1818,7 +1821,7 @@ should render with default values:
18181821
valueFrom:
18191822
fieldRef:
18201823
fieldPath: spec.nodeName
1821-
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
1824+
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.14.0
18221825
imagePullPolicy: IfNotPresent
18231826
lifecycle:
18241827
preStop:
@@ -1841,7 +1844,7 @@ should render with default values:
18411844
- --driver_name=csi.quobyte.com
18421845
- --driver_version=v2.1.6
18431846
- --enable_access_key_mounts=false
1844-
- --quobyte_version=3
1847+
- --quobyte_version=4
18451848
- --immediate_erase=false
18461849
- --use_k8s_namespace_as_tenant=false
18471850
- --enable_volume_metrics=true

‎csi-driver-templates/values.yaml‎

Lines changed: 28 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
quobyte:
2-
# Quobyte cluster version (valid values - 2 for Quobyte 2.x/3 for Quobyte 3.x)
3-
version: 3
2+
# Quobyte cluster version (valid values - 2 for Quobyte 2.x/3 for Quobyte 3.x/4 for 4.x)
3+
version: 4
44
# apiURL should be of the form http(s)://<ip or resolvable host>:<port>
55
# Example Quobyte API: http://hydrogen.quobyte.com:26801
66
# The default below is ready to connect to a Quobyte api
@@ -51,13 +51,21 @@ quobyte:
5151
# Set to true to schedule erase volume task immediately (supported by Quobyte 3.x)
5252
immediateErase: false
5353

54-
# Required only for Quobyte 2.x
55-
# Specify list of shared volumes. Quobyte CSI driver allows volumes not listed here
56-
# to be used as shared volumes (StorageClass.parameters.sharedVolumeName), however,
57-
# driver does not cleanup volumes not mentioned here.
58-
# Example, sharedVolumesList: "sharedVolume1,mySharedVolume,...,sharedVolumeN"
59-
# Name or UUID of the shared volume (UUIDs are preferred to avoid name collisions)
60-
sharedVolumesList: ""
54+
# If you are creating PVCs at a rapid rate with reclaimPolicy: Delete,
55+
# that leads to too many DELETE_FILES_IN_VOLUMES against the same Quobyte volume.
56+
# In such setups set this flag to false, Quobyte CSI Driver removes the PVC using
57+
# rm -rf via mount point (only few rm -rf run at a time)
58+
useDeleteFilesTaskForSharedVolumeCleanup: true
59+
60+
# Specify list of shared volume UUIDs if useDeleteFilesTaskForSharedVolumeCleanup is set to false.
61+
# Quobyte CSI driver allows volumes not listed here
62+
# to be used as shared volumes (StorageClass.parameters.sharedVolumeName). However,
63+
# driver does not cleanup volumes not mentioned here i.e. does not delete PVCs
64+
# created inside the shared volumes that are not configured here.
65+
sharedVolumesList:
66+
# - <volumeUUID1>
67+
# - <volumeUUID2>
68+
# - <volumeUUIDN>
6169

6270
# Set to 'false' if PVC/volume metrics (used/available/total bytes and inodes)
6371
# are not required to be exported as Prometheus metrics.
@@ -66,8 +74,12 @@ quobyte:
6674
# Quobyte CSI driver is only deployed on nodes with the specified label.
6775
# Empty means all the nodes in the k8s cluster i.e. no node selector will be used.
6876
# nodeSelector: "<node-label-name>: '<node-label-value>'"
69-
nodeSelector: ""
70-
77+
# Example: csiDriverNodeSelector: "quobyteCsiDriverNode: 'true'"
78+
csiDriverNodeSelector: ""
79+
# Pod killer cache node selector - leaving it empty creates pod killer cache pod
80+
# on any of the k8s node
81+
# Example: podKillerCacheNodeSelector: "quobytePodCacheNode: 'true'"
82+
podKillerCacheNodeSelector: ""
7183
podKiller:
7284
# To disable pod killer, uninstall current CSI driver (helm uninstall <chart-name>)
7385
# set enable: false and install CSI driver again
@@ -96,13 +108,13 @@ quobyte:
96108
# k8s sidecar containers (https://github.com/kubernetes-csi/)
97109
# Updating k8s...Image might require RBAC files update
98110
# https://github.com/quobyte/quobyte-csi/tree/master/quobyte-csi-driver/templates/pods/rbac
99-
k8sProvisionerImage: "registry.k8s.io/sig-storage/csi-provisioner:v4.0.1"
100-
k8sResizerImage: "registry.k8s.io/sig-storage/csi-resizer:v1.8.1"
101-
k8sNodeRegistrarImage: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1"
102-
k8sAttacherImage: "registry.k8s.io/sig-storage/csi-attacher:v4.5.1"
111+
k8sProvisionerImage: "registry.k8s.io/sig-storage/csi-provisioner:v5.3.0"
112+
k8sResizerImage: "registry.k8s.io/sig-storage/csi-resizer:v1.14.0"
113+
k8sNodeRegistrarImage: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.14.0"
114+
k8sAttacherImage: "registry.k8s.io/sig-storage/csi-attacher:v4.9.0"
103115
# when updating image for snapshotter, update snaptshotter setup CRD with
104116
# instructions in README (CRD should be pulled from matched release).
105117
# Additionally, ./quobyte-csi-driver/k8s-snapshot-controller.yaml
106118
# (see this file for source link) should be updated with appropriate version
107119
# files (Do NOT forget updating namespace to kube-system)
108-
k8sSnapshotterImage: "registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2"
120+
k8sSnapshotterImage: "registry.k8s.io/sig-storage/csi-snapshotter:v8.3.0"

‎kind-cluster/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ The aim of these set of scripts is to enable CSI e2e test runs against internal
5050
or
5151

5252
You can run with `TEST_CASE_DIR` that contains only CSI driver values.yaml to deploy the driver
53-
(note that some defined values such as CSI image/pod killer images are overriden)
53+
(note that some defined values such as CSI image/pod killer images are overridden)
5454

5555
## Cleanup
5656

0 commit comments

Comments
 (0)