From 0755545b06352deaedde138954ecd698923d97a9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Fri, 18 Jun 2021 18:17:55 +0200 Subject: [PATCH 01/14] Update the release process notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- docs/TESTING-DEPLOYER.md | 60 ++++++++++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 21 deletions(-) diff --git a/docs/TESTING-DEPLOYER.md b/docs/TESTING-DEPLOYER.md index 1b264a3..2c4033f 100644 --- a/docs/TESTING-DEPLOYER.md +++ b/docs/TESTING-DEPLOYER.md @@ -10,10 +10,10 @@ **Contents:** +- [Cutting a new release](#cutting-a-new-release) - [Pricing mechanism](#pricing-mechanism) - [Creating and testing the deployer image](#creating-and-testing-the-deployer-image) - [mpdev install on your own cluster](#mpdev-install-on-your-own-cluster) -- [Cutting a new release](#cutting-a-new-release) - [Testing the application without having access to the Billing API](#testing-the-application-without-having-access-to-the-billing-api) - [How the Application object "wrangles" its components](#how-the-application-object-wrangles-its-components) - [Installing and manually testing the deployer image](#installing-and-manually-testing-the-deployer-image) @@ -21,6 +21,40 @@ - [Debugging deployer and smoke-tests when run in Cloud Build](#debugging-deployer-and-smoke-tests-when-run-in-cloud-build) - [Updating the upstream cert-manager chart version](#updating-the-upstream-cert-manager-chart-version) +## Cutting a new release + +Since the process is manual and evolves from release to release, we document all +the steps that were taken in each release directly on the GitHub Release itself +in a `
` block that looks like this: + +> ▶ 📦 Recording of the manual steps of the release process + +Imagining that you want to release `1.1.0-gcm.5`, the steps are: + +1. Copy the `
` block from the previous release [1.1.0-gcm.4](https://github.com/jetstack/jetstack-secure-gcm/releases/tag/1.1.0-gcm.4) +2. In an editor, change the references to `1.1.0-gcm.4`. +3. Follow the steps and tick the checkboxes. +4. After the `1.1.0-gcm.5` is pushed to GitHub, create a GitHub Release for that + tag and paste the content into the `
` block into the GitHub Release + you just created (see `PASTE HERE` below). The GitHub Release description + should look like this: + + ```md + ## Changelog + + + + ## Notes + +
+ + 📦 Recording of the manual steps of the release process + + + +
+ ``` + ## Pricing mechanism Each cluster is priced at $50 a month, billed hourly ($0.07/hour). The way the @@ -32,10 +66,10 @@ of `1` to the `time` value. The unit for `time` is something we have configured in the [pricing panel](https://console.cloud.google.com/partner/editor/jetstack-public/jetstack-secure-for-cert-manager?project=jetstack-public&authuser=4&form=saasK8sPricingPanel). -| Field | Value | -| -------------- | ------ | -| ID | `time` | -| Unit | `h` | +| Field | Value | +| ----- | ------ | +| ID | `time` | +| Unit | `h` | Note that the cert-manager deployment should always be run with replicas=1. High-availability (replicas > 1) is not supported yet, and the application will @@ -298,22 +332,6 @@ see everything green: The application page for test-1 shows that all the deployments are green. This screenshot is stored in this issue: https://github.com/jetstack/jetstack-secure-gcm/issues/21 -## Cutting a new release - -Since the process is manual and evolves from release to release, we document all -the steps taken in each release directly on the GitHub Release itself in a -`
` block that looks like this: - -> ▶ 📦 Recording of the manual steps of the release process - -For example, when releasing `1.1.0-gcm.5`, the steps were: - -1. Copy the `
` block from the previous release [1.1.0-gcm.4](https://github.com/jetstack/jetstack-secure-gcm/releases/tag/1.1.0-gcm.4) -2. In an editor, change the references to `1.1.0-gcm.4`. -3. Follow the steps and tick the checkboxes. -4. After the `1.1.0-gcm.5` is pushed to GitHub, create a GitHub Release for that - tag and paste the content the `
` block to the GitHub Release. - ## Testing the application without having access to the Billing API Jetstack members do not have access to the Billing API. In order to test From d92cbebed76b9a24d6b7c58c62bfb1019d9382e5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Fri, 18 Jun 2021 18:37:08 +0200 Subject: [PATCH 02/14] schema: add ClusterRoles added with the 1.4 release MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- schema.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/schema.yaml b/schema.yaml index 9e17038..5e28584 100644 --- a/schema.yaml +++ b/schema.yaml @@ -242,6 +242,21 @@ properties: - googlecasclusterissuers.cas-issuer.jetstack.io/* - googlecasissuers.cas-issuer.jetstack.io/* + # Added in https://github.com/jetstack/cert-manager/pull/4064 + - apiGroups: ["certificates.k8s.io"] + resources: ["certificatesigningrequests"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["certificates.k8s.io"] + resources: ["certificatesigningrequests/status"] + verbs: ["update"] + - apiGroups: ["certificates.k8s.io"] + resources: ["signers"] + resourceNames: ["issuers.cert-manager.io/*", "clusterissuers.cert-manager.io/*"] + verbs: ["sign"] + - apiGroups: ["authorization.k8s.io"] + resources: ["subjectaccessreviews"] + verbs: ["create"] + cert-manager.webhook.serviceAccount.name: type: string title: Name of the Kubernetes service account for the cert-manager webhook From 79544ce48880168d5989d875d6301be19359e076 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Fri, 18 Jun 2021 18:43:16 +0200 Subject: [PATCH 03/14] bump version to 1.4.0-gcm.0 --- schema.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/schema.yaml b/schema.yaml index 5e28584..4eaf5dd 100644 --- a/schema.yaml +++ b/schema.yaml @@ -14,7 +14,7 @@ x-google-marketplace: # # Important: it must have the same value as the version in the # Application manifest. - publishedVersion: 1.3.1-gcm.0 + publishedVersion: 1.4.0-gcm.0 publishedVersionMetadata: releaseNote: >- Initial release. From 9adc68c9d7e9180b348081baa2977d4eb670cecb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Fri, 18 Jun 2021 19:38:37 +0200 Subject: [PATCH 04/14] app-crd: the Application CRD breaks under CRD v1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For some reason, the Application that we have had a field that does not exist on the v1beta1 spec. More specifically, we had: apiVersion: app.k8s.io/v1beta1 kind: Application spec: descriptor: ... info: [] It should have been: apiVersion: app.k8s.io/v1beta1 kind: Application spec: descriptor: ... info: [] It seems like when Google upgraded [1] the Application CRD from v1beta1 to v1 (this in the version of the CRD object, not the version of the Application itself). After this change, the above Application manifest could not be applied anymore. The error looked like this: error: error validating "/data/resources.yaml": error validating data: ValidationError(Application.spec.descriptor): unknown field "info" in io.k8s.app.v1beta1.Application.spec.descriptor; if you choose to ignore these errors, turn validation off with --validate=false My guess is that before this change, the faulty "info" field was not being validated, and the new v1 CRD version started validating it. [1]: https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/pull/562 Signed-off-by: Maël Valais --- chart/jetstack-secure-gcm/templates/application.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/chart/jetstack-secure-gcm/templates/application.yaml b/chart/jetstack-secure-gcm/templates/application.yaml index 34fedac..d6e00ce 100644 --- a/chart/jetstack-secure-gcm/templates/application.yaml +++ b/chart/jetstack-secure-gcm/templates/application.yaml @@ -44,7 +44,6 @@ spec: url: https://cert-manager.io/docs/usage/ - description: Getting Started with the Jetstack Secure Platform url: https://platform.jetstack.io/docs - info: [] notes: |- ### Create your first certificate From 238bcc67491b51e247be496226c04762f7011914 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Mon, 21 Jun 2021 11:08:49 +0200 Subject: [PATCH 05/14] cas-issuer: upgrade googlecasissuers: v1alpha1 -> v1beta1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- .../google-cas-issuer/templates/crds.yaml | 381 +++++++++--------- docs/TESTING-DEPLOYER.md | 2 +- smoke-test.yaml | 10 +- 3 files changed, 194 insertions(+), 199 deletions(-) diff --git a/chart/jetstack-secure-gcm/charts/google-cas-issuer/templates/crds.yaml b/chart/jetstack-secure-gcm/charts/google-cas-issuer/templates/crds.yaml index 8c1a25f..c911f06 100644 --- a/chart/jetstack-secure-gcm/charts/google-cas-issuer/templates/crds.yaml +++ b/chart/jetstack-secure-gcm/charts/google-cas-issuer/templates/crds.yaml @@ -1,12 +1,13 @@ {{- if .Values.installCRDs }} -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.2.2 - name: googlecasclusterissuers.cas-issuer.jetstack.io labels: {{- include "google-cas-issuer.labels" . | nindent 4 }} + annotations: + controller-gen.kubebuilder.io/version: v0.5.0 + creationTimestamp: null + name: googlecasclusterissuers.cas-issuer.jetstack.io spec: group: cas-issuer.jetstack.io names: @@ -15,106 +16,103 @@ spec: plural: googlecasclusterissuers singular: googlecasclusterissuer scope: Cluster - subresources: - status: {} - validation: - openAPIV3Schema: - description: GoogleCASClusterIssuer is the Schema for the googlecasclusterissuers - API - properties: - apiVersion: - description: "APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources" - type: string - kind: - description: "Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds" - type: string - metadata: - type: object - spec: - description: GoogleCASIssuerSpec defines the desired state of GoogleCASIssuer - properties: - certificateAuthorityID: - description: CertificateAuthorityID is The ID of the Google Private - certificate authority that will sign certificates - type: string - credentials: - description: Credentials is a reference to a Kubernetes Secret Key that - contains Google Service Account Credentials - properties: - key: - description: The key of the entry in the Secret resource's `data` - field to be used. Some instances of this field may be defaulted, - in others it may be required. - type: string - name: - description: "Name of the resource being referred to. More info: - https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names" - type: string - required: - - name - type: object - location: - description: Location is the Google Cloud Project Location - type: string - project: - description: Project is the Google Cloud Project ID - type: string - type: object - status: - description: GoogleCASIssuerStatus defines the observed state of GoogleCASIssuer - properties: - conditions: - items: - description: IssuerCondition contains condition information for a - CAS Issuer. + versions: + - additionalPrinterColumns: + - jsonPath: .status.conditions[?(@.type=='Ready')].status + name: ready + type: string + - jsonPath: .status.conditions[?(@.type=='Ready')].reason + name: reason + type: string + - jsonPath: .status.conditions[?(@.type=='Ready')].message + name: message + type: string + name: v1beta1 + schema: + openAPIV3Schema: + description: GoogleCASClusterIssuer is the Schema for the googlecasclusterissuers API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: GoogleCASIssuerSpec defines the desired state of GoogleCASIssuer + properties: + caPoolId: + description: CaPoolId is the id of the CA pool to issue certificates from + type: string + certificateAuthorityId: + description: CertificateAuthorityId is specific certificate authority to use to sign. Omit in order to load balance across all CAs in the pool + type: string + credentials: + description: Credentials is a reference to a Kubernetes Secret Key that contains Google Service Account Credentials properties: - lastTransitionTime: - description: LastTransitionTime is the timestamp corresponding - to the last status change of this condition. - format: date-time - type: string - message: - description: Message is a human readable description of the details - of the last transition, complementing reason. + key: + description: The key of the entry in the Secret resource's `data` field to be used. Some instances of this field may be defaulted, in others it may be required. type: string - reason: - description: Reason is a brief machine readable explanation for - the condition's last transition. + name: + description: 'Name of the resource being referred to. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' type: string - status: - allOf: + required: + - name + type: object + location: + description: Location is the Google Cloud Project Location + type: string + project: + description: Project is the Google Cloud Project ID + type: string + type: object + status: + description: GoogleCASIssuerStatus defines the observed state of GoogleCASIssuer + properties: + conditions: + items: + description: IssuerCondition contains condition information for a CAS Issuer. + properties: + lastTransitionTime: + description: LastTransitionTime is the timestamp corresponding to the last status change of this condition. + format: date-time + type: string + message: + description: Message is a human readable description of the details of the last transition, complementing reason. + type: string + reason: + description: Reason is a brief machine readable explanation for the condition's last transition. + type: string + status: + allOf: - enum: - - "True" - - "False" - - Unknown + - "True" + - "False" + - Unknown - enum: - - "True" - - "False" - - Unknown - description: Status of the condition, one of ('True', 'False', - 'Unknown'). - type: string - type: - description: Type of the condition, currently ('Ready'). - enum: + - "True" + - "False" + - Unknown + description: Status of the condition, one of ('True', 'False', 'Unknown'). + type: string + type: + description: Type of the condition, currently ('Ready'). + enum: - Ready - type: string - required: + type: string + required: - status - type - type: object - type: array - type: object - type: object - version: v1alpha1 - versions: - - name: v1alpha1 - served: true - storage: true + type: object + type: array + type: object + type: object + served: true + storage: true + subresources: + status: {} status: acceptedNames: kind: "" @@ -122,14 +120,13 @@ status: conditions: [] storedVersions: [] --- -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.2.2 + controller-gen.kubebuilder.io/version: v0.5.0 + creationTimestamp: null name: googlecasissuers.cas-issuer.jetstack.io - labels: - {{- include "google-cas-issuer.labels" . | nindent 4 }} spec: group: cas-issuer.jetstack.io names: @@ -137,110 +134,108 @@ spec: listKind: GoogleCASIssuerList plural: googlecasissuers singular: googlecasissuer - scope: "" - subresources: - status: {} - validation: - openAPIV3Schema: - description: GoogleCASIssuer is the Schema for the googlecasissuers API - properties: - apiVersion: - description: "APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources" - type: string - kind: - description: "Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds" - type: string - metadata: - type: object - spec: - description: GoogleCASIssuerSpec defines the desired state of GoogleCASIssuer - properties: - certificateAuthorityID: - description: CertificateAuthorityID is The ID of the Google Private - certificate authority that will sign certificates - type: string - credentials: - description: Credentials is a reference to a Kubernetes Secret Key that - contains Google Service Account Credentials - properties: - key: - description: The key of the entry in the Secret resource's `data` - field to be used. Some instances of this field may be defaulted, - in others it may be required. - type: string - name: - description: "Name of the resource being referred to. More info: - https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names" - type: string - required: - - name - type: object - location: - description: Location is the Google Cloud Project Location - type: string - project: - description: Project is the Google Cloud Project ID - type: string - type: object - status: - description: GoogleCASIssuerStatus defines the observed state of GoogleCASIssuer - properties: - conditions: - items: - description: IssuerCondition contains condition information for a - CAS Issuer. + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .status.conditions[?(@.type=='Ready')].status + name: ready + type: string + - jsonPath: .status.conditions[?(@.type=='Ready')].reason + name: reason + type: string + - jsonPath: .status.conditions[?(@.type=='Ready')].message + name: message + type: string + name: v1beta1 + schema: + openAPIV3Schema: + description: GoogleCASIssuer is the Schema for the googlecasissuers API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: GoogleCASIssuerSpec defines the desired state of GoogleCASIssuer + properties: + caPoolId: + description: CaPoolId is the id of the CA pool to issue certificates from + type: string + certificateAuthorityId: + description: CertificateAuthorityId is specific certificate authority to use to sign. Omit in order to load balance across all CAs in the pool + type: string + credentials: + description: Credentials is a reference to a Kubernetes Secret Key that contains Google Service Account Credentials properties: - lastTransitionTime: - description: LastTransitionTime is the timestamp corresponding - to the last status change of this condition. - format: date-time - type: string - message: - description: Message is a human readable description of the details - of the last transition, complementing reason. + key: + description: The key of the entry in the Secret resource's `data` field to be used. Some instances of this field may be defaulted, in others it may be required. type: string - reason: - description: Reason is a brief machine readable explanation for - the condition's last transition. + name: + description: 'Name of the resource being referred to. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' type: string - status: - allOf: + required: + - name + type: object + location: + description: Location is the Google Cloud Project Location + type: string + project: + description: Project is the Google Cloud Project ID + type: string + type: object + status: + description: GoogleCASIssuerStatus defines the observed state of GoogleCASIssuer + properties: + conditions: + items: + description: IssuerCondition contains condition information for a CAS Issuer. + properties: + lastTransitionTime: + description: LastTransitionTime is the timestamp corresponding to the last status change of this condition. + format: date-time + type: string + message: + description: Message is a human readable description of the details of the last transition, complementing reason. + type: string + reason: + description: Reason is a brief machine readable explanation for the condition's last transition. + type: string + status: + allOf: - enum: - - "True" - - "False" - - Unknown + - "True" + - "False" + - Unknown - enum: - - "True" - - "False" - - Unknown - description: Status of the condition, one of ('True', 'False', - 'Unknown'). - type: string - type: - description: Type of the condition, currently ('Ready'). - enum: + - "True" + - "False" + - Unknown + description: Status of the condition, one of ('True', 'False', 'Unknown'). + type: string + type: + description: Type of the condition, currently ('Ready'). + enum: - Ready - type: string - required: + type: string + required: - status - type - type: object - type: array - type: object - type: object - version: v1alpha1 - versions: - - name: v1alpha1 - served: true - storage: true + type: object + type: array + type: object + type: object + served: true + storage: true + subresources: + status: {} status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] -{{- end }} +{{- end }} \ No newline at end of file diff --git a/docs/TESTING-DEPLOYER.md b/docs/TESTING-DEPLOYER.md index 2c4033f..144c9ac 100644 --- a/docs/TESTING-DEPLOYER.md +++ b/docs/TESTING-DEPLOYER.md @@ -570,7 +570,7 @@ the example [suite.yaml](https://github.com/GoogleCloudPlatform/marketplace-test ```yaml actions: - - name: { { .Env.TEST_NAME } } + - name: "{{ .Env.TEST_NAME }}" httpTest: url: http://{{ .Var.MainVmIp }}:9012 expect: diff --git a/smoke-test.yaml b/smoke-test.yaml index c1bde10..a131700 100644 --- a/smoke-test.yaml +++ b/smoke-test.yaml @@ -4,9 +4,9 @@ # To see how this testrunner works, the only way is to read the bash.go: # https://github.com/GoogleCloudPlatform/marketplace-testrunner/blob/4245fa9/tests/bash.go actions: - # - name: hang for debugging purposes - # bashTest: - # script: sleep 1200 # 20 minutes + - name: hang for debugging purposes + bashTest: + script: sleep 1200 # 20 minutes - name: kubectl smoke test bashTest: script: kubectl version @@ -64,14 +64,14 @@ actions: bashTest: script: | kubectl apply -n ${NAMESPACE} -f - < Date: Mon, 21 Jun 2021 12:19:29 +0200 Subject: [PATCH 06/14] schema: lease was missing from the controller's service account MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- schema.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/schema.yaml b/schema.yaml index 4eaf5dd..0b569b3 100644 --- a/schema.yaml +++ b/schema.yaml @@ -232,6 +232,9 @@ properties: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "create", "update", "patch"] + - apiGroups: [coordination.k8s.io] + resources: [leases] + verbs: [create, get, update, patch] - apiGroups: ["cert-manager.io"] resources: ["signers"] verbs: ["approve"] @@ -301,7 +304,7 @@ properties: verbs: [create] - apiGroups: [coordination.k8s.io] resources: [leases] - verbs: [create, get, list, update] + verbs: [create, get, update, patch] - type: ClusterRole rulesType: CUSTOM rules: From 800baa0048bdbda3f330b7a580e7e6399a20615f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Mon, 21 Jun 2021 13:22:25 +0200 Subject: [PATCH 07/14] schema: hardcode leader elect namespace to kube-system, schema.yaml limitations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- schema.yaml | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/schema.yaml b/schema.yaml index 0b569b3..af10e06 100644 --- a/schema.yaml +++ b/schema.yaml @@ -235,6 +235,10 @@ properties: - apiGroups: [coordination.k8s.io] resources: [leases] verbs: [create, get, update, patch] + # Although the leader election namespace is usually + # configurable, we can't make it configurable due to + # schema.yaml. So we default to kube-system. + namespaces: [kube-system] - apiGroups: ["cert-manager.io"] resources: ["signers"] verbs: ["approve"] @@ -305,6 +309,11 @@ properties: - apiGroups: [coordination.k8s.io] resources: [leases] verbs: [create, get, update, patch] + # Although the leader election namespace is usually + # configurable, we can't make it configurable due to + # schema.yaml. So we default to kube-system. + namespaces: [kube-system] + - type: ClusterRole rulesType: CUSTOM rules: @@ -353,7 +362,11 @@ properties: verbs: [create] - apiGroups: [coordination.k8s.io] resources: [leases] - verbs: [create, get, list, update] + verbs: [create, get, update, patch] + # Although the leader election namespace is usually + # configurable, we can't make it configurable due to + # schema.yaml. So we default to kube-system. + namespaces: [kube-system] - type: ClusterRole rulesType: CUSTOM rules: From 3812adcf4ad7646e16bc409abfad277840e150b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Mon, 21 Jun 2021 14:03:44 +0200 Subject: [PATCH 08/14] schema: move lease and configmap RBACs to ClusterRole, events to Role MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- schema.yaml | 135 ++++++++++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 62 deletions(-) diff --git a/schema.yaml b/schema.yaml index af10e06..5281853 100644 --- a/schema.yaml +++ b/schema.yaml @@ -107,6 +107,12 @@ properties: serviceAccount: description: Service account used by cert-manager Controller Deployment roles: + - type: Role + rulesType: CUSTOM + rules: + - apiGroups: [""] + resources: ["events"] + verbs: ["create", "patch"] - type: ClusterRole rulesType: CUSTOM rules: @@ -120,9 +126,6 @@ properties: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch", "create", "update", "delete"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "patch"] # ClusterIssuer controller role - apiGroups: ["cert-manager.io"] resources: ["clusterissuers", "clusterissuers/status"] @@ -133,9 +136,6 @@ properties: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch", "create", "update", "delete"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "patch"] # Certificates controller role - apiGroups: ["cert-manager.io"] resources: ["certificates", "certificates/status", "certificaterequests", "certificaterequests/status"] @@ -152,9 +152,6 @@ properties: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch", "create", "update", "delete"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "patch"] # Orders controller role - apiGroups: ["acme.cert-manager.io"] resources: ["orders", "orders/status"] @@ -174,9 +171,6 @@ properties: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "patch"] # Challenges controller role - apiGroups: ["acme.cert-manager.io"] resources: ["challenges", "challenges/status"] @@ -190,9 +184,6 @@ properties: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "patch"] - apiGroups: [""] resources: ["pods", "services"] verbs: ["get", "list", "watch", "create", "delete"] @@ -226,19 +217,7 @@ properties: - apiGroups: ["networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list", "watch"] - - apiGroups: [""] - resources: ["events"] - verbs: ["create", "patch"] - - apiGroups: [""] - resources: ["configmaps"] - verbs: ["get", "create", "update", "patch"] - - apiGroups: [coordination.k8s.io] - resources: [leases] - verbs: [create, get, update, patch] - # Although the leader election namespace is usually - # configurable, we can't make it configurable due to - # schema.yaml. So we default to kube-system. - namespaces: [kube-system] + - apiGroups: ["cert-manager.io"] resources: ["signers"] verbs: ["approve"] @@ -264,6 +243,28 @@ properties: resources: ["subjectaccessreviews"] verbs: ["create"] + # Leader election roles. Although leader election only operates on + # one single namespace (often kube-system), we have to use a + # ClusterRole due to schema.yaml limitations: + # https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/issues/564 + - apiGroups: [""] + verbs: [create] + resources: [configmaps] + - apiGroups: [""] + resources: [configmaps] + verbs: [get, list, watch, update, patch, delete] + resourceNames: ["cert-manager-controller"] + - apiGroups: [""] + resources: [configmaps/status] + verbs: [get, update, patch] + - apiGroups: [coordination.k8s.io] + resources: [leases] + verbs: [create] + - apiGroups: [coordination.k8s.io] + resources: [leases] + verbs: [get, update, patch] + resourceNames: ["cert-manager-controller"] + cert-manager.webhook.serviceAccount.name: type: string title: Name of the Kubernetes service account for the cert-manager webhook @@ -296,23 +297,9 @@ properties: - type: Role rulesType: CUSTOM rules: - # Leader election roles. - - apiGroups: [""] - resources: [configmaps] - verbs: [get, list, watch, create, update, patch, delete] - - apiGroups: [""] - resources: [configmaps/status] - verbs: [get, update, patch] - apiGroups: [""] resources: [events] verbs: [create] - - apiGroups: [coordination.k8s.io] - resources: [leases] - verbs: [create, get, update, patch] - # Although the leader election namespace is usually - # configurable, we can't make it configurable due to - # schema.yaml. So we default to kube-system. - namespaces: [kube-system] - type: ClusterRole rulesType: CUSTOM @@ -339,34 +326,36 @@ properties: resources: ["configmaps"] verbs: ["get", "create", "update", "patch"] - google-cas-issuer.serviceAccount.name: - type: string - title: Name of the Kubernetes service account for the google-cas-issuer - x-google-marketplace: - type: SERVICE_ACCOUNT - serviceAccount: - description: Service account used by google-cas-issuer - roles: - - type: Role - rulesType: CUSTOM - rules: - # Leader election roles. + # Leader election roles. Although leader election only operates on + # one single namespace (often kube-system), we have to use a + # ClusterRole due to schema.yaml limitations: + # https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/issues/564 + - apiGroups: [""] + verbs: [create] + resources: [configmaps] - apiGroups: [""] resources: [configmaps] - verbs: [get, list, watch, create, update, patch, delete] + verbs: [get, list, watch, update, patch, delete] + resourceNames: ["cert-manager-cainjector-leader-election", "cert-manager-cainjector-leader-election-core"] - apiGroups: [""] resources: [configmaps/status] verbs: [get, update, patch] - - apiGroups: [""] - resources: [events] + - apiGroups: [coordination.k8s.io] + resources: [leases] verbs: [create] - apiGroups: [coordination.k8s.io] resources: [leases] - verbs: [create, get, update, patch] - # Although the leader election namespace is usually - # configurable, we can't make it configurable due to - # schema.yaml. So we default to kube-system. - namespaces: [kube-system] + verbs: [get, update, patch] + resourceNames: ["cert-manager-cainjector-leader-election", "cert-manager-cainjector-leader-election-core"] + + google-cas-issuer.serviceAccount.name: + type: string + title: Name of the Kubernetes service account for the google-cas-issuer + x-google-marketplace: + type: SERVICE_ACCOUNT + serviceAccount: + description: Service account used by google-cas-issuer + roles: - type: ClusterRole rulesType: CUSTOM rules: @@ -395,6 +384,28 @@ properties: resources: [certificaterequests/status] verbs: [get, patch, update] + # Leader election roles. Although leader election only operates on + # one single namespace (often kube-system), we have to use a + # ClusterRole due to schema.yaml limitations: + # https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/issues/564 + - apiGroups: [""] + verbs: [create] + resources: [configmaps] + - apiGroups: [""] + resources: [configmaps] + verbs: [get, list, watch, update, patch, delete] + resourceNames: ["cm-google-cas-issuer"] + - apiGroups: [""] + resources: [configmaps/status] + verbs: [get, update, patch] + - apiGroups: [coordination.k8s.io] + resources: [leases] + verbs: [create] + - apiGroups: [coordination.k8s.io] + resources: [leases] + verbs: [get, update, patch] + resourceNames: ["cm-google-cas-issuer"] + preflight.serviceAccount.name: type: string title: Name of the Kubernetes service account for the Jetstack Secure agent From 85b9287f8e90aa84bc84513e779d03280bfc6118 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Wed, 23 Jun 2021 16:08:51 +0200 Subject: [PATCH 09/14] smoke-test: disable the debugging step MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- smoke-test.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/smoke-test.yaml b/smoke-test.yaml index a131700..445f518 100644 --- a/smoke-test.yaml +++ b/smoke-test.yaml @@ -4,9 +4,9 @@ # To see how this testrunner works, the only way is to read the bash.go: # https://github.com/GoogleCloudPlatform/marketplace-testrunner/blob/4245fa9/tests/bash.go actions: - - name: hang for debugging purposes - bashTest: - script: sleep 1200 # 20 minutes + # - name: hang for debugging purposes + # bashTest: + # script: sleep 1200 # 20 minutes - name: kubectl smoke test bashTest: script: kubectl version From 4a03e1161a23d602f36bf99cf9eccebeff0fb1e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Tue, 29 Jun 2021 13:35:41 +0200 Subject: [PATCH 10/14] smoke-tests: clarify the test case descriptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- smoke-test.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/smoke-test.yaml b/smoke-test.yaml index 445f518..451dc01 100644 --- a/smoke-test.yaml +++ b/smoke-test.yaml @@ -13,7 +13,7 @@ actions: expect: exitCode: equals: 0 - - name: Create test issuer and self signed cert + - name: Create a self-signed Issuer and a Certificate bashTest: script: | kubectl apply -n ${NAMESPACE} -f - < Date: Tue, 29 Jun 2021 21:01:49 +0200 Subject: [PATCH 11/14] cloudbuild: only display the logs, not when waiting for the pod MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- cloudbuild.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cloudbuild.yaml b/cloudbuild.yaml index 1261736..f66e78c 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -144,7 +144,7 @@ steps: name: gcr.io/cloud-builders/gcloud entrypoint: bash args: - - -exc + - -euc - | while : ; do kubectl get ns -oname 2>/dev/null | grep apptest- && break @@ -158,6 +158,7 @@ steps: done pod=$(kubectl -n "$ns" get pods -oname | grep "apptest-.*-deployer" | cut -d/ -f2) + set -x kubectl wait -n "$ns" --for=condition=ready --timeout=5m pod $pod kubectl logs -n "$ns" $pod -f --tail=-1 waitFor: @@ -168,7 +169,7 @@ steps: name: gcr.io/cloud-builders/gcloud entrypoint: bash args: - - -exc + - -euc - | while : ; do kubectl get ns -oname 2>/dev/null | grep apptest- && break @@ -181,6 +182,7 @@ steps: sleep 10 done + set -x kubectl wait -n "$ns" --for=condition=ready --timeout=5m pod smoke-test-pod kubectl logs -n "$ns" smoke-test-pod -f --tail=-1 waitFor: From 6193a40ebdfaa762f47bc1563abfadf6268bcad7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Tue, 29 Jun 2021 20:36:31 +0200 Subject: [PATCH 12/14] smoke-test: use our own bash-based runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I went with cram because it gives excellent stderr and stdout diagnostic in case of error. You can run it locally with: docker build --file smoke-test.Dockerfile -t runner . docker run -it -v ~/.kube:/root/.kube -v $PWD:/opt runner cram smoke-test.t To update the stdout/stderr and status codes interactively, run: docker run -it -v ~/.kube:/root/.kube -v $PWD:/opt runner cram smoke-test.t -i Signed-off-by: Maël Valais --- .gitignore | 1 + smoke-test.Dockerfile | 23 +++++---- smoke-test.t | 105 +++++++++++++++++++++++++++++++++++++++ smoke-test.yaml | 112 ------------------------------------------ 4 files changed, 117 insertions(+), 124 deletions(-) create mode 100644 smoke-test.t delete mode 100644 smoke-test.yaml diff --git a/.gitignore b/.gitignore index 2ebd490..f912bf2 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ ca.crt ca.key +*.t.err diff --git a/smoke-test.Dockerfile b/smoke-test.Dockerfile index 9b8e3fe..f21e0bd 100644 --- a/smoke-test.Dockerfile +++ b/smoke-test.Dockerfile @@ -7,19 +7,18 @@ # certificate using the Google CAS issuer?" # Dockerfile: https://github.com/GoogleCloudPlatform/marketplace-testrunner/blob/master/Dockerfile -FROM gcr.io/cloud-marketplace-tools/testrunner:0.1.2 +FROM python:alpine -RUN apt-get update && apt-get install -y --no-install-recommends \ - curl wget dnsutils netcat jq \ - && rm -rf /var/lib/apt/lists/* +RUN apk add curl patch +RUN pip3 install cram +RUN curl -L https://github.com/stern/stern/releases/download/v1.19.0/stern_1.19.0_linux_amd64.tar.gz | tar xz -C /tmp \ + && mv /tmp/stern_1.19.0_linux_amd64/stern /usr/local/bin \ + && curl -L https://storage.googleapis.com/kubernetes-release/release/v1.19.6/bin/linux/amd64/kubectl --output-dir /usr/local/bin -O \ + && chmod 755 /usr/local/bin/kubectl -RUN mkdir -p /opt/kubectl/1.19 \ - && wget -q -O /opt/kubectl/1.19/kubectl \ - https://storage.googleapis.com/kubernetes-release/release/v1.19.6/bin/linux/amd64/kubectl \ - && chmod 755 /opt/kubectl/1.19/kubectl \ - && ln -s /opt/kubectl/1.19/kubectl /usr/bin/kubectl +WORKDIR /opt +COPY smoke-test.t . -COPY smoke-test.yaml /smoke-test.yaml +ENV NAMESPACE=test -WORKDIR / -ENTRYPOINT ["testrunner", "-logtostderr", "--test_spec=/smoke-test.yaml"] +CMD ["sh", "-c", "stern -A -l app.kubernetes.io/name=jetstack-secure-gcm & cram smoke-test.t"] diff --git a/smoke-test.t b/smoke-test.t new file mode 100644 index 0000000..723313f --- /dev/null +++ b/smoke-test.t @@ -0,0 +1,105 @@ +This file is written using the "cram" format. We could have used a simple Bash +script, but cram tests make it very easy to spot when something "changes" e.g. +stderr or stdout mismatch. + +This file is used in the smoke-test image (smoke-test.Dockerfile) to run the +test suite. This image is used in a Job in data-test. mpdev verify detects the +presence of this Job thanks to the annotation + "marketplace.cloud.google.com/verification: test" + +First, let us create a self-signed Issuer and a Certificate. This is a good way +to spot webhook misconfigurations. + + $ kubectl apply -n ${NAMESPACE} -f - < apiVersion: cert-manager.io/v1 + > kind: Issuer + > metadata: + > name: selfsigned-issuer + > spec: + > selfSigned: {} + > --- + > apiVersion: cert-manager.io/v1 + > kind: Certificate + > metadata: + > name: selfsigned-cert + > spec: + > dnsNames: + > - example.com + > secretName: selfsigned-cert-tls + > issuerRef: + > name: selfsigned-issuer + > EOF + issuer.cert-manager.io/selfsigned-issuer created + certificate.cert-manager.io/selfsigned-cert created + +Now, let us wait until the Certificate becomes Ready. An error here might +indicate that the cert-manager-controller is running but keeps failing at the +leader election step. We know that cert-manager-controller is available because +"mpdev verify" only runs this test suite after cert-manager-controller becomes +available. + + $ kubectl wait -n ${NAMESPACE} --for=condition=Ready --timeout=2m certificate selfsigned-cert + certificate.cert-manager.io/selfsigned-cert condition met + +Get the Secret associated to the Certificate: + + $ kubectl get secret -n ${NAMESPACE} selfsigned-cert-tls + NAME TYPE DATA AGE + selfsigned-cert-tls kubernetes.io/tls 3 * (glob) + +Delete the self-signed Issuer and the Certificate: + + $ kubectl delete -n ${NAMESPACE} issuer selfsigned-issuer + issuer.cert-manager.io "selfsigned-issuer" deleted + + $ kubectl delete -n ${NAMESPACE} certificate selfsigned-cert + certificate.cert-manager.io "selfsigned-cert" deleted + + +Create a GoogleCASIssuer and a Certificate: + + $ kubectl apply -n ${NAMESPACE} -f - < apiVersion: cas-issuer.jetstack.io/v1beta1 + > kind: GoogleCASIssuer + > metadata: + > name: googlecas-issuer + > spec: + > project: "todo" + > location: "todo" + > caPoolId: "todo" + > --- + > apiVersion: cert-manager.io/v1 + > kind: Certificate + > metadata: + > name: googlecas-cert + > spec: + > secretName: demo-cert-tls + > commonName: cert-manager.io.demo + > dnsNames: + > - cert-manager.io + > - jetstack.io + > duration: 24h + > renewBefore: 8h + > issuerRef: + > group: cas-issuer.jetstack.io + > kind: GoogleCASIssuer + > name: googlecas-issuer + > EOF + googlecasissuer.cas-issuer.jetstack.io/googlecas-issuer created + certificate.cert-manager.io/googlecas-cert created + +We don't know yet how, as part of this smoke test file, to create a GCP service +account and then create a Kubernetes service account and bind the two together +using the "workload identity" feature. + +Right now, the above GoogleCASIssuer does nothing, and the certificate will +never be issued. This TODO is tracked at +https://github.com/jetstack/jetstack-secure-gcm/issues/19. + +Now, let us delete the GoogleCASIssuer and Certificate. + + $ kubectl delete -n ${NAMESPACE} googlecasissuer googlecas-issuer + googlecasissuer.cas-issuer.jetstack.io "googlecas-issuer" deleted + + $ kubectl delete -n ${NAMESPACE} certificate googlecas-cert + certificate.cert-manager.io "googlecas-cert" deleted diff --git a/smoke-test.yaml b/smoke-test.yaml deleted file mode 100644 index 451dc01..0000000 --- a/smoke-test.yaml +++ /dev/null @@ -1,112 +0,0 @@ -# This smoke test manifest is used in the smoke-test image to run the test -# suite. The testrunner is run in smoke-test.Dockerfile. -# -# To see how this testrunner works, the only way is to read the bash.go: -# https://github.com/GoogleCloudPlatform/marketplace-testrunner/blob/4245fa9/tests/bash.go -actions: - # - name: hang for debugging purposes - # bashTest: - # script: sleep 1200 # 20 minutes - - name: kubectl smoke test - bashTest: - script: kubectl version - expect: - exitCode: - equals: 0 - - name: Create a self-signed Issuer and a Certificate - bashTest: - script: | - kubectl apply -n ${NAMESPACE} -f - < Date: Tue, 29 Jun 2021 21:36:58 +0200 Subject: [PATCH 13/14] smoke-test: only "kubectl logs" on failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running stern immediately (kubectl logs was giving similar results), the following error would show: Error: failed to set up watch: failed to set up watch: unknown (get pods) To get around this issue, the logs are now only fetched on error. Signed-off-by: Maël Valais --- smoke-test.Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/smoke-test.Dockerfile b/smoke-test.Dockerfile index f21e0bd..7467263 100644 --- a/smoke-test.Dockerfile +++ b/smoke-test.Dockerfile @@ -21,4 +21,4 @@ COPY smoke-test.t . ENV NAMESPACE=test -CMD ["sh", "-c", "stern -A -l app.kubernetes.io/name=jetstack-secure-gcm & cram smoke-test.t"] +CMD ["sh", "-c", "cram smoke-test.t || (stern -A -l app.kubernetes.io/name=jetstack-secure-gcm; exit 1)"] From d745a4a989d1a56c016066fc9e98d4e5e292ca85 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ma=C3=ABl=20Valais?= Date: Wed, 30 Jun 2021 13:22:21 +0200 Subject: [PATCH 14/14] cram tests: disable GitHub highlighting for .t files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Maël Valais --- .gitattributes | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..4e0abbc --- /dev/null +++ b/.gitattributes @@ -0,0 +1,6 @@ +# The .t files are cram tests. The GitHub syntax highlighter [1] does not know +# about the cram format and thinks they are Perl files. So we disabled the +# GitHub highlighter altogether for these files. + +*.t text linguist-language=Text +