diff --git a/README.md b/README.md index 3b07610..79a13eb 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ Table of Contents: | **[Serverless MLOps](jobs/ml-ops/README.md)**
An example of running a Serverless Machine Leaning workflow. | Python | [Terraform]-[Console]-[CLI] | | **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)**
Use Serverless Jobs to create snapshots of your instances | Go | [Console] | | **[Instance Snapshot Cleaner](jobs/instances-snapshot-cleaner/README.md)**
Use Serverless Jobs to clean old instances snapshots | Go | [Console] | +| **[Registry Tag Cleaner](jobs/registry-version-based-retention/README.md)**
Use Serverless Jobs to keep a desired amount of tags for each image | Go | [Console] | +| **[Registry Empty Image Cleaner](jobs/registry-empty-ressource-cleaner/README.md)**
Use Serverless Jobs to clean container registry empty namespaces and images | Go | [Console] | ### 💬 Messaging and Queueing diff --git a/jobs/registry-empty-ressource-cleaner/Dockerfile b/jobs/registry-empty-ressource-cleaner/Dockerfile new file mode 100644 index 0000000..0546f2e --- /dev/null +++ b/jobs/registry-empty-ressource-cleaner/Dockerfile @@ -0,0 +1,18 @@ +# Use the alpine version of the golang image as the base image +FROM golang:1.24-alpine + +# Set the working directory inside the container to /app +WORKDIR /app + +# Copy the go.mod and go.sum files to the working directory +COPY go.mod ./ +COPY go.sum ./ + +# Copy the Go source files to the working directory +COPY *.go ./ + +# Build the executable named reg-clean from the Go source files +RUN go build -o /reg-namespace-clean +# Set the default command to run the reg-clean executable when the container starts + +CMD ["/reg-namespace-clean"] diff --git a/jobs/registry-empty-ressource-cleaner/README.md b/jobs/registry-empty-ressource-cleaner/README.md new file mode 100644 index 0000000..dafddf5 --- /dev/null +++ b/jobs/registry-empty-ressource-cleaner/README.md @@ -0,0 +1,72 @@ +# Scaleway Container Registry Cleaner + +This project helps you clean up your Container Registry by deleting namespaces that do not contain any images. + +## Requirements + +- Scaleway Account +- Docker daemon running to build the image +- Container registry namespace created, for this example we assume that your namespace name is `registry-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/) +- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/) + +## Step 1: Build and Push to Container Registry + +Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works +with containers. So first, use your terminal reach this folder and run the following commands: + +```shell +# The first command logs in to the container registry; you can find it in the Scaleway console +docker login rg.fr-par.scw.cloud/registry-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY" + +# The next command builds the image to push +docker build -t rg.fr-par.scw.cloud/registry-cleaner/empty-namespaces:v1 . + +## TIP: For Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture +# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/registry-cleaner/empty-namespaces:v1 . + +# This command pushes the image online to be used on Serverless Jobs +docker push rg.fr-par.scw.cloud/registry-cleaner/empty-namespaces:v1 +``` + +> [!TIP] +> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case. + +To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry. + +## Step 2: Creating the Job Definition + +On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par + +1. On Container image, select the image you created in the step before. +2. You can set the image name to something clear like `registry-namespace-cleaner` too. +3. For the region you can select the one you prefer :) +4. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory. +5. To schedule your job for example every night at 2am, you can set the cron to `0 2 * * *`. +6. Important: advanced option, you need to set the following environment variables: + +> [!TIP] +> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/). + +- **Environment Variables**: Set the required environment variables: + - `SCW_DEFAULT_ORGANIZATION_ID`: Your Scaleway organization ID. + - `SCW_ACCESS_KEY`: Your Scaleway API access key. + - `SCW_SECRET_KEY`: Your Scaleway API secret key. + - `SCW_PROJECT_ID`: Your Scaleway project ID. + - `SCW_NO_DRY_RUN`: Set to `true` to delete namespaces; otherwise, it will perform a dry run. + +* Then click "Create Job" + +## Step 3: Run the job + +On your created Job Definition, just click the button "Run Job" and within seconds it should be successful. + +## Troubleshooting + +If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error. + +# Additional content + +- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/) +- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/) +- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/) +- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/) diff --git a/jobs/registry-empty-ressource-cleaner/go.mod b/jobs/registry-empty-ressource-cleaner/go.mod new file mode 100644 index 0000000..7bfa4ac --- /dev/null +++ b/jobs/registry-empty-ressource-cleaner/go.mod @@ -0,0 +1,12 @@ +module github.com/scaleway/serverless-examples/jobs/registry-empty-ressource-cleaner + +go 1.24.0 + +require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 + +require ( + github.com/kr/pretty v0.3.1 // indirect + github.com/rogpeppe/go-internal v1.13.1 // indirect + gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect + gopkg.in/yaml.v2 v2.4.0 // indirect +) diff --git a/jobs/registry-empty-ressource-cleaner/go.sum b/jobs/registry-empty-ressource-cleaner/go.sum new file mode 100644 index 0000000..e86e798 --- /dev/null +++ b/jobs/registry-empty-ressource-cleaner/go.sum @@ -0,0 +1,19 @@ +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= +github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= +github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= +github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= diff --git a/jobs/registry-empty-ressource-cleaner/main.go b/jobs/registry-empty-ressource-cleaner/main.go new file mode 100644 index 0000000..e964255 --- /dev/null +++ b/jobs/registry-empty-ressource-cleaner/main.go @@ -0,0 +1,75 @@ +package main + +import ( + "log/slog" + "os" + "strings" + + "github.com/scaleway/scaleway-sdk-go/scw" +) + +// Constants for environment variable names used to configure the application +const ( + envOrgID = "SCW_DEFAULT_ORGANIZATION_ID" // Scaleway organization ID + envAccessKey = "SCW_ACCESS_KEY" // Scaleway API access key + envSecretKey = "SCW_SECRET_KEY" // Scaleway API secret key + envProjectID = "SCW_PROJECT_ID" // Scaleway project ID + + // If set to "true", older tags will be deleted. + // Otherwise, only a dry run will be performed + envNoDryRun = "SCW_NO_DRY_RUN" +) + +// Check for mandatory variables before starting to work. +func init() { + // Slice of environmental variables that must be set for the application to run + mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envProjectID} + + // Iterate through the slice and check if any variables are not set + for idx := range mandatoryVariables { + if os.Getenv(mandatoryVariables[idx]) == "" { + panic("missing environment variable " + mandatoryVariables[idx]) + } + } +} + +func main() { + slog.Info("cleaning container registry tags...") + + // Create a Scaleway client with credentials provided via environment variables. + // The client is used to interact with the Scaleway API + client, err := scw.NewClient( + // Get your organization ID at https://console.scaleway.com/organization/settings + scw.WithDefaultOrganizationID(os.Getenv(envOrgID)), + + // Get your credentials at https://console.scaleway.com/iam/api-keys + scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)), + + // Get more about our availability + // zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/ + scw.WithDefaultRegion(scw.RegionFrPar), + ) + if err != nil { + panic(err) + } + + // Create a new instance of RegistryAPI, passing the Scaleway client and the project ID + // RegistryAPI is a custom interface for interacting with the Scaleway container registry + regAPI := NewRegistryAPI(client, os.Getenv(scw.ScwDefaultProjectIDEnv)) + + // Determine whether to perform a dry run or delete the tags + // Default behavior is to perform a dry run (no deletion) + dryRun := true + noDryRunEnv := os.Getenv(envNoDryRun) + + // If the SCW_NO_DRY_RUN environment variable is set to "true", + // the tags will be deleted; otherwise, only a dry run will be performed + if strings.EqualFold(noDryRunEnv, "true") { + dryRun = false + } + + // Delete the tags or perform a dry run, depending on the dryRun flag + if err := regAPI.DeleteEmptyNamespace(dryRun); err != nil { + panic(err) + } +} diff --git a/jobs/registry-empty-ressource-cleaner/registry.go b/jobs/registry-empty-ressource-cleaner/registry.go new file mode 100644 index 0000000..c1dd886 --- /dev/null +++ b/jobs/registry-empty-ressource-cleaner/registry.go @@ -0,0 +1,57 @@ +package main + +import ( + "fmt" + "log/slog" + + registry "github.com/scaleway/scaleway-sdk-go/api/registry/v1" + "github.com/scaleway/scaleway-sdk-go/scw" +) + +// RegistryAPI represents a Scaleway Container Registry accessor and extends +// capabilities to clean images. It allows you to manage container images and tags +// across one or more projects, and provides options to delete image tags safely or +// with caution. +type RegistryAPI struct { + // regClient Scaleway Container Registry accessor. + regClient *registry.API + + // projectID specifies a project to be scoped for operations. If this field + // is nil, operations will be performed on all available projects. + projectID *string +} + +// NewRegistryAPI creates a new RegistryAPI to manage the Scaleway Container Registry API. +// It initializes the RegistryAPI struct with the provided Scaleway SDK client and a project +// ID. If the projectID is empty, it will not be passed to the Scaleway SDK, allowing operations +// on all projects. +func NewRegistryAPI(client *scw.Client, projectID string) *RegistryAPI { + return &RegistryAPI{ + regClient: registry.NewAPI(client), + projectID: scw.StringPtr(projectID), + } +} + +func (r *RegistryAPI) DeleteEmptyNamespace(dryRun bool) error { + namespaces, err := r.regClient.ListNamespaces(®istry.ListNamespacesRequest{ProjectID: r.projectID}, scw.WithAllPages()) + if err != nil { + return fmt.Errorf("error listing registry namespaces %w", err) + } + slog.Info("DryRun ENABLED") + + for _, namespace := range namespaces.Namespaces { + if namespace.Status == registry.NamespaceStatusReady && namespace.ImageCount == 0 { + slog.Info("deleteing namespace", slog.String("name", namespace.Name), slog.String("id", namespace.ID)) + if !dryRun { + _, err := r.regClient.DeleteNamespace(®istry.DeleteNamespaceRequest{ + NamespaceID: namespace.ID, + }) + if err != nil { + return fmt.Errorf("error deleting namesapce %w", err) + } + } + } + } + + return nil +} diff --git a/jobs/registry-version-based-retention/Dockerfile b/jobs/registry-version-based-retention/Dockerfile new file mode 100644 index 0000000..d85ab86 --- /dev/null +++ b/jobs/registry-version-based-retention/Dockerfile @@ -0,0 +1,18 @@ +# Use the alpine version of the golang image as the base image +FROM golang:1.24-alpine + +# Set the working directory inside the container to /app +WORKDIR /app + +# Copy the go.mod and go.sum files to the working directory +COPY go.mod ./ +COPY go.sum ./ + +# Copy the Go source files to the working directory +COPY *.go ./ + +# Build the executable named reg-clean from the Go source files +RUN go build -o /reg-clean + +# Set the default command to run the reg-clean executable when the container starts +CMD ["/reg-clean"] diff --git a/jobs/registry-version-based-retention/README.md b/jobs/registry-version-based-retention/README.md new file mode 100644 index 0000000..368b9a2 --- /dev/null +++ b/jobs/registry-version-based-retention/README.md @@ -0,0 +1,73 @@ +# Scaleway Container Registry Tag Cleaner + +This project aims to clean up Scaleway Container Registry tags to keep only the N latest tags for each image, which is useful for managing disk space and organizing the registry. + +## Requirements + +- Scaleway Account +- Docker daemon running to build the image +- Container registry namespace created, for this example we assume that your namespace name is `registry-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/) +- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/) + +## Step 1: Build and Push to Container Registry + +Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works +with containers. So first, use your terminal reach this folder and run the following commands: + +```shell +# First, log in to the container registry. You can find your login details in the Scaleway console. +docker login rg.fr-par.scw.cloud/registry-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY" + +# Build the image to push. +docker build -t rg.fr-par.scw.cloud/registry-cleaner/versions-retention:v1 . + +## TIP: For Apple Silicon or other ARM processors, use the following command as Serverless Jobs supports the amd64 architecture. +# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/registry-cleaner/versions-retention:v1 . + +# Push the image online to be used on Serverless Jobs. +docker push rg.fr-par.scw.cloud/registry-cleaner/versions-retention:v1 +``` + +> [!TIP] +> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case. + +To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry. + +## Step 2: Creating the Job Definition + +On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par + +1. On Container image, select the image you created in the step before. +2. You can set the image name to something clear like `registry-version-retention` too. +3. For the region you can select the one you prefer :) +4. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory. +5. To schedule your job for example every night at 2am, you can set the cron to `0 2 * * *`. +6. Important: advanced option, you need to set the following environment variables: + +> [!TIP] +> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/). + +- **Environment Variables**: Set the required environment variables: + - `SCW_DEFAULT_ORGANIZATION_ID`: Your Scaleway organization ID. + - `SCW_ACCESS_KEY`: Your Scaleway API access key. + - `SCW_SECRET_KEY`: Your Scaleway API secret key. + - `SCW_PROJECT_ID`: Your Scaleway project ID. + - `SCW_NUMBER_VERSIONS_TO_KEEP`: The number of latest tags to keep for each image. + - `SCW_NO_DRY_RUN`: Set to `true` to delete namespaces; otherwise, it will perform a dry run. + +* Then click "Create Job" + +## Step 3: Run the job + +On your created Job Definition, just click the button "Run Job" and within seconds it should be successful. + +## Troubleshooting + +If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error. + +# Additional content + +- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/) +- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/) +- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/) +- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/) diff --git a/jobs/registry-version-based-retention/go.mod b/jobs/registry-version-based-retention/go.mod new file mode 100644 index 0000000..417f265 --- /dev/null +++ b/jobs/registry-version-based-retention/go.mod @@ -0,0 +1,12 @@ +module github.com/scaleway/serverless-examples/jobs/registry-version-based-retention + +go 1.24.0 + +require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 + +require ( + github.com/kr/pretty v0.3.1 // indirect + github.com/rogpeppe/go-internal v1.13.1 // indirect + gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect + gopkg.in/yaml.v2 v2.4.0 // indirect +) diff --git a/jobs/registry-version-based-retention/go.sum b/jobs/registry-version-based-retention/go.sum new file mode 100644 index 0000000..e86e798 --- /dev/null +++ b/jobs/registry-version-based-retention/go.sum @@ -0,0 +1,19 @@ +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= +github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs= +github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= +github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= diff --git a/jobs/registry-version-based-retention/main.go b/jobs/registry-version-based-retention/main.go new file mode 100644 index 0000000..a49ad3d --- /dev/null +++ b/jobs/registry-version-based-retention/main.go @@ -0,0 +1,90 @@ +package main + +import ( + "log/slog" + "os" + "strconv" + "strings" + + "github.com/scaleway/scaleway-sdk-go/scw" +) + +// Constants for environment variable names used to configure the application +const ( + envOrgID = "SCW_DEFAULT_ORGANIZATION_ID" // Scaleway organization ID + envAccessKey = "SCW_ACCESS_KEY" // Scaleway API access key + envSecretKey = "SCW_SECRET_KEY" // Scaleway API secret key + envProjectID = "SCW_PROJECT_ID" // Scaleway project ID + + envNTagsToKeep = "SCW_NUMBER_VERSIONS_TO_KEEP" // Number of container registry tags to keep + // If set to "true", older tags will be deleted. + // Otherwise, only a dry run will be performed + envNoDryRun = "SCW_NO_DRY_RUN" +) + +// Check for mandatory variables before starting to work. +func init() { + // Slice of environmental variables that must be set for the application to run + mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envProjectID, envNTagsToKeep} + + // Iterate through the slice and check if any variables are not set + for idx := range mandatoryVariables { + if os.Getenv(mandatoryVariables[idx]) == "" { + panic("missing environment variable " + mandatoryVariables[idx]) + } + } +} + +func main() { + slog.Info("cleaning container registry tags...") + + // Create a Scaleway client with credentials provided via environment variables. + // The client is used to interact with the Scaleway API + client, err := scw.NewClient( + // Get your organization ID at https://console.scaleway.com/organization/settings + scw.WithDefaultOrganizationID(os.Getenv(envOrgID)), + + // Get your credentials at https://console.scaleway.com/iam/api-keys + scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)), + + // Get more about our availability + // zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/ + scw.WithDefaultRegion(scw.RegionFrPar), + ) + if err != nil { + panic(err) + } + + // Create a new instance of RegistryAPI, passing the Scaleway client and the project ID + // RegistryAPI is a custom interface for interacting with the Scaleway container registry + regAPI := NewRegistryAPI(client, os.Getenv(scw.ScwDefaultProjectIDEnv)) + + // Parse the number of tags to keep from the environment variable, which should be a string + numberTagsToKeep, err := strconv.Atoi(os.Getenv(envNTagsToKeep)) + if err != nil { + panic(err) + } + + // Get the tags to delete by specifying the number of tags to keep + // The function returns both the tags to be deleted and an error, if any + tagsToDelete, err := regAPI.GetTagsAfterNVersions(numberTagsToKeep) + if err != nil { + panic(err) + } + + // Determine whether to perform a dry run or delete the tags + // Default behavior is to perform a dry run (no deletion) + dryRun := true + noDryRunEnv := os.Getenv(envNoDryRun) + + // If the SCW_NO_DRY_RUN environment variable is set to "true", + // the tags will be deleted; otherwise, only a dry run will be performed + if strings.EqualFold(noDryRunEnv, "true") { + dryRun = false + } + + // Delete the tags or perform a dry run, depending on the dryRun flag + if err := regAPI.DeleteTags(tagsToDelete, dryRun); err != nil { + panic(err) + } +} diff --git a/jobs/registry-version-based-retention/registry.go b/jobs/registry-version-based-retention/registry.go new file mode 100644 index 0000000..be3043b --- /dev/null +++ b/jobs/registry-version-based-retention/registry.go @@ -0,0 +1,120 @@ +package main + +import ( + "fmt" + "log/slog" + "strings" + + registry "github.com/scaleway/scaleway-sdk-go/api/registry/v1" + "github.com/scaleway/scaleway-sdk-go/scw" +) + +// RegistryAPI represents a Scaleway Container Registry accessor and extends +// capabilities to clean images. It allows you to manage container images and tags +// across one or more projects, and provides options to delete image tags safely or +// with caution. +type RegistryAPI struct { + // regClient Scaleway Container Registry accessor. + regClient *registry.API + + // projectID specifies a project to be scoped for operations. If this field + // is nil, operations will be performed on all available projects. + projectID *string + + // disableProtection if set to true, it will allow deletion of images that + // might be in use by Serverless Jobs, Functions, or Containers. This should + // be used with caution. + disableProtection bool +} + +// NewRegistryAPI creates a new RegistryAPI to manage the Scaleway Container Registry API. +// It initializes the RegistryAPI struct with the provided Scaleway SDK client and a project +// ID. If the projectID is empty, it will not be passed to the Scaleway SDK, allowing operations +// on all projects. +func NewRegistryAPI(client *scw.Client, projectID string) *RegistryAPI { + return &RegistryAPI{ + regClient: registry.NewAPI(client), + projectID: scw.StringPtr(projectID), + disableProtection: false, + } +} + +// GetTagsAfterNVersions returns a list of image tags that should be deleted, based on the number of +// versions to keep. This function lists all container images and their associated tags, and determines +// which tags are beyond the specified count of versions to retain. +// +// The numberVersionsToKeep parameter specifies how many versions of each image should be preserved. +// If this value is less than or equal to 1, an error is returned to prevent accidental deletion of +// all image tags. +// +// The function returns a slice of pointers to registry.Tag structures representing the tags to be +// deleted, or an error if any issues arise during the process. +func (r *RegistryAPI) GetTagsAfterNVersions(numberVersionsToKeep int) ([]*registry.Tag, error) { + images, err := r.regClient.ListImages(®istry.ListImagesRequest{ProjectID: r.projectID}, scw.WithAllPages()) + if err != nil { + return nil, fmt.Errorf("error listing container images: %w", err) + } + + if numberVersionsToKeep <= 1 { + return nil, fmt.Errorf("number of versions to keep <= 1 is dangerous") + } + + tagsToDelete := make([]*registry.Tag, 0) + + for _, image := range images.Images { + tags, err := r.regClient.ListTags(®istry.ListTagsRequest{ + ImageID: image.ID, + OrderBy: registry.ListTagsRequestOrderByCreatedAtDesc, + }, scw.WithAllPages()) + if err != nil { + return nil, fmt.Errorf("error listing tags for image %s: %w", image.Name, err) + } + + if len(tags.Tags) <= numberVersionsToKeep { + // not enough versions to delete, skipping + continue + } + + slog.Info("appending tags for image: " + image.Name) + + tagsToDelete = append(tagsToDelete, tags.Tags[numberVersionsToKeep:]...) + } + + return tagsToDelete, nil +} + +// DeleteTags deletes the specified image tags. If the dryRun parameter is set to true, +// the function will log the tags that would be deleted without actually performing the +// deletion. If dryRun is false, the function will proceed to delete the tags. +// +// The tagsToDelete parameter is a slice of pointers to registry.Tag structures representing +// the tags to be deleted. +// +// The function logs informational messages about the operations being performed and returns +// an error if any issues arise during the process. +func (r *RegistryAPI) DeleteTags(tagsToDelete []*registry.Tag, dryRun bool) error { + if dryRun { + slog.Info("Dry run mode ENABLED") + + for _, tag := range tagsToDelete { + slog.Info("dry-run: deleting tag:", slog.String("tag name", tag.Name), slog.String("tagID", tag.ID)) + } + } else { + slog.Warn("Dry run DISABLED") + + for _, tag := range tagsToDelete { + if strings.EqualFold(tag.Name, "latest") { + slog.Info("skipping deletion of latest tag", slog.String("tag name", tag.Name)) + + continue + } + + _, err := r.regClient.DeleteTag(®istry.DeleteTagRequest{TagID: tag.ID}) + if err != nil { + return fmt.Errorf("error deleting registry tag %s (id %s): %w", tag.Name, tag.ID, err) + } + } + } + + return nil +}