diff --git a/README.md b/README.md index c17d2cf..3b07610 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,7 @@ Table of Contents: | **[Serverless Jobs Hello World](jobs/terraform-hello-world/README.md)**
An example of building a container image and running it as a Serverless Job using Terraform. | N/A | [Terraform]-[Console] | | **[Serverless MLOps](jobs/ml-ops/README.md)**
An example of running a Serverless Machine Leaning workflow. | Python | [Terraform]-[Console]-[CLI] | | **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)**
Use Serverless Jobs to create snapshots of your instances | Go | [Console] | +| **[Instance Snapshot Cleaner](jobs/instances-snapshot-cleaner/README.md)**
Use Serverless Jobs to clean old instances snapshots | Go | [Console] | ### 💬 Messaging and Queueing diff --git a/jobs/instances-snapshot-cleaner/Dockerfile b/jobs/instances-snapshot-cleaner/Dockerfile new file mode 100644 index 0000000..7e2e2cc --- /dev/null +++ b/jobs/instances-snapshot-cleaner/Dockerfile @@ -0,0 +1,16 @@ +# Using apline/golang image +FROM golang:1.23-alpine + +# Set destination for COPY +WORKDIR /app + +# Copy required files +COPY go.mod ./ +COPY go.sum ./ +COPY *.go ./ + +# Build the executable +RUN go build -o /jobs-snapshot-cleaner + +# Run the executable +ENTRYPOINT [ "/jobs-snapshot-cleaner" ] diff --git a/jobs/instances-snapshot-cleaner/README.md b/jobs/instances-snapshot-cleaner/README.md new file mode 100644 index 0000000..c3009db --- /dev/null +++ b/jobs/instances-snapshot-cleaner/README.md @@ -0,0 +1,84 @@ +# Serverless Jobs for cleaning old snapshots + +This project shows how it's possible to automate tasks using Serverless Jobs. + +This simple example shows how to clean up snapshots after X days, it's useful to avoid a growing list of snapshots. + +# Set-up + +## Requirements + +- Scaleway Account +- Docker daemon running to build the image +- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/) +- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/) + +## Step 1 : Build and push to Container registry + +Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works +with containers. So first, use your terminal reach this folder and run the following commands: + +```shell +# First command is to login to container registry, you can find it in Scaleway console +docker login rg.fr-par.scw.cloud/jobs-snapshot-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY" + +# Here we build the image to push +docker build -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 . + +## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture +# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 . + +# Push the image online to be used on Serverless Jobs +docker push rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 +``` +> [!TIP] +> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case. + +To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry. + +## Step 2: Creating the Job Definition + +On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par + +1. On Container image, select the image you created in the step before. +1. You can set the image name to something clear like `jobs-snapshot-cleaner` too. +1. For the region you can select the one you prefer :) +1. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory. +1. To schedule your job for example every two days at 2am, you can set the cron to `0 2 */2 * *`. +1. Important: advanced option, you need to set the following environment variables: + +> [!TIP] +> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/). + +- `SCW_DELETE_AFTER_DAYS`: number of days after the snapshots will be deleted +- `SCW_PROJECT_ID`: project you want to clean up +- `SCW_ZONE`: you need to give the ZONE of your snapshot you want to clean, like `fr-par-2` +- `SCW_ACCESS_KEY`: your access key +- `SCW_SECRET_KEY`: your secret key +- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID + +* Then click "create job" + +## Step 3: Run the job + +On your created Job Definition, just click the button "Run Job" and within seconds it should be successful. + +## Troubleshooting + +If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error. + +# Possible improvements + +You can exercice by adding the following features: + +- Add tags to exclude +- Add alerts if a Job goes in error +- Use Secret Manager instead of job environment variables +- Support multiple zones dans projects + +# Additional content + +- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/) +- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/) +- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/) +- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/) diff --git a/jobs/instances-snapshot-cleaner/go.mod b/jobs/instances-snapshot-cleaner/go.mod new file mode 100644 index 0000000..06ae8bf --- /dev/null +++ b/jobs/instances-snapshot-cleaner/go.mod @@ -0,0 +1,7 @@ +module github.com/scaleway/serverless-examples/jobs/instances-snapshot + +go 1.23 + +require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 + +require gopkg.in/yaml.v2 v2.4.0 // indirect diff --git a/jobs/instances-snapshot-cleaner/go.sum b/jobs/instances-snapshot-cleaner/go.sum new file mode 100644 index 0000000..e287b70 --- /dev/null +++ b/jobs/instances-snapshot-cleaner/go.sum @@ -0,0 +1,8 @@ +github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI= +github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c= +github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= diff --git a/jobs/instances-snapshot-cleaner/main.go b/jobs/instances-snapshot-cleaner/main.go new file mode 100644 index 0000000..99db6de --- /dev/null +++ b/jobs/instances-snapshot-cleaner/main.go @@ -0,0 +1,130 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "strconv" + "time" + + "github.com/scaleway/scaleway-sdk-go/api/instance/v1" + "github.com/scaleway/scaleway-sdk-go/scw" +) + +const ( + envOrgID = "SCW_DEFAULT_ORGANIZATION_ID" + envAccessKey = "SCW_ACCESS_KEY" + envSecretKey = "SCW_SECRET_KEY" + envProjectID = "SCW_PROJECT_ID" + envZone = "SCW_ZONE" + + // envDeleteAfter name of env variable to deleter older images. + envDeleteAfter = "SCW_DELETE_AFTER_DAYS" + + // defaultDaysDeleteAfter is the default days value for older images to be deleted. + defaultDaysDeleteAfter = int(90) +) + +func main() { + fmt.Println("cleaning instances snapshots...") + + // Create a Scaleway client with credentials from environment variables. + client, err := scw.NewClient( + // Get your organization ID at https://console.scaleway.com/organization/settings + scw.WithDefaultOrganizationID(os.Getenv(envOrgID)), + + // Get your credentials at https://console.scaleway.com/iam/api-keys + scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)), + + // Get more about our availability + // zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/ + scw.WithDefaultRegion(scw.RegionFrPar), + ) + if err != nil { + panic(err) + } + + // Create SDK objects for Scaleway Instance product + instanceAPI := instance.NewAPI(client) + + deleteAfterDays := defaultDaysDeleteAfter + + deleteAfterDaysVar := os.Getenv(envDeleteAfter) + + if deleteAfterDaysVar != "" { + deleteAfterDays, err = strconv.Atoi(deleteAfterDaysVar) + if err != nil { + panic(err) + } + } + + if err := cleanSnapshots(deleteAfterDays, instanceAPI); err != nil { + var precondErr *scw.PreconditionFailedError + + if errors.As(err, &precondErr) { + fmt.Println("\nExtracted Error Details:") + fmt.Println("Precondition:", precondErr.Precondition) + fmt.Println("Help Message:", precondErr.HelpMessage) + + // Decode RawBody (if available) + if len(precondErr.RawBody) > 0 { + var parsedBody map[string]interface{} + if json.Unmarshal(precondErr.RawBody, &parsedBody) == nil { + fmt.Println("RawBody (Decoded):", parsedBody) + } else { + fmt.Println("RawBody (Raw):", string(precondErr.RawBody)) + } + } + } + panic(err) + } +} + +// cleanSnapshots when called will clean snapshots in the project (if specified) +// that are older than the number of days. +func cleanSnapshots(days int, instanceAPI *instance.API) error { + // Get the list of all snapshots + snapshotsList, err := instanceAPI.ListSnapshots(&instance.ListSnapshotsRequest{ + Zone: scw.Zone(os.Getenv(envZone)), + Project: scw.StringPtr(os.Getenv(envProjectID)), + }, + scw.WithAllPages()) + if err != nil { + return fmt.Errorf("error while listing snapshots %w", err) + } + + const hoursPerDay = 24 + + currentTime := time.Now() + + // For each snapshot, check conditions + for _, snapshot := range snapshotsList.Snapshots { + // Check if snapshot is in ready state and if it's older than the number of days definied. + if snapshot.State == instance.SnapshotStateAvailable && (currentTime.Sub(*snapshot.CreationDate).Hours()/hoursPerDay) > float64(days) { + fmt.Printf("\nDeleting snapshot <%s>:%s created at: %s\n", snapshot.ID, snapshot.Name, snapshot.CreationDate.Format(time.RFC3339)) + + // Delete snapshot found. + err := instanceAPI.DeleteSnapshot(&instance.DeleteSnapshotRequest{ + SnapshotID: snapshot.ID, + Zone: snapshot.Zone, + }) + if err != nil { + return fmt.Errorf("error while deleting snapshot: %w", err) + } + } + } + + return nil +} + +// Check for mandatory variables before starting to work. +func init() { + mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envZone, envProjectID} + + for idx := range mandatoryVariables { + if os.Getenv(mandatoryVariables[idx]) == "" { + panic("missing environment variable " + mandatoryVariables[idx]) + } + } +} diff --git a/jobs/instances-snapshot/Dockerfile b/jobs/instances-snapshot/Dockerfile index bc9c84e..7e8f9a0 100644 --- a/jobs/instances-snapshot/Dockerfile +++ b/jobs/instances-snapshot/Dockerfile @@ -13,4 +13,4 @@ COPY *.go ./ RUN go build -o /jobs-snapshot # Run the executable -CMD [ "/jobs-snapshot" ] +ENTRYPOINT [ "/jobs-snapshot" ] diff --git a/jobs/instances-snapshot/README.md b/jobs/instances-snapshot/README.md index bf6b62f..ab426a9 100644 --- a/jobs/instances-snapshot/README.md +++ b/jobs/instances-snapshot/README.md @@ -11,6 +11,7 @@ This example is very simple, it generates snapshots of your desired Instance. - Scaleway Account - Docker daemon running to build the image - Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/) +- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/) ## Step 1 : Build and push to Container registry