Skip to content

Commit 0c2d35a

Browse files
Jobs snapshot cleaner (#94)
* feat(jobs):snapshot cleaner * readme * having fun with sdk wrapped errors * improvements * review * update readme
1 parent 0a2c4f2 commit 0c2d35a

File tree

8 files changed

+248
-1
lines changed

8 files changed

+248
-1
lines changed

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ Table of Contents:
8383
| **[Serverless Jobs Hello World](jobs/terraform-hello-world/README.md)** <br/> An example of building a container image and running it as a Serverless Job using Terraform. | N/A | [Terraform]-[Console] |
8484
| **[Serverless MLOps](jobs/ml-ops/README.md)** <br/> An example of running a Serverless Machine Leaning workflow. | Python | [Terraform]-[Console]-[CLI] |
8585
| **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)** <br/> Use Serverless Jobs to create snapshots of your instances | Go | [Console] |
86+
| **[Instance Snapshot Cleaner](jobs/instances-snapshot-cleaner/README.md)** <br/> Use Serverless Jobs to clean old instances snapshots | Go | [Console] |
8687

8788
### 💬 Messaging and Queueing
8889

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Using apline/golang image
2+
FROM golang:1.23-alpine
3+
4+
# Set destination for COPY
5+
WORKDIR /app
6+
7+
# Copy required files
8+
COPY go.mod ./
9+
COPY go.sum ./
10+
COPY *.go ./
11+
12+
# Build the executable
13+
RUN go build -o /jobs-snapshot-cleaner
14+
15+
# Run the executable
16+
ENTRYPOINT [ "/jobs-snapshot-cleaner" ]
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Serverless Jobs for cleaning old snapshots
2+
3+
This project shows how it's possible to automate tasks using Serverless Jobs.
4+
5+
This simple example shows how to clean up snapshots after X days, it's useful to avoid a growing list of snapshots.
6+
7+
# Set-up
8+
9+
## Requirements
10+
11+
- Scaleway Account
12+
- Docker daemon running to build the image
13+
- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
14+
- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/)
15+
16+
## Step 1 : Build and push to Container registry
17+
18+
Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
19+
with containers. So first, use your terminal reach this folder and run the following commands:
20+
21+
```shell
22+
# First command is to login to container registry, you can find it in Scaleway console
23+
docker login rg.fr-par.scw.cloud/jobs-snapshot-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
24+
25+
# Here we build the image to push
26+
docker build -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
27+
28+
## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
29+
# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
30+
31+
# Push the image online to be used on Serverless Jobs
32+
docker push rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1
33+
```
34+
> [!TIP]
35+
> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
36+
37+
To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
38+
39+
## Step 2: Creating the Job Definition
40+
41+
On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
42+
43+
1. On Container image, select the image you created in the step before.
44+
1. You can set the image name to something clear like `jobs-snapshot-cleaner` too.
45+
1. For the region you can select the one you prefer :)
46+
1. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
47+
1. To schedule your job for example every two days at 2am, you can set the cron to `0 2 */2 * *`.
48+
1. Important: advanced option, you need to set the following environment variables:
49+
50+
> [!TIP]
51+
> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
52+
53+
- `SCW_DELETE_AFTER_DAYS`: number of days after the snapshots will be deleted
54+
- `SCW_PROJECT_ID`: project you want to clean up
55+
- `SCW_ZONE`: you need to give the ZONE of your snapshot you want to clean, like `fr-par-2`
56+
- `SCW_ACCESS_KEY`: your access key
57+
- `SCW_SECRET_KEY`: your secret key
58+
- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID
59+
60+
* Then click "create job"
61+
62+
## Step 3: Run the job
63+
64+
On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
65+
66+
## Troubleshooting
67+
68+
If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
69+
70+
# Possible improvements
71+
72+
You can exercice by adding the following features:
73+
74+
- Add tags to exclude
75+
- Add alerts if a Job goes in error
76+
- Use Secret Manager instead of job environment variables
77+
- Support multiple zones dans projects
78+
79+
# Additional content
80+
81+
- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
82+
- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
83+
- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
84+
- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
module github.com/scaleway/serverless-examples/jobs/instances-snapshot
2+
3+
go 1.23
4+
5+
require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32
6+
7+
require gopkg.in/yaml.v2 v2.4.0 // indirect
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
2+
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
3+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c=
4+
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks=
5+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
6+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
7+
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
8+
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
package main
2+
3+
import (
4+
"encoding/json"
5+
"errors"
6+
"fmt"
7+
"os"
8+
"strconv"
9+
"time"
10+
11+
"github.com/scaleway/scaleway-sdk-go/api/instance/v1"
12+
"github.com/scaleway/scaleway-sdk-go/scw"
13+
)
14+
15+
const (
16+
envOrgID = "SCW_DEFAULT_ORGANIZATION_ID"
17+
envAccessKey = "SCW_ACCESS_KEY"
18+
envSecretKey = "SCW_SECRET_KEY"
19+
envProjectID = "SCW_PROJECT_ID"
20+
envZone = "SCW_ZONE"
21+
22+
// envDeleteAfter name of env variable to deleter older images.
23+
envDeleteAfter = "SCW_DELETE_AFTER_DAYS"
24+
25+
// defaultDaysDeleteAfter is the default days value for older images to be deleted.
26+
defaultDaysDeleteAfter = int(90)
27+
)
28+
29+
func main() {
30+
fmt.Println("cleaning instances snapshots...")
31+
32+
// Create a Scaleway client with credentials from environment variables.
33+
client, err := scw.NewClient(
34+
// Get your organization ID at https://console.scaleway.com/organization/settings
35+
scw.WithDefaultOrganizationID(os.Getenv(envOrgID)),
36+
37+
// Get your credentials at https://console.scaleway.com/iam/api-keys
38+
scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)),
39+
40+
// Get more about our availability
41+
// zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/
42+
scw.WithDefaultRegion(scw.RegionFrPar),
43+
)
44+
if err != nil {
45+
panic(err)
46+
}
47+
48+
// Create SDK objects for Scaleway Instance product
49+
instanceAPI := instance.NewAPI(client)
50+
51+
deleteAfterDays := defaultDaysDeleteAfter
52+
53+
deleteAfterDaysVar := os.Getenv(envDeleteAfter)
54+
55+
if deleteAfterDaysVar != "" {
56+
deleteAfterDays, err = strconv.Atoi(deleteAfterDaysVar)
57+
if err != nil {
58+
panic(err)
59+
}
60+
}
61+
62+
if err := cleanSnapshots(deleteAfterDays, instanceAPI); err != nil {
63+
var precondErr *scw.PreconditionFailedError
64+
65+
if errors.As(err, &precondErr) {
66+
fmt.Println("\nExtracted Error Details:")
67+
fmt.Println("Precondition:", precondErr.Precondition)
68+
fmt.Println("Help Message:", precondErr.HelpMessage)
69+
70+
// Decode RawBody (if available)
71+
if len(precondErr.RawBody) > 0 {
72+
var parsedBody map[string]interface{}
73+
if json.Unmarshal(precondErr.RawBody, &parsedBody) == nil {
74+
fmt.Println("RawBody (Decoded):", parsedBody)
75+
} else {
76+
fmt.Println("RawBody (Raw):", string(precondErr.RawBody))
77+
}
78+
}
79+
}
80+
panic(err)
81+
}
82+
}
83+
84+
// cleanSnapshots when called will clean snapshots in the project (if specified)
85+
// that are older than the number of days.
86+
func cleanSnapshots(days int, instanceAPI *instance.API) error {
87+
// Get the list of all snapshots
88+
snapshotsList, err := instanceAPI.ListSnapshots(&instance.ListSnapshotsRequest{
89+
Zone: scw.Zone(os.Getenv(envZone)),
90+
Project: scw.StringPtr(os.Getenv(envProjectID)),
91+
},
92+
scw.WithAllPages())
93+
if err != nil {
94+
return fmt.Errorf("error while listing snapshots %w", err)
95+
}
96+
97+
const hoursPerDay = 24
98+
99+
currentTime := time.Now()
100+
101+
// For each snapshot, check conditions
102+
for _, snapshot := range snapshotsList.Snapshots {
103+
// Check if snapshot is in ready state and if it's older than the number of days definied.
104+
if snapshot.State == instance.SnapshotStateAvailable && (currentTime.Sub(*snapshot.CreationDate).Hours()/hoursPerDay) > float64(days) {
105+
fmt.Printf("\nDeleting snapshot <%s>:%s created at: %s\n", snapshot.ID, snapshot.Name, snapshot.CreationDate.Format(time.RFC3339))
106+
107+
// Delete snapshot found.
108+
err := instanceAPI.DeleteSnapshot(&instance.DeleteSnapshotRequest{
109+
SnapshotID: snapshot.ID,
110+
Zone: snapshot.Zone,
111+
})
112+
if err != nil {
113+
return fmt.Errorf("error while deleting snapshot: %w", err)
114+
}
115+
}
116+
}
117+
118+
return nil
119+
}
120+
121+
// Check for mandatory variables before starting to work.
122+
func init() {
123+
mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envZone, envProjectID}
124+
125+
for idx := range mandatoryVariables {
126+
if os.Getenv(mandatoryVariables[idx]) == "" {
127+
panic("missing environment variable " + mandatoryVariables[idx])
128+
}
129+
}
130+
}

jobs/instances-snapshot/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,4 @@ COPY *.go ./
1313
RUN go build -o /jobs-snapshot
1414

1515
# Run the executable
16-
CMD [ "/jobs-snapshot" ]
16+
ENTRYPOINT [ "/jobs-snapshot" ]

jobs/instances-snapshot/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ This example is very simple, it generates snapshots of your desired Instance.
1111
- Scaleway Account
1212
- Docker daemon running to build the image
1313
- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
14+
- API keys generated, Access Key and Secret Key [doc here](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/)
1415

1516
## Step 1 : Build and push to Container registry
1617

0 commit comments

Comments
 (0)