Skip to content

Prune empty directories before uploading #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Prune empty directories before uploading #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3dcdb5a
prune empty dirs before upload. observed a case where file by file cl…
seanshahkarami Oct 6, 2022
b96b248
added mindepth 1 to ensure we don't accidentally delete working direc…
seanshahkarami Oct 6, 2022
ce6cd7a
leave all directory cleanup to pre-step to avoid mutating tree during…
seanshahkarami Oct 6, 2022
4650c6b
removed duplicate find_uploads_in_cwd func
seanshahkarami Oct 6, 2022
be77707
only find nonempty dirs in find_uploads_in_cwd to eliminate need for …
seanshahkarami Oct 6, 2022
66e43e2
combined liveness touch
seanshahkarami Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM alpine:3.12
FROM alpine:3.16.2
WORKDIR /app
RUN apk add --no-cache openssh-client rsync bash
COPY . .
Expand Down
25 changes: 6 additions & 19 deletions main.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,6 @@ attempt_to_cleanup_dir() {
# * shared pid sidecar (wolfgang shared this with me: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/)
rsync_supervisor &

find_uploads_in_cwd() {
find . -mindepth 3 -maxdepth 3 -type d
}

find_uploads_in_cwd() {
# NOTE(sean) upload data is mounted at /uploads with leaf files like:
#
Expand All @@ -162,7 +158,7 @@ find_uploads_in_cwd() {
# with job
# path: ./Pluginctl/test-pipeline/0.2.8/1649746359093671949-a31446e4291ac3a04a3c331e674252a63ee95604/data
# depth: 1 2 3 4 5 files
find . -mindepth 3 -maxdepth 4 -type d | awk -F/ '
find . -mindepth 3 -maxdepth 4 -type d ! -empty | awk -F/ '
# match paths which ends in version/timestamp-shasum
$3 ~ /[0-9]+\.[0-9]+\.[0-9]+/ && $4 ~ /[0-9]+-[0-9a-f]+/
$4 ~ /[0-9]+\.[0-9]+\.[0-9]+/ && $5 ~ /[0-9]+-[0-9a-f]+/
Expand Down Expand Up @@ -216,26 +212,17 @@ while true; do
fatal "failed to resolve upload server and update /etc/hosts."
fi

echo "scanning and uploading files..."
cd /uploads

find_uploads_in_cwd | while read -r dir; do
if ! ls "${dir}" | grep -q .; then
echo "skipping dir with no uploads: ${dir}"
attempt_to_cleanup_dir "${dir}"
continue
fi
echo "cleaning up empty upload dirs..."
find . -mindepth 1 -type d -empty -delete

echo "scanning and uploading files..."
find_uploads_in_cwd | while read -r dir; do
echo "uploading: ${dir}"
upload_dir "${dir}"
touch /tmp/rsync_healthy

echo "cleaning up: ${dir}"
attempt_to_cleanup_dir "${dir}"

# indicate that we are healthy and making progress after each transfer completes
touch /tmp/healthy

touch /tmp/rsync_healthy /tmp/healthy
echo "done: ${dir}"
done

Expand Down