Skip to content

Add EC2 deployment configuration for Celery workers and Flower, and update Docker Compose for Flower service #388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add EC2 deployment configuration for Celery workers and Flower, and update Docker Compose for Flower service #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c27cfa4
Add EC2 deployment configuration for Celery workers and Flower, and u…
kartpop Sep 25, 2025
37111e5
Refactor deployment scripts for EC2: clean up whitespace and improve …
kartpop Sep 30, 2025
14db057
Refactor EC2 deployment scripts for Celery and Flower: streamline var…
kartpop Oct 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,18 @@ CELERY_ENABLE_UTC=true
# India Standard Time (UTC+05:30)
CELERY_TIMEZONE=Asia/Kolkata

# Flower Configuration (Celery Monitoring)
FLOWER_PORT=5555
FLOWER_BASIC_AUTH=admin:changethis
FLOWER_URL_PREFIX=
FLOWER_MAX_TASKS=10000
Comment on lines +77 to +81
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Flower defaults: warn and align with CLI usage.

Good to document, but note Flower won’t auto-read these names; they must be passed via CLI flags (addressed in workflows/compose). Also, don’t ship weak defaults.

Apply this diff to discourage weak creds:

-FLOWER_BASIC_AUTH=admin:changethis
+FLOWER_BASIC_AUTH=please-change-user:please-change-password
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Flower Configuration (Celery Monitoring)
FLOWER_PORT=5555
FLOWER_BASIC_AUTH=admin:changethis
FLOWER_URL_PREFIX=
FLOWER_MAX_TASKS=10000
# Flower Configuration (Celery Monitoring)
FLOWER_PORT=5555
FLOWER_BASIC_AUTH=please-change-user:please-change-password
FLOWER_URL_PREFIX=
FLOWER_MAX_TASKS=10000
🧰 Tools
🪛 dotenv-linter (3.3.0)

[warning] 79-79: [UnorderedKey] The FLOWER_BASIC_AUTH key should go before the FLOWER_PORT key

(UnorderedKey)

[warning] 81-81: [UnorderedKey] The FLOWER_MAX_TASKS key should go before the FLOWER_PORT key

(UnorderedKey)

🤖 Prompt for AI Agents 
In .env.example around lines 77 to 81, replace the weak default
FLOWER_BASIC_AUTH and add a warning that these env vars are not automatically
applied by Flower and must be passed to the Flower CLI via flags (e.g.,
--basic_auth) in compose/workflow files; remove the insecure default value and
replace it with a blank or a strong placeholder like "admin:CHANGE_ME" and add a
short comment telling users to set a strong password and not to ship with
defaults, keeping FLOWER_PORT, FLOWER_URL_PREFIX and FLOWER_MAX_TASKS but
documenting their CLI-equivalent usage.


# Callback Timeouts (in seconds)
CALLBACK_CONNECT_TIMEOUT = 3
CALLBACK_READ_TIMEOUT = 10


# EC2 Deployment Configuration for Celery Workers
EC2_HOST=production-celery-host.example.com
EC2_USER=ubuntu
EC2_KEY=production-ec2-key
Comment on lines +88 to +91
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t suggest storing SSH private keys in env files.

EC2_KEY in an env example nudges insecure practices. Use GitHub Actions secrets or SSM Parameter Store; keep only host/user hints in .env.example.

-# EC2 Deployment Configuration for Celery Workers
-EC2_HOST=production-celery-host.example.com
-EC2_USER=ubuntu
-EC2_KEY=production-ec2-key
+# EC2 Deployment Configuration (reference only; use GitHub Secrets/SSM in CI)
+EC2_HOST=production-celery-host.example.com
+EC2_USER=ubuntu
+# EC2_KEY is intentionally not listed; store as a secret (not in .env files).
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# EC2 Deployment Configuration for Celery Workers
EC2_HOST=production-celery-host.example.com
EC2_USER=ubuntu
EC2_KEY=production-ec2-key
# EC2 Deployment Configuration (reference only; use GitHub Secrets/SSM in CI)
EC2_HOST=production-celery-host.example.com
EC2_USER=ubuntu
# EC2_KEY is intentionally not listed; store as a secret (not in .env files).
🧰 Tools
🪛 dotenv-linter (3.3.0)

[warning] 91-91: [UnorderedKey] The EC2_KEY key should go before the EC2_USER key

(UnorderedKey)

🤖 Prompt for AI Agents 
In .env.example around lines 88 to 91, remove the EC2_KEY entry (do not suggest
storing SSH private keys in env files) and replace it with a short hint or
placeholder indicating that SSH keys should be managed via a secrets store
(e.g., GitHub Actions Secrets, AWS SSM/Secrets Manager) or referenced by a path
to a key stored outside source control; keep EC2_HOST and EC2_USER as examples
only, and update the project README or a deployment docs section to explicitly
instruct contributors to use secrets management for private keys and not to
commit keys to the repository.

98 changes: 95 additions & 3 deletions .github/workflows/cd-production.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Deploy AI Platform to ECS Production
name: Deploy AI Platform to Production

on:
push:
Expand Down Expand Up @@ -29,7 +29,7 @@ jobs:
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2

- name: Build and Push Docker Image
- name: Build and Push Backend Docker Image
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-repo
Expand All @@ -38,9 +38,101 @@ jobs:
docker build -t $REGISTRY/$REPOSITORY:latest ./backend
docker push $REGISTRY/$REPOSITORY:latest

- name: Deploy to ECS
- name: Build and Push Celery Docker Image
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-celery-repo
run: |
docker build -f ./backend/Dockerfile.celery -t $REGISTRY/$REPOSITORY:production ./backend
docker push $REGISTRY/$REPOSITORY:production

- name: Deploy Backend to ECS
run: |
aws ecs update-service \
--cluster ${{ github.event.repository.name }}-cluster \
--service ${{ github.event.repository.name }}-service \
--force-new-deployment

- name: Deploy Celery Workers to EC2
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-celery-repo
run: |
# Create deployment script
cat > deploy-celery.sh << 'EOF'
#!/bin/bash
set -e

# Stop existing containers
docker stop celery-prod-high celery-prod-other 2>/dev/null || true
docker rm celery-prod-high celery-prod-other 2>/dev/null || true

# Pull latest image
aws ecr get-login-password --region $3 | docker login --username AWS --password-stdin $1
docker pull $1/$2:production

# Start high priority worker
docker run -d --name celery-prod-high --restart unless-stopped \
--env-file /home/ubuntu/.env.production \
$1/$2:production \
uv run celery -A app.celery.celery_app worker --queues=high_priority --loglevel=info --concurrency=2 --hostname=prod-high@%h

# Verify high priority worker started
sleep 2
docker ps | grep celery-prod-high || { echo "High priority worker failed to start"; exit 1; }

# Start other queues worker
docker run -d --name celery-prod-other --restart unless-stopped \
--env-file /home/ubuntu/.env.production \
$1/$2:production \
uv run celery -A app.celery.celery_app worker --queues=default,low_priority,cron --loglevel=info --concurrency=2 --hostname=prod-other@%h

# Verify other queues worker started
sleep 2
docker ps | grep celery-prod-other || { echo "Other queues worker failed to start"; exit 1; }
EOF

# Copy script to EC2 and execute
echo "${{ secrets.EC2_KEY }}" > production_key.pem
chmod 600 production_key.pem
trap 'rm -f production_key.pem deploy-celery.sh' EXIT
scp -i production_key.pem -o StrictHostKeyChecking=no deploy-celery.sh ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }}:/tmp/
ssh -i production_key.pem -o StrictHostKeyChecking=no ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} "chmod +x /tmp/deploy-celery.sh && /tmp/deploy-celery.sh $REGISTRY $REPOSITORY ap-south-1"
rm -f production_key.pem deploy-celery.sh

- name: Deploy Flower to EC2
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-celery-repo
run: |
# Create flower deployment script
cat > deploy-flower.sh << 'EOF'
#!/bin/bash
set -e

# Stop existing flower container
docker stop flower-production 2>/dev/null || true
docker rm flower-production 2>/dev/null || true

# Start flower using the same celery image
docker run -d --name flower-production --restart unless-stopped \
--env-file /home/ubuntu/.env.production \
-p 5555:5555 \
$1/$2:production \
uv run celery -A app.celery.celery_app flower --port=5555

# Verify Flower started
sleep 2
docker ps | grep flower-production || { echo "Flower failed to start"; exit 1; }

# Health check
curl -f http://localhost:5555/healthcheck || { echo "Flower health check failed"; exit 1; }
EOF

# Copy script to EC2 and execute
echo "${{ secrets.EC2_KEY }}" > production_key.pem
chmod 600 production_key.pem
trap 'rm -f production_key.pem deploy-flower.sh' EXIT
scp -i production_key.pem -o StrictHostKeyChecking=no deploy-flower.sh ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }}:/tmp/
ssh -i production_key.pem -o StrictHostKeyChecking=no ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} "chmod +x /tmp/deploy-flower.sh && /tmp/deploy-flower.sh $REGISTRY $REPOSITORY"
rm -f production_key.pem deploy-flower.sh
96 changes: 94 additions & 2 deletions .github/workflows/cd-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,106 @@ jobs:
uses: aws-actions/amazon-ecr-login@v2


- name: Build and Push Docker Image
- name: Build and Push Backend Docker Image
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-staging-repo
run: |
docker build -t $REGISTRY/$REPOSITORY:latest ./backend
docker push $REGISTRY/$REPOSITORY:latest

- name: Deploy to ECS
- name: Build and Push Celery Docker Image
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-celery-repo
run: |
docker build -f ./backend/Dockerfile.celery -t $REGISTRY/$REPOSITORY:staging ./backend
docker push $REGISTRY/$REPOSITORY:staging

- name: Deploy Backend to ECS
run: |
aws ecs update-service --cluster ${{ github.event.repository.name }}-staging-cluster --service ${{ github.event.repository.name }}-staging-service --force-new-deployment

- name: Deploy Celery Workers to EC2
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-celery-repo
run: |
# Create deployment script
cat > deploy-celery.sh << 'EOF'
#!/bin/bash
set -e

# Stop existing containers
docker stop celery-stg-high celery-stg-other 2>/dev/null || true
docker rm celery-stg-high celery-stg-other 2>/dev/null || true

# Pull latest image
aws ecr get-login-password --region $3 | docker login --username AWS --password-stdin $1
docker pull $1/$2:staging

# Start high priority worker
docker run -d --name celery-stg-high --restart unless-stopped \
--env-file /home/ubuntu/.env.staging \
$1/$2:staging \
uv run celery -A app.celery.celery_app worker --queues=high_priority --loglevel=info --concurrency=1 --hostname=stg-high@%h

# Verify high priority worker started
sleep 2
docker ps | grep celery-stg-high || { echo "High priority worker failed to start"; exit 1; }

# Start other queues worker
docker run -d --name celery-stg-other --restart unless-stopped \
--env-file /home/ubuntu/.env.staging \
$1/$2:staging \
uv run celery -A app.celery.celery_app worker --queues=default,low_priority,cron --loglevel=info --concurrency=1 --hostname=stg-other@%h

# Verify other queues worker started
sleep 2
docker ps | grep celery-stg-other || { echo "Other queues worker failed to start"; exit 1; }
EOF

# Copy script to EC2 and execute
echo "${{ secrets.EC2_KEY }}" > staging_key.pem
chmod 600 staging_key.pem
trap 'rm -f staging_key.pem deploy-celery.sh' EXIT
scp -i staging_key.pem -o StrictHostKeyChecking=no deploy-celery.sh ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }}:/tmp/
ssh -i staging_key.pem -o StrictHostKeyChecking=no ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} "chmod +x /tmp/deploy-celery.sh && /tmp/deploy-celery.sh $REGISTRY $REPOSITORY ap-south-1"
rm -f staging_key.pem deploy-celery.sh

- name: Deploy Flower to EC2
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ github.event.repository.name }}-celery-repo
run: |
# Create flower deployment script
cat > deploy-flower.sh << 'EOF'
#!/bin/bash
set -e

# Stop existing flower container
docker stop flower-staging 2>/dev/null || true
docker rm flower-staging 2>/dev/null || true

# Start flower using the same celery image
docker run -d --name flower-staging --restart unless-stopped \
--env-file /home/ubuntu/.env.staging \
-p 5555:5555 \
$1/$2:staging \
uv run celery -A app.celery.celery_app flower --port=5555

# Verify Flower started
sleep 2
docker ps | grep flower-staging || { echo "Flower failed to start"; exit 1; }

# Health check
curl -f http://localhost:5555/healthcheck || { echo "Flower health check failed"; exit 1; }
EOF

# Copy script to EC2 and execute
echo "${{ secrets.EC2_KEY }}" > staging_key.pem
chmod 600 staging_key.pem
trap 'rm -f staging_key.pem deploy-flower.sh' EXIT
scp -i staging_key.pem -o StrictHostKeyChecking=no deploy-flower.sh ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }}:/tmp/
ssh -i staging_key.pem -o StrictHostKeyChecking=no ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} "chmod +x /tmp/deploy-flower.sh && /tmp/deploy-flower.sh $REGISTRY $REPOSITORY"
rm -f staging_key.pem deploy-flower.sh
25 changes: 25 additions & 0 deletions docker-compose.dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,31 @@ services:
- backend
command: ["uv", "run", "celery", "-A", "app.celery.celery_app", "worker", "--loglevel=info", "--concurrency=2"]

flower:
build:
context: ./backend
dockerfile: Dockerfile.celery
environment:
- ENVIRONMENT=development
- REDIS_HOST=redis
- RABBITMQ_HOST=rabbitmq
- RABBITMQ_USER=guest
- RABBITMQ_PASSWORD=guest
env_file:
- ./.env
ports:
- "5555:5555"
volumes:
- ./backend:/app # Mount for live code changes
- /app/.venv # Exclude .venv from volume mount
networks:
- app-network
depends_on:
- redis
- rabbitmq
- celery-worker
command: ["uv", "run", "celery", "-A", "app.celery.celery_app", "flower", "--port=5555"]

Comment on lines +118 to +119
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Secure Flower: enforce auth and use env-driven flags (switch to shell form).

Flower is exposed on 0.0.0.0:5555 without basic auth or URL prefix; env vars in .env.example are not consumed by Flower unless passed as CLI flags. Also, array-form command won’t reliably expand env vars. Use shell-form, add auth, and honor FLOWER_*.

Apply this diff:

-    command: ["uv", "run", "celery", "-A", "app.celery.celery_app", "flower", "--port=5555"]
+    command: uv run celery -A app.celery.celery_app flower \
+      --port=${FLOWER_PORT:-5555} \
+      --basic_auth="${FLOWER_BASIC_AUTH:-}" \
+      --url_prefix="${FLOWER_URL_PREFIX:-}" \
+      --max_tasks=${FLOWER_MAX_TASKS:-10000}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
command: ["uv", "run", "celery", "-A", "app.celery.celery_app", "flower", "--port=5555"]
command: uv run celery -A app.celery.celery_app flower \
--port=${FLOWER_PORT:-5555} \
--basic_auth="${FLOWER_BASIC_AUTH:-}" \
--url_prefix="${FLOWER_URL_PREFIX:-}" \
--max_tasks=${FLOWER_MAX_TASKS:-10000}
🤖 Prompt for AI Agents 
In docker-compose.dev.yml around lines 118-119, the current array-form command
launches Flower without basic auth, without honoring .env vars, and array form
prevents shell expansion; change to shell form so env vars expand and add flags
to enforce auth and optional URL prefix. Replace the array with a single shell
command that runs uv run celery -A app.celery.celery_app flower and passes flags
using environment variables (e.g. --port=${FLOWER_PORT:-5555},
--basic_auth=${FLOWER_BASIC_AUTH}, --url_prefix=${FLOWER_URL_PREFIX:-/flower})
and include a passthrough ${FLOWER_EXTRA_ARGS:-} so additional flags from .env
can be consumed; ensure FLOWER_BASIC_AUTH is set in .env (or fail-fast/validate)
so Flower is not exposed without credentials.

networks:
app-network:
driver: bridge
Expand Down