Skip to content

Update nwp-consumer container tag to 1.1.32 #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Sep 16, 2025
Merged

Update nwp-consumer container tag to 1.1.32 #349

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
7b7c794
Update nwp-consumer container tag to 1.1.30
peterdudfield Sep 1, 2025
ad1f32e
add cpu and memory overrides
peterdudfield Sep 1, 2025
57d3f42
use 1.1.31, and increase gfs nan threshold
peterdudfield Sep 2, 2025
106a571
create two different container definitions
peterdudfield Sep 2, 2025
ffe3392
lint
peterdudfield Sep 2, 2025
d724d34
lint
peterdudfield Sep 2, 2025
1a38815
use 0 and 12 run times, and 6 hour delay
peterdudfield Sep 2, 2025
4410e9c
Merge branch 'main' into india-nwp-1.1.30
peterdudfield Sep 2, 2025
2a2c79c
Merge branch 'main' into india-nwp-1.1.30
peterdudfield Sep 5, 2025
1a367e8
remove cpu and memoery override
peterdudfield Sep 5, 2025
5e981f2
tidy up
peterdudfield Sep 5, 2025
bae9ada
Merge commit 'd2187a023b624f640bcb1601d9ed527f57b7db6f' into india-nw…
peterdudfield Sep 15, 2025
a3344ea
update to 1.1.32
peterdudfield Sep 15, 2025
7bdb941
Merge branch 'main' into india-nwp-1.1.30
peterdudfield Sep 15, 2025
4e5adce
Merge branch 'main' into india-nwp-1.1.30
peterdudfield Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 24 additions & 12 deletions src/airflow_dags/dags/india/consume-nwp-dag.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,26 +25,36 @@
"max_active_tasks": 10,
}

nwp_consumer = ContainerDefinition(
name="nwp-consumer",
container_image="ghcr.io/openclimatefix/nwp-consumer",
container_tag="1.1.10",
container_env={
default_args = {
"name":"nwp-consumer",
"container_image":"ghcr.io/openclimatefix/nwp-consumer",
"container_tag":"1.1.32",
"container_env":{
"CONCURRENCY": "false",
"LOGLEVEL": "DEBUG",
},
container_secret_env={
"container_secret_env":{
f"{env}/data/nwp-consumer": [
"ECMWF_REALTIME_S3_ACCESS_KEY",
"ECMWF_REALTIME_S3_ACCESS_SECRET",
"METOFFICE_API_KEY",
],
},
container_command=["consume"],
container_cpu=512,
container_memory=1024,
domain="india",
)
"container_command":["consume"],
"container_cpu":512,
"container_memory":1024,
"domain":"india",
}

# GFS and MetOffice consumers
nwp_consumer = ContainerDefinition(**default_args)

# ECWMF consumer
ecmwf_args = default_args.copy()
ecmwf_args["name"] = "nwp-consumer-ecmwf"
ecmwf_args["container_memory"] = 2048
ecmwf_args["container_cpu"] = 1024
nwp_consumer_ecwmf = ContainerDefinition(**ecmwf_args)


@dag(
Expand All @@ -61,7 +71,7 @@ def nwp_consumer_dag() -> None:

consume_ecmwf_op = EcsAutoRegisterRunTaskOperator(
airflow_task_id="consume-ecmwf-nwp",
container_def=nwp_consumer,
container_def=nwp_consumer_ecwmf,
env_overrides={
"MODEL_REPOSITORY": "ecmwf-realtime",
"MODEL": "hres-ifs-india",
Expand All @@ -86,6 +96,8 @@ def nwp_consumer_dag() -> None:
env_overrides={
"MODEL_REPOSITORY": "gfs",
"ZARRDIR": f"s3://india-nwp-{env}/gfs/data",
# SDE has nans
"ALLOWED_VALIDATION_FAILURE_PERCENTAGE": "0.07",
},
on_failure_callback=slack_message_callback(
f"⚠️🇮🇳 The {get_task_link()} failed."
Expand Down