Reduction in Ocean and Sea Ice Output #4200

NeilBarton-NOAA · 2025-10-30T21:41:43Z

Description

Addressing
NOAA-EMC/GFS#37
NOAA-EMC/GFS#36
NOAA-EMC/GFS#4
#4163

Reducing the I/O burden by reducing the number of variables, layers, and adding compression.
Also adds archiving for native subsets

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Tested using C192mx025 SFS forecasts. This will be a draft until GFS testing can be done.

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
My changes generate no new warnings
New and existing tests pass with my changes
[] This change is covered by an existing CI test or a new one has been added
Any new scripts have been added to the .github/CODEOWNERS file with owners
I have made corresponding changes to the system documentation if necessary

NeilBarton-NOAA · 2025-11-03T18:44:53Z

GFS forecast tested with archiving changes. New interpolated_zgrid file needed, and is at

gaeac6:/gpfs/f6/sfs-emc/scratch/Neil.Barton/CODE/gw_ocnice_products_NeilBarton-NOAA/fix/interpolate_zgrid_30L.nc

…orkflow into ocnice_products

parm/archive/ice_native.yaml.j2

parm/archive/ocean_native.yaml.j2

aerorahul · 2025-11-03T21:13:40Z

parm/post/oceanice_products_gfs.yaml

    copy:
-      - ["{{ EXECgfs }}/ocnicepost.x", "{{ DATA }}/"]
      - ["{{ PARMgfs }}/post/ocnicepost.nml.jinja2", "{{ DATA }}/"]
+      {% if write_grib2 and write_netcdf %}


Shouldn't this be or instead of and?
Also, shouldn't the ocnicepost.nml.jinja2 file be conditioned based on the if-block?
I think the reason it it outside because the yaml wants atleast one item in the copy block.
In that case, the copy block itself should be conditioned on the if statement, no?

yes, I uploaded the change

aerorahul · 2025-11-03T21:14:56Z

parm/post/oceanice_products_gfs.yaml

-      - ["{{ DATA }}/ocean_subset.nc", "{{ COMOUT_OCEAN_NETCDF }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.tripolar.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ocean_subset.nc", "{{ COMOUT_OCEAN_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ocean_compressed.nc", "{{ COMIN_OCEAN_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]
+      {% if write_grib2 %}


Where is the data from write_netcdf being copied to?

I don't know. I don't think it was defined in the yaml file before my edits.

Correct. The requirement was to provide grib2 output to the users. The netcdf was considered as an intermediate by-product of ocnicepost.x
write_netcdf=True is only needed if we are going to ever give out interpolated netcdf products.

reminder that we need to add the section when write_netcdf = True to copy the interpolated netcdf files

aerorahul · 2025-11-03T21:15:53Z

parm/post/oceanice_products_gfs.yaml

+      - ["{{ DATA }}/ice_subset.nc", "{{ COMOUT_ICE_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ice_compressed.nc", "{{ COMIN_ICE_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]
      {% for grid in product_grids %}
+      {% if write_grib2 %}


Shouldn't the interpolated output data from write_netcdf=True be copied to COMOUT_ICE_NETCDF/grid?

same as above

same reminder as above.

aerorahul · 2025-11-03T21:18:21Z

ush/python/pygfs/task/oceanice_products.py

-            # save subsetted variables to a new netcdf file
-            ds_subset.to_netcdf(output_file)
+            # save subsetted variables to a new netcdf file and compress
+            default_compression = {"zlib": True, "complevel": 8}


This should be configured in the oceanice_products.yaml

the compression level? or to compress or not?

namelist options added

aerorahul · 2025-11-03T21:23:02Z

parm/post/oceanice_products_sfs.yaml

    copy:
-      - ["{{ DATA }}/ice_subset.nc", "{{ COMOUT_ICE_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.tripolar.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ice_subset.nc", "{{ COMOUT_ICE_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ice_compressed.nc", "{{ COMIN_ICE_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]


Over-writing model history output?
I don't think this should be allowed because it looses provenance. Perhaps consider the model to write out compressed output in the first place.

I don't think NCO would accept that. I don't think we can overwrite what's in COM right?

Would it be possible to just compress what we subset in the product directory ?

Compression to the products is included in this PR. Compression for the history files can be removed.

The model uses pnetcdf2, which to my knowledge cannot be compressed.

compression of history files removed

Still see the compressed files are overwriting history files.

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

aerorahul · 2025-11-03T21:39:15Z

parm/archive/ocean_native.yaml.j2

@@ -1,10 +1,7 @@
 ocean_native:
-    {% set NET = NET %}
-    {% set FREQ = FHOUT_ICE_GFS %}


Why is this FHOUT_ICE_GFS? Should it be FHOUT_OCN_GFS (if that exists)?

changed with update. This was a copy from the other file

…orkflow into ocnice_products

parm/post/oceanice_products_gfs.yaml

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

Removed unnecessary variables from the subset list in oceanice_products_gfs.yaml. remove duplicate variables

aerorahul · 2025-11-06T19:57:11Z

parm/archive/ocean_native.yaml.j2

+    name: "OCEAN_NATIVE"
+    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/ocean_native_subset.tar"
+    required:
+        {% for fhr in range(FHOUT_ICE_GFS, FHMAX_GFS + FHOUT_ICE_GFS, FHOUT_ICE_GFS) %}


Shouldn't these be FHOUT_OCN_GFS?

yes, change uploaded

aerorahul · 2025-11-06T19:59:49Z

parm/post/oceanice_products_gfs.yaml

-    write_grib2: True
-    write_netcdf: False
+    write_grib2: {{ write_grib2 }}
+    write_netcdf: {{ write_netcdf }}


This should stay False. When this is True, the program will run and create the interpolated netcdf files, but they remain in DATA, which gets removed when the job ends.

Suggested change

write_netcdf: {{ write_netcdf }}

write_netcdf: False

these variables are now defined in https://github.com/NeilBarton-NOAA/global-workflow/blob/ocnice_products/dev/parm/config/gfs/config.oceanice_products

parm/post/oceanice_products_gfs.yaml

parm/post/oceanice_products_sfs.yaml

dev/parm/config/sfs/config.nsst.j2

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

aerorahul · 2025-11-12T20:47:47Z

parm/post/oceanice_products_gfs.yaml

    copy:
-      - ["{{ EXECgfs }}/ocnicepost.x", "{{ DATA }}/"]
      - ["{{ PARMgfs }}/post/ocnicepost.nml.jinja2", "{{ DATA }}/"]
+      {% if write_grib2 %}


These are needed when write_netcdf = True as well.

aerorahul · 2025-11-12T20:49:02Z

parm/post/oceanice_products_gfs.yaml

-      - ["{{ DATA }}/ocean_subset.nc", "{{ COMOUT_OCEAN_NETCDF }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.tripolar.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ocean_subset.nc", "{{ COMOUT_OCEAN_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ocean_compressed.nc", "{{ COMIN_OCEAN_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]
+      {% if write_grib2 %}


reminder that we need to add the section when write_netcdf = True to copy the interpolated netcdf files

aerorahul · 2025-11-12T20:49:17Z

parm/post/oceanice_products_gfs.yaml

+      - ["{{ DATA }}/ice_subset.nc", "{{ COMOUT_ICE_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ice_compressed.nc", "{{ COMIN_ICE_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]
      {% for grid in product_grids %}
+      {% if write_grib2 %}


same reminder as above.

aerorahul · 2025-11-12T20:49:52Z

parm/post/oceanice_products_sfs.yaml

    copy:
-      - ["{{ DATA }}/ocean_subset.nc", "{{ COMOUT_OCEAN_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.tripolar.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ocean_subset.nc", "{{ COMOUT_OCEAN_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ocean_compressed.nc", "{{ COMIN_OCEAN_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]


Should not overwrite model output.

aerorahul · 2025-11-12T20:50:46Z

parm/post/oceanice_products_sfs.yaml

+      {% if write_netcdf %}
      - "{{ COMOUT_OCEAN_NETCDF }}/{{ grid }}"
+      {% endif %}


write_netcdf = False. This section is irrelevant.

aerorahul · 2025-11-12T20:51:14Z

parm/post/oceanice_products_sfs.yaml

+      {% if write_netcdf %}
      - "{{ COMOUT_ICE_NETCDF }}/{{ grid }}"
+      {% endif %}


This section is irrelevant because write_netcdf = False

added write_netcdf: {{ write_netcdf }}

sfs products are requested as lat/lon as netCDF

aerorahul · 2025-11-12T20:51:41Z

parm/post/oceanice_products_sfs.yaml

    copy:
-      - ["{{ DATA }}/ice_subset.nc", "{{ COMOUT_ICE_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.tripolar.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ice_subset.nc", "{{ COMOUT_ICE_NETCDF }}/native/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.native.f{{ '%03d' % forecast_hour }}.nc"]
+      - ["{{ DATA }}/ice_compressed.nc", "{{ COMIN_ICE_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.{{ interval }}hr_avg.f{{ '%03d' % forecast_hour }}.nc"]


Still see the compressed files are overwriting history files.

…orkflow into ocnice_products

initial commits for reducing size of ocean and sea ice output

90f0c6d

NeilBarton-NOAA requested review from ChristopherHill-NOAA, DavidNew-NOAA, JesseMeng-NOAA and aerorahul as code owners October 30, 2025 21:41

NeilBarton-NOAA marked this pull request as draft October 30, 2025 21:42

NeilBarton-NOAA added 4 commits October 30, 2025 17:44

edits to MOM6 diag_table

c9483c1

edits for file name differences

76f89d0

changing to 30L zgrid MOM6 file

f8f6fe8

gfs test and edits

5ad905c

NeilBarton-NOAA marked this pull request as ready for review November 3, 2025 18:43

NeilBarton-NOAA requested a review from DavidHuber-NOAA as a code owner November 3, 2025 18:43

Merge branch 'develop' into ocnice_products

94237f7

NeilBarton-NOAA added 7 commits November 3, 2025 13:54

fix python code style

d7abd79

remove trailing white space

d2ed372

Merge branch 'ocnice_products' of github.com:NeilBarton-NOAA/global-w…

90875e1

…orkflow into ocnice_products

fixing python errors

46b0bc2

fix python style errors

b7b220f

try again

6fcf46f

white space

c9ad5e3

aerorahul requested changes Nov 3, 2025

View reviewed changes

NeilBarton-NOAA and others added 2 commits November 3, 2025 16:34

Update parm/archive/ice_native.yaml.j2

156580a

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

Update parm/archive/ocean_native.yaml.j2

486f690

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

aerorahul reviewed Nov 3, 2025

View reviewed changes

NeilBarton-NOAA added 2 commits November 3, 2025 16:39

corrected and to or

386a01d

Merge branch 'ocnice_products' of github.com:NeilBarton-NOAA/global-w…

b41f6b6

…orkflow into ocnice_products

aerorahul reviewed Nov 3, 2025

View reviewed changes

parm/post/oceanice_products_gfs.yaml Show resolved Hide resolved

Update parm/post/oceanice_products_gfs.yaml

1943a47

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

NeilBarton-NOAA changed the title ~~Reduction on Ocean and Sea Ice Output~~ Reduction in Ocean and Sea Ice Output Nov 5, 2025

NeilBarton-NOAA and others added 2 commits November 5, 2025 16:04

Update subset list in oceanice_products_gfs.yaml

3091fa1

Removed unnecessary variables from the subset list in oceanice_products_gfs.yaml. remove duplicate variables

removing compressing of COMROT files and adding namelist options

93a10fe

aerorahul reviewed Nov 6, 2025

View reviewed changes

NeilBarton-NOAA and others added 2 commits November 6, 2025 15:10

Update dev/parm/config/sfs/config.nsst.j2

fecb2ab

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

addressing comments

b3f10b3

NeilBarton-NOAA requested a review from aerorahul November 12, 2025 20:43

Update parm/post/oceanice_products_sfs.yaml

b7e343a

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

aerorahul requested changes Nov 12, 2025

View reviewed changes

NeilBarton-NOAA and others added 6 commits November 12, 2025 16:01

changing to or

0246dd6

remove compressed in COMOUT overwrite

29f7357

defining variable outside of yaml

2b705db

Merge branch 'develop' into ocnice_products

98b205b

add option to copy netcdf lat/lon products to COMROT

08f4a5a

Merge branch 'ocnice_products' of github.com:NeilBarton-NOAA/global-w…

e09e4d1

…orkflow into ocnice_products

NeilBarton-NOAA requested a review from aerorahul November 12, 2025 21:47

Reduction in Ocean and Sea Ice Output #4200

Are you sure you want to change the base?

Reduction in Ocean and Sea Ice Output #4200

Uh oh!

Conversation

NeilBarton-NOAA commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

NeilBarton-NOAA commented Nov 3, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

NeilBarton-NOAA commented Oct 30, 2025 •

edited

Loading